Controlled abstention neural networks for identifying skillful predictions for classification problems. (arXiv:2104.08281v1 [physics.ao-ph])

The earth system is exceedingly complex and often chaotic in nature, making
prediction incredibly challenging: we cannot expect to make perfect predictions
all of the time. Instead, we look for specific states of the system that lead
to more predictable behavior than others, often termed “forecasts of
opportunity.” When these opportunities are not present, scientists need
prediction systems that are capable of saying “I don’t know.” We introduce a
novel loss function, termed the “NotWrong loss”, that allows neural networks to
identify forecasts of opportunity for classification problems. The NotWrong
loss introduces an abstention class that allows the network to identify the
more confident samples and abstain (say “I don’t know”) on the less confident
samples. The abstention loss is designed to abstain on a user-defined fraction
of the samples via a PID controller. Unlike many machine learning methods used
to reject samples post-training, the NotWrong loss is applied during training
to preferentially learn from the more confident samples. We show that the
NotWrong loss outperforms other existing loss functions for multiple climate
use cases. The implementation of the proposed loss function is straightforward
in most network architectures designed for classification as it only requires
the addition of an abstention class to the output layer and modification of the
loss function.