Loss function for log-scoring probabilistic forecasts

1

I’m completely new to PyTorch, so apologies in advance if this is very silly.
My goal is to post-process (evolutions of) probabilistic forecasts for which the outcome is known. E.g. I have forecasts of the form [.5, .6, .8] and the information that this forecast resolved positively, i.e. resolution=1. (In this example, the forecast started off as uncertain with 50% and converged towards 80%, so not too bad.)

The input to my network is a probability x_1 and more relevant information x_2,x_3,... accessible at the time this forecast was made. Its output should be a different probability y_hat. The loss function I’m interested in is y_true*log(y_hat) + (1-y_true)*log(1-y_hat).

Since y_true is either 0 or 1, this can be seen as the Kullback-Leibler divergence loss if we consider not only the KL distance between y_true and y_hat, but also add the KL distance between 1-y_true and 1-y_hat.

(I’d also like to weigh these contributions to the log score by the time elapsed between to consecutive features, but unlike other losses, KLDivLoss — PyTorch 1.11.0 documentation doesn’t seem to take a weight argument.)

Is there a neat way to get the loss function I described above by somehow mirroring output data or do I have to write it “from scratch” myself?

2

Hi Peter!

This is pytorch’s BCELoss.

However, for reasons of numerical stability, you will be better off using
BCEWithLogitsLoss. To do this, you will have to modify your network to
predict logits that run from -inf to inf (rather than probabilities that run
from 0.0 to 1.0).

You would typically do this by having the final layer of your network be
a Linear, not followed by a sigmoid() (that would have been used to
convert logits into probabilities).

It’s worth noting that both BCELoss and BCEWithLogitsLoss take weight
constructor arguments.)

As an aside, cross entropy is related to the Kullback-Leibler divergence,
although the two are not identical.

Best.

K. Frank

1 Like
3

Hi Frank,

That’s exactly what I was looking for, I don’t know how I overlooked it until now. Thank you very much!

Best,
Peter