# Loss function for log-scoring probabilistic forecasts

I’m completely new to PyTorch, so apologies in advance if this is very silly.
My goal is to post-process (evolutions of) probabilistic forecasts for which the outcome is known. E.g. I have forecasts of the form `[.5, .6, .8]` and the information that this forecast resolved positively, i.e. `resolution=1`. (In this example, the forecast started off as uncertain with 50% and converged towards 80%, so not too bad.)

The input to my network is a probability `x_1` and more relevant information `x_2,x_3,...` accessible at the time this forecast was made. Its output should be a different probability `y_hat`. The loss function I’m interested in is `y_true*log(y_hat) + (1-y_true)*log(1-y_hat)`.

Since `y_true` is either 0 or 1, this can be seen as the Kullback-Leibler divergence loss if we consider not only the KL distance between `y_true` and `y_hat`, but also add the KL distance between `1-y_true` and `1-y_hat`.

(I’d also like to weigh these contributions to the log score by the time elapsed between to consecutive features, but unlike other losses, KLDivLoss — PyTorch 1.11.0 documentation doesn’t seem to take a `weight` argument.)

Is there a neat way to get the loss function I described above by somehow mirroring output data or do I have to write it “from scratch” myself?

Hi Peter!

This is pytorch’s BCELoss.

However, for reasons of numerical stability, you will be better off using
BCEWithLogitsLoss. To do this, you will have to modify your network to
predict logits that run from `-inf` to `inf` (rather than probabilities that run
from `0.0` to `1.0`).

You would typically do this by having the final layer of your network be
a `Linear`, not followed by a `sigmoid()` (that would have been used to
convert logits into probabilities).

It’s worth noting that both `BCELoss` and `BCEWithLogitsLoss` take `weight`
constructor arguments.)

As an aside, cross entropy is related to the Kullback-Leibler divergence,
although the two are not identical.

Best.

K. Frank

1 Like

Hi Frank,

That’s exactly what I was looking for, I don’t know how I overlooked it until now. Thank you very much!

Best,
Peter