Manual Calculation of Binary Cross Entropy with logits

[KFrank]

Hi Arif!

@arif_saeed

In order to ensure that I understood how BCE with logits loss works in pytorch, I tried to manually calculate the loss, however I cannot reconcile my manual calculation with the loss generated by the pytorch function F.binary_cross_entropy_with_logits.

p1=y*(math.log(sigmoid(x),2))
p0=(1-y)*math.log(1-sigmoid(x),2)

You need to use the natural logarithm (log-base-e) in your
cross-entropy formula. You are using log-base-2. Remove
the second argument in your calls to python’s math.log().
Thus:

    p1=y*(math.log(sigmoid(x)))
    p0=(1-y)*math.log(1-sigmoid(x))

As a side note, calculating log() and sigmoid() separately
can amplify numerical errors. You can avoid this by using
something like pytorch’s logsigmoid() that uses a better
algorithm internally.

Good luck!

K. Frank