Does NLLLoss handle Log-Softmax and Softmax in the same way?


(Rian Adam) #1

from documentation here : http://pytorch.org/docs/master/nn.html#torch.nn.LogSoftmax, log-softmax is defined as:

f(x)=log(softmax(x))

as I know, using log in probabilities will change the high-low value, so biggest value in softmax(x) will be the smallest value in log(softmax(x)).

Will it change the way Negative Log Likelihood compute loss when it implements it together in CrossEntropy?


(Rian Adam) #2

Oh., I’ve just read from NLLLoss documentation, that NLLLoss is implemented differently than what I’ve known before:

loss(x, class) = -x[class]

it doesn’t use log function. So, does it means I can’t use Softmax and NLLLoss together?


#3

so biggest value in softmax(x) will be the smallest value in log(softmax(x)).

The softmax function returns probabilities between [0, 1].
The log of these probabilities returns values between [-inf, 0], since log(0) = -inf and log(1) = 0.
That is why the order won’t change.

However, you should use the NLLLoss with a log_softmax output
or CrossEntropyLoss with logits if you prefer not to add an extra log_softmax layer into your model.


(Rian Adam) #4

oh… that’s right! I thought log(0) = inf

Thanks,