# Does NLLLoss handle Log-Softmax and Softmax in the same way? **URL:** \ **Category:** Uncategorized\ **Created:** [October 19, 2017, 3:34am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835 "2017-10-19T03:34:41Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![malioboro](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/malioboro/32/1660_2.png) [@malioboro](https://discuss.pytorch.org/u/malioboro)\ **Post date:** [October 19, 2017, 3:34am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/1 "2017-10-19T03:34:41Z")
from documentation here : [http://pytorch.org/docs/master/nn.html#torch.nn.LogSoftmax](http://pytorch.org/docs/master/nn.html#torch.nn.LogSoftmax), log-softmax is defined as: ```auto f(x)=log(softmax(x)) ``` as I know, using log in probabilities will change the high-low value, so biggest value in softmax(x) will be the smallest value in log(softmax(x)). Will it change the way Negative Log Likelihood compute loss when it implements it together in `CrossEntropy`? ---
**Author:** ![malioboro](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/malioboro/32/1660_2.png) [@malioboro](https://discuss.pytorch.org/u/malioboro)\ **Post date:** [October 19, 2017, 3:54am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/2 "2017-10-19T03:54:25Z")
Oh., I’ve just read from [NLLLoss documentation](http://pytorch.org/docs/master/nn.html#torch.nn.NLLLoss), that NLLLoss is implemented differently than what I’ve known before: ```auto loss(x, class) = -x[class] ``` it doesn’t use `log` function. So, does it means I can’t use `Softmax` and `NLLLoss` together? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [October 19, 2017, 12:34pm UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/3 "2017-10-19T12:34:31Z")
> so biggest value in softmax(x) will be the smallest value in log(softmax(x)). The `softmax` function returns probabilities between [0, 1]. The log of these probabilities returns values between [-inf, 0], since `log(0) = -inf` and `log(1) = 0`. That is why the order won’t change. However, you should use the `NLLLoss` with a `log_softmax` output or `CrossEntropyLoss` with logits if you prefer not to add an extra `log_softmax` layer into your model. ---
**Author:** ![malioboro](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/malioboro/32/1660_2.png) [@malioboro](https://discuss.pytorch.org/u/malioboro)\ **Post date:** [October 19, 2017, 11:00pm UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/4 "2017-10-19T23:00:34Z")
oh… that’s right! I thought `log(0) = inf` Thanks, ---
**Author:** ![ANKITA\_DAS](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ankita_das/32/19126_2.png) [@ANKITA\_DAS](https://discuss.pytorch.org/u/ANKITA_DAS)\ **Post date:** [January 10, 2020, 12:30am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/5 "2020-01-10T00:30:40Z")
If the order doesn’t change, why do we have to use a ‘log-softmax’ with NLLLoss? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [January 10, 2020, 1:03am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/6 "2020-01-10T01:03:18Z")
By “order” I meant the range of the outputs will still be in the same order. I.e. if `p0 < p1` created by `softmax` , then `log(p0) < log(p1)`. `nn.NLLLoss` expects the log probabilities, as the loss will be calculated as described [here](https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss). Note that `nn.CrossEntropyLoss` applies internally `F.log_softmax` and `nn.NLLLoss` afterwards, which is why it expects raw logits instead. ---
**Author:** ![ANKITA\_DAS](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ankita_das/32/19126_2.png) [@ANKITA\_DAS](https://discuss.pytorch.org/u/ANKITA_DAS)\ **Post date:** [January 10, 2020, 2:30am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/7 "2020-01-10T02:30:24Z")
Yes I do get that. Yet I have another question if log-softmax gives values in the range [-infinity,0] that means that the values are negative,and the NLLLoss functon is -log(y), where y=log-softmax(x) , but log of some negative value isn’t defined so how does is work? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [January 10, 2020, 2:40am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/8 "2020-01-10T02:40:54Z")
The formula in the docs is the negative log softmax written as: ```python - log ( exp(x[class]) / sum(exp(x[j])) ``` `x` are the logits here while `exp()/sum(exp())` is the softmax function. ---
**Author:** ![ANKITA\_DAS](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ankita_das/32/19126_2.png) [@ANKITA\_DAS](https://discuss.pytorch.org/u/ANKITA_DAS)\ **Post date:** [January 10, 2020, 3:26am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/9 "2020-01-10T03:26:09Z")
Sorry but, I searched for it in the documentation,I didn’t find it explicitly mentioned anywhere that negative of log\_softmax goes in as input to the nllloss function.I don’t know if I’m missing something important, but please check the docs once. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [January 10, 2020, 4:45am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/10 "2020-01-10T04:45:54Z")
From the [nn.NLLLoss docs](https://pytorch.org/docs/stable/nn.html?highlight=nllloss#torch.nn.NLLLoss): > The input given through a forward call is expected to contain log-probabilities of each class. input has to be a Tensor of size either […]. > Obtaining log-probabilities in a neural network is easily achieved by adding a LogSoftmax layer in the last layer of your network. You may use CrossEntropyLoss instead, if you prefer not to add an extra layer. ```auto Examples: m = nn.LogSoftmax(dim=1) loss = nn.NLLLoss() # input is of size N x C = 3 x 5 input = torch.randn(3, 5, requires_grad=True) # each element in target has to have 0 <= value < C target = torch.tensor([1, 0, 4]) output = loss(m(input), target) output.backward() ``` ---
**Author:** ![ANKITA\_DAS](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ankita_das/32/19126_2.png) [@ANKITA\_DAS](https://discuss.pytorch.org/u/ANKITA_DAS)\ **Post date:** [January 10, 2020, 5:11am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/11 "2020-01-10T05:11:12Z")
But, there isn’t any mention of **negative** log\_softmax here! as you mentioned in this :[Does NLLLoss handle Log-Softmax and Softmax in the same way?](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/8) ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [January 10, 2020, 5:14am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/12 "2020-01-10T05:14:53Z")
Ah, sorry for the confusion, as I can see the misunderstanding now. `nn.NLLLoss` expects log probabilities, so you should just apply `F.log_softmax` on your model output (not multiplying with `-1`!). The formula posted in my previous post is, how the loss can be calculated, but you shouldn’t worry about the minus sign, as it will be applied internally for you. The posted example shows, how to apply the criterion. ---
**Author:** ![ANKITA\_DAS](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ankita_das/32/19126_2.png) [@ANKITA\_DAS](https://discuss.pytorch.org/u/ANKITA_DAS)\ **Post date:** [January 10, 2020, 5:23am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/14 "2020-01-10T05:23:51Z")
If the minus will be applied internally my doubt has been cleared, but they haven’t mentioned that in the documentation 🙁 as per my findings. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [January 10, 2020, 5:26am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/15 "2020-01-10T05:26:24Z")
The documentation mentions that log probabilities are expected and gives a code example. What is missing and what statement do you think might have been helpful to solve your misunderstanding? The docs are by far perfect, so feedback is always welcome. 😉 ---
**Author:** ![ANKITA\_DAS](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ankita_das/32/19126_2.png) [@ANKITA\_DAS](https://discuss.pytorch.org/u/ANKITA_DAS)\ **Post date:** [January 10, 2020, 5:40am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/16 "2020-01-10T05:40:19Z")
What needs to be given as input and what we get as output is fine, there is no problem in understanding that, but, if a minus is introduced internally I think there should be mention of that, as the screenshot of the documentation that you provided above it says > [@ptrblck](#): > > The input given through a forward call is expected to contain log-probabilities of each class. input has to be a Tensor of size either […]. If it was mentioned somewhere that “nn.NLLLoss internally multiplies the log-probabilities by a minus”, it would have cleared the confusions! ---
**Author:** ![Ashima\_Garg](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/bc8723/32.png) [@Ashima\_Garg](https://discuss.pytorch.org/u/Ashima_Garg)\ **Post date:** [March 1, 2020, 10:23pm UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/17 "2020-03-01T22:23:06Z")
As you said `so biggest value in softmax(x) will be the smallest value in log(softmax(x)).` Does that mean to calculate accuracy, `argmin` of predictions will be used for classification in Sentimental analysis? I am using NLLLoss to calculate the loss. Thankyou. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [March 2, 2020, 12:03am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/18 "2020-03-02T00:03:00Z")
No, I just quoted this sentence and corrected it in [this post](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/3). ---
**Author:** ![Ashima\_Garg](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/bc8723/32.png) [@Ashima\_Garg](https://discuss.pytorch.org/u/Ashima_Garg)\ **Post date:** [March 2, 2020, 11:20am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/19 "2020-03-02T11:20:17Z")
Thanks for the reply. ---
**Author:** ![ADONAI\_TZEVAOT](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adonai_tzevaot/32/31089_2.png) [@ADONAI\_TZEVAOT](https://discuss.pytorch.org/u/ADONAI_TZEVAOT)\ **Post date:** [November 16, 2020, 6:40pm UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/20 "2020-11-16T18:40:39Z")
Hi, pls could you clear my doubt, I have a multi-label classification problem, where each label can either be positive or negative, so each label on its own is binary in nature. The last layer in my network is LogSoftmax , so what loss function should I use ? BCEloss ? And how do i calculate the class weights? Pls help. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [November 17, 2020, 6:41am UTC](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835/21 "2020-11-17T06:41:36Z")
When each label is either positive or negative it would be a binary classification or a (2 class) multi-class classification. You can either return two logits (or log probabilities) from the model and use `nn.CrossEntropyLoss` (or `nn.NLLLoss`) or alternatively you can return a single logit and use `nn.BCEWithLogitsLoss`. [Next page](https://discuss.pytorch.org/t/does-nllloss-handle-log-softmax-and-softmax-in-the-same-way/8835.md?page=2)