# Confusion understanding nll\_loss

**URL:** <https://discuss.pytorch.org/t/confusion-understanding-nll-loss/43303>\
**Category:** vision\
**Created:** [April 23, 2019, 7:27am UTC](https://discuss.pytorch.org/t/confusion-understanding-nll-loss/43303 "2019-04-23T07:27:09Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jayesh\_Saita](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jayesh_saita/32/10695_2.png) [@Jayesh\_Saita](https://discuss.pytorch.org/u/Jayesh_Saita)\
**Post date:** [April 23, 2019, 7:27am UTC](https://discuss.pytorch.org/t/confusion-understanding-nll-loss/43303/1 "2019-04-23T07:27:10Z")

</div>

Theoretically, a loss function should return 0 when target=prediction, and some positive number when target and prediction and different. In the simple example below, when the prediction=target, I am getting a loss value of -1, and when prediction and target are different, I am getting a loss value of 0. How are these values calculated? What is the intuition behind these values?

 ![Screenshot%20from%202019-04-22%2021-16-33](https://discuss.pytorch.org/uploads/default/original/2X/b/b7310a7fac7714c5935049ed06d3211f95a10dc5.png)  
I also tried passing the log\_softmax of input, but then also, I am getting a loss value of 2.1232. Again, intuitively, the value of loss in this case should be 0.  
 ![image](https://discuss.pytorch.org/uploads/default/original/2X/2/23409a9c445d09d9c515d317144fe0ad245170a7.png)  
What is the intuition/reason behind these loss values?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [April 23, 2019, 9:43am UTC](https://discuss.pytorch.org/t/confusion-understanding-nll-loss/43303/2 "2019-04-23T09:43:43Z")

</div>

`F.log_softmax` expects logits which might have arbitrarily large and small numbers.  
If you just call `F.softmax` on your `input` tensor, you’ll see that the probability is not exactly one for the desired class:

```python
x = torch.zeros(1, 21)
x[0, 3] = 1.
print(F.softmax(x, 1))
> tensor([[0.0440, 0.0440, 0.0440, 0.1197, 0.0440, 0.0440, 0.0440, 0.0440, 0.0440,
         0.0440, 0.0440, 0.0440, 0.0440, 0.0440, 0.0440, 0.0440, 0.0440, 0.0440,
         0.0440, 0.0440, 0.0440]])

```

As you can see, class3 has a probability of `0.1197`, so you’ll get a positive loss.

---

<div class="post-metadata">

**Author:** ![Jayesh\_Saita](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jayesh_saita/32/10695_2.png) [@Jayesh\_Saita](https://discuss.pytorch.org/u/Jayesh_Saita)\
**Post date:** [April 23, 2019, 3:45pm UTC](https://discuss.pytorch.org/t/confusion-understanding-nll-loss/43303/3 "2019-04-23T15:45:43Z")

</div>

> [@ptrblck](#):
>
> `F.log_softmax` expects logits which might have arbitrarily large and small numbers.

Ahh, got it.

> [@ptrblck](#):
>
> As you can see, class3 has a probability of `0.1197` , so you’ll get a positive loss.

After applying F.nll\_loss to this softmax output, I get a loss value of -0.1197 which is close to 0 but not exactly zero. Shouldn’t loss functions return exactly zero when prediction equals target like in this case?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [April 23, 2019, 4:06pm UTC](https://discuss.pytorch.org/t/confusion-understanding-nll-loss/43303/4 "2019-04-23T16:06:35Z")

</div>

You will get a loss close to zero (in this case you’ll see the print outputs an exact zero, although the value might just underflow), if the logit for the target class is very high:

```python
x = torch.ones(1, 21) * (-10000)
x[0, 3] = 10000
target = torch.tensor([3])

loss = F.nll_loss(F.log_softmax(x, 1), target)
print(loss)
> tensor(0.)

```

---

<div class="post-metadata">

**Author:** ![Jayesh\_Saita](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jayesh_saita/32/10695_2.png) [@Jayesh\_Saita](https://discuss.pytorch.org/u/Jayesh_Saita)\
**Post date:** [April 23, 2019, 4:33pm UTC](https://discuss.pytorch.org/t/confusion-understanding-nll-loss/43303/5 "2019-04-23T16:33:09Z")

</div>

> [@ptrblck](#):
>
> if the logit for the target class is very high

Great, understood it! This also forces the network to output larger logit value for the corresponding neuron to reduce the loss during backprop!  
Thanks for the explanation 😃