# Difference between Cross-Entropy Loss or Log Likelihood Loss?

**URL:** <https://discuss.pytorch.org/t/difference-between-cross-entropy-loss-or-log-likelihood-loss/38816>\
**Category:** Uncategorized\
**Created:** [March 4, 2019, 12:53am UTC](https://discuss.pytorch.org/t/difference-between-cross-entropy-loss-or-log-likelihood-loss/38816 "2019-03-04T00:53:35Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![DoubtWang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/doubtwang/32/8307_2.png) [@DoubtWang](https://discuss.pytorch.org/u/DoubtWang)\
**Post date:** [March 4, 2019, 12:53am UTC](https://discuss.pytorch.org/t/difference-between-cross-entropy-loss-or-log-likelihood-loss/38816/1 "2019-03-04T00:53:35Z")

</div>

I’m very confused the difference between cross-entropy loss or log likelihood loss when dealing with Multi-Class Classification (including Binary Classification) or Multi-Label Classification ?  
Could you explain the difference ?  
Thanks.

---

<div class="post-metadata">

**Author:** ![K\_Frank](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/k_frank/32/10119_2.png) [@K\_Frank](https://discuss.pytorch.org/u/K_Frank)\
**Post date:** [March 4, 2019, 9:55pm UTC](https://discuss.pytorch.org/t/difference-between-cross-entropy-loss-or-log-likelihood-loss/38816/2 "2019-03-04T21:55:42Z")

</div>

Hello Doubt -

This may be a duplicate; see below\*.

> [@DoubtWang](#):
>
> I’m very confused the difference between cross-entropy loss or log likelihood loss when dealing with Multi-Class Classification (including Binary Classification) or Multi-Label Classification ?  
> Could you explain the difference ?  
> Thanks.

The cross-entropy loss and the (negative) log-likelihood are  
the same in the following sense:

If you apply Pytorch’s `CrossEntropyLoss` to your output layer,  
you get the same result as applying Pytorch’s `NLLLoss` to a  
`LogSoftmax` layer added after your original output layer.

(I suspect – but don’t know for a fact – that using  
`CrossEntropyLoss` will be more efficient because it  
can collapse some calculations together, and doesn’t  
introduce an additional layer.)

You are trying to maximize the “likelihood” of your model  
parameters (weights) having the right values. Maximizing  
the likelihood is the same as maximizing the log-likelihood,  
which is the same as minimizing the negative-log-likelihood.  
For the classification problem, the cross-entropy is the  
negative-log-likelihood. (The “math” definition of cross-entropy  
applies to your output layer being a (discrete) probability  
distribution. Pytorch’s `CrossEntropyLoss` implicitly adds  
a soft-max that “normalizes” your output layer into such a  
probability distribution.)

Wikipedia has some explanation of the equivalence of  
negative-log-likelihood and [Cross entropy](https://en.wikipedia.org/wiki/Cross_entropy).

\*Possible duplicate:

> [@Is log\_softmax + NLLLoss == CrossEntropyLoss?](https://discuss.pytorch.org/t/is-log-softmax-nllloss-crossentropyloss/9352):
>
> If I’m not missing something, they should be the same. However, I tried the follow snippet, but they are not equal. #!/usr/bin/env python3 # -\*- coding: utf-8 -\*- import numpy as np import torch import torch.nn.functional as F from torch import nn from torch.autograd import Variable class Net(nn.Module): def \_\_init\_\_(self, n\_features, n\_hiddens, n\_classes): super(Net, self).\_\_init\_\_() self.gru = torch.nn.GRU(n\_features, n\_hiddens) self.linear = torch.nn.Linear(n\_…

Best.

K. Frank

---

<div class="post-metadata">

**Author:** ![DoubtWang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/doubtwang/32/8307_2.png) [@DoubtWang](https://discuss.pytorch.org/u/DoubtWang)\
**Post date:** [March 5, 2019, 12:41am UTC](https://discuss.pytorch.org/t/difference-between-cross-entropy-loss-or-log-likelihood-loss/38816/3 "2019-03-05T00:41:20Z")

</div>

Thanks,  
I have understood your words.

---

<div class="post-metadata">

**Author:** ![halahup](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/halahup/32/9100_2.png) [@halahup](https://discuss.pytorch.org/u/halahup)\
**Post date:** [March 5, 2019, 12:46am UTC](https://discuss.pytorch.org/t/difference-between-cross-entropy-loss-or-log-likelihood-loss/38816/4 "2019-03-05T00:46:59Z")

</div>

Hi, I have wrote a little post about KL divergence, Cross-Entropy and Negative Log-Likelihood Loss a few weeks ago: [https://medium.com/@stepanulyanin/notes-on-deep-learning-theory-part-1-data-generating-process-31fdda2c8941](https://medium.com/@stepanulyanin/notes-on-deep-learning-theory-part-1-data-generating-process-31fdda2c8941). I hope you can find a few answers there too.

---

<div class="post-metadata">

**Author:** ![rasbt](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rasbt/32/60_2.png) [@rasbt](https://discuss.pytorch.org/u/rasbt)\
**Post date:** [March 5, 2019, 1:24am UTC](https://discuss.pytorch.org/t/difference-between-cross-entropy-loss-or-log-likelihood-loss/38816/5 "2019-03-05T01:24:59Z")

</div>

As pointed out above, conceptually negative log likelihood and cross entropy are the same. And cross entropy is a generalization of binary cross entropy if you have multiple classes and use one-hot encoding. The confusion is mostly due to the naming in PyTorch namely that it expects different input representations. While it’s conceptually the same, the way it’s used implementation-wise is something to memorize/be aware of. I made a quick cheatsheet for my students because of that: [https://github.com/rasbt/stat479-deep-learning-ss19/blob/master/other/pytorch-lossfunc-cheatsheet.md](https://github.com/rasbt/stat479-deep-learning-ss19/blob/master/other/pytorch-lossfunc-cheatsheet.md)

---

<div class="post-metadata">

**Author:** ![DoubtWang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/doubtwang/32/8307_2.png) [@DoubtWang](https://discuss.pytorch.org/u/DoubtWang)\
**Post date:** [March 7, 2019, 8:51am UTC](https://discuss.pytorch.org/t/difference-between-cross-entropy-loss-or-log-likelihood-loss/38816/7 "2019-03-07T08:51:14Z")

</div>

I have come to understand this problem 😄

Thanks @K_Frank, @halahup and @rasbt.
