How can I perform "Weighted NLLLoss" calculations manually?

1

Hello
Im wonder why NLLLoss doesn’t change when using [2,2] weights in example below?
There is really incomplete documentation related to details of calculations.

log_prob = torch.tensor([[0.1, 0.5],
                         [0.8, 0.8],
                         [0.8, 0.1]])

target = torch.tensor([0, 0, 1])

weight = torch.tensor([2.0, 2.0])

criterion = nn.NLLLoss()
criterion_weighted = nn.NLLLoss(weight=weight)

print('NLLLoss = {}'.format(criterion(log_prob, target)))
#NLLLoss = -0.3333333432674408

print('Weighted NLLLoss = {}'.format(criterion_weighted(log_prob, target)))
#Weighted NLLLoss = -0.3333333432674408

print('Manualy calculated weighted NLLLoss = {}'.format((-2*np.log(0.1) - 2*np.log(0.8) - 2*np.log(0.1)) / (3 * 4)))
Manualy calculated weighted NLLLoss = 0.8047189562170501

I guess there is some mistakes in my manual calculations. Any suggestion how can I calculate weighted NLLLoss manuall?

2

Hi Андрей!

You have two mistakes in your calculation: First, NLLLoss does not
take the log() of its input, but you do in your manual calculation.
Second, when you manually calculate the weighted average, you
should be dividing by 3 * 2 (not (3 * 4)). Thus:

>>> import math
>>> import torch
>>> torch.__version__
'1.9.0'
>>> log_prob = torch.tensor([[0.1, 0.5],
...                          [0.8, 0.8],
...                          [0.8, 0.1]])
>>> target = torch.tensor([0, 0, 1])
>>> weight = torch.tensor([2.0, 2.0])
>>> criterion = torch.nn.NLLLoss()
>>> criterion_weighted = torch.nn.NLLLoss(weight=weight)
>>> criterion (log_prob, target)
tensor(-0.3333)
>>> criterion_weighted (log_prob, target)
tensor(-0.3333)
>>> (-2 * math.log (0.1) - 2 * math.log (0.8) - 2 * math.log (0.1)) / (3 * 4)
0.8047189562170501
>>> (-2 * (0.1) - 2 * (0.8) - 2 * (0.1)) / (3 * 2)
-0.3333333333333333

Best.

K. Frank

1 Like