Needing clarity for equivalent of Categoricalcrossentropy as CrossEntropyLoss

1

I wanted to have CategoricalCrossentropyLoss as performed in TensorFlow but while
I’ve read answers from discuss but it’s I am not getting clearly what to do

here is what I tried

my outputs =

output_truth = np.ones((1,1024,15))

if we tried to find loss for above shape using 
tf.keras.losses.CategoricalCrossentropy()(output_from_model, output_truth) 
# we get answer with no error 

but if we same tried with pytorch 
torch.nn.CrossEntropyLoss()(torch.tensor(output_from_model) torch.tensor(output_truth))

we get following error

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_33/1032697456.py in <module>
----> 1 torch.nn.CrossEntropyLoss()(torch.tensor(tf_outs.numpy()), torch.tensor(x[2].numpy()))

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
   1119     def forward(self, input: Tensor, target: Tensor) -> Tensor:
   1120         return F.cross_entropy(input, target, weight=self.weight,
-> 1121                                ignore_index=self.ignore_index, reduction=self.reduction)
   1122 
   1123 

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2822     if size_average is not None or reduce is not None:
   2823         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2824     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2825 
   2826 

RuntimeError: Expected target size [1, 15], got [1, 1024, 15]
2

image
image698×205 25.7 KB

read CrossEntropyLoss.
if you want target of shape 1x15x1024, then your model output should have 1xCx15x1024 shape.

3

15 were the number of classes in my case

4

then reshape target to be of size 1x15x1024
and model output of size 1x15x1024

If target containing class probabilities, same shape as the input.

5

the output and target are already having same shape

6

I am just not getting why it’s throwing me an error

7 
import torch

import torch.nn as nn
t = torch.ones(1,1024,15)
o = torch.randn(1,1024,15)
loss_fn = nn.CrossEntropyLoss()
loss_fn(o,t)

i see no error.

8

while I was trying I got this

image
image1243×438 88.8 KB

9

I even tried on my local environment still getting that error

image
image1072×491 85.1 KB

what’s the pytorch version you are using ?

10

check torch version. 1.10

torch.__version__
11

1.9.0+cu111
it’s my version

12

it seems that only crossentorpy in version 1.10 can get Probabilities for each class as target.

13

ohh I see thanks for help I think I can get that running
the problem was with my version

again there is no space left on my device so have to install that after freeing up some more space

14 

from torch.nn.modules.loss import _WeightedLoss
import torch.nn.functional as F

class SmoothCrossEntropyLoss(_WeightedLoss):
    def __init__(self, weight=None, reduction='mean'):
        super().__init__(weight=weight, reduction=reduction)
        self.weight = weight
        self.reduction = reduction

    def reduce_loss(self, loss):
        return loss.mean() if self.reduction == 'mean' else loss.sum() \
        if self.reduction == 'sum' else loss

    def forward(self, inputs, targets):

        log_preds = F.log_softmax(inputs, -1)

        if self.weight is not None:
            log_preds = log_preds * self.weight.unsqueeze(0)

        return self.reduce_loss(-(targets * log_preds).sum(dim=-1))

maybe work. ¯\_(ツ)_/¯
class Probabilities shoud be at last dim.

import torch
import torch.nn as nn
t = torch.ones(1,1024,15)
o = torch.randn(1,1024,15)
loss_fn = SmoothCrossEntropyLoss()
loss_fn(o,t) #-> 47.3582
loss_fn1 = nn.CrossEntropyLoss()
loss_fn1(o.transpose(1,2),t.transpose(1,2)) #-> 47.3582