Alternating runtime errors: RuntimeError: expected scalar type Long but found Float and RuntimeError: Expected floating point type for target with class probabilities, got Long

#1

Hi, I am currently training a NN to classify inputs into 199 categories using pytorch cross_entropy loss. If I try using the probabilities for each class as the targets and cast the targets to float:

def cross_entropy_loss(sender_input, _message, _receiver_input, receiver_output, _labels, _aux_input=None):
    _labels =  F.one_hot(_labels.long(),receiver_output.shape[-1])
    loss = F.cross_entropy(receiver_output.squeeze(), _labels.float(), reduction='none',label_smoothing=0.1)
    return loss, {}

I get the output for one epoch:

{"loss": 5.204253673553467, "baseline": 5.205080986022949, "sender_entropy": 0.001109135104343295, "receiver_entropy": 0.0, "mode": "train", "epoch": 1}

and then the following error:

<ipython-input-91-936be60121ba> in cross_entropy_loss(sender_input, _message, _receiver_input, receiver_output, _labels, _aux_input)
      5 def cross_entropy_loss(sender_input, _message, _receiver_input, receiver_output, _labels, _aux_input=None):
      6     _labels =  F.one_hot(_labels.long(),receiver_output.shape[-1])
----> 7     loss = F.cross_entropy(receiver_output.squeeze(), _labels.float(), reduction='none',label_smoothing=0.1)
      8     return loss, {}

~\Anaconda3\envs\egg36\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   2844     if size_average is not None or reduce is not None:
   2845         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2846     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
   2847 
   2848 

RuntimeError: expected scalar type Long but found Float

If, on the other hand, I cast the target to long:

def cross_entropy_loss(sender_input, _message, _receiver_input, receiver_output, _labels, _aux_input=None):
    _labels =  F.one_hot(_labels.long(),receiver_output.shape[-1])
    loss = F.cross_entropy(receiver_output.squeeze(), _labels.long(), reduction='none',label_smoothing=0.1)
    return loss, {}

I inmediately get the following error:

<ipython-input-94-e319bc27aeaa> in cross_entropy_loss(sender_input, _message, _receiver_input, receiver_output, _labels, _aux_input)
      5 def cross_entropy_loss(sender_input, _message, _receiver_input, receiver_output, _labels, _aux_input=None):
      6     _labels =  F.one_hot(_labels.long(),receiver_output.shape[-1])
----> 7     loss = F.cross_entropy(receiver_output.squeeze(), _labels.long(), reduction='none',label_smoothing=0.1)
      8     return loss, {}

~\Anaconda3\envs\egg36\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   2844     if size_average is not None or reduce is not None:
   2845         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2846     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
   2847 
   2848 

RuntimeError: Expected floating point type for target with class probabilities, got Long

As you can see, in both cases the error is being triggered by the loss function I defined (in line 7), and internally in line 2846 of the torch.nn.functional. And it is asking for contradictory target formats.

If I change the loss function to work with index targets instead of probabilities:

def cross_entropy_loss(sender_input, _message, _receiver_input, receiver_output, _labels, _aux_input=None):
    #_labels =  F.one_hot(_labels.long(),receiver_output.shape[-1])
    loss = F.cross_entropy(receiver_output.squeeze(), _labels.long(), reduction='none',label_smoothing=0.1)
    return loss, {}

I also get the first batch output, but after that I get this error (in the same line as the others):

~\Anaconda3\envs\egg36\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   2844     if size_average is not None or reduce is not None:
   2845         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2846     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
   2847 
   2848 

RuntimeError: Expected target size [32, 199], got [32]

Which acording to the documentation should not be a problem for index targets:

image
image726×233 21.2 KB

I am completely at a loss of what is happening here, I tried the same exact loss with reduction='mean' and it worked fine, so maybe the issue is there. Still I need this part to have reduction='none' so I have to fix it.

#3

Could you check which PyTorch version you are using?
If I’m not mistaken nn.CrossEntropyLoss accepts probabilities as a FloatTensor for the targets since 1.10.0.

#4

Hi, I am using version 1.10.1+cpu, but as you can see in the first block of code, I use _labels.float() as targets, It runs for one epoch and then it outputs the error in the third block of code.

#5

If the code runs for some iterations and raises an error later, check the shape as well as dtype of all used tensors as one of them seems to have changed.