'RuntimeError: Expected object of scalar type Long but got scalar' for torch.nn.CrossEntropyLoss()

Hi
I’m using this loss function for xlm-roberta-large-longformer and it gives me this error:

    import torch.nn.functional as f
    from scipy.special import softmax
    
    loss_func = torch.nn.CrossEntropyLoss()
    output = torch.softmax(logits.view(-1,num_labels), dim=0).float()
    target = b_labels.type_as(logits).view(-1,num_labels)
    loss = loss_func(output, target)
    train_loss_set.append(loss.item()) 

when I try

b_labels.type_as(logits).view(-1,num_labels).long()

it tells me

RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15

What should I do?

For a multi-class classification use case nn.CrossEntropyLoss expects logits in the shape [batch_size, nb_classes, *] and targets as a LongTensor in [batch_size, *] containing class indices in [0, nb_clsses-1]. Newer PyTorch versions also allow you to pass “soft” labels as the target, but let’s skip it here.
The error indicates that your target shape is wrong which is also shown via:

target = b_labels.type_as(logits).view(-1,num_labels)

since the target should not contain a class dimension as previously explained.