Nl_loss expects different targets size

nlp
1

I am trying to train my neuronal network. Train in the model is correct, but I can’t calculate loss. The output and the target have the same dimension.

I had tried to use torch.stack, but I can’t because the size of the each input is (252, x) where x is the same in the 252 elements, but is different for the others inputs.

I use a custom Dataset:

class MusicDataSet(Dataset):
def __init__(self, transform=None):
    self.ms, self.target, self.tam = sd.cargarDatos()  
    self.mean, self.std = self.NormalizationValues()                    
def __len__(self):
    return self.tam

def __getitem__(self, idx):
    #Normalize
    inp = (self.ms[idx]-self.mean)/self.std
    inp = torch.from_numpy(inp).float()    
    inp = inp.t()
    inp = inp.to('cuda')

    target= torch.from_numpy(self.target[idx])
    target = target.long()
    target = target.t()
    target = target.to('cuda')

    return inp, target

I must say that list can’t be cast with something like: target = torch.Tensor() or torch.stack() because this (252, x), as I have already said.

def music_collate_fn(batch):
    data = [item[0] for item in batch]
    data = pad_sequence(data, batch_first=True)
    target = [item[0] for item in batch]
    target = pad_sequence(target, batch_first=True)
    return data, target


musicSet = mds.MusicDataSet()
train_loader = torch.utils.data.DataLoader(musicSet,batch_size=50, collate_fn = music_collate_fn, shuffle=False)

input_dim = 252
hidden_dim = (512,1024,512)
output_dim = 88
mlp = rn.MLP(input_dim, hidden_dim, output_dim).to(device)

optimizer = torch.optim.RMSprop(mlp.parameters(), lr = learning_rate)
criterion = nn.CrossEntropyLoss()
for batch_idx, (x,y) in enumerate(train_loader):
    outputs = mlp(x.to(device))
    loss = criterion(outputs, y)
    optimizer.zero_grad()           
    loss.backward()                 
    optimizer.step()

The size of output and target is the same:

output: torch.Size([50, 288, 88])
target:  torch.Size([50, 288, 88])

But the next error apears when I try to calculate loss:

 File "<ipython-input-205-3c47d7aa11a4>", line 32, in <module>
    loss = criterion(outputs, y)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 904, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\functional.py", line 1970, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch
\nn\functional.py", line 1800, in nll_loss
    out_size, target.size()))

ValueError: Expected target size (50, 88), got torch.Size([50, 288, 88])
2

nn.NLLLoss is used for a multi-class classification use case.
While the model output should contain the log probabilities for all classes, the target should be a LongTensor containing the class indices.
Based on the shape it looks like your target might be a one-hot encoded tensor.
If that’s the case (and the number of classes is encoded in dim1), you could use

target = torch.argmax(target, 1)

to get the class indices in the right shape.

3

My target is not encoded.

I am trying to classify isolated musical notes(88 exactly). Each note has input of 252 tensors, and each tensor has a variable size. That is way I can’t use the default collate_fn. You can see in getitem() function target is convert to LongTensor.

The target of each note is a tensor. If we have the note 0, tensor number 0 is filled with 1 and the rest of tensors are filled with 0. Target for note 0:

tensor([[1., 0., 0., …, 0., 0., 0.],
[1., 0., 0., …, 0., 0., 0.],
[1., 0., 0., …, 0., 0., 0.],
…,
[1., 0., 0., …, 0., 0., 0.],
[1., 0., 0., …, 0., 0., 0.],
[1., 0., 0., …, 0., 0., 0.]])

Thanks so much!

4

Thanks for the information!
Based on the description, I assume the output of [50, 288, 88] corresponds to [batch_size, seq_length, nb_classes]. Is this correct?

If so, the class dimension should be in dim1: [batch_size, nb_classes, seq_length].
Also, it does look as if your target is one-hot encoded, if you are setting a 1 for the active class.
If that’s the case, permute the target like the output (target = target.permute(0, 2, 1)) and use torch.argmax(target, 1) to get the class index.

5

Hi @ptrblck
I have a question about loss. CrossEntropyLoss gives loss of all of classes, but I want to get loss of class 1 ,how can I get loss of each class?

N, C = 5, 4
loss = nn.CrossEntropyLoss()
data = torch.randn(N, 16, 10, 10)
conv = nn.Conv2d(16, C, (3, 3))
target = torch.empty(N, 8, 8, dtype=torch.long).random_(0, C)
pr=conv(data)
output = loss(pr, target)
output.backward()
6

Would using nn.CrossEntropyLoss(reduction='none') work for you?
This will return the unreduced loss and you could then print or process the row corresponding to your desired label.

7

It works without loss.backward() and the shape of output for following code is (8,64,64,64), but I can’t get loss of each class.

import torch
N, C = 8, 21
loss = torch.nn.CrossEntropyLoss(reduction='none')
data = torch.randn(8,21 ,64, 64, 64)
target = torch.empty(8, 64, 64,64, dtype=torch.long).random_(0, 21)
# target.requires_grad = True
output = loss(data, target)
# output.backward()
print(output.shape)
### torch.Size([8, 64, 64, 64])
8

You could create masks based on the target and index the loss with it to get the losses for all classes in the current batch.

9

Thanks for reply.
I write this code ,is it correct?
I have another question, if i use some losses in my code, is it necessary to use loss.backward() for each of losses?

import torch
N, C = 8, 21
loss = torch.nn.MSELoss()
data = torch.randn(8,21 ,64, 64, 64)
target = torch.empty(8, 64, 64,64, dtype=torch.long).random_(0, 21)

_, preds_argmax = torch.max(data.data, 1)
labels3 = target.eq(3).float()
preds3 = preds_argmax.eq(3).float()
loss1 = loss(preds3, labels3)
# loss for class 3
print(loss1)
10

This code will most likely not work as intended, since you are passing the prediction indices after torch.max to the criterion, which will break the computation graph.
Also, don’t use the .data attribute.

Could you please give me an example, what the loss of a particular class would look like?
Do you want to get the predictions of your model for class 3 and compare it as a binary classification?

11

I use CrossEntropy for calculating loss of 21 classes.(my first post in this topic).
But in my project , I must calculate loss of each class(for example class 1).
And for it,I use MSEloss for it.
I extract the parameters of class 1 in preds and target,then I used binary loss for it.
this post shows my project better.