Multi-target not supported, Multi-class classification

#1

Hi,
I have a problem when I want to obtain a loss value.
Firstly, for example, I design my own DataLoader including 3 inputs (Each input has 4 one-hot vector) and 3 outputs (Each output has a vector with size = 33), The problem is a multiclass classification (just one class must be activated!):

class MyDataset(Dataset):
def init(self):
self.X = Variable(torch.Tensor([
[[0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1], [0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],

[[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0]],

[[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0]]
]))
self.Y = Variable(torch.Tensor([[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0], [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0]]))
self.len = 3

def __getitem__(self, index):
    return self.X[index], self.Y[index]

def __len__(self):
    return self.len

and in the forward() method of the Model class, the return line is a 33-dimensional vector :
Z = torch.nn.functional.softmax(self.linear_final(Z), dim=1)

and with the following criterion:
loss = criterion(y_hat, labels.long())
where,
criterion = torch.nn.CrossEntropyLoss()
Then, it launches the following error:
RuntimeError: multi-target not supported at /opt/conda/conda-bld/pytorch-cpu_1556653093101/work/aten/src/THNN/generic/ClassNLLCriterion.c:20
Does anyone knows the problem?
When I print the y_hat and the labels, I get the following result (batch_size=2):

tensor([[0.0126, 0.0251, 0.0233, 0.0266, 0.0150, 0.0274, 0.0565, 0.0332, 0.0490,
0.0282, 0.0231, 0.0313, 0.0362, 0.0102, 0.0111, 0.0923, 0.0161, 0.0842,
0.0192, 0.0235, 0.0163, 0.0274, 0.0200, 0.0190, 0.0223, 0.0361, 0.0098,
0.0532, 0.0126, 0.0279, 0.0771, 0.0254, 0.0085],
[0.0276, 0.0481, 0.0292, 0.0282, 0.0292, 0.0101, 0.0470, 0.0183, 0.0195,
0.0154, 0.0347, 0.0329, 0.0257, 0.0327, 0.0100, 0.0998, 0.0284, 0.0368,
0.0137, 0.0426, 0.0376, 0.0172, 0.0099, 0.0166, 0.0305, 0.0196, 0.0275,
0.0179, 0.0287, 0.0394, 0.0295, 0.0697, 0.0259]],
grad_fn=)
tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])

Thank you very much in advance.

#2

Hello Sina!

The problem is that CrossEntropyLoss takes “labels” (often
called “targets”) that are integer class labels (in the range
[0, nClass), i.e., 0 <= label < nClass), rather than
one-hot encoded vectors.

Your example y_hat and labels correspond to a batch size
of two, and a number of classes nClass = 33. Therefore a
batch of labels should be a tensor of shape (2, ), i.e., just
two class-label integers in the range [0, 33). You are passing
CrossEntropyLoss a (one-hot encoded)labels vector of shape
(2, 33). (Your y_hat has the correct shape, (2, 33), i.e.,
(batchSize, nClass), that is expected by CrossEntropyLoss.)

Edit:

Also, CrossEntropyLoss expects “raw scores” (logits) for its
predictions (your y_hat), not probabilities. So you do not want
a softmax() for the output of your model. Just use the output
of your last linear layer:

Z = self.linear_final(Z)
return Z

rather than:

Z = torch.nn.functional.softmax(self.linear_final(Z), dim=1)
return Z

Best.

K. Frank

#3

Thanks Frank for your help.

#4

Dear Frank,
How about for testing phase?
How can I find accuracy and error for test set?

#5

Hello Sina!

The conventional way to calculate the accuracy:

y_hat_test = model (inputs_test)
dummy, preds_test = torch.max (y_hat_test, dim = 1)
accuracy_test = (preds_test == labels_test).long().sum().float() /  preds_test.size()[0]

That is, you need to convert the raw scores (logits) for each
class output by your model into predictions for a single, specific
class. Usually you just use the class with the largest logit as
the prediction.

(In newer versions of pytorch you can use the torch.argmax()
function, preds_test = torch.argmax (y_hat_test, dim = 1)
(and dispense with the explicit conversion to .long() before the
.sum()).

I’m not sure what you mean by test-set error. If you mean your loss
function,you calculate it the same way you would for a training batch
(but wrap it in a volatile = True Variable, or, with newer versions,
use with torch.no_grad():).

Note, you do not want softmax() as the final layer of your model.
I’ve edited my previous post to clarify this.

Good luck.

K. Frank