# Multi Label Classification Evaluation and Training Loop

Hi
I’m currently doing a multi label classification problem
As far as I know using BCELogitsLoss() function is used as a loss function for such type of problems

I have images and one hot vectors and the image ids as input
Example
imagetensor => tensor([[[-1.2406, -1.6744, -1.8826, …, -1.9694, -1.9347, -1.8826],
[-1.8306, -1.9694, -1.9867, …, -1.9867, -1.9867, -1.9867],
[-1.9867, -1.9867, -1.9867, …, -1.9867, -1.9867, -1.9867],
…,
[-1.9867, -1.9867, -1.9867, …, -1.9867, -1.9867, -1.9867],
[-1.9867, -1.9867, -1.9867, …, -1.9867, -1.9867, -1.9867],
[-1.7264, -1.8653, -1.8826, …, -1.9867, -1.9867, -1.9867]]])
one hot vector => [1. 0. 1. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]

This is the evaluation function

``````loss_fn = nn.BCEWithLogitsLoss()
opt = optim.SGD(resnet.parameters(), lr = 0.01)
``````
``````def evaluation(dataloader, model):
# model added as argument - 1 change from lenet
total, correct = 0,0
inputs, labels,ids = data
inputs, labels, ids = inputs.to(device), labels.to(device), ids.to(device)
# inputs are put in the gpu
outputs = model(inputs)
print(outputs)
# from net to model - 2 change from lenet
_, pred = torch.max(outputs.data,1)
print(pred)
total += labels.size(0)
correct +=(pred==labels).sum().item()
return 100 * correct/total
``````

Training loop

``````loss_epoch_arr = []
max_epochs = 10
min_loss = 1000

n_iters = np.ceil(len(trainset)/batch_size)

for epoch in range(max_epochs):
for i, data in enumerate(trainloader, 0):

inputs, labels , imgindex = data
inputs, labels, imgindex  = inputs.to(device), labels.to(device), imgindex.to(device)

#print(inputs.shape)

outputs = resnet(inputs)
loss = loss_fn(outputs, labels)
loss.backward()
opt.step()

if min_loss> loss.item():
min_loss = loss.item()
best_model = copy.deepcopy(resnet.state_dict())
print('Min loss %0.2f' % min_loss)

if i%100 == 0:
print('Iteration : %d/%d, Loss : %0.2f' % (i, n_iters, loss.item()))

del inputs, labels, imgindex, outputs
torch.cuda.empty_cache()

loss_epoch_arr.append(loss.item())

print('Epoch: %d/%d, Train acc : %0.2f, Test acc : %0.2f' % (epoch, max_epochs, evaluation(trainloader,resnet), evaluation(testloader,resnet)) )
plt.plot(loss_epoch_arr)
plt.show()
``````

I’m using resnet pretrained with imagenet weights and have modified the first layer to accept grey scale images and the number of output classes to 15 in the last layer.

On running the training loop, the print statement calling the evaluation(trainloader,resnet) causes an error, on running with the following runtime settings
CPU -> RuntimeError: The size of tensor a (250) must match the size of tensor b (15) at non-singleton dimension 1, the batch size I’m using is 250, 15 is number of classes

GPU -> 9 **
** 10 inputs, labels , imgindex = data

—> 11 inputs, labels, imgindex = inputs.to(device), labels.to(device), imgindex.to(device)

AttributeError: ‘tuple’ object has no attribute 'to’
I’m new to multi label classification…So any help would be appreciated. Hello Salome!

It looks as if you might be mixing together things for multi-label
(multi-class) and single-label (multi-class) problems.

One-hot vectors are typically be used to encode the labels for
a single-label, multi-class problem. (But you normally use integer
class labels instead of one-hot encoded labels.)

Note, this is not a one-hot vector because multiple elements have
the value `1`, not just one. (Also, as written, it has no comma
separators.)

This is how you would compute the number of correct predictions is
a single-label, multi-class problem. Note that `pred`, the second
element of the tuple returned by `torch.max()` is `argmax()`, that is,
the index of the largest value (along dimension 1) in `output`. This
would be a single class label for your sample, not a set of multiple
active class labels.

Could you print out the `shape` and `type()` of `inputs`, `labels`,
`imgindex`, and `outputs`? Could you also show the code for the last
`Linear` layer of your model and any activations that follow it?

Best.

K. Frank

Hi K. Frank,
I want to do a multi-label (multiple class) problem
one sample can belong to more than 1 class
so this means [1. 0. 1. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.] that the sample belongs to 3 classes

(tensor([[[-1.9000, -1.9000, -1.9000, …, -1.9000, -1.9000, -1.9000],
[-1.9000, -1.9000, -1.9000, …, -1.8826, -1.8826, -1.8826],
[-1.9000, -1.9000, -1.9000, …, -1.8826, -1.8826, -1.8826],
…,
[ 0.2864, 0.5467, 0.6334, …, -0.5118, -0.4771, -0.5465],
[ 0.2864, 0.5293, 0.6508, …, -0.5292, -0.4597, -0.5118],
[ 0.0955, 0.2864, 0.4079, …, -0.6506, -0.5986, -0.6159]]]), array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
dtype=float32), ‘00001250_006.png’)

this is a tuple(input_img tensor, output_vector, image_id)

Image type and shape => <class ‘torch.Tensor’> torch.Size([1, 224, 224])
Output type and shape =><class ‘numpy.ndarray’> (15,)
Image id/imgindex(name of the image file) type=><class ‘str’>

(1): BasicBlock(
(conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(fc): Linear(in_features=512, out_features=15, bias=True)
)

Last layers of Resnet

And I also want to get the probabilities for the predicted classes like 70% it belongs to class1, 50% it belongs to class 5, etc for a given sample image.

Hi Salome!

Well, this is confusing. In the code you posted, you had the variables
`inputs`, `labels`, and `imgindex`. But now you seem to be calling
them “input_img tensor, output_vector, image_id”. I will guess that
these are the same things.

Now I’m guessing that “Image” is what you just called “input_img
tensor", “Output” is “output_vector”, and “Image id/imgindex(name
of the image file)” is “image_id”.

(And I’m also guessing that you haven’t given the `shape` and `type()`
of the variable `outputs` in your code, that is, the object returned by
your `model`.)

So it looks like what your code calls `inputs` is a single 224x224 image
with a leading singleton (`size = 1`) dimension, and that what your
code calls `labels` is a single numpy array of (floating-point) binary
labels, one for each of your fifteen classes. Note, I don’t see your
batch size of 250 anywhere.

Your `inputs` tensor should be a batch of `250` images. It should
most likely be a tensor of `shape` `[250, 224, 244]`, unless your
`model` is expecting its input images to carry a “channel” dimension
of `nChannel = 1`, in which case your input tensor should have
`shape` `[250, 1, 224, 244]`.

Your `labels` tensor should be a pytorch tensor (not a numpy array),
and should be a batch of sets of (floating-point) binary class labels,
one for each of your 15 classes, and therefore should have `shape`
`[250, 15]`.

Although you didn’t provide the information for what your `outputs`
variable is, logically, and based on the last layer of your `model`, it
should have `shape` `[batch size, nClass]`, that is, `[250, 15]`.
It will be your `model`'s predicted raw-score logits for each of your
250 samples being in each of your 15 classes.

This is what your data should look like. But it looks like the `shape` of
your `model` `outputs` and the shape of your `labels` aren’t matching
up properly, presumably somewhere in your `evaluation()` function.

Could you call something like

``````trainAcc = evaluation (trainloader, resnet)
print (trainAcc)
print (testAcc)
``````

before you run any training? (And call `evaluation()` in two separate
lines of code, as above.)

Good luck.

K. Frank

Hi K. Frank,

print(‘Epoch: %d/%d, Train acc : %0.2f, Test acc : %0.2f’ % (epoch, max_epochs, evaluation(trainloader,resnet), evaluation(testloader,resnet)) )

when i run this, i get the following output

I’m printing outputs = model(inputs)
print(outputs)
_, pred = torch.max(outputs.data,15)
print(pred) within the evaluation function

## tensor([[-1.3976, -1.9027, -0.8711, …, -1.4458, -1.2116, -0.8598], [-0.3187, -0.8117, -1.0978, …, -2.2132, -1.3070, -0.9956], [-0.8397, -1.6915, -0.6634, …, -1.8321, -1.0346, -1.2636], …, [-1.4526, -1.1693, -1.0159, …, -1.3135, -1.1257, -0.9665], [-1.3092, -1.4912, -1.3867, …, -2.0305, -1.1680, -1.3192], [-0.5745, -1.1265, -0.9143, …, -2.1783, -0.6507, -1.4715]], grad_fn=) tensor([ 3, 3, 5, 3, 3, 3, 3, 7, 3, 3, 3, 3, 3, 3, 7, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 13, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 9, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 3, 3, 5, 3, 3, 3, 3, 3, 3, 3, 3, 3, 8, 5, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 8, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 5, 3, 5, 3, 3, 3, 3, 3, 3, 3, 3, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 8, 3, 3, 3, 3, 3, 5, 2, 3, 3, 3, 3, 3, 3, 5, 5, 3, 2, 3, 3, 3, 3, 3, 9, 3, 3, 3, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 13, 5, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 8, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 3, 3, 3, 3, 3, 3, 3])

RuntimeError Traceback (most recent call last)
in ()
----> 1 print(‘Epoch: %d/%d, Train acc : %0.2f, Test acc : %0.2f’ % (epoch, max_epochs, evaluation(trainloader,resnet), evaluation(testloader,resnet)) )

1 frames
12 print(pred)
13 total += labels.size(0)
—> 14 correct +=(pred==labels).sum().item()
15 return 100 * correct/total
16 #preds = torch.tensor([])

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in wrapped(*args, **kwargs)
26 def wrapped(*args, **kwargs):
27 try:
—> 28 return f(*args, **kwargs)
29 except TypeError:
30 return NotImplemented

RuntimeError: The size of tensor a (250) must match the size of tensor b (15) at non-singleton dimension 1

The dataset is imbalanced hence it predicts ‘3’ in most of the cases as there are more samples of class corresponding to the index ‘3’

I think from the output that I get it is of shape [250, 1] and not [250, 15]. It should be [250, 15]. I’m not understanding where I have to make the change within the training loop or the evaluation function.

Yes it is the same thing

Ya this is just for a single sample and not for the batch

So do I need to change the last year out_features to [250, 15] or something else?

Hello Salome!

It’s very hard to know what you are doing. You post some code, and
then you post some data that might or might not be related, and the
data you post has been renamed and edited.

I’ve asked you to post some specific values. You post things that might
or might not be related, but usually differ at least somewhat from what

You don’t reflect on things I’ve tried to explain.

For example, in my first reply I noted that using:

``````    _, pred = torch.max(outputs.data,1)
``````

to convert your `outputs` to `pred` is appropriate for “a single-label,
multi-class problem.” You reiterate that yours is a multi-label problem.
But the tensor you just posted for `pred` looks like the result of the
`torch.max()` statement. But I don’t know, because I can only guess
what code produced the `pred` result you just posted.

No.

It is important to understand that you write pytorch models from the
perspective of processing a single sample. But the pytorch framework
actually processes samples in batches.

So, for example, if you write your model to take as input a tensor of
`shape` `[244, 244]` (that is a single sample), and produce as output
a tensor of `shape` `` (that are the fifteen binary class labels it
predicts for that single sample), then when you run your model, you
have to pass in a batch of samples of `shape` `[nBatch, 244, 244]`,
and the model will return a batch of predicted class labels of `shape`
`[nBatch, 15]`.

Note, the value of `nBatch` is not wired into your model. You can pass
in a batch of 250 (`[250, 244, 244]`), getting out a batch of 250
(`[250, 15]`), and then, without changing the model pass in a batch
of 1 (`[1, 244, 244]` --> `[1, 15]`). (Also note, that if you want to apply
your model to a single sample, that sample has to be wrapped in
a batch of 1. That is, you can’t pass in `[244, 244]`; you have to pass
in `[1, 244, 244]`).

So, to repeat, your batch size of 250 does not appear in the code for
your model. And even though the last linear layer looks as if it outputs
a tensor of shape `` (`out_features=15`), because pytorch models
always work with batches, your model will output a tensor of `shape`
`[nBatch, 15]`, where `nBatch` is the size of the batch you passed

Good luck.

K. Frank

inputs.shape & type torch.Size([64, 1, 224, 224]) <class ‘torch.Tensor’>
labels.shape & type torch.Size([64, 15]) <class ‘torch.Tensor’>
imgindex & type (‘00001151_002.png’, ‘00000583_010.png’, ‘00000500_000.png’, ‘00000271_000.png’, …, ‘00000032_031.png’, ‘00000913_000.png’, ‘00001075_024.png’)
outputs, shape & type tensor([[-2.1863, -2.8136, -2.0755, 0.1303, -2.4541, -2.1647, -3.0948, -2.5282,
-1.9832, -2.4096, -2.6687, -2.7546, -2.1767, -2.6486, -3.0040],
[-2.3346, -1.7464, -1.3218, -0.2523, -1.8560, -0.8745, -1.6130, -1.5004,
-1.5930, -1.9782, -2.1910, -2.3470, -1.8963, -2.1992, -1.6291],
[-2.0328, -2.0936, -1.9872, 0.6333, -2.5061, -1.2418, -2.6371, -3.0564,
-2.0829, -2.0732, -2.0986, -2.5197, -2.8750, -2.4518, -2.9555],

[-2.5923, -2.6953, -1.5379, 0.2798, -2.7280, -1.5506, -2.4956, -1.8402,
-2.2393, -2.3018, -2.3598, -1.9540, -3.1476, -2.3570, -1.6612]],
device=‘cuda:0’, grad_fn=) torch.Size([64, 15]) <built-in method type of Tensor object at 0x7feedc76ec18>

Hi K. Frank,
The value of nBatch is being incorporated in the model. Currently I have set nBatch to 64.

## After 1 epoch is completed When I run this statement evaluation(trainloader, resnet)

I get the following error
pred : tensor([3, 3, 3, 3, 3, 3, 3, 5, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3], device=‘cuda:0’)

RuntimeError Traceback (most recent call last)
in ()
35 loss_epoch_arr.append(loss.item())
36
—> 37 print(‘Epoch: %d/%d, Train acc : %0.2f’ % (epoch, max_epochs, evaluation(trainloader,resnet)) )
38 plt.plot(loss_epoch_arr)
39 plt.show()

1 frames
/usr/local/lib/python3.6/dist-packages/torch/tensor.py in wrapped(*args, **kwargs)
26 def wrapped(*args, **kwargs):
27 try:
—> 28 return f(*args, **kwargs)
29 except TypeError:
30 return NotImplemented

RuntimeError: The size of tensor a (64) must match the size of tensor b (15) at non-singleton dimension 1

Actually I’m doing multilabel(multi-class) classifcation for the first time, so I dont know what changes need to be made in the evaluation function…I just know that BCEWithLogitLoss/BCELoss is used

Can someone help me out with this…cause its urgent It seems that your evaluation code creates a shape mismatch.
For a multi-label classification the shape of the model output and target should be equal, while it seems that one tensor might be missing the batch size.
Could you print the shapes of the model output and target before the error is raised (in the validation code)?

Hi @ptrblck!

Her problem (among other things) is that she calls `argmax()` to get
a single class label (for the accuracy calculation in the evaluation
code), but it’s a multi-label problem, so she needs a vector of length
`nClass` binary predictions.

I wasn’t able to explain this clearly.

Best.

K. Frank

1 Like

Hi @KFrank!

Ah OK, didn’t realize it and the posted code snippets were a bit hard to follow.

@ssalome
You can post code snippets by wrapping them into three backticks ```, which makes debugging easier 