Multi Label Classification Evaluation and Training Loop

Hi
I’m currently doing a multi label classification problem
As far as I know using BCELogitsLoss() function is used as a loss function for such type of problems

I have images and one hot vectors and the image ids as input
Example
imagetensor => tensor([[[-1.2406, -1.6744, -1.8826, …, -1.9694, -1.9347, -1.8826],
[-1.8306, -1.9694, -1.9867, …, -1.9867, -1.9867, -1.9867],
[-1.9867, -1.9867, -1.9867, …, -1.9867, -1.9867, -1.9867],
…,
[-1.9867, -1.9867, -1.9867, …, -1.9867, -1.9867, -1.9867],
[-1.9867, -1.9867, -1.9867, …, -1.9867, -1.9867, -1.9867],
[-1.7264, -1.8653, -1.8826, …, -1.9867, -1.9867, -1.9867]]])
one hot vector => [1. 0. 1. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]

This is the evaluation function

loss_fn = nn.BCEWithLogitsLoss()
opt = optim.SGD(resnet.parameters(), lr = 0.01)
def evaluation(dataloader, model):
  # model added as argument - 1 change from lenet
  total, correct = 0,0
  for data in dataloader:
    inputs, labels,ids = data
    inputs, labels, ids = inputs.to(device), labels.to(device), ids.to(device)
    # inputs are put in the gpu
    outputs = model(inputs)
    print(outputs)
    # from net to model - 2 change from lenet
    _, pred = torch.max(outputs.data,1)
    print(pred)
    total += labels.size(0)
    correct +=(pred==labels).sum().item()
  return 100 * correct/total

Training loop

loss_epoch_arr = []
max_epochs = 10
min_loss = 1000

n_iters = np.ceil(len(trainset)/batch_size)

for epoch in range(max_epochs):
    for i, data in enumerate(trainloader, 0):

        inputs, labels , imgindex = data
        inputs, labels, imgindex  = inputs.to(device), labels.to(device), imgindex.to(device)

        #print(inputs.shape)
        opt.zero_grad()

        outputs = resnet(inputs)
        loss = loss_fn(outputs, labels)
        loss.backward()
        opt.step()

        if min_loss> loss.item():
            min_loss = loss.item()
            best_model = copy.deepcopy(resnet.state_dict())
            print('Min loss %0.2f' % min_loss)

        if i%100 == 0:
            print('Iteration : %d/%d, Loss : %0.2f' % (i, n_iters, loss.item()))
    
        del inputs, labels, imgindex, outputs
        torch.cuda.empty_cache()

    loss_epoch_arr.append(loss.item())

    print('Epoch: %d/%d, Train acc : %0.2f, Test acc : %0.2f' % (epoch, max_epochs, evaluation(trainloader,resnet), evaluation(testloader,resnet)) )   
plt.plot(loss_epoch_arr)
plt.show()

I’m using resnet pretrained with imagenet weights and have modified the first layer to accept grey scale images and the number of output classes to 15 in the last layer.

On running the training loop, the print statement calling the evaluation(trainloader,resnet) causes an error, on running with the following runtime settings
CPU -> RuntimeError: The size of tensor a (250) must match the size of tensor b (15) at non-singleton dimension 1, the batch size I’m using is 250, 15 is number of classes

GPU -> 9 **
** 10 inputs, labels , imgindex = data

—> 11 inputs, labels, imgindex = inputs.to(device), labels.to(device), imgindex.to(device)

AttributeError: ‘tuple’ object has no attribute 'to’
I’m new to multi label classification…So any help would be appreciated. :smiley:

Hello Salome!

It looks as if you might be mixing together things for multi-label
(multi-class) and single-label (multi-class) problems.

One-hot vectors are typically be used to encode the labels for
a single-label, multi-class problem. (But you normally use integer
class labels instead of one-hot encoded labels.)

Note, this is not a one-hot vector because multiple elements have
the value 1, not just one. (Also, as written, it has no comma
separators.)

This is how you would compute the number of correct predictions is
a single-label, multi-class problem. Note that pred, the second
element of the tuple returned by torch.max() is argmax(), that is,
the index of the largest value (along dimension 1) in output. This
would be a single class label for your sample, not a set of multiple
active class labels.

Could you print out the shape and type() of inputs, labels,
imgindex, and outputs? Could you also show the code for the last
Linear layer of your model and any activations that follow it?

Best.

K. Frank

Hi K. Frank,
I want to do a multi-label (multiple class) problem
one sample can belong to more than 1 class
so this means [1. 0. 1. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.] that the sample belongs to 3 classes

(tensor([[[-1.9000, -1.9000, -1.9000, …, -1.9000, -1.9000, -1.9000],
[-1.9000, -1.9000, -1.9000, …, -1.8826, -1.8826, -1.8826],
[-1.9000, -1.9000, -1.9000, …, -1.8826, -1.8826, -1.8826],
…,
[ 0.2864, 0.5467, 0.6334, …, -0.5118, -0.4771, -0.5465],
[ 0.2864, 0.5293, 0.6508, …, -0.5292, -0.4597, -0.5118],
[ 0.0955, 0.2864, 0.4079, …, -0.6506, -0.5986, -0.6159]]]), array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
dtype=float32), ‘00001250_006.png’)

this is a tuple(input_img tensor, output_vector, image_id)

Image type and shape => <class ‘torch.Tensor’> torch.Size([1, 224, 224])
Output type and shape =><class ‘numpy.ndarray’> (15,)
Image id/imgindex(name of the image file) type=><class ‘str’>

(1): BasicBlock(
(conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
(fc): Linear(in_features=512, out_features=15, bias=True)
)

Last layers of Resnet

And I also want to get the probabilities for the predicted classes like 70% it belongs to class1, 50% it belongs to class 5, etc for a given sample image.

Hi Salome!

Well, this is confusing. In the code you posted, you had the variables
inputs, labels, and imgindex. But now you seem to be calling
them “input_img tensor, output_vector, image_id”. I will guess that
these are the same things.

Now I’m guessing that “Image” is what you just called “input_img
tensor", “Output” is “output_vector”, and “Image id/imgindex(name
of the image file)” is “image_id”.

(And I’m also guessing that you haven’t given the shape and type()
of the variable outputs in your code, that is, the object returned by
your model.)

So it looks like what your code calls inputs is a single 224x224 image
with a leading singleton (size = 1) dimension, and that what your
code calls labels is a single numpy array of (floating-point) binary
labels, one for each of your fifteen classes. Note, I don’t see your
batch size of 250 anywhere.

Your inputs tensor should be a batch of 250 images. It should
most likely be a tensor of shape [250, 224, 244], unless your
model is expecting its input images to carry a “channel” dimension
of nChannel = 1, in which case your input tensor should have
shape [250, 1, 224, 244].

Your labels tensor should be a pytorch tensor (not a numpy array),
and should be a batch of sets of (floating-point) binary class labels,
one for each of your 15 classes, and therefore should have shape
[250, 15].

Although you didn’t provide the information for what your outputs
variable is, logically, and based on the last layer of your model, it
should have shape [batch size, nClass], that is, [250, 15].
It will be your model's predicted raw-score logits for each of your
250 samples being in each of your 15 classes.

This is what your data should look like. But it looks like the shape of
your model outputs and the shape of your labels aren’t matching
up properly, presumably somewhere in your evaluation() function.

Could you call something like

trainAcc = evaluation (trainloader, resnet)
print (trainAcc)
testAcc = evaluation (testloader, resnet)
print (testAcc)

before you run any training? (And call evaluation() in two separate
lines of code, as above.)

Good luck.

K. Frank

Hi K. Frank,

print(‘Epoch: %d/%d, Train acc : %0.2f, Test acc : %0.2f’ % (epoch, max_epochs, evaluation(trainloader,resnet), evaluation(testloader,resnet)) )

when i run this, i get the following output

I’m printing outputs = model(inputs)
print(outputs)
_, pred = torch.max(outputs.data,15)
print(pred) within the evaluation function

tensor([[-1.3976, -1.9027, -0.8711, …, -1.4458, -1.2116, -0.8598],
[-0.3187, -0.8117, -1.0978, …, -2.2132, -1.3070, -0.9956],
[-0.8397, -1.6915, -0.6634, …, -1.8321, -1.0346, -1.2636],
…,
[-1.4526, -1.1693, -1.0159, …, -1.3135, -1.1257, -0.9665],
[-1.3092, -1.4912, -1.3867, …, -2.0305, -1.1680, -1.3192],
[-0.5745, -1.1265, -0.9143, …, -2.1783, -0.6507, -1.4715]],
grad_fn=)
tensor([ 3, 3, 5, 3, 3, 3, 3, 7, 3, 3, 3, 3, 3, 3, 7, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
13, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 9, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 2, 3, 3, 5, 3, 3, 3, 3, 3, 3,
3, 3, 3, 8, 5, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 8, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 5, 3, 5, 3, 3,
3, 3, 3, 3, 3, 3, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 8, 3, 3, 3, 3, 3, 5, 2, 3, 3,
3, 3, 3, 3, 5, 5, 3, 2, 3, 3, 3, 3, 3, 9, 3, 3, 3, 2,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 13, 5, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 8, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 4, 3, 3, 3, 3, 3, 3, 3])

RuntimeError Traceback (most recent call last)
in ()
----> 1 print(‘Epoch: %d/%d, Train acc : %0.2f, Test acc : %0.2f’ % (epoch, max_epochs, evaluation(trainloader,resnet), evaluation(testloader,resnet)) )

1 frames
in evaluation(dataloader, model)
12 print(pred)
13 total += labels.size(0)
—> 14 correct +=(pred==labels).sum().item()
15 return 100 * correct/total
16 #preds = torch.tensor([])

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in wrapped(*args, **kwargs)
26 def wrapped(*args, **kwargs):
27 try:
—> 28 return f(*args, **kwargs)
29 except TypeError:
30 return NotImplemented

RuntimeError: The size of tensor a (250) must match the size of tensor b (15) at non-singleton dimension 1

The dataset is imbalanced hence it predicts ‘3’ in most of the cases as there are more samples of class corresponding to the index ‘3’

I think from the output that I get it is of shape [250, 1] and not [250, 15]. It should be [250, 15]. I’m not understanding where I have to make the change within the training loop or the evaluation function.

Yes it is the same thing

Ya this is just for a single sample and not for the batch

So do I need to change the last year out_features to [250, 15] or something else?

Hello Salome!

It’s very hard to know what you are doing. You post some code, and
then you post some data that might or might not be related, and the
data you post has been renamed and edited.

I’ve asked you to post some specific values. You post things that might
or might not be related, but usually differ at least somewhat from what
I’ve asked you to post.

You don’t reflect on things I’ve tried to explain.

For example, in my first reply I noted that using:

    _, pred = torch.max(outputs.data,1)

to convert your outputs to pred is appropriate for “a single-label,
multi-class problem.” You reiterate that yours is a multi-label problem.
But the tensor you just posted for pred looks like the result of the
torch.max() statement. But I don’t know, because I can only guess
what code produced the pred result you just posted.

No.

It is important to understand that you write pytorch models from the
perspective of processing a single sample. But the pytorch framework
actually processes samples in batches.

So, for example, if you write your model to take as input a tensor of
shape [244, 244] (that is a single sample), and produce as output
a tensor of shape [15] (that are the fifteen binary class labels it
predicts for that single sample), then when you run your model, you
have to pass in a batch of samples of shape [nBatch, 244, 244],
and the model will return a batch of predicted class labels of shape
[nBatch, 15].

Note, the value of nBatch is not wired into your model. You can pass
in a batch of 250 ([250, 244, 244]), getting out a batch of 250
([250, 15]), and then, without changing the model pass in a batch
of 1 ([1, 244, 244] --> [1, 15]). (Also note, that if you want to apply
your model to a single sample, that sample has to be wrapped in
a batch of 1. That is, you can’t pass in [244, 244]; you have to pass
in [1, 244, 244]).

So, to repeat, your batch size of 250 does not appear in the code for
your model. And even though the last linear layer looks as if it outputs
a tensor of shape [15] (out_features=15), because pytorch models
always work with batches, your model will output a tensor of shape
[nBatch, 15], where nBatch is the size of the batch you passed
into your model.

Good luck.

K. Frank

inputs.shape & type torch.Size([64, 1, 224, 224]) <class ‘torch.Tensor’>
labels.shape & type torch.Size([64, 15]) <class ‘torch.Tensor’>
imgindex & type (‘00001151_002.png’, ‘00000583_010.png’, ‘00000500_000.png’, ‘00000271_000.png’, …, ‘00000032_031.png’, ‘00000913_000.png’, ‘00001075_024.png’)
outputs, shape & type tensor([[-2.1863, -2.8136, -2.0755, 0.1303, -2.4541, -2.1647, -3.0948, -2.5282,
-1.9832, -2.4096, -2.6687, -2.7546, -2.1767, -2.6486, -3.0040],
[-2.3346, -1.7464, -1.3218, -0.2523, -1.8560, -0.8745, -1.6130, -1.5004,
-1.5930, -1.9782, -2.1910, -2.3470, -1.8963, -2.1992, -1.6291],
[-2.0328, -2.0936, -1.9872, 0.6333, -2.5061, -1.2418, -2.6371, -3.0564,
-2.0829, -2.0732, -2.0986, -2.5197, -2.8750, -2.4518, -2.9555],

[-2.5923, -2.6953, -1.5379, 0.2798, -2.7280, -1.5506, -2.4956, -1.8402,
-2.2393, -2.3018, -2.3598, -1.9540, -3.1476, -2.3570, -1.6612]],
device=‘cuda:0’, grad_fn=) torch.Size([64, 15]) <built-in method type of Tensor object at 0x7feedc76ec18>

Hi K. Frank,
The value of nBatch is being incorporated in the model. Currently I have set nBatch to 64.

After 1 epoch is completed When I run this statement evaluation(trainloader, resnet)

I get the following error
pred : tensor([3, 3, 3, 3, 3, 3, 3, 5, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3], device=‘cuda:0’)

RuntimeError Traceback (most recent call last)
in ()
35 loss_epoch_arr.append(loss.item())
36
—> 37 print(‘Epoch: %d/%d, Train acc : %0.2f’ % (epoch, max_epochs, evaluation(trainloader,resnet)) )
38 plt.plot(loss_epoch_arr)
39 plt.show()

1 frames
/usr/local/lib/python3.6/dist-packages/torch/tensor.py in wrapped(*args, **kwargs)
26 def wrapped(*args, **kwargs):
27 try:
—> 28 return f(*args, **kwargs)
29 except TypeError:
30 return NotImplemented

RuntimeError: The size of tensor a (64) must match the size of tensor b (15) at non-singleton dimension 1

Actually I’m doing multilabel(multi-class) classifcation for the first time, so I dont know what changes need to be made in the evaluation function…I just know that BCEWithLogitLoss/BCELoss is used

Can someone help me out with this…cause its urgent :slightly_frowning_face:

It seems that your evaluation code creates a shape mismatch.
For a multi-label classification the shape of the model output and target should be equal, while it seems that one tensor might be missing the batch size.
Could you print the shapes of the model output and target before the error is raised (in the validation code)?

Hi @ptrblck!

Her problem (among other things) is that she calls argmax() to get
a single class label (for the accuracy calculation in the evaluation
code), but it’s a multi-label problem, so she needs a vector of length
nClass binary predictions.

I wasn’t able to explain this clearly.

Best.

K. Frank

1 Like

Hi @KFrank!

Ah OK, didn’t realize it and the posted code snippets were a bit hard to follow.

@ssalome
You can post code snippets by wrapping them into three backticks ```, which makes debugging easier :wink: