model = model.cuda() # move the model parameters to CPU/GPU
for e in range(epochs):
print("Epoch: ", e)
for t, (x, y) in enumerate(train_loader):
model.train() # put model to training mode
x = x.cuda() # move to device, e.g. GPU
y = y.type(torch.LongTensor)
y = y.cuda()
scores = model(x)
loss = F.cross_entropy(scores, y)
# Zero out all of the gradients for the variables which the optimizer
# will update.
optimizer.zero_grad()
# This is the backwards pass: compute the gradient of the loss with
# respect to each parameter of the model.
loss.backward()
# Actually update the parameters of the model using the gradients
# computed by the backwards pass.
optimizer.step()
if t % 100 == 0:
print('Iteration %d, loss = %.4f' % (t, loss.item()))
check_val_accuracy(val_loader, model)
print()
Shape of x: torch.Size([3, 3, 224, 224])
Shape of y: torch.Size([3, 1, 1024, 2048])
I assume scores has the same spatial shape as your target.
If so, you would need to get rid of the channel dimension in y. nn.CrossEntropyLoss expects the targets to contain the class indices in the range [0, nb_classes-1] for a multi-class segmentation use case.
You could try y = y.squeeze(1), if y already contains the class indices.
Could you explain your use case a bit?
Based on the target shape and the criterion it seems you are working on a multi-class segmentation use case.
However in this case the model output should have the shape [batch_size, nb_classes, height, width], while your output seems to come from a multi-class classification with 30 classes?
Then the spatial dimensions of your output are missing.
Are you using a linear layer as the last layer in your model?
If so, then you would have to swap it for a conv layer with out_channels=nb_classes.
Worked, but ran into a new error,
RuntimeError: Expected 4-dimensional input for 4-dimensional weight 30 2048 3 3, but got 2-dimensional input of size [3, 2048] instead.
Some conv layer is expecting a 4-dimensional input as [batch_size, channels, height, width], while you are passing a 2-dim tensor. If you are using a flattening operation, you would have to remove it as well.
In that case you would have to change the final linear layer(s) to conv layers, so that your model output has spatial dimensions.
Alternatively you could also reshape the last linear output to match the target dimension, but I’m not sure how well this would work.
Also, you would have to make sure that the spatial shapes of the output and target match, since each “pixel” represents a logit.
so I passed this dataset through my network then it produces softmax_output for first sample as: tensor([[0.600, 0.400]])
Now I want to pass target labels of dataset and the output generated by network into a crossentropyloss function. When I pass first sample’s label (label=1) and first sample’s output (i.e. tensor([[0.600,0.400]])) into the loss function, how is it able to understand that label=1 (means one-hot encode [[0,1]]) when I am not converting it into one-hot?
Secondly, is it correct to pass an integer label (i.e. = 1) in longtensor form and the tensor output (predicted by model) into loss function? Won’t it broadcast label=1 tensor into [[1,1]], which we don’t need?
This sounds wrong, as nn.CrossEntropyLoss expects raw logits, so you should not apply softmax on your outputs.
To “select” the right output, you could either multiply the output with a one-hot encoded target tensor or alternatively index the output with a target containing the class indices. Since the first approach is wasteful in memory (you are using a sparse tensor with a single 1), PyTorch uses the class index approach.
Yes, it’s correct as also described in the docs. The criterion will not apply any broadcasting.
So i have read through the solutions here and none of them help solve my
IndexError: Target 9 is out of bounds. issue
class RFCLF1(nn.Module):
def __init__(self):
super().__init__()
self.linear1 = nn.Linear(input_size ,9 )
def forward(self, xb):
xb = self.linear1(xb)
return xb
log_reg = RFCLF1()
y.ndim
1
loss_func = F.cross_entropy
lr = 0.084
def trainer(model , epochs , lr , val_dl , train_dl ,loss_func , min_loss, Opt_func=opt.SGD):
min_l = min_loss
Op = Opt_func(model.parameters() , lr)
for epoch in range(epochs):
#training
for xb , yb in train_dl:
yhat = model(xb)
tloss = loss_func(yhat , yb.long())
tloss.backward()
Op.step()
Op.zero_grad()
train_loss.append(tloss)
if tloss < min_l:
print("{} number of epochs and loss of {}".format(epoch , tloss))
torch.save(model.state_dict() , 'm14.pth')
#validation
for xb,yb in val_dl:
yval = model(xb)
vloss = loss_func(yval , yb.long())
val_loss.append(vloss)
reading through the recent comments i changed the y to int64 tensors for integer labels but nothing seems to work, num of classes are 10; 1,2,3,4,5,6,7,8,9,10
@ZzyZx What is the size of yhat? If you have 10 classes then its shape should be (minibatch,10). Secondly, your target labels should start from [0-9] if there are 10 classes. So try to manage that or reduce the integer value by one.
Note: I can see you using Op.zero_grad after tloss.backward which I think is not suggested. Recommendation as far as I know , is to use zero_grad before loss.backward.
How did you change it, using a linear layer? What is 512, is it batchsize? Can you please put that snippet where you are doing all these implementations? And I didn’t say to reshape, I meant the labels should be 0-9 and not 1-10.
