Loss function that is weighted by the label

autograd
1

Hi !

I want to make a loss function that depends on the label (some dimensions are somehow co-linear). For example, I would like to weight the loss of feature 2 by the norm of feature 1. The issue here is that autograd says that it cannot compute the gradient

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [20, 224, 224]] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

for the code

    outputs[:, 1] *=  labels[:,0]
    labels[:, 1] *= labels[:,0]

    loss_parameters = F.smooth_l1_loss(outputs, labels)

Do you have any idea on how to make that happen ?

Thanks
Adrian

2

You need to avoid the inplace operation.
If the size along the second dimension is 2, you could use

mult = torch.stack([torch.ones_like(labels[:,0]), labels[:, 0]], dim=1)
outputs = outputs * mult
labels = labels * mult

or so.

Best regards

Thomas

3

Great ! I could copy labels[:,0] to mult to make it not requiring the gradient. Thanks

Finally I did this:

    loss_parameters = F.smooth_l1_loss(outputs, labels, reduction='none')
    loss_parameters[:, 1, :, :] *= labels[:, 0, :, :]

It’s slightly different, but since it’s a L1 norm, it’s almost the same.