Can't find Inplace operation: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

1

I know this error is because inplace operation, but I don’t know where is the operation in my code.
I define my model forward function as follow:

def forward(self, input):
    x = input[0]
    y = input[1]
    x = self.feat(x)
    x_proj = self.proj(x)
    x = x_proj
    x = self.toimg(x, y)
    x = self.concat(x, y)
    x = self.resnet(x)
    return x, x_proj

And I want to use x_proj to compute a extra loss:

def feature_regularizer(x_proj):
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    min = torch.zeros(x_proj.shape).to(device).to(torch.float32)
    max = torch.ones(x_proj.shape).to(device).to(torch.float32)
    max[:, 0, :] = max[:, 0, :] * 1920
    max[:, 1, :] = max[:, 1, :] * 1200
    max[:, 2, :] = max[:, 2, :] * 200
    min = min - x_proj
    max = x_proj - max
    loss_min = F.relu(min)
    loss_max = F.relu(max)
    loss = torch.sum(loss_min) + torch.sum(loss_max)
    return loss

Finally I add this extra loss to the main loss:

pred, x_proj = classifier(input)
loss = scheduler(pred, target)
reg = feature_regularizer(x_proj) * 1e-8
loss = loss + reg

I’m very certain the inplace operation is in def feature_regularizer(x_proj), because when I delete this loss, I can run the code.
But I can’t find the inplace operation in def feature_regularizer(x_proj), so I’m confused, pls help me

2

Hi,

As mentioned in the error message, you should be able to enable anomaly mode to get a better idea where the error comes from.
In the function you mention, doing max[:, 0, :] = XXX is an inplace operation. So most likely the issue?
You can try to replace this with:
max = max * torch.tensor([[[1920,], [1200,], [200,]],]) (weird brackets to make it a tensor of size [1, 3, 1] to make sure broadcasting does what you want.

3

sorry the solution is wrong, I still get the inplace error.
I think this no-inplace constraint targets the tensor which is forward along the model, only this tensor needs back-propagation, so you can’t change it. But in my code the “max” tensor is just a tensor.

4

Ho right it does not require gradients.
In that case you should enable anomaly mode to get more details on where the error comes from by following the instructions in the error message.