One of the variables needed for gradient computation has been modified by an inplace operation in PixelCNN

vision
1

Hi everybody,
I have an image and want to calculate loss just for 1 pixel on it, therefore I use a mask array and masked the other value that I want to be shown in the loss . my code is just below, but I got an error and I couldn’t fix it . just somebody can please help to know me whats wrong with my code?

fm = 64
net = nn.Sequential(
    MaskedConv2d('A', 3,  fm, 3, 1, 1, bias=False), nn.BatchNorm2d(fm), nn.ReLU(True),
    MaskedConv2d('A', fm, fm, 3, 1, 1, bias=False), nn.BatchNorm2d(fm), nn.ReLU(True),
    MaskedConv2d('A', fm, fm, 3, 1, 1, bias=False), nn.BatchNorm2d(fm), nn.ReLU(True),
    MaskedConv2d('A', fm, fm, 3, 1, 1, bias=False), nn.BatchNorm2d(fm), nn.ReLU(True),
    MaskedConv2d('A', fm, fm, 3, 1, 1, bias=False), nn.BatchNorm2d(fm), nn.ReLU(True),
    MaskedConv2d('A', fm, fm, 3, 1, 1, bias=False), nn.BatchNorm2d(fm), nn.ReLU(True),
    MaskedConv2d('A', fm, fm, 3, 1, 1, bias=False), nn.BatchNorm2d(fm), nn.ReLU(True),
    MaskedConv2d('A', fm, fm, 3, 1, 1, bias=False), nn.BatchNorm2d(fm), nn.ReLU(True),
    nn.Conv2d(fm, 3, 1))


sample = V(torch.zeros(bs, 3, picSize,picSize))
residual = V(torch.zeros(bs, 3, picSize, picSize))
res_mask = V(torch.zeros_like(residual.data))
for batch_idx, (images, labels) in enumerate(dataloaders):
    images = V(images, requires_grad = False)

    for i in range(picSize):
        for j in range(picSize):
            optimizer.zero_grad()
            out = net(sample)
            residual[:,:,i,j] = (images[:,:,i,j] - out[:,:,i,j])**2
            res_mask = V(torch.zeros_like(residual.data))
            res_mask[:,:,i,j] = 1
            residual = torch.mul(residual,res_mask)
            loss = (residual.sum()/(bs * 3))
            pdb.set_trace()
            loss.backward(retain_graph = True)
            optimizer.step()
            sample[:,:,i,j] = images[:,:,i,j]

Error: one of the variables needed for gradient computation has been modified by an inplace operation
@jpeg729

2

I suspect that some_var[:,:,i,j] = stuff counts as an inplace operation.
If I have understood correctly what your code is doing then most of them are unnecessary, and the last one sample[:,:,i,j] = images[:,:,i,j] doesn’t need gradients.
Would this work?

for i in range(picSize):
    for j in range(picSize):
        optimizer.zero_grad()
        out = net(sample)
        residual = (images[:,:,i,j] - out[:,:,i,j])**2
        loss = (residual.sum()/(bs * 3))
        pdb.set_trace()
        loss.backward(retain_graph = True)
        optimizer.step()
        sample.data[:,:,i,j] = images[:,:,i,j]
1 Like
3

Tnx for your reply, I did those things because I want just a specific pixel of my output to be in the loss cost and not the whole of the output. for that I masked my output.
Is there any option to my code do this work without any error?

4

I believe my proposal does just that.

residual is calculated as the mse of a single pixel, and loss is calculated from the residual of that single pixel (I assume bs is the batch size), so when you do loss.backward() it will backpropagate through the calculations that lead to that pixel.

5

Yeah your right, I did the modification. and it worked very well. Tnx again for your reply :slight_smile:

1 Like