One Error: inplace operation

1

I get one error:
one of the variables needed for gradient computation has been modified by an inplace operation
thanks
my model:

class SCNN(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(SCNN, self).__init__()
        self.down = nn.ModuleList(
            [nn.Conv2d(in_ch, out_ch, kernel_size=(1, 9), stride=1, padding=(0, 4)) for i in range(35)])
        self.up = nn.ModuleList(
            [nn.Conv2d(in_ch, out_ch, kernel_size=(1, 9), stride=1, padding=(0, 4)) for i in range(36)])
        self.left = nn.ModuleList(
            [nn.Conv2d(in_ch, out_ch, kernel_size=(9, 1), stride=1, padding=(4, 0)) for i in range(99)])
        self.right = nn.ModuleList(
            [nn.Conv2d(in_ch, out_ch, kernel_size=(9, 1), stride=1, padding=(4, 0)) for i in range(100)])

    def forward(self, x):
        h = x.shape[2]
        w = x.shape[3]
        for i in range(h - 1):
            x_h = F.relu(self.down[i](x[:, :, [i], :]), inplace=True)
            x[:, :, [i + 1], :].add_(x_h)
        for i in range(h - 1, 0, -1):
            x_h = F.relu(self.up[i](x[:, :, [i], :]), inplace=True)
            x[:, :, [i - 1], :].add_(x_h)
        x[:, :, [0], :] = F.relu(self.up[0](x[:, :, [0], :]), inplace=True)
        for i in range(w - 1):
            x_h = F.relu(self.left[i](x[:, :, :, [i]]), inplace=True)
            x[:, :, :, [i + 1]].add_(x_h)
        for i in range(w - 1, 0, -1):
            x_h = F.relu(self.right[i](x[:, :, :, [i]]), inplace=True)
            x[:, :, :, [i - 1]].add_(x_h)
        x[:, :, :, [0]] = F.relu(self.up[0](x[:, :, :, [0]]), inplace=True)
        return x
2

Hi,

You cannot always do inplace operations as some values are needed to compute gradients and so the backward cannot happen if you changed these values with an inplace operation.

To find which one is the problem, you should remove all the inplace operations and then add them back one by one.

3

In my model, those are inplace operations?
thanks

4

Hi,

all the operations with the argument inplace=True are inplace operations. You can set the argument to False to disable it.
Also operations with trailing underscores (like add_) are inplace operation. The out of place operations are without the underscore (like add).
Finally operations like x[smth] = foo are inplace operations as well.

5

HI,
Replace them, will it affect the performance of the model?
thanks

6

I won’t affect the result, only the memory comsumption.

7

@albanD is there any way to avoid x[smth] = foo inplace operations in pytorch?

8

The canonical thing to do is x_new = where(smth, foo, x) will do, but then foo has to be constructed differently (to be of the same shape as x, so this is in fact similar to x[smth] = foo[smth] out of place).

1 Like