Guidelines for when and why one should set inplace = True?

[Nikronic (Nikan Doosti)]

Hello,

First, there is an important thing you have to consider; you only can use inplace=True when you are sure your model won’t cause any error. For example, if you trying to train a CNN, in the time of backpropagation, autograd needs all the values, but inplace=True operation can cause a change so your backprop is no longer valid. Actually, this kind of error has been handled by PyTorch, so you’ll be noticed about it.

Second, if you do not have any error, it is better to use inplace=True operation because it won’t allocate new memory for the output of your layer. So it can prevent from Out of memory error.

Finally, as far as I know developers usually use inplace=True unless they do not get any error.

I got 'RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation' error

Would you please show me how to fix this? I changed it like this: def forward(self, logits, labels): N, C, H, W = logits.size() n_pixs = N * H * W logits = logits.permute(0, 2, 3, 1).contiguous().view(-1, C) with torch.no_grad(): scores = F.softmax(logits, dim=1).cpu().detach() labels = labels.view(-1) labels_cpu = labels.cpu().detach() invalid_mask = labels_cpu==self.ignore_lb labels_cpu[invalid_mask]…

What's the difference between nn.ReLU() and nn.ReLU(inplace=True)?

I implemented generative adversarial network using both nn.ReLU() and nn.ReLU(inplace=True). It seems that nn.ReLU(inplace=True) saved very small amount of memory. What’s the purpose of the using inplace=True? Is the behavior different in backpropagation?

bests