Guidelines for when and why one should set inplace = True?

[John_Deterious]

I know what inplace does, but, can you please explain why should I opt to use it or not. Are there any caveats and what are the scenarios in which this option is important, sepecifically when building a NN. Take ReLU layer as an example in which this option is available.

[Nikronic (Nikan Doosti)]

Hello,

First, there is an important thing you have to consider; you only can use inplace=True when you are sure your model won’t cause any error. For example, if you trying to train a CNN, in the time of backpropagation, autograd needs all the values, but inplace=True operation can cause a change so your backprop is no longer valid. Actually, this kind of error has been handled by PyTorch, so you’ll be noticed about it.

Second, if you do not have any error, it is better to use inplace=True operation because it won’t allocate new memory for the output of your layer. So it can prevent from Out of memory error.

Finally, as far as I know developers usually use inplace=True unless they do not get any error.

I got 'RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation' error

Would you please show me how to fix this? I changed it like this: def forward(self, logits, labels): N, C, H, W = logits.size() n_pixs = N * H * W logits = logits.permute(0, 2, 3, 1).contiguous().view(-1, C) with torch.no_grad(): scores = F.softmax(logits, dim=1).cpu().detach() labels = labels.view(-1) labels_cpu = labels.cpu().detach() invalid_mask = labels_cpu==self.ignore_lb labels_cpu[invalid_mask]…

What's the difference between nn.ReLU() and nn.ReLU(inplace=True)?

I implemented generative adversarial network using both nn.ReLU() and nn.ReLU(inplace=True). It seems that nn.ReLU(inplace=True) saved very small amount of memory. What’s the purpose of the using inplace=True? Is the behavior different in backpropagation?

bests

[tom (Thomas V)]

Indeed @Nikronic nails it with the rule of thumb You can use inplace for memory efficiency unless you it breaks.
You might also be less eager to use inplace when planning to use the JIT, as it will fuse pointwise non-inplace operations like ReLU if there are several in a row.

The two things to avoid are:

• you move a leaf tensor into the graph (using inplace on something that you just defined with requires_grad=True),
• when the operation before the inplace wants to have its result to compute the backward. Whether this is the case is not easy to tell with from the outside, unfortunately.

As a corollary, you should avoid using inplace on the inputs of your re-usable module in lest the future use could be in one of the two situations.

Best regards

Thomas