[Autograd] In-place operations, clone and slicing?

[KFrank]

Hi Nick!

@technick

This leads me to my current understanding of autograd:

• use clone() when I want to do inplace operations on my tensor with grad history, which I want to keep. I can also assign my cloned tensor to the original one, as it has the same grad history.
• use detach().clone() when I want to have a copy of my tensor that uses new memory and has no grad history
• inplace operations without clone() can cause issues with autograd and my grad results could be wrong even though there is no grad error

Let me make some comments that are relevant to your questions:

Inplace operations can cause “inplace-modification” errors during
backpropagation, but they don’t necessarily cause such errors.

The following post of mine tries to explain what is going on with some
examples:

When inplace operation are allowed and when not

Hi Jie (and Srishti)! The part that can be confusing is that a given tensor used in the forward pass may or may not be used in the backward pass. It depends on the math of the gradient computation (and the details of how the relevant backward() function is implemented). It’s not always obvious. Here are some explanatory examples: The derivative of exp (x) is exp (x). So it makes sense that pytorch saves the output of exp (x) when it is computed during the forward pass so that it can …

Note, your “grad results” won’t be wrong because if they would have been
wrong, autograd will raise a “grad error.”

Tensors without a “grad history” (that is, with requires_grad = False)
are not immune to inplace-modification errors. A tensor which is not
having its own gradient computed can still be used in the computation
of some other tensor. Consider:

>>> import torch
>>> torch.__version__
'1.13.0'
>>> a = torch.tensor ([2])   # a tensor with requires_grad = False
>>> t = torch.ones (1, requires_grad = True)
>>> l = a * t
>>> l.backward()
>>> t.grad
tensor([2.])
>>> t.grad = None
>>> l = a * t
>>> a[0] = 99   # modify inplace
>>> l.backward()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<path_to_pytorch_install>\torch\_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "<path_to_pytorch_install>\torch\autograd\__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.LongTensor [1]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Yes, if you need to modify a tensor inplace and doing so causes an
inplace-modification error during backpropagation, then cloning the
tensor before modifying it is often an appropriate solution.

Best.

K. Frank