# \[Autograd\] In-place operations, clone and slicing?

**URL:** <https://discuss.pytorch.org/t/autograd-in-place-operations-clone-and-slicing/171456>\
**Category:** autograd\
**Created:** [January 31, 2023, 9:57am UTC](https://discuss.pytorch.org/t/autograd-in-place-operations-clone-and-slicing/171456 "2023-01-31T09:57:07Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![KFrank](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/32.png) [@KFrank](https://discuss.pytorch.org/u/KFrank)\
**Post date:** [February 1, 2023, 1:50am UTC](https://discuss.pytorch.org/t/autograd-in-place-operations-clone-and-slicing/171456/2 "2023-02-01T01:50:48Z")

</div>

Hi Nick!

> [@technick](#):
>
> This leads me to my current understanding of autograd:
> 
> - use clone() when I want to do inplace operations on my tensor with grad history, which I want to keep. I can also assign my cloned tensor to the original one, as it has the same grad history.
> - use detach().clone() when I want to have a copy of my tensor that uses new memory and has no grad history
> - inplace operations without clone() can cause issues with autograd and my grad results could be wrong even though there is no grad error

Let me make some comments that are relevant to your questions:

Inplace operations _can_ cause “inplace-modification” errors during  
backpropagation, but they don’t _necessarily_ cause such errors.

The following post of mine tries to explain what is going on with some  
examples:

> [@When inplace operation are allowed and when not](https://discuss.pytorch.org/t/when-inplace-operation-are-allowed-and-when-not/169583/3):
>
> Hi Jie (and Srishti)! The part that can be confusing is that a given tensor used in the forward pass may or may not be used in the backward pass. It depends on the math of the gradient computation (and the details of how the relevant backward() function is implemented). It’s not always obvious. Here are some explanatory examples: The derivative of exp (x) is exp (x). So it makes sense that pytorch saves the output of exp (x) when it is computed during the forward pass so that it can …

Note, your “grad results” won’t be wrong because if they would have been  
wrong, autograd _will_ raise a “grad error.”

Tensors without a “grad history” (that is, with `requires_grad = False`)  
are not immune to inplace-modification errors. A tensor which is not  
having its own gradient computed can still be used in the computation  
of some other tensor. Consider:

```plaintext
>>> import torch
>>> torch. __version__'1.13.0'
>>> a = torch.tensor ([2]) # a tensor with requires_grad = False
>>> t = torch.ones (1, requires_grad = True)
>>> l = a * t
>>> l.backward()
>>> t.grad
tensor([2.])
>>> t.grad = None
>>> l = a * t
>>> a[0] = 99 # modify inplace
>>> l.backward()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<path_to_pytorch_install>\torch\_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "<path_to_pytorch_install>\torch\autograd\ __init__.py", line 197, in backward
    Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.LongTensor [1]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

```

Yes, if you need to modify a tensor inplace _and_ doing so causes an  
inplace-modification error during backpropagation, then cloning the  
tensor before modifying it is often an appropriate solution.

Best.

K. Frank

---

_[View the full topic](https://discuss.pytorch.org/t/autograd-in-place-operations-clone-and-slicing/171456)._