Some in-place operations work with autograd?

[ArchieGertsman]

The following example does not result in an autograd error:

lin = nn.Linear(2,2)
y = lin(torch.rand(2))
y[1] = 0 # in-place operation
y.sum().backward()

I’m just curious, how come this works? When, in general, do in-place operations not interfere with autograd? Thanks!

[KFrank]

Hi Archie!

@ArchieGertsman

When, in general, do in-place operations not interfere with autograd?

Here is a post with some examples that illustrate when inplace operations
will and won’t break autograd:

When inplace operation are allowed and when not

Hi Jie (and Srishti)! The part that can be confusing is that a given tensor used in the forward pass may or may not be used in the backward pass. It depends on the math of the gradient computation (and the details of how the relevant backward() function is implemented). It’s not always obvious. Here are some explanatory examples: The derivative of exp (x) is exp (x). So it makes sense that pytorch saves the output of exp (x) when it is computed during the forward pass so that it can …

Best.

K. Frank