How to understand 'calling .backward() clears the computation graph'

[dolaameng]

I understand that calling backward will clear the computation graph and as a result, a second call of it will throw an exception. e.g.,

x = Variable(torch.Tensor(5, 3), requires_grad=True)
y = Variable(torch.Tensor(5, 3), requires_grad=True)
z = torch.mm(x, torch.transpose(y, 0, 1))
z.backward(torch.ones(5, 5), retain_graph=False)
z.backward(torch.ones(5, 5), retain_graph=False)) # >> Throws RuntimeError

However, this seems to work differently for some operators, e.g., elementwise addition, e.g.,

x = Variable(torch.Tensor(5, 3), requires_grad=True)
y = Variable(torch.Tensor(5, 3), requires_grad=True)
z = x + y
z.backward(torch.ones(5, 3), retain_graph=False)
# You can call it multiple times and the grad will just accumulate
z.backward(torch.ones(5, 3), retain_graph=False) 

I am using pytorch ‘0.2.0_4’, can some one explain why the second case works differently? Thanks!

[colesbury (Sam Gross)]

See the answer here:

Multiple call to backward still works when retain_graph=false, why?

In this specific case you do not need retain_graph=True, but in general you may need it. As you compute the forward pass, PyTorch saves variables that will be needed to compute the gradients in the backward pass. For example, z = y * y needs to save the value of y, because dz/dy = 2*y (or y + y). However, y = x + 2 doesn’t need to save anything because dy/dx = 1 which doesn’t depend on x. When you call backwards() with retain_graph=False (or without specifying it), the automatic differentiation…

If retain_graph=False, intermediate outputs needed for the backwards computation are freed. If there are no saved intermediate outputs, (like in the case of addition) then subsequent calls may still work, but you should not rely on that behavior.

[dolaameng]

Cool. That makes sense. Thank you for the clarification.