How to understand 'calling .backward() clears the computation graph'

[colesbury (Sam Gross)]

See the answer here:

Multiple call to backward still works when retain_graph=false, why?

In this specific case you do not need retain_graph=True, but in general you may need it. As you compute the forward pass, PyTorch saves variables that will be needed to compute the gradients in the backward pass. For example, z = y * y needs to save the value of y, because dz/dy = 2*y (or y + y). However, y = x + 2 doesn’t need to save anything because dy/dx = 1 which doesn’t depend on x. When you call backwards() with retain_graph=False (or without specifying it), the automatic differentiation…

If retain_graph=False, intermediate outputs needed for the backwards computation are freed. If there are no saved intermediate outputs, (like in the case of addition) then subsequent calls may still work, but you should not rely on that behavior.