@albanD why do we need to use save_for_backwards for input tensors only ? I just tried to pass one input tensor from forward() to backward() using ctx.tensor = inputTensor in forward() and inputTensor = ctx.tensor in backward() and it seemed to work.
I appreciate your answer since I’m currently trying to really understand when to use ctx rather than save_for_backward . Another post tells us memory leak isn’t an issue any more , and concludes on my question.
In addition, reading this other post suggests “earlier torch versions it was not possible to save_for_backward an intermediate result, only inputs to the forward function, but that seems to be no longer the case.”
An actual limit would be the need for the saved value to be a variable (in the sense of pytorch tensor) in the case of save_for_backward .
So what happens is that save_for_backward stores the tensors in SavedVariables. This is what does the sanity checks for the backward (one of the variables needed for gradient computation has been modified by an inplace operation being the perhaps most (in)famous).
Now variables that aren’t input or output of your function (and don’t share storage…) are not visible to the outside world and thus cannot be changed in ways that would trigger those sanity checks.
Now you can safely store other tensors, but SavedVariable won’t work for non-tensors.