For a custom layer like that:
def forward(ctx, input, int1, int2):
ctx.save_for_backward(input, int1, int2)
out = some_c_function(input, int1, int2)
def backward(ctx, grad_output):
input, int1, int2 = ctx.saved_tensors
grad_input = torch.zeros_like(input)
backward_func(input, grad_output, grad_input, int1, int2)
return grad_input, None, None
But when I call this layer, error occurred:
TypeError: save_for_backward can only save variables, but argument 3 is of type int
Now the question is, how can I save those integers for backward ?
You need to use
save_for_backward only for inputs and outputs.
All other tensors or python objects can be saved in the
ctx.in1 = int1.
@albanD why do we need to use
save_for_backwards for input tensors only ? I just tried to pass one input tensor from
ctx.tensor = inputTensor in
inputTensor = ctx.tensor in
backward() and it seemed to work.
I appreciate your answer since I’m currently trying to really understand when to use
ctx rather than
save_for_backward . Another post tells us memory leak isn’t an issue any more , and concludes on my question.
In addition, reading this other post suggests “earlier torch versions it was not possible to
save_for_backward an intermediate result, only inputs to the
forward function, but that seems to be no longer the case.”
An actual limit would be the need for the saved value to be a variable (in the sense of pytorch tensor) in the case of
Oh! PyTorch internals time!
So what happens is that
save_for_backward stores the tensors in
SavedVariables. This is what does the sanity checks for the backward (one of the variables needed for gradient computation has been modified by an inplace operation being the perhaps most (in)famous).
Now variables that aren’t input or output of your function (and don’t share storage…) are not visible to the outside world and thus cannot be changed in ways that would trigger those sanity checks.
Now you can safely store other tensors, but SavedVariable won’t work for non-tensors.
I see, sanity checks. Thanks !