Grad_output : tensor or variable?

In the backward method when implementing autograd, we must have two arguments : ctx and grad_output

def backward(ctx, grad_output)

In the doc, it says that grad_output is a tensor, but by running the code, I see it is a variable. Is there a bug somewhere on my side ?

Assuming you use new style Function (your backward functions is defined with @staticmethod), then there are two cases:

  • If you use the @oncedifferentiable (not sure if that is the exact naming, there is possibly an _ in it) decorator, you will get grad_output as a Tensor as the backward cannot be differentiated anyway.
  • If you don’t use this decorator, then you will get a Variable and you need to make sure that your backward is building a proper graph in case you will backpropagate through your backward pass. (If you don’t need the double backward feature, just add the above-mentionned decorator to just work with Tensors).

Thanks.

I don’t see mentions once_differentiable anywhere in the documentation, is this function really supported ? Is it really recommended to use it outside pytorch code ?

It is possibly an undocumented feature :slight_smile:
I quite like using it because that ensure that you won’t silently break gradients with your custom Functions that are not implemented to be double differentiable!
Also the doc mentioning that you’re getting a Tensor is possibly something that has not been changed from the old style Function doc.

1 Like