I find some codes in this and become curious about the function of @once_differentiable.
2 Likes
If you wrap your Function's backward method with this wrapper, the Tensors that you will get as input will never require gradients and you don’t have to write a backward function that computes the gradients in a differentiable manner. For example, you can use other libraries to do the computation.
If you try to backward through the backward pass of such Function, an error will be raised stating that this Function is only differentiable once.
1 Like
Thx for your reply but it’s a bit confusing…Is that meaning we cannot compute high-order gradients from this function?
Yes.
You should use it when you write a backward that cannot be used for high-order gradients.
5 Likes
Got it!
Very helpful!
I don’t think the decorator once_differentiable does anything here. The decorator checks whether the input requires_grad (and err’s if yes). However the grad computation logic has been fully encapsulated in the backward function body (with context from forward), and the only part exposed to the outside is the grad_output, which will always not require grad in higher order derivative computations, hence making the decorator pretty much useless.
I highly doubt if there is a way to trigger a double backward for functions like this. Can someone give an example where it may happen?