Hi,
I have problems understanding this tutorial: Double Backward with Custom Functions — PyTorch Tutorials 2.5.0+cu124 documentation
could someone simplify this a bit?
how do you know a function support double backward or not?
best
Hi @bpfrd,
A good example for defining a DoubleBackward can be found here: How to customize the double backward? - #5 by amensch
and this thread shows an example I wrote explaining a custom torch.autograd.Function object here: Customizing torch.autograd.Function - #4 by AlphaBetaGamma96
In short, you’ll need to manually define a derivative of the output of your function with respect to all inputs. Then define how to combine the grad_output (the gradient of the loss with respect to the output of your function) with your manually defined derivative (gradient of output with respect to the input) via the chain rule to get the gradient of the loss with respect to your inputs, which is what autograd is looking for.
In the case of the second-order derivative (or DoubleBackward), you just need to repeat this process of defining the derivative of the first Backward call with respect to its inputs, and how to combined them with the grad_output term of the second derivative.
The lingo’s a bit confusing, but do look at the examples I’ve listed above and see if the examples explain this process in more detail.