Combine autograd with finite difference


I am wondering if there are ways to combine autograd and numerical gradients with finite difference method.

Specifically, for Modified Bessel Function, I also want to compute the gradient with respect to $\nu$. It’s simple to compute its gradient with finite difference while hard with autograd. How to wrap finite difference as an autograd function and use it in torch.nn.Module subclasses?


You’d calculate & store dBessel(…)/dorder (generally, dF(x)/dx). This can be done in [autograd.Function’s] forward() or postponed to backward(). Irrespective of how you obtained a gradient, in backward you do multiplication: dLoss/dx = dLoss/dF(x) * dF(x)/dx = grad_out * local_grad and return that for each function input.

nn.Module is orthogonal to this, it would only be an additional wrapper around a function (or rather, a call to Function.apply). That is mainly useful for adding local nn.Parameters or if you use nn.Sequential declarative pipelines.

1 Like

Thank you Alex! Yes I’d like to register $\nu$ as a nn.Parameter and optimize it. Will take a try, thanks :slight_smile: