Compute the Hessian matrix of a network

As previously mentioned in the thread, computing the Hessian via autograd is expected to be slow. The main idea is to call grad() on grad(). The snippet mentioned above makes two nested calls of the Jacobian, which is conceptually less clear when you first encounter the question of how to compute the Hessian. The main thing to avoid is to call backward() and then grad() on the result, instead of calling grad() twice in a nested fashion, since the former approach leads to memory leakage.

Here are two methods of a model, that can be used for trying to understand how to compute the Hessian:

The grad_log_target_val input argument to hess_log_target() is the output of grad_log_target().

If you want a standalone script without involving a model, you may check my jupyter notebook below, where I explain step by step via a walk-through example how to compute the Hessian:

Use PyTorch’s autograd.functional library:

torch.autograd.functional.hessian(func, inputs)
1 Like

@iacob Hi, would you please share us an example please?