Compute hessian vector product

[Hoang_Tran]

Hi, I’m trying to write an optimizer using pytorch that uses the information from Hessian Vector Product. Usually, I can easily compute Hessian Vector product in the training loop by calling torch.autograd.grad and set the attribute create_graph to True. However, since I want to compute it inside an optimizer, I have no access to loss and can’t change create_graph to True. Therefore, I’ve been getting this error: “element 0 of tensors does not require grad and does not have a grad_fn”. May I ask if there’s any way that I can get around this? Thank you!

[Yaroslav_Bulatov]

You can pass loss into your optimizer by using a closure. Couple of examples: look at the “step” function:

• https://github.com/fmeirinhos/pytorch-hessianfree/blob/master/hessianfree.py
• https://github.com/pytorch/pytorch/blob/master/torch/optim/lbfgs.py