R-operator in pytorch?

[arogozhnikov (Alex Rogozhnikov)]

Is it possible to obtain somewhat like R-op in pytorch (jacobian times vector product)?

See e.g. theano documenation http://deeplearning.net/software/theano/tutorial/gradients.html#r-operator

[albanD]

Hi,

The backward engine in pytorch only allows you to perform L-op using backward mode AD.
R-op is tricky for neural nets when using backward mode AD because you don’t want to compute the full Jacobian for memory size reasons. You would need forward mode AD to implement that efficiently which is not implemented.
I am not sure how hard it would be to do a forward mode AD engine for pytorch. But definitely non trivial.

[tor]

Hi:

Is there any example of L-op implementation in Pytorch?

It seems there is a hope on R-op in Pytorch because
we can implement R-op using twice L-op, as in here.

[tom (Thomas V)]

In fact, @PiotrSokol was kind enough to share his implementation:

[Adding functionality] Hessian and Fisher Information vector products

Hi, I implemented Hessian and Fisher Information matrix (FIM) vector products and was wondering if there’d be interest in adding this functionality. The FIM products are optimized, in the sense that they analytically compute the Hessian matrix of the logarithm of the loss wrt. the output layer for a set of commonly used losses. Best, P. Edit: Tagging you in @smth, @apaszke Editing for posterity since no one wants to discuss how to submit this PR. import numpy as np import torch def _chec…

Best regards

Thomas

[tor]

Awesome. It works like a charm
Thanks a lot @tom @PiotrSokol