Get parameters gradient after every layer's backward computation

autograd
1

Hi,

I wonder if there is any way that can directly report parameters gradient after each layer’s backward computation. Through the hook function on tensor or module, we are only able to get the intermediate results of the current layer but not the parameters’ gradient.

Thanks in advance.

2

Hi,

You can add a hook on the Parameter itself with register_hook(). This will be used when the gradients are computed (and just before the .grad field is populated).

3

Thanks for the prompt reply. I register the hook on the weights (e.g., module.weight.register_hook()). However, I got nothing when trying to print weight.grad in this hook function. I think this is because the hook function is called before .grad field is populated as you mentioned. I wonder if the computed gradient is the input of this tensor hook operation.

Thanks again!

4

Yes, the hook is given the grad as the argument.
Note that if you want to modify the gradient that is saved in the .grad field, you can return the new value from the hook and it will be used.