Deprecation of register_backward_hook problematic

I’ve started recently needing and using the register_backward_hook and I found the warning that this function is deprecated in favor of the full backwards hook, which I find confusing because using the full backwards hook in various applications would require more redundant code and be more error prone.

Consider that one wants to manipulate the gradients of the output layer of their neural network, and only the output layer. A registered backward hook makes this simple, but a full backward hook requires one to add a redundantly running if statement to verify the correct module is being targeted.

Am I perhaps missing something that makes the use of a full backward hook concise and neat for this application that I’m unaware of?.. To me it seems like the simple backward hook is very useful.

You can also register a full backward hook to a specific module, perhaps you were using global hooks?

One difference between a regular backwards hook and a full backwards hook is that full backwards hook fires when the gradient wrt the inputs the module are computed, so you’d need to register another dummy module and then register hook if you’d like mutate gradients before running backward for that module. (We also have a new type of hook available on the latest nightly that fires when the gradients for the module outputs are computed, but you would be unable to modify the gradients).

Another difference to note for full backwards (pre)hook is that if your module returns multiple outputs, only a single hook would be registered and when that hook is called you’d be able to see gradients wrt all the outputs.

1 Like