How to clip the gradient in a model during training alone ? Please help
tom
(Thomas V)
2
Gradient clipping is always only done in training (because you ordinarily don’t do backprop in evaluation).
There are two ways:
- backpropagate, then clip gradients using the utility functions,
- add backward hooks doing the clipping in flight.
You might see this somewhat old thread for a discussion of the details:
Best regards
Thomas