Gradient clipping

Indeed. The original code seems to be available, too, and the function used for clipping is bound_range:

$ git clone https://github.com/szcom/rnnlib
$ rgrep bound_range  rnnlib/
rnnlib/src/MixtureOutputLayer.hpp:    bound_range(paramSigmaXY, almostZero, realMax);
rnnlib/src/MixtureOutputLayer.hpp:        bound_range(inputErrors[pt], -100.0, 100.0);
rnnlib/src/Helpers.hpp:static void bound_range(R &r,
rnnlib/src/Lstm1dLayer.hpp:      bound_range(inErrs, -10.0, 10.0);
rnnlib/src/LstmLayer.hpp:      bound_range(inErrs, -10.0, 10.0);
rnnlib/src/CharWindowLayer.hpp:      bound_range(inputErrors[coords], -10.0, 10.0);

I do look forward looking at pytorch code instead. :slight_smile:

Best regards

Thomas

2 Likes