Question about changing optimizer

chunchun · January 11, 2020, 9:13am

I get the error ‘nan or inf for input tensor’ when I change SGD to RMS,Why?

ptrblck · January 12, 2020, 1:47am

Could you post the error message, please?
Do you get this error immediately after changing the optimizer?
Is the output of your model NaN or Inf?

chunchun · January 13, 2020, 11:37am

I just change the optimizer,there will be the error like this.

chunchun · January 13, 2020, 1:02pm

and the loss becomes nan.
Why?

ptrblck · January 14, 2020, 5:05am

Once the parameters become NaN, e.g. due to a high learning rate, your output will also be NaN.
Could you try to initialize the new optimizer with a smaller learning rate and retry the code again?

chunchun · January 14, 2020, 6:06am

Yes,I have tried 0.00001 but it doesn’t work.
I try to print the gradient ,it’s sometimes very small(about 0),sometimes very large(about 100000).
Could you please tell me why?

ptrblck · January 14, 2020, 6:18am

I’m not sure, but my guess would be the internal functionality of RMSProp.
This optimizer divides the gradient by a running average of its recent magnitude.
If your gradients are quite small, since you’ve already trained your model for a few epochs, I assume this division might blow up.

chunchun · January 16, 2020, 6:46am

Thanks a lot for your answer.