CTCLoss predicts all blank characters

I am using crnn.pytorch (https://github.com/meijieru/crnn.pytorch), Pytorch 0.4.0 and SeanNaren/warp-ctc to recognize handwritten documents. The training loss does not decrease and the model predicts all BLANK characters.

It turns out that the learning rate should be relatively small (e.g. 0.001 instead of 0.01).