Speech recognition gets stuck

Im trying to run a model similar to the one shown here Building an end-to-end Speech Recognition model in PyTorch
It keeps getting stuck at 2.8 loss with 5 rnn’s but if i decrease it to 4 it actually learns but with a terrible error rate of 70 percent. Is there any way i could fix this.