Also, you could take a look in this thread. Perhaps, you are doing something similar with your model or optimizer.