Simon,
Thank you for you answer.
I have modified the code for the toy example. If you could take a look at the full code and see if there’s something immediately obvious, that would be great!
I have inspected all the weights during each time step and none of them has nan in it. But the softmax or logsoftmax layer output is nan. This is very bizarre. Here is the link to the code: it should be runnable: Embeddings not getting updated
Thanks in advance.