I realize the op posted a long time ago but to keep people from getting confused by this.
These lines need to be removed
also these.
edit:
also remove this
self.hidden = self.initHidden()
from the forward pass, it’s reseting your hidden state to zero every time you make a forward pass.
and really just refer to the example language model for the rest! https://github.com/pytorch/examples/blob/master/word_language_model/model.py