This thread may be relevant: [Solved] Training a simple RNN
A key point is that to keep hidden state Variable across batches without having to specify retain_graph=True you need to detach the Variable: hidden.detach().
2 Likes