How to initialize the hidden state of a LSTM?

[tom (Thomas V)]

Even though that is in the context of BPTT, I think similar considerations apply as discussed the first item in:

Initialization of first hidden state in LSTM and truncated BPTT

Yes, zero initial hiddenstate is standard so much so that it is the default in nn.LSTM if you don’t pass in a hidden state (rather than, e.g. throwing an error). Random initialization could also be used if zeros don’t work. Two basic ideas here: If your hidden state evolution is “ergodic”, the state will move closer to some “steady distribution” anyways, so it doesn’t matter as much. You want the initial hidden state handling to be somewhat consistent between training and inference. The fan…

Best regards

Thomas