How to initialize the hidden state of a LSTM?

[Peter_Ham]

in order to use LSTM, you need a hidden state and a cell state, which is not provided in the first place. My question is how to you initialize the hidden state and the cell state for the first input? If it is randomly initialized then if I feed into the second input, the same initialization should also work to predict the next output. But it does not make sense to me that inputting different hidden/cell state would come up with the same output.

[tom (Thomas V)]

Even though that is in the context of BPTT, I think similar considerations apply as discussed the first item in:

Initialization of first hidden state in LSTM and truncated BPTT

Yes, zero initial hiddenstate is standard so much so that it is the default in nn.LSTM if you don’t pass in a hidden state (rather than, e.g. throwing an error). Random initialization could also be used if zeros don’t work. Two basic ideas here: If your hidden state evolution is “ergodic”, the state will move closer to some “steady distribution” anyways, so it doesn’t matter as much. You want the initial hidden state handling to be somewhat consistent between training and inference. The fan…

Best regards

Thomas