Should LSTM/RNN state be kept when inferencing?

Say I am doing time series prediction which predict some value for next time step with past T inputs from historical inputs. Say I am using a RNN module like LSTM or GRU.

In trainning/validation, I’ll feed RNN module with batch of shape (batch_size, T, *) data to get a model.

When inferencing, I can either:

  1. Always use past T inputs to get next step prediction, then discard the state of the RNN module. That is: use input from time -T to -1 to get prediction at t=0, then discard the final state of the RNN module and use input from time -T+1 to 0 to get prediction at t=1 etc.
  2. keep the RNN state, and each time feed only one input to get the prediction. That is:
    first use input from time -T to -1 to get prediction at t=1 like above. Then feed the RNN with input at t=1 to get the prediction at t=2 etc.

Which one is better? Or It depends on specific problems? Thanks

Usually, you use the hidden state of the last layer for each time step t, what corresponds to your second option, I suppose. What is the idea behind option 1?