I am using LSTM to model time-series data. Since the data that I am using are long and of variable lengths depending on the day (5000 ~ 20,000 data points per day depending on the day), I would like to limit back-propagation through time to a controlled maximum number, let’s say 100.
That is, regardless of where you stand in time, back-prop is 100 time steps backward at most.
Is there a way to do that ?
you can call detach() every hundred timesteps to detach the hidden state to it’s previous states.
That way, you will end up with:
without detatch
x -> x -> x -> x-> x-> x-> x-> … -> x -> x-> x -> x -> x -> x-> x-> x-> x->
after detaching every T timesteps
x -> x -> x -> x-> x x -> x -> x -> x-> x x -> x -> x -> x-> x x -> x -> x -> x-> x x -> x -> x -> x-> x
Thank you. Sounds great.
Just one more question. The final hidden state (from the final time step) is fed to the next layer of my network and eventually fed to the final loss function.
Detaching the hidden states at every T steps wouldn’t distort this in any strange way, would it ?