Variational RNN

1

Original LSTM cell uses dropout that uses different mask at every time step which is ad-Hoc and it leads to unstable results.According this paper we should use same dropout masks at every time step. Variational RNN

Here is the screenshot what should ideally happen

Screenshot_2020-05-03 1512 05287 pdf
Screenshot_2020-05-03 1512 05287 pdf704×270

Keras supports this with (dropout and recurrent dropout)
Is there any neat implementation for this pytorch? Thanks for Helping

2

I think the answer is no. I have also looked for this feature but not found anything. In my Keras model it improves accuracy by about 5% so it’s the main reason I haven’t been able to port my model to PyTorch yet.

3

I spoke to soon. Seems like this LSTM supports variational dropout: https://github.com/keitakurita/Better_LSTM_PyTorch