Hello,
I’d like to build with Pytorch a Bidirectional Stacked LSTM ( Stacked DT-RNN)with fully connected layers between the hidden states as suggested in Pascanu, Razvan, Gulcehre, Caglar, Cho, Kyunghyun, and Bengio, Yoshua. How to Construct Deep Recurrent Neural Networks. arXiv
preprint arXiv:1312.6026, pp. 1–10, 2013.
Pythorch nn.LSTM don’t support this features: you don’t have the control on each time step. So you can’t add a fully connected network which takes in input the hidden state at time t and gives an output for the hidden state at time t+1 . So i suppose that the usage of nn.LSTMCell is the only option. At this point many issues born:
- LSTM cell implementation is not bidirectional and it works on single timesteps. So if i’d like to build a 2 layer LSTM i need 2 LSTM cell and 4 for loops in the forward method to iterate over the sequences ( one for the original sequence and one for the reversed one for each layer). Operations which i am constrained to do sequentially, but that are strongly parallelizable. It would be very slow.
- The input sequences, which i use, have different lenghts; So i have padded them. LSTM cell don’t support pad_packed_sequences, because it works on a single time step. I’d like to work with batch samples to have reasonable time performances. I have tried to get the the resulting tensor from torch.nn.utils.rnn.pack_padded_sequence(myBatch) and feed it into a single LSTM cell, grouping them in smaller batches (let us call them “mini_batches”), whose length vary with the length of sequences. In this case the issue is that the mini_batches have different sizes and consequently the hidden state size to feed in the hidden-to-hidden transition too.
Do you have any suggestions to solve these issues?
Thank you in advance