DataParallel for RNN (using packing)

[fashandge]

This answer might be helpful,

Multi layer RNN with DataParallel

Here is one solution for using nn.DataParallel I find works well. It can return both rnn output and hidden states from your module, using batch_first = False mode (which is a popular mode). Use batch_first = False for seq2seq modeling (e.g., encoder-decoder architecture), which seems to the be the popular way for feeding the input data, so we get input batch with shape (max_sequence_length, batch_size, num_embeddings). Then set dim=1 to scatter the data the along the dimension for batches …