RNN Pack Padded Sequence Hidden State

1

Hi,

When using the pack padded sequence, what should be the final lstm state that is passed to the decoder?

Should it be hidden or output?

packed = torch.nn.utils.rnn.pack_padded_sequence(embedded, input_lengths, batch_first=True)
output, hidden = self.text_LSTM(packed, None)
output, _ = torch.nn.utils.rnn.pad_packed_sequence(output, batch_first=True)

Cheers!

1 Like
2

The output contains the output of LSTM for all timesteps for all sequences.

The hidden has two components.

  1. The output of last timestep (of the individual sequences).
  2. The cell state after last timestep

In my understanding, you may want to pass the first component of the hidden variable (hidden[0]) to the decoder, as it represents the encoding of the whole sentence after having seen all the words in the sentence. Again, its a design choice about your network.

1 Like
3

thanks @InnovArul at some places like here i see output[:,-1,:] passed, are hidden[0] and output[:, -1,:] the same?

4

Yes, they are same. In addition, output contains the output from all the timesteps, but hidden[0] has only the output of the last timestep.

1 Like