I was reading Seq2seq. The below code is for getting output from LSTM for various time steps. Is it possible to eliminate this loop and get the output in one line?
for ei in range(input_length):
encoder_output, encoder_hidden = encoder(
encoder_outputs[ei] = encoder_output[0, 0]
Look at how the
forward() method of
Encoder is implemented. it embeds a single vector, and then runs it through the GRU (rather than LSTM here) stored on that class. the pytorch GRU and LSTM classes can, by default take a sequence in the form of [batch_length, sequence_length, embedding_dim]. You can therefore embed a sequence of
input_length to be a tensor with dimensions [1, sequence_length, embedding_dim], and run that through the LSTM in one go, allowing the efficient pytorch code to iterate over it, rather than a python loop. You can even batch your sequences and embed those, and pass them through an LSTM or GRU.
So in batch case how would you ensure that decoder terminates when EOS token is an output of current time step?
Since in loop case we can know the output at each time step and stop when EOS is output as below:
topv, topi = decoder_output.topk(1)
decoder_input = topi.squeeze().detach()
if decoder_input.item() == EOS_token:
Hi, Above you have the encoder, which can be done without the loop.
For decoding, I don’t think you can simply pass an entire sequence, because you need to make a prediction at each timestep before moving to the next one. The RNN is informed by both the encoder, and the previously predicted item in the sequence. You can, of course, still use a batch, just not a batch of an entire sequence.
E.g. a tensor of (batch_size, 1, dims) should be doable.