TL;DR: How do I use nn.RNN() to generate sequences word by word? Do I feed it a seq_len = 1 word with a GO_TOKEN, and keep iterating that way? My experience with doing that is the RNN module just returns an output of zeros for some reason (after it has been properly trained).
Longer version
I trained a model using pytorch’s nn.RNN() module by feeding it batches of sequences (of varying seq_len) and properly padding when sequences in the batch were of different lengths.
I’m now trying to generate sequences using the trained RNN model by feeding a GO_TOKEN and iterating until an END_TOKEN is reached. I make a sequence of length one containing the index of my GO_TOKEN and feed that to my trained RNN. Unfortunately what I’m getting back are all 0s. Has anyone done this before?
Some code in case that helps:
class Model(nn.Module):
def __init__(self,
inputs_vocab_size,
embedding_dim,
out_units=128,
hidden_size=128,
nlayers=1,
num_directions=1,
dropout=0.1):
super(Model, self).__init__()
...
self.encoder = nn.Embedding(inputs_vocab_size, embedding_dim)
self.rnn = nn.RNN(embedding_dim, hidden_size, nlayers, dropout=0.5)
self.decoder = nn.Linear(hidden_size * num_directions, inputs_vocab_size)
...
Any ideas?