I’m trying to make a basic RNN model to use on some torchtext datasets, initially to try and complete an assignment in the Duke University ML course but having to piece together ideas from the internet because the instruction there is very lacking.
The problem I have is that there doesn’t appear to be any learning happening. After each epoch, the output accuracy is equal to chance, and loss does not seem to decrease at all. (It is a classification problem with 4 possible outputs and the correct predictions are about 1 in 4.)
The first thing of note is that it uses a word embedding layer, and the intent is that it learns this layer alongside the RNN itself. The Duke instructors suggest this is common but most of the examples I found online suggest otherwise, though the nn.Embedding layer does seem to support learning weights.
The next thing is that I am very confused about how (and if) to use the hidden argument to the RNN. Many examples online are using an init_hidden member function (often copy-and-pasted from somewhere else apparently), and imply that setting the hidden input to the RNN is important. However the NLP tutorial that Pytorch provides (NLP From Scratch: Classifying Names with a Character-Level RNN — PyTorch Tutorials 2.8.0+cu128 documentation) leaves it entirely unspecified.
I won’t post the whole code as I suspect most of it is irrelevant, but the model and the training loop are here:
class MyRNN(nn.Module):
def __init__(self, vocab_size: int, embedding_size: int, hidden_dim: int, num_outputs: int):
super().__init__()
self._hidden_dim = hidden_dim # remember this value in case we need it
self.embedding = nn.Embedding(num_embeddings=vocab_size, embedding_dim=embedding_size)
self.rnn = nn.RNN(input_size=embedding_size, hidden_size=hidden_dim, num_layers=1)
self.hidden_to_output = nn.Linear(hidden_dim, num_outputs)
self.softmax = nn.LogSoftmax(dim=1)
def forward(self, x):
embed = self.embedding(x)
rnn_out, hidden = self.rnn(embed) # no Hx provided, "defaults to zeros". Is this right?!
hidden_one_layer = hidden[0] # only one layer, so extract that
pre_output = self.hidden_to_output(hidden_one_layer)
output = self.softmax(pre_output)
return output
model = MyRNN(VOCAB_SIZE, EMBED_DIM, HIDDEN_DIM, NUM_OUTPUTS)
model.cuda()
model.train() # no effect here apparently, but could matter in other contexts
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.005)
# Iterate through training set minibatches
for epoch in range(NUM_EPOCHS):
print(f"Starting epoch {epoch}")
epoch_loss = 0
model.zero_grad() # clear the gradients each epoch
# train_loader is an instance of torch.utils.data.DataLoader,
# wrapping a torchtext.datasets.TextClassificationDataset
for inputs, labels in tqdm(train_loader):
# push cpu stuff to cuda
inputs = inputs.to(device)
labels = labels.to(device)
# Forward pass, measure the result loss
y = model(inputs)
loss = criterion(y, labels)
# Backward pass
loss.backward()
optimizer.step()
optimizer.zero_grad()
epoch_loss += loss.item()
print(f'Epoch [{epoch + 1}/{NUM_EPOCHS}], Loss: {epoch_loss / len(train_loader):.4f}')
I’d be grateful to hear about what I am doing wrong or suboptimally, or things I could try if it’s not quite as obvious as that.