Hello
I’m training my small many-to-one LSTM network to generate chars one at a time.
my training data are splited into mini-batches, each batch has the following shape
[batch size, sequence Length, Number of feature]
with batch_first=True in my LSTM unit
Now, after forward feeding my mini-batch to the network and calculating CrossEntropyLoss(), i call
Loss.backward() just one time so the gradients gets calculated and stored in .grad attribute.
My questions is :
consider a batch size of 16, isn’t gradient should be calculated 16 times (after each sequence in mini-batch) and accumulated then Averaged before updating the parameters using (optimiser.step()).
do Loss.backward() calculate the gradients across each mini-batch sequence or just calculate it one time after last sequence in batch. if so, how can i calculate gradient after each sequence in mini-batch then update weights by average gradients at the end.
Here is my training loop:
for epoch in range(num_epochs):
for startidx in range(0, num_batches, batch_size):
endidx = startidx + batch_size
step = startidx//batch_size
xbatch = xhot_seq[startidx:endidx]
ybatch = yhot_seq[startidx:endidx]
# Forward pass
# Initialise hidden state
hidden = model.init_hidden()
# Clear stored gradient
model.zero_grad()
y_pred, hidden = model(xbatch, hidden)
target = torch.argmax(ybatch.long(),dim=1)
loss = loss_fn(y_pred, target )
loss_hist[epoch] = loss.item()
# Backward pass
loss.backward()
# Update parameters
optimiser.step()
Thanks
