I am now trying to train a 3-layer LSTM model with sequence samples of various length. The tools that I use are pack_padded_sequence and pad_packed_sequence. I refer the examples in (About the variable length input in RNN scenario) to implement my own version. However I met a similar situation as posted in the link. My network does not converge. My code is shown below:
The code used for batch sorting : (to make sure the samples in the batch are sorted based on length, in descending order )
def sort_batch(x, length):
batch_size = x.size(0) # get size of batch
sorted_length, sorted_idx = length.sort() # sort the length of sequence samples
reverse_idx = torch.linspace(batch_size-1,0,batch_size).long()
reverse_idx = reverse_idx.cuda(GPU_ID)
sorted_length = sorted_length[reverse_idx] # for descending order
sorted_idx = sorted_idx[reverse_idx]
sorted_data = x[sorted_idx] # sorted in descending order
return sorted_data, sorted_length
forward function:
def forward(self, x, l):
x = pack_padded_sequence(x, list(l.data), batch_first = True) # pack batch
r_out, (h_n, h_c) = self.rnn(x, None) # pass in LSTM model
r_out, _ = pad_packed_sequence(r_out, batch_first = True) # unpack batch
idx = (l-1).view(-1,1).expand(r_out.size(0), r_out.size(2)).unsqueeze(1).long()
r_out = r_out.gather(1, idx).squeeze().unsqueeze(1) # get last hidden output of each sequence
out = self.out(r_out[:, -1, :])
return out
training :
for step, (x, y, l) in enumerate(dset_loaders['Train']): # gives batch data
b_x = Variable(x.cuda(GPU_ID).view(-1, TIME_STEP, INPUT_SIZE)) # reshape x to (batch, time_step, input_size)
b_y = torch.squeeze(Variable(y.cuda(GPU_ID))) # batch y
b_l = torch.squeeze(Variable(l.cuda(GPU_ID)))
b_x, b_l = sort_batch(b_x, b_l)
output = model(b_x, b_l) # rnn output
loss = loss_func(output, b_y) # cross entropy loss
optimizer.zero_grad() # clear gradients for this training step
loss.backward() # backpropagation, compute gradients
optimizer.step() # apply gradients
The output is :
Data Loading : 100% [==========================================================================================] Elapsed Time: 0:01:15 | Time: 0:01:15
Train Set : 60 Categories, 33212 Samples in total
Data Loading : 100% [==========================================================================================] Elapsed Time: 0:00:42 | Time: 0:00:42
Test Set : 60 Categories, 17885 Samples in total
('\n', RNN (
(rnn): LSTM(75, 100, num_layers=3, batch_first=True)
(out): Linear (100 -> 60)
))
LR is set to 0.01
Epoch: 0 | train loss: 4.0995 | test accuracy: 0.0167
Epoch: 0 | train loss: 4.1131 | test accuracy: 0.0165
Epoch: 0 | train loss: 4.1193 | test accuracy: 0.0161
Epoch: 0 | train loss: 4.1162 | test accuracy: 0.0208
Epoch: 0 | train loss: 4.1005 | test accuracy: 0.0166
Epoch: 0 | train loss: 4.1043 | test accuracy: 0.0164
Epoch: 0 | train loss: 4.0987 | test accuracy: 0.0167
Epoch: 0 | train loss: 4.1065 | test accuracy: 0.0167
Epoch: 0 | train loss: 4.1033 | test accuracy: 0.0167
I am not sure where the problem is. Am I using the pack_padded_sequence and pad_packed_sequence in a wrong way? Is there any way that I can debug this model?
Besides, I want to use gradcheck to debug my code. However I don’t know how to use it in my case. Anyone can help?
Thank you so much.
