Trouble with packed_padded_sequence and LSTM for variable length inputs

1

Its been months I’ve been trying to use pack_padded_sequence with LSTM. My current setup I’m working with data that is in a python list of tensors shape 2x(some variable length) such as torch.Size([2, 2466]).

I have a data loader with a custom collate_fn that is pretty much same as found here: Use PyTorch’s DataLoader with Variable Length Sequences for LSTM/GRU with the exception I don’t sort in order of length. I’m using pytorch 1.1.0.

def padSequence(batch):
  # each element in "batch" is a tuple (data, label)
  # Get each sequence and pad it
  sequences = [x[0] for x in batch] #sorted_batch]
  sequences_padded = torch.nn.utils.rnn.pad_sequence(sequences, batch_first=True) 
  # Also need to store the length of each sequence
  # This is later needed in order to unpad the sequences
  seq_lengths = [len(x) for x in sequences]
  # need to pad labels too
  labels = [x[1] for x in batch]
  labels_padded = torch.nn.utils.rnn.pad_sequence(labels,batch_first=True)
  label_lengths = [len(x) for x in labels] 
  return sequences_padded, seq_lengths, labels_padded, label_lengths

The data loader’s output with batch=5 gives:

tensor([[8.2430, 8.2793, 8.3186,  ..., 0.0000, 0.0000, 0.0000],
        [6.6331, 6.6288, 6.6292,  ..., 0.0000, 0.0000, 0.0000],
        [4.2062, 4.2408, 4.2675,  ..., 3.4694, 3.4807, 3.4933],
        [3.5047, 3.5154, 3.5313,  ..., 0.0000, 0.0000, 0.0000],
        [5.4685, 5.4138, 5.3533,  ..., 0.0000, 0.0000, 0.0000]],
       dtype=torch.float64)

And a list of sequence lengths: [474, 473, 1160, 533, 555]

Feeding it to torch.nn.utils.rnn.pack_padded_sequence(sequences_padded, seq_lengths, batch_first=True, enforce_sorted=False) I get an output something like this:

PackedSequence(data=tensor([4.2062, 5.4685, 3.5047,  ..., 3.4694, 3.4807, 3.4933],
       dtype=torch.float64), batch_sizes=tensor([5, 5, 5,  ..., 1, 1, 1]), sorted_indices=tensor([2, 4, 3, 0, 1]), unsorted_indices=tensor([3, 4, 0, 2, 1]))

And then passing to my lstm I get the error:
RuntimeError: input must have 2 dimensions, got 1

Which I found similar in this post: Pytorch passing PackSequence argument to LSTM

Now I tried making my data in the same shape with my collate_fn:

def padSequence(batch):
  sequences = [x[0].reshape((x[0].shape[0],1)) for x in batch]
  sequences_padded = torch.nn.utils.rnn.pad_sequence(sequences, batch_first=True)
  seq_lengths = [len(x) for x in sequences] 
  labels = [x[1].reshape((x[1].shape[0],1)) for x in batch]
  labels_padded = torch.nn.utils.rnn.pad_sequence(labels,batch_first=True)
  label_lengths = [len(x) for x in labels] #torch.LongTensor([len(x) for x in labels])
  return sequences_padded, seq_lengths, labels_padded, label_lengths

However then I keep getting the error: RuntimeError: input.size(-1) must be equal to input_size. Expected 2, got 1

I’ve tried changing the shape of the original data making it a list of tensors with variable length x 2 and so many other things getting same errors. At this point I don’t know what is wrong and where to go from here in understanding what pytorch input wants for a simple LSTM w/ packed_padded sequence… I’ve read through so many posts and searched a lot these past few months with no help at all. Any direction would help.

Including this if could give any clues:

input_size = 1 
hidden_size = 1 
output_size = 1 
num_layers = 2
num_classes = 4

class Model(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(Model, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True, bidirectional=False)
        self.fc = nn.Linear(hidden_size, num_classes)
    
    def forward(self, x, X_lengths):
      
      # Set initial hidden and cell states 
      h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).cuda()
      c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).cuda()
      
      packed = torch.nn.utils.rnn.pack_padded_sequence(x, X_lengths, batch_first=True, enforce_sorted=False)
      # Forward propagate LSTM
      X, _ = self.lstm((packed), (h0,c0))
      #X, _ = torch.nn.utils.rnn.pad_packed_sequence(X, batch_first=True)

      out = self.fc(X) #[:, -1, :])
      return nn.functional.softmax(out,dim=2)

Thanks!

2

Turns out I didn’t know how to assign variables to cuda or cpu and things got all out of whack. Turns out my packed padded sequence was not assigned to a cuda device. Here is a simple example I used to test things that helped me figure it out.

import torch 
import torch.nn.utils.rnn as rnn_utils 
a = torch.Tensor([[1], [2], [3]]) 
b = torch.Tensor([[4], [5]]) 
c = torch.Tensor([[6]]) 
d = torch.Tensor([[7],[8],[9],[10]])
batch = [a,b,c,d]
padded= rnn_utils.pad_sequence(batch, batch_first=True)
sorted_batch_lengths = [len(x) for x in padded]
packed = rnn_utils.pack_padded_sequence(padded, sorted_batch_lengths, batch_first=True, enforce_sorted=False).cuda()
lstm = torch.nn.LSTM(input_size=1, hidden_size=3, batch_first=True).cuda()
lstm(packed)

Hopefully this will help someone else out in the future so you don’t spend months stuck.