Its been months I’ve been trying to use pack_padded_sequence with LSTM. My current setup I’m working with data that is in a python list of tensors shape 2x(some variable length) such as torch.Size([2, 2466]).
I have a data loader with a custom collate_fn that is pretty much same as found here: Use PyTorch’s DataLoader with Variable Length Sequences for LSTM/GRU with the exception I don’t sort in order of length. I’m using pytorch 1.1.0.
def padSequence(batch): # each element in "batch" is a tuple (data, label) # Get each sequence and pad it sequences = [x for x in batch] #sorted_batch] sequences_padded = torch.nn.utils.rnn.pad_sequence(sequences, batch_first=True) # Also need to store the length of each sequence # This is later needed in order to unpad the sequences seq_lengths = [len(x) for x in sequences] # need to pad labels too labels = [x for x in batch] labels_padded = torch.nn.utils.rnn.pad_sequence(labels,batch_first=True) label_lengths = [len(x) for x in labels] return sequences_padded, seq_lengths, labels_padded, label_lengths
The data loader’s output with batch=5 gives:
tensor([[8.2430, 8.2793, 8.3186, ..., 0.0000, 0.0000, 0.0000], [6.6331, 6.6288, 6.6292, ..., 0.0000, 0.0000, 0.0000], [4.2062, 4.2408, 4.2675, ..., 3.4694, 3.4807, 3.4933], [3.5047, 3.5154, 3.5313, ..., 0.0000, 0.0000, 0.0000], [5.4685, 5.4138, 5.3533, ..., 0.0000, 0.0000, 0.0000]], dtype=torch.float64)
And a list of sequence lengths:
[474, 473, 1160, 533, 555]
Feeding it to
torch.nn.utils.rnn.pack_padded_sequence(sequences_padded, seq_lengths, batch_first=True, enforce_sorted=False) I get an output something like this:
PackedSequence(data=tensor([4.2062, 5.4685, 3.5047, ..., 3.4694, 3.4807, 3.4933], dtype=torch.float64), batch_sizes=tensor([5, 5, 5, ..., 1, 1, 1]), sorted_indices=tensor([2, 4, 3, 0, 1]), unsorted_indices=tensor([3, 4, 0, 2, 1]))
And then passing to my lstm I get the error:
RuntimeError: input must have 2 dimensions, got 1
Which I found similar in this post: Pytorch passing PackSequence argument to LSTM
Now I tried making my data in the same shape with my collate_fn:
def padSequence(batch): sequences = [x.reshape((x.shape,1)) for x in batch] sequences_padded = torch.nn.utils.rnn.pad_sequence(sequences, batch_first=True) seq_lengths = [len(x) for x in sequences] labels = [x.reshape((x.shape,1)) for x in batch] labels_padded = torch.nn.utils.rnn.pad_sequence(labels,batch_first=True) label_lengths = [len(x) for x in labels] #torch.LongTensor([len(x) for x in labels]) return sequences_padded, seq_lengths, labels_padded, label_lengths
However then I keep getting the error:
RuntimeError: input.size(-1) must be equal to input_size. Expected 2, got 1
I’ve tried changing the shape of the original data making it a list of tensors with variable length x 2 and so many other things getting same errors. At this point I don’t know what is wrong and where to go from here in understanding what pytorch input wants for a simple LSTM w/ packed_padded sequence… I’ve read through so many posts and searched a lot these past few months with no help at all. Any direction would help.
Including this if could give any clues:
input_size = 1 hidden_size = 1 output_size = 1 num_layers = 2 num_classes = 4 class Model(nn.Module): def __init__(self, input_size, hidden_size, num_layers, num_classes): super(Model, self).__init__() self.hidden_size = hidden_size self.num_layers = num_layers self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True, bidirectional=False) self.fc = nn.Linear(hidden_size, num_classes) def forward(self, x, X_lengths): # Set initial hidden and cell states h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).cuda() c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).cuda() packed = torch.nn.utils.rnn.pack_padded_sequence(x, X_lengths, batch_first=True, enforce_sorted=False) # Forward propagate LSTM X, _ = self.lstm((packed), (h0,c0)) #X, _ = torch.nn.utils.rnn.pad_packed_sequence(X, batch_first=True) out = self.fc(X) #[:, -1, :]) return nn.functional.softmax(out,dim=2)