How to turn list of varying length tensor into a tensor

1

Hi, I am currently trying to do batch training on RNN.
The first step is to pad the batch of sequence using pack_padded_sequence().
But the function seems to take Variable as input, which means it need to be a tensor.

Currently, my input format is a list of tensors with varying length.
When I try to turn that list into a tensor, it throws me the error: ‘FloatTensor’ object does not support indexing

It seems that I cannot create tensor with varying length on any dimension.

My goal is to created a batch of padded sequence from list of tensors with varying length.
What may be the correct way of implementation ?

2 Likes
2

The code here will show you how to pad your tensors of different lengths into a single tensor used by pack_padded_sequence:

5 Likes
3

If I understand correctly, that means I need to go through some procedure to pad the sequence myself.
In this case, what are the purpose of pack_padded_seqence() and pad_packed_sequence() functions ?
I thought they were used somehow to pad sequence with varying length automatically.

2 Likes
4

These two functions help you pack (already) padded sequences and pad (already) packed sequences. I think the purpose is not to help you pad sequences, but to save some spaces. Check out the following example:

import itertools
import torch
import torch.nn as nn
import torch.nn.utils.rnn as rnn_utils
from torch.autograd import Variable, gradcheck
def test_pack_padded_sequence():
def pad(tensor, length):
return torch.cat([tensor, tensor.new(length - tensor.size(0), *tensor.size()[1:]).zero_()])
lengths = [2,1]#[10, 8, 4, 2, 2, 2, 1]
max_length = lengths[0]
batch_sizes = [sum(map(bool, filter(lambda x: x >= i, lengths))) for i in range(1, max_length + 1)]
offset = 0
padded = torch.cat([pad(i * 100 + torch.range(1, 5 * l).view(l, 1, 5), max_length) for i, l in enumerate(lengths, 1)], 1)
padded = Variable(padded, requires_grad=True)
expected_data = [[torch.range(1, 5) + i * 100 for i in range(batch_size)] for batch_size in batch_sizes]
expected_data = list(itertools.chain.from_iterable(expected_data))
expected_data = torch.cat(expected_data)
for batch_first in (True, False):
src = padded
if batch_first:
src = src.transpose(0, 1)
print(“SRCSRCSRC”)
print(src)
print(“LENGTHS”)
print(lengths)
packed = rnn_utils.pack_padded_sequence(src, lengths, batch_first=batch_first)
print(“PACKED”)
print(packed)
unpacked, unpacked_len = rnn_utils.pad_packed_sequence(packed, batch_first=batch_first)
print(“UNPACKED”)
print(unpacked)
print(“UNPACKED_LEN”)
print(unpacked_len)
test_pack_padded_sequence()

1 Like
5

And apart from saving space, it allows to use nn.LSTM with variable length sequences, which can speed up your model quite a lot if you’re using bidirectional RNNs, or using the last hidden states for some purposes. We don’t have a function that takes in a list of tensors and pad then, but it sounds useful. Can you open a feature request?

9 Likes
6

Any detailed examples on these 2 functions, pack_padded pad_packed? Seems confusing to me.

I thought, pack_padded would turn an already padded tensor (i.e. with each row having the same length, suppose 2d tensors here), into another one with each row having different length (just like sparse matrix?)

7

@ecolss see the examples in the release notes here: https://github.com/pytorch/pytorch/releases/tag/v0.1.10

3 Likes
8

As illustrated in the release notes, PackedSequence holds data and a list of sequence sizes of a packed sequence batch.

Then why would we have to first pad the seq to equal length manually and then pass it to pack_padded func to generate this PackedSequence object? Wouldn’t it be easier to just construct the object directly from the original seq without manual padding?

9

@ecolss , @apaszke suggested making a feature request for this as it seems useful. was wondering if you followed up on it?

10

I’m not following up on this, but I agree with you on this.

11

just to give a consistent link for those lines, you can press ‘y’ in github and get the link.
here is the updated link:

2 Likes
13

For anyone wondering if PyTorch has created a function that takes a list of tensors and pads them to be the same length, the following function does the trick:

https://pytorch.org/docs/stable/nn.html#torch.nn.utils.rnn.pad_sequence

Example:

>>> import torch
>>> l = [torch.Tensor([1., 2.]), torch.Tensor([3.])]
>>> torch.nn.utils.rnn.pad_sequence(l, batch_first=True, padding_value=0)
tensor([[1., 2.],
        [3., 0.]])
5 Likes