Is there an inverse of rnn.pack_sequence?

nlp
1

When we feed sentences into LSTM

# Variable length input sequence 'a' (without pad)
# 10 sentences, embedding size 5
a = [torch.randn(random.randint(1, 4), 5), 10]
a = torch.nn.utils.rnn.pack_sequence(a, enforce_sorted=True)
...
# here `out` is packed_sequence..
out, _ = model.lstm(a)

To unpack out,
Pytorch has torch.nn.utils.rnn.pad_packed_sequence which is an inverse operation to pack_padded_sequence. It gives unpacked and also padded sequences…
sure we can easily remove pads…
but sometimes people want nice and simple way… like an inverse of pack_sequence.
is there anything like this?

thanks

2

instead of sorting the sequences : using torch.nn.utils.rnn.pad_packed_sequences
you can try sorted(key, ‘reverse’) for sequence sorting without changing it actually.
and you can pass it directly into the input model and no need to reverse it accordingly

3

I was faced with this usecase and ended up writing a function to convert a packed sequence into a list of (unpadded) sequences. Sharing it here just in case it’s useful for someone else:

import torch.nn.utils.rnn as rnnutils

def unpack_sequence(packed_sequence, lengths):
    assert isinstance(packed_sequence, rnnutils.PackedSequence)
    head = 0
    trailing_dims = packed_sequence.data.shape[1:]
    unpacked_sequence = [torch.zeros(l, *trailing_dims) for l in lengths]
    # l_idx - goes from 0 - maxLen-1
    for l_idx, b_size in enumerate(packed_sequence.batch_sizes):
        for b_idx in range(b_size):
            unpacked_sequence[b_idx][l_idx] = packed_sequence.data[head]
            head += 1
    return unpacked_sequence
1 Like
4

I expanded on the code of @shaabhishek to allow sequences of length zero, but ended up not using it anyway. Hope this might help someone else.

from typing import List, Tuple
import torch
import torch.nn.utils.rnn as rnn


def pack(sequence: List[torch.Tensor]) -> Tuple[rnn.PackedSequence, List[int]]:
    lengths = list(map(len, sequence))
    tensors = [tensor for length, tensor in zip(lengths, sequence) if length > 0]
    packed_sequence = rnn.pack_sequence(tensors)
    return packed_sequence, lengths


def unpack(packed_sequence: rnn.PackedSequence, lengths: List[int]) -> List[torch.Tensor]:
    trailing_dims = packed_sequence.data.shape[1:]
    unpacked_sequence = []
    idx_map = {}
    head = 0
    for b_idx, length in enumerate(lengths):
        unpacked_sequence.append(torch.zeros(length, *trailing_dims))
        if length > 0:
            idx_map[head] = b_idx
            head += 1
    head = 0
    for l_idx, b_size in enumerate(packed_sequence.batch_sizes):
        for b_idx in range(b_size):
            unpacked_sequence[idx_map[b_idx]][l_idx] = packed_sequence.data[head]
            head += 1
    return unpacked_sequence