When should we use pack_sequence, pack_padded_sequence and pad_packed_sequence

jef · April 17, 2018, 2:51pm

Recently, I found pack_sequence, pack_padded_sequence, and pad_packed_sequence for RNN modules. But I am not sure when these functions are useful.

Q1. To use pack_padded_sequence, sorting tensors by length is needed for every mini-batch. This is cost. On the other hand, I heard pack_padded_sequence skip calculation of padded elements. Compared them, skipping calculation is much beneficial? The below link is an example of pack_padded_sequence.

gist.github.com

https://gist.github.com/Tushar-N/dfca335e370a2bc3bc79876e6270099e

pad_packed_demo.py

import torch
import torch.nn as nn
from torch.autograd import Variable
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
import torch.nn.functional as F
import numpy as np
import itertools

def flatten(l):
	return list(itertools.chain.from_iterable(l))

This file has been truncated. show original

Q2. Although pack_sequence looks similar to pack_sequence, pack_sequence doesn’t need to use padded data. But what kind of situations can we use this function?

Thank you!

eugene · July 23, 2018, 6:04am

With pack_sequence in PyTorch 0.4, you can directly call it without padding zeros.

>>> import torch
>>> import torch.nn.utils.rnn as rnn_utils
>>> a = torch.Tensor([1, 2, 3])
>>> b = torch.Tensor([4, 5])
>>> c = torch.Tensor([6])
>>> packed = rnn_utils.pack_sequence([a, b, c])

yihan0512 · August 1, 2018, 2:55am

I’m having the same confusion as your Q2. The following link is the only use case about pack_sequence I found: Code.