Packing in RNNs

nlp
1

Hi,
For the following data:

Input(names) - [‘Griffiths’, ‘Jemaldinov’, ‘Naifeh’]
True Output(Language) - [‘English’, ‘Russian’, ‘Arabic’]

When used pack_padded_sequence for the above names the output generated is

J G N e r a m i i a f f l f e d i h i t n h o s v

which is clearly not in the order in which it were sent because of their varying lengths.

The question is, in what order should I arrange the true outputs so that they are compared to their corresponding predicted output for the computation of loss.

I hope I’m able to explain my doubt :sweat_smile:

2

short answer:
before using pack_padded_sequence you need to sort the inputs by length and pass that to the function.
sorting will return the permutation index and you can use that to permute your output

x = ['aaa', 'bbbb', 'cc']
lengths = torch.tensor([3,  4, 2])
lengths_permuted, perm_index = length.sort(0, descending=True)
x_permuted = x[perm_index]
x_packed = torch.nn.utils.pack_padded_sequence(x_permuted, lengths)
lstm_out_packed, h = LSTM(x_packed, h)
output_permuted = output[perm_index]

longer answer:
when using packed sequence the strings are rearranged by length
then they are stacked vertically
in your example it will be
Jemaldinov
Griffiths
Naifeh
each iteration it will process the i’th column across all rows if exists and pass it to the next rnn cell
that’s why it seems to you like the letters got scrambled.

2 Likes
3

Thanks, I liked the shorter answer :+1: