Problem UnderstandingThe PackedSequence

nlp
1 
import torch
import torch.nn as nn


batch_size = 3
max_length = 3

batch_in = torch.LongTensor(batch_size, max_length).zero_()
batch_in[0] = torch.LongTensor([1, 2, 3])
batch_in[1] = torch.LongTensor([4, 5, 0])
batch_in[2] = torch.LongTensor([6, 0, 0])
seq_lengths = [3, 2, 1]
pack = torch.nn.utils.rnn.pack_padded_sequence(batch_in, seq_lengths, batch_first=True)
print(pack)

then the output is

PackedSequence(data=tensor([ 1,  4,  6,  2,  5,  3]), batch_sizes=tensor([ 3,  2,  1]))

From what I under stand,if we put sentences into RNN or LSTM, the sentence should be in order,but from the output, the data is 1,4,6,2,5,3,which pack the sequence in a vertical way.If we put this packed sequence into RNN, how does the RNN read the data?

2 
import torch
import torch.nn as nn


batch_size = 3
max_length = 3

batch_in = torch.LongTensor(batch_size, max_length).zero_()
batch_in[0] = torch.LongTensor([1, 2, 3])
batch_in[1] = torch.LongTensor([4, 5, 0])
batch_in[2] = torch.LongTensor([6, 0, 0])
seq_lengths = [3, 2, 1]
pack = torch.nn.utils.rnn.pack_padded_sequence(batch_in, seq_lengths, batch_first=True)
print(pack)
pack = torch.nn.utils.rnn.pad_packed_sequence(pack, batch_first=True)[0]
print(pack)


PackedSequence(data=tensor([ 1,  4,  6,  2,  5,  3]), batch_sizes=tensor([ 3,  2,  1]))
tensor([[ 1,  2,  3],
        [ 4,  5,  0],
        [ 6,  0,  0]])

That is right.
LSTM is essentially a recurrent neural network, which first inputs the first step, then the second step,…
And it gets the input value based on the length

3

From what I understand, my input is 1 2 3,4 5 6 ,but from the tensor in the data, it is 1,4,6,2,5,3 which is in a different order from what I expected which is 1 2 3 4 5 6. The batch_sizes also confuses me.Am I under standing this properly?

4

I think:
the first-step : get [1, 4, 6] according to length 3
the second-step : get [2, 5] according to length 2

recurrent neural network run step by step, instead of batch by batch
But every step its input is a batch size data (e.g., 3 and 2).

5

So it deals with data in a vertical way?
if we store the data in a vertical way, and input it in the RNN, like[I like your cooking]and [i like apple], if we pack this 2 sequence. Then the input to the RNN will be [I I] at the first step, and not the[ I like your cooking] as the first batch. Is this the right understanding?
But if it is like this, the input order is not correct.

6

I think:
the first step, LSTM cell can not process the one batch data, so the input data is [i, i].
why do you say that the input order is not correct ?
So what is the correct input order ?

7

I think the input is the sequence of words,like[ I love your cooking ],as a sequence of words,not [I I]

8

the entire input sequence is definitely [ I love your cooking ],
but the input of every step is [I], [love], …
Then consider batch size

9

But if we pack these 2 sequence,the result is [I,I,love,eat,your,apple,cooking],if we put this into RNN,how does the batch_size work to reconstruct a right input order?Why don’t this function directly pack the sentence like[ I ,love ,your ,cooking, I, eat ,apple]

10

I guess I understand your point now,it deals witl coloumn just to make sure it reads data step by step,like [i,i], however, the LSTM gets the input still in the sequence of words,the final input is still [I,love, your ,cooking],the batch sizes tells how many data we have at each step then it knows how to input the data.Is this right way to understand this?

11

Right, you finally understand my point.
This is my means.
:smile:

1 Like
12

Thank you.I was really confused about this several days ago.:grinning: