Merge two variable length sequence together?

1

Hi, all

How can I merge two variable sequences together? Like the example below, with word and image token sequence (batch_first=False) and their length

w_input = Variable( torch.randn( 20, 4, 50 ) )
w_len = Variable( torch.LongTensor( [ 20, 15, 10, 5 ] ) )
i_input = Variable( torch.randn( 3, 4, 50 ) )
i_len = Variable( torch.LongTensor( [ 3, 2, 1, 1 ] ) )

What I want is a new variable:
token_input = Variable( torch.FloatTensor( 23, 4, 50 ).fill_(0) )
whose data come from the above two sequences, with their lengths
token_len = w_len + i_len

Any hints here?

Thanks so much for your help.

2 Likes
2

You’re probably looking for torch.cat

3

Thanks for the note.

Yeah. torch.cat can help. But what I further need to do is to copy the last few tokens’ embedding to previous slots if w_len is less than the max_len (20 here) in first token sequence. That may needs a for loop. So do you know any other efficient ways of dong this. Thanks for your precious time.

4

um no, there’s no automatic way of padding the tensors. I’d just fill them with 0s and copy only the relevant parts in a for loop :confused:

5

@apaszke Thanks.

BTW, do we need to fill them with 0’s after ‘torch.cat’ if we’ve provided the length for each sequence? I don’t think RNN will compute sequence tokens after the timestep of sequence length , right?

6

Oh yeah actually if you use torch.nn.utils.rnn.pack_padded_sequence then you can just put any random values in the padding. They will be ignored anyway

7

Hi @Yufeng_Ma,

did you solve your problem? I have the same problem. I have two tensors with variable sequences and I want to concat them, but I want to put the zeros in the end.
For example, I have a sequence 5 6 7 8 0 0 and another sequence 6 9 1 2 3 4 0 0. I need a merged sequence as follows: 5 6 7 8 6 9 1 2 3 4 0 0 0 0. I don’t want just to concat, because if I concat I would have 5 6 7 8 0 0 6 9 1 2 3 4 0 0.

Does your problem is similar to mine?

1 Like
8

Hi,

Does this fit your requirement? This may be not the efficient way, but it works.

seq_a = torch.tensor([5,6,7,8,0,0])
seq_b = torch.tensor([6,9,1,2,3,4,0,0])
result = torch.cat([seq_a[seq_a.nonzero()], seq_b[seq_b.nonzero()]])
zeros = len(seq_a) - seq_a.nonzero().numel() + len(seq_b) - seq_b.nonzero().numel()
result = torch.cat([result, torch.zeros(zeros).unsqueeze(-1).long()]).squeeze(-1)
9

I think this works, but it seems only works for one-dimension tensor

10

Hi @leoribeiro I think this general solution helps your case.

x = torch.tensor(
       [[  -2.0813,   -1.3431, 0],
        [   1.7704, 0, 0],
        [   0.1229,    1.2149,    0.5455]])
y= torch.tensor(
        [[   1.6999,   -0.7491,   -1.9227, 0],
        [   0.4784,   -0.2545, 0, 0],
        [  -1.5127,   -0.1439,   -0.1829,   -0.3703]])

## first we concat the tensor
concated_tensor = torch.cat([x,y], dim=-1)
## The key is to create this index:
index = torch.tensor([[2,3,4,5], [1,2,3,4], [3,4,5,6]])
## Finally we call
concated_tensor.scatter_(1, index, y)

We finally obtain something like this:

tensor([[  -2.0813,   -1.3431,    1.6999,   -0.7491,   -1.9227, 0, 0],
        [   1.7704,    0.4784,   -0.2545, 0,0,0,0],
        [   0.1229,    1.2149,    0.5455,   -1.5127,   -0.1439,   -0.1829, -0.3703]])

How to create that index tensor?

tmp_idx = torch.bincount((x!=0).nonzero()[:,0], minlength=3) ##3 is batch size
## tmp_idx = tensor([2, 1, 3])
## Create a ones tensor
pad_ones = torch.ones(y.size(0), y.size(1)-1)
##>>> pad_ones
##tensor([[1., 1., 1.],
  ##      [1., 1., 1.],
  ##      [1., 1., 1.]])
update_index = torch.cat([num.unsqueeze(-1), pad_ones.long()], dim=-1)
index = torch.cumsum(update_index,dim=1)

You will end up with this index:

tensor([[2, 3, 4, 5],
        [1, 2, 3, 4],
        [3, 4, 5, 6]])
11

Hi @Allan_Jie thanks for this solution, it’s very nice!
However, there is a small bug, you can try it for the input:

x = torch.tensor(
       [[  -2.0813,   -1.3431, 0],
        [   1.7704, 0, 0],
        [   0.1229,    1.2149,   **0**]])
y= torch.tensor(
        [[   1.6999,   -0.7491,   -1.9227, 0],
        [   0.4784,   -0.2545, 0, 0],
        [  -1.5127,   -0.1439,   -0.1829,   -0.3703]])

this output will be:

tensor([[-2.0813, -1.3431,  1.6999, -0.7491, -1.9227,  0.0000,  0.0000],
        [ 1.7704,  0.4784, -0.2545,  0.0000,  0.0000,  0.0000,  0.0000],
        [ 0.1229,  1.2149, -1.5127, -0.1439, -0.1829, -0.3703, -0.3703]])

while the correct output should be:

tensor([[-2.0813, -1.3431,  1.6999, -0.7491, -1.9227,  0.0000,  0.0000],
        [ 1.7704,  0.4784, -0.2545,  0.0000,  0.0000,  0.0000,  0.0000],
        [ 0.1229,  1.2149, -1.5127, -0.1439, -0.1829, -0.3703, 0]])

(see bottom right).

The solution is to zero-out the remainder of the sequence after the scatter operation:

def length_to_mask(length, max_len=None):
        assert len(length.shape) == 1, 'Length shape should be 1 dimensional.'
        max_len = length.max() if not max_len else max_len
        return torch.arange(max_len)[None, :].to(length.device) < length[:, None]

...
merged_len = length_to_mask(len_x + lens_y, max_len=concated_tensor.shape[1])
concated_tensor.scatter_(1, index, y)
concated_tensor[~merged_len] = 0

(assuming you know lens_x and lens_y)

Hope that helps!

1 Like
12

Yes, you are right. Didn’t consider that edge case :smiley:

1 Like