Hey, I have one question about matrix operations

1

My target is pooling(max, min, avg, etc.) every sequence to one vector.

For example, given two sentences “I love Outman. I love Superman, too.”, after encoding, the first sentence length is 5(include punctuation) second is 6. I want to pool these two sentences to 2 vectors. And these sentences must encode in one round.

A simple idea is using the ‘scatter_’ function, but I have no idea about this.

# To simplify this question, I don't use batch.
# suppose we have the sentences embedding, 5+6=11 means the sum of the lengths of the sentences, 20 means the embedding dim.

sentence_embedding = torch.Tensor(11, 20)
# I have the split flag like this:
sentence_split_flag = torch.LongTensor([1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0])

....
# After some oprations(depends on the pooling method), I want to get:
pooled_sentence_embedding = torch.Tensor(2, 20)

And I’m sorry for my poor English, if you are perplexed for anything, I will try my best to explain it.

2

Hi, If I understand correctly want you want to do, you could do this:

sentence_embedding = torch.Tensor(11, 20)

# We the equivalent indices instead of flags
# sentence_split_flag = torch.LongTensor([1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0]) 
# which you can also get with:
# sentence_split_idx = (sentence_split_flag == 1).nonzero().view(-1)
sentence_split_idx =  torch.LongTensor([0, 5])

pooled_sentence_embedding  = sentence_embedding.index_select(dim=0, index=sentence_split_idx)
4

Thanks for your reply.

I want to pool all word of one sentence to one, not just get the first.
And in the sentence_split_flag, 1 means the start of one sentence.

# the sentence segment could be represented like this :
sentence_segment = torch.LongTensor([0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1])
5

My bad, I misunderstood.

So I’m not sure of the optimal way of doing it.
But maybe doing something like this might help you:

segments_sizes = (5, 6)
pooled_segs = []
for seg in torch.split(sentence_embedding, segments_sizes, dim=0):
    pooled_segs.append(pooling_op(seg))
pooled_sentence_embedding = torch.cat(pooled_segs, dim=0)
7

Yes, it works. But I think there is a more efficient method. :grinning:
Maybe I could scatter the sentences which dim is (11, 20) to (2, 6, 10), which sentence length is smaller than 6 should be padded. Then pooling (2, 6, 10) to (2, 1, 10), and then reshape to (2, 10).
So I want to know one quick method to get the scatter index from sentence_split_flag or sentence_segment.