In exploring a word2vec variant, I’d like to find a way to do the following bmm like operation which bmm does not seem suited for. So far I can only write it with a for-loop. Is there a way to do this with pytorch tensor operations?
def combine(ivectors, ovectors):
(batch_sz, sense_sz, context_sz, embedding_sz) = ovectors.shape
LL = torch.zeros((batch_sz, sense_sz, context_sz))
for i in range(batch_sz):
LL[i,:,:] = torch.matmul(ovectors[i,:,:,:], ivectors[i,:].T)
return LL
My problem seems to boil down to this: I want to take an input tensor of shape, say, 2x2x3x5 (ovectors) and combine it with a tensor of shape 5x2 (ivectors.T) and output a tensor of shape (2,2,3). What bmm and matmul seem to be designed for is a reduction that gives (2,2,3,2), 24 elements, but I want the reduction in the above for-loop, which gives only 12.
Is there a way to get rid of the for loop?