How to multiply a 4D tensor with 3D tensor according to code below?
qr = torch.einsum('b h i d, i d j → b h d j ', q, r_q)
b = 64
h = 8
i = 8
d = 64
j = 64
Hi @Jayden9912,
When you say “multiply” do you mean matrix-multiplication? Or do you want to use element-wise multiplication instead?
Your code example works and returns a Tensor of shape [b,h,d,j]
import torch
b=64
h=8
i=8
d=64
j=64
q=torch.randn(b,h,i,d)
r_q=torch.randn(i,d,j)
qr=torch.einsum("bhid, idj->bhdj", q, r_q)
print(qr.shape) #returns torch.Size([64, 8, 64, 64])
I assume you want to multiply a batch of Tensors (the 4D tensor) with a single 3D tensor for all tensors in your batch?
The einsum expression ("bhid, idj -> bhdj") you have is equivalent to taking a matmul over the i-th index, then taking the trace over the d-th index. There some examples of einsum notation in numpy’s documentation (see here)