For high-dimensional tensors, the matrix multiplication can only be operated on the last two dimensions, which requires the previous dimensions to be equal. But in my research, matrix multiplication on the former dimensions also makes sense.
For example, we can do this:
# Mat_A's size is (10, 20, 2, 32)
# Mat_B's size is (10, 20, 32, 3)
# torch.matmul(Mat_A, Mat_b)'s size is (10, 20, 2, 3)
torch.matmul(Mat_A, Mat_b)
But how can we implement this:
# Mat_a's size is (10, 20, 30)
# Mat_b's size is (5, 10)
# torch.some_operate(Mat_b, Mat_a)'s size is (5, 20, 30)
torch.some_operate(Mat_b, Mat_a)
I can use loops to do the calculations, but it’s not very elegant. This is a simple example:
Mat_a is:
[[a, b],
[c, d]]
Mat_b is:
[[A, B],
[C, D]]
then the result is:
[[aA+bC, aB+bD],
[cA+dC, cB+dD]]
where a, b, c, d are scalars and A, B, C, D are vectors.