I am trying to calculate number of MACs in multihead attention layer, using thop tool. Below is the code used. I am gettings macs = 0! Shouldn’t this number be non zero?
Can someone help me understand why macs=0? And how I can get the exact count?
#MultiheadAttention
import torch
m3=torch.nn.MultiheadAttention(1024, 8)
query = torch.randn(1,1024);
key = torch.randn(1,1024);
value = torch.randn(1,1024);
o3,mask=m3(query, key, value)
print(o3)
print(o3.shape)
output :
tensor([[ 0.5605, -0.1514, -0.5640, ..., -0.0213, -0.4984, -0.2785]],
grad_fn=<SqueezeBackward1>)
torch.Size([1, 1024])
‘thop’ prints number of macs as 0!
from thop import profile
macs, params = profile(m3, inputs=(query, key, value))
macs
output:
0.0