nn.MultiHeadAttention attn_output_weights become the same during evaluation only

[Brian1]

I used nn.MultiHeadAttention like below and output attn_output_weights during both training and evaluation:
class New_attention(nn.Module):
def init(self,num_heads):
super(New_attention, self).init()
self.multi_attn = nn.MultiheadAttention(3000, 5,dropout = 0.3, batch_first = True)

def forward(self, q, k, v):
    attn_output, attn_output_weights = self.multi_attn(q, k, v)
    print(attn_output_weights)
    return attn_output, attn_output_weights

The weights look good during training mode. However, when I perform evaluation with model.eval() and with torch.no_grad(), the value in tensor become the same:

size of q is (128,1,3000)
size of k is (128, 4,3000)
size of v is (128,4, 3000)

Does anybody know why?

[liyunhan1993 (云汉 李)]

I have the same problem as yours but not the same, did you solve it?

MultiheadAttention after LSTM returns the same output for all input

Hi, I would like to use MultiheadAttention as self-attention after applying LSTM on a single sequence. import torch # shape: (sequence length, batch size, embedding dimension) inp = torch.randn(5, 3, 10) lstm = torch.nn.LSTM(input_size=10, hidden_size=10, num_layers=2) self_attn = torch.nn.MultiheadAttention(embed_dim=10, num_heads=2) x, _ = lstm(input=inp) # x is query, key, value at the same time out, weight = self_attn(query=x, key=x, value=x) out tensor([[[ 0.0137, 0.0457, -0.0169, -0.…