nn.MultiHeadAttention attn_output_weights become the same during evaluation only

[liyunhan1993 (云汉 李)]

I have the same problem as yours but not the same, did you solve it?

MultiheadAttention after LSTM returns the same output for all input

Hi, I would like to use MultiheadAttention as self-attention after applying LSTM on a single sequence. import torch # shape: (sequence length, batch size, embedding dimension) inp = torch.randn(5, 3, 10) lstm = torch.nn.LSTM(input_size=10, hidden_size=10, num_layers=2) self_attn = torch.nn.MultiheadAttention(embed_dim=10, num_heads=2) x, _ = lstm(input=inp) # x is query, key, value at the same time out, weight = self_attn(query=x, key=x, value=x) out tensor([[[ 0.0137, 0.0457, -0.0169, -0.…