MultiheadAttention after LSTM returns the same output for all input, please watch me!

[liyunhan1993 (云汉 李)]

MultiheadAttention after LSTM returns the same output for all input

Hi, I would like to use MultiheadAttention as self-attention after applying LSTM on a single sequence. import torch # shape: (sequence length, batch size, embedding dimension) inp = torch.randn(5, 3, 10) lstm = torch.nn.LSTM(input_size=10, hidden_size=10, num_layers=2) self_attn = torch.nn.MultiheadAttention(embed_dim=10, num_heads=2) x, _ = lstm(input=inp) # x is query, key, value at the same time out, weight = self_attn(query=x, key=x, value=x) out tensor([[[ 0.0137, 0.0457, -0.0169, -0.…

Here is the question，I meet the same question too. I have no idea why this happen and how to solve it, After a few rounds of training, I also observed the attention socres and found that

[liyunhan1993 (云汉 李)]

And here is my attention out