Nvm, you take care of my intended padding with a -inf because of how the softmax works in MHA, see: How to add padding mask to nn.TransformerEncoder module?
Nvm, you take care of my intended padding with a -inf because of how the softmax works in MHA, see: How to add padding mask to nn.TransformerEncoder module?