Pytorch Transformers

[vainaijr]

I created one post for a similar question here,

How to add padding mask to nn.TransformerEncoder module?

I want to use vanilla transformer(only the encoder side), but I don’t know how&where to add the padding mask.

see if it helps, regarding math.sqrt(self.ninp)
in the paper https://arxiv.org/pdf/1706.03762.pdf, they use a scaling factor of math.sqrt(self.ninp), maybe cancelling this scaling factor (or not considering to divide by math.sqrt(self.ninp)), gives a better accuracy in the tutorial.