# Pytorch Transformers

**URL:** <https://discuss.pytorch.org/t/pytorch-transformers/76993>\
**Category:** nlp\
**Created:** [April 16, 2020, 6:12am UTC](https://discuss.pytorch.org/t/pytorch-transformers/76993 "2020-04-16T06:12:08Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![vainaijr](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/e19b73/32.png) [@vainaijr](https://discuss.pytorch.org/u/vainaijr)\
**Post date:** [April 17, 2020, 12:51pm UTC](https://discuss.pytorch.org/t/pytorch-transformers/76993/2 "2020-04-17T12:51:51Z")

</div>

I created one post for a similar question here,

> [@How to add padding mask to nn.TransformerEncoder module?](https://discuss.pytorch.org/t/how-to-add-padding-mask-to-nn-transformerencoder-module/63390):
>
> I want to use vanilla transformer(only the encoder side), but I don’t know how&where to add the padding mask.

see if it helps, regarding `math.sqrt(self.ninp)`  
in the paper [https://arxiv.org/pdf/1706.03762.pdf](https://arxiv.org/pdf/1706.03762.pdf), they use a scaling factor of `math.sqrt(self.ninp)`, maybe cancelling this scaling factor (or not considering to divide by `math.sqrt(self.ninp)`), gives a better accuracy in the tutorial.

---

_[View the full topic](https://discuss.pytorch.org/t/pytorch-transformers/76993)._