# Pytorch Transformers

**URL:** <https://discuss.pytorch.org/t/pytorch-transformers/76993>\
**Category:** nlp\
**Created:** [April 16, 2020, 6:12am UTC](https://discuss.pytorch.org/t/pytorch-transformers/76993 "2020-04-16T06:12:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Anandh\_Perumal\_Konar](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anandh_perumal_konar/32/12027_2.png) [@Anandh\_Perumal\_Konar](https://discuss.pytorch.org/u/Anandh_Perumal_Konar)\
**Post date:** [April 16, 2020, 6:12am UTC](https://discuss.pytorch.org/t/pytorch-transformers/76993/1 "2020-04-16T06:12:08Z")

</div>

The below code is from pytorch [doc](https://pytorch.org/tutorials/beginner/transformer_tutorial.html)

```auto
    def forward(self, src):
        if self.src_mask is None or self.src_mask.size(0) != len(src):
            device = src.device
            mask = self._generate_square_subsequent_mask(len(src)).to(device)
            self.src_mask = mask

        src = self.encoder(src) * math.sqrt(self.ninp)
        src = self.pos_encoder(src)
        output = self.transformer_encoder(src, self.src_mask)
        output = self.decoder(output)
        return output

```

I don’t understand why there is a mask in the transformer encoder?  
In decoder, we don’t want the decoder to look at the future and it makes sense to mask but why encoder?

and what is the use of the below code?  
`math.sqrt(self.ninp)` in

```auto
src = self.encoder(src) * math.sqrt(self.ninp)

```

Thanks in advance

---

<div class="post-metadata">

**Author:** ![vainaijr](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/e19b73/32.png) [@vainaijr](https://discuss.pytorch.org/u/vainaijr)\
**Post date:** [April 17, 2020, 12:51pm UTC](https://discuss.pytorch.org/t/pytorch-transformers/76993/2 "2020-04-17T12:51:51Z")

</div>

I created one post for a similar question here,

> [@How to add padding mask to nn.TransformerEncoder module?](https://discuss.pytorch.org/t/how-to-add-padding-mask-to-nn-transformerencoder-module/63390):
>
> I want to use vanilla transformer(only the encoder side), but I don’t know how&where to add the padding mask.

see if it helps, regarding `math.sqrt(self.ninp)`  
in the paper [https://arxiv.org/pdf/1706.03762.pdf](https://arxiv.org/pdf/1706.03762.pdf), they use a scaling factor of `math.sqrt(self.ninp)`, maybe cancelling this scaling factor (or not considering to divide by `math.sqrt(self.ninp)`), gives a better accuracy in the tutorial.

---

<div class="post-metadata">

**Author:** ![Anandh\_Perumal\_Konar](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anandh_perumal_konar/32/12027_2.png) [@Anandh\_Perumal\_Konar](https://discuss.pytorch.org/u/Anandh_Perumal_Konar)\
**Post date:** [April 17, 2020, 11:06pm UTC](https://discuss.pytorch.org/t/pytorch-transformers/76993/3 "2020-04-17T23:06:02Z")

</div>

> [@vainaijr](#):
>
> not considering to divide by `math.sqrt(self.ninp)`

Well, actually my question is different from the one you asked.  
My question is what’s the need for mask in the encoder layer?  
But I think it makes sense when you want to use an architecture like GPT 2, I guess. Correct me if I’m wrong.

Regarding the scaling factor, even if we don’t scale it shouldn’t affect the results a lot, I can’t recollect the paper but in the earlier version of dot product attention even they didn’t use the scale factor. But obviously scaling helps.

But I don’t understand why are we multiplying it? it should be divided based on the attention paper.

---

<div class="post-metadata">

**Author:** ![ShaneB](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shaneb/32/8266_2.png) [@ShaneB](https://discuss.pytorch.org/u/ShaneB)\
**Post date:** [May 20, 2021, 2:50pm UTC](https://discuss.pytorch.org/t/pytorch-transformers/76993/4 "2021-05-20T14:50:56Z")

</div>

> [@Anandh\_Perumal\_Konar](#):
>
> Regarding the scaling factor, even if we don’t scale it shouldn’t affect the results a lot, I can’t recollect the paper but in the earlier version of dot product attention even they didn’t use the scale factor. But obviously scaling helps.
> 
> But I don’t understand why are we multiplying it? it should be divided based on the attention paper.

Again, regarding the scaling factor:

The [paper](https://arxiv.org/pdf/1706.03762.pdf) does say:

> In the embedding layers, we multiply those weights by `sqrt(d_model)`.

(Note [The Annotated Transformer](https://nlp.seas.harvard.edu/2018/04/03/attention.html) directly connects the paper to PyTorch code line-by-line).

However, the paper does not say _why_ this is done.

This [post](https://towardsdatascience.com/how-to-code-the-transformer-in-pytorch-24db27c8f9ec) says the following:

> The reason we increase the embedding values [by adding `math.sqrt(self.d_model)`] before addition is to make the positional encoding relatively smaller. This means the original meaning in the embedding vector won’t be lost when we add them together.