# Masking in torch.nn.MultiheadAttention

**URL:** <https://discuss.pytorch.org/t/masking-in-torch-nn-multiheadattention/132060>\
**Category:** Uncategorized\
**Created:** [September 15, 2021, 1:39pm UTC](https://discuss.pytorch.org/t/masking-in-torch-nn-multiheadattention/132060 "2021-09-15T13:39:20Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![legoh](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/82dd89/32.png) [@legoh](https://discuss.pytorch.org/u/legoh)\
**Post date:** [September 15, 2021, 1:39pm UTC](https://discuss.pytorch.org/t/masking-in-torch-nn-multiheadattention/132060/1 "2021-09-15T13:39:20Z")

</div>

Hi there! I am using the nn.MultiheadAttention to construct a transformer encoder layer. Suppose my queries, keys, and values, are all the same (e.g. `h`), meaning I call it via `h, score = MHA(h, h, h)`. This means that for some of the computation, there is some form of self-attention going on. Is there a way to mask this away?

---

<div class="post-metadata">

**Author:** ![AbdulsalamBande](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abdulsalambande/32/40450_2.png) [@AbdulsalamBande](https://discuss.pytorch.org/u/AbdulsalamBande)\
**Post date:** [October 14, 2021, 8:56am UTC](https://discuss.pytorch.org/t/masking-in-torch-nn-multiheadattention/132060/2 "2021-10-14T08:56:59Z")

</div>

I think the [torch.nn.MultiheadAttention](https://discuss.pytorch.org/t/masking-in-torch-nn-multiheadattention/132060) has a mask argument

> `forward` (_query_ , _key_ , _value_ , _key\_padding\_mask=None_ , _need\_weights=True_ , _attn\_mask=None_ )

You can watch this video for an explanation [https://www.youtube.com/watch?v=\_iDanMWVj98&ab\_channel=PytorchModulesExplained](https://www.youtube.com/watch?v=_iDanMWVj98&ab_channel=PytorchModulesExplained)