# Do i need to learn RNN and LSTMs prior to learn Transformer Model?

**URL:** <https://discuss.pytorch.org/t/do-i-need-to-learn-rnn-and-lstms-prior-to-learn-transformer-model/186648>\
**Category:** nlp\
**Created:** [August 18, 2023, 4:10pm UTC](https://discuss.pytorch.org/t/do-i-need-to-learn-rnn-and-lstms-prior-to-learn-transformer-model/186648 "2023-08-18T16:10:27Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Teddy\_Salas](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/teddy_salas/32/64758_2.png) [@Teddy\_Salas](https://discuss.pytorch.org/u/Teddy_Salas)\
**Post date:** [August 18, 2023, 4:10pm UTC](https://discuss.pytorch.org/t/do-i-need-to-learn-rnn-and-lstms-prior-to-learn-transformer-model/186648/1 "2023-08-18T16:10:27Z")

</div>

Do i need to learn RNN and LSTMs prior to learn Transformer Model?

---

<div class="post-metadata">

**Author:** ![vdw](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vdw/32/10074_2.png) [@vdw](https://discuss.pytorch.org/u/vdw)\
**Post date:** [August 19, 2023, 6:28am UTC](https://discuss.pytorch.org/t/do-i-need-to-learn-rnn-and-lstms-prior-to-learn-transformer-model/186648/2 "2023-08-19T06:28:11Z")

</div>

Well, strictly speaking, an LSTM is special instance of an RNN. So getting a basic grasp about the ideas behind and inner workings of RNNs would be enough.

In my opinion, it’s probably not really needed to learn about RNNs, but I would still recommended it – again, at least the basic concepts. Firstly, the limitations of RNNs – mainly the problem long-term dependencies and the lack of parallelism – are one of the goals to address in the original [Transformer paper](https://arxiv.org/abs/1706.03762). And secondly, the concept of attention, as least as far as I know, has first been proposed (or at least popularized) by RNNs in [this paper](https://arxiv.org/abs/1409.0473). The Transformer paper generalized the idea of attention, but I think the concept is a more intuitive in the context of RNNs.

If you just want to train a Transformer model, just go for it. For some deeper understanding, it’s always good to learn about (closely) related ideas to put things into better context.

---

<div class="post-metadata">

**Author:** ![Teddy\_Salas](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/teddy_salas/32/64758_2.png) [@Teddy\_Salas](https://discuss.pytorch.org/u/Teddy_Salas)\
**Post date:** [August 19, 2023, 4:06pm UTC](https://discuss.pytorch.org/t/do-i-need-to-learn-rnn-and-lstms-prior-to-learn-transformer-model/186648/3 "2023-08-19T16:06:34Z")

</div>

@vdw Thank you for your suggestion