# LSTM dropout - Clarification of Last Layer

**URL:** <https://discuss.pytorch.org/t/lstm-dropout-clarification-of-last-layer/5588>\
**Category:** Uncategorized\
**Created:** [July 30, 2017, 9:54am UTC](https://discuss.pytorch.org/t/lstm-dropout-clarification-of-last-layer/5588 "2017-07-30T09:54:50Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ronrest](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ronrest/32/1134_2.png) [@ronrest](https://discuss.pytorch.org/u/ronrest)\
**Post date:** [July 30, 2017, 9:54am UTC](https://discuss.pytorch.org/t/lstm-dropout-clarification-of-last-layer/5588/1 "2017-07-30T09:54:51Z")

</div>

In the [documentation for LSTM](http://pytorch.org/docs/master/nn.html#torch.nn.LSTM), for the `dropout` argument, it states:

> introduces a dropout layer on the outputs of each RNN layer except the last layer

I just want to clarify what is meant by “everything except the last layer”.

Below I have an image of two possible options for the meaning.

- **Option 1:** The final cell is the one that does not have dropout applied for the output.
- **Option 2:** In a multi-layer LSTM, all the connections between layers have dropout applied, except the very top layer. So a single layer LSTM would not have any dropout applied.

 ![](https://discuss.pytorch.org/uploads/default/original/2X/6/62f94ceee433b693ef73be231f51ae4291e53880.png)

I tried looking at the source code for LSTM and RNNBase, but I can’t figure out how it is being applied.

---

<div class="post-metadata">

**Author:** ![dgriff](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dgriff/32/1361_2.png) [@dgriff](https://discuss.pytorch.org/u/dgriff)\
**Post date:** [July 30, 2017, 10:52am UTC](https://discuss.pytorch.org/t/lstm-dropout-clarification-of-last-layer/5588/2 "2017-07-30T10:52:56Z")

</div>

LSTM is a rolled up LSTMCell and to think of each layer as Performing one LSTMCell action and that should help make sense of it. Number layers being the number of time steps performed

---

<div class="post-metadata">

**Author:** ![tom](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/32/3162_2.png) [@tom](https://discuss.pytorch.org/u/tom)\
**Post date:** [July 30, 2017, 1:02pm UTC](https://discuss.pytorch.org/t/lstm-dropout-clarification-of-last-layer/5588/3 "2017-07-30T13:02:08Z")

</div>

Hello,

from both the text you quoted and the code, I’d say likely option 2. This is my understanding of layers.

If you believe that, after disabling cudnn, this line is relevant, it might also give a hint.

> <https://github.com/pytorch/pytorch/blob/master/torch/nn/_functions/rnn.py#L91>

  
My understanding of this code is that it does a single timestep (your horizontal axis) for many layers.

Best regards

Thomas

---

<div class="post-metadata">

**Author:** ![dgriff](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dgriff/32/1361_2.png) [@dgriff](https://discuss.pytorch.org/u/dgriff)\
**Post date:** [July 30, 2017, 2:01pm UTC](https://discuss.pytorch.org/t/lstm-dropout-clarification-of-last-layer/5588/4 "2017-07-30T14:01:28Z")

</div>

Time steps I’m referring from going from each element in the input. Like if you fed a sentence as sequence the order of the words in that sentence matters. Hence why the pytorch docs say:

“For each element in the input sequence, each layer computes the following function”

Nn.LSTM is a window LSTM Aka rolled up LSTM

And would be option 2

Which sorry I misread you Tom thought you were disagreeing with that but read incorrectly and what I wrote was wrong it is multilayer for each time step😲

---

<div class="post-metadata">

**Author:** ![STU](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stu/32/16640_2.png) [@STU](https://discuss.pytorch.org/u/STU)\
**Post date:** [September 29, 2020, 6:37am UTC](https://discuss.pytorch.org/t/lstm-dropout-clarification-of-last-layer/5588/5 "2020-09-29T06:37:57Z")

</div>

Hi, if it means the Option-1, why does the num\_layers must great than 1?