# Simple encoder decoder model is overfitting

**URL:** <https://discuss.pytorch.org/t/simple-encoder-decoder-model-is-overfitting/74632>\
**Category:** Uncategorized\
**Created:** [March 28, 2020, 8:51pm UTC](https://discuss.pytorch.org/t/simple-encoder-decoder-model-is-overfitting/74632 "2020-03-28T20:51:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rohan\_Kumar](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rohan_kumar/32/10228_2.png) [@Rohan\_Kumar](https://discuss.pytorch.org/u/Rohan_Kumar)\
**Post date:** [March 28, 2020, 8:51pm UTC](https://discuss.pytorch.org/t/simple-encoder-decoder-model-is-overfitting/74632/1 "2020-03-28T20:51:14Z")

</div>

I have this simple gru based encoder decoder model pretty, its the same model as  
[seq2seq implementation](https://github.com/bentrevett/pytorch-seq2seq/blob/master/2%20-%20Learning%20Phrase%20Representations%20using%20RNN%20Encoder-Decoder%20for%20Statistical%20Machine%20Translation.ipynb) and yet it is over fitting only after 10 epochs.

![image](https://discuss.pytorch.org/uploads/default/original/3X/8/6/86aea1d4c08464185390bd02a7810f6bbdcf38aa.png)

the translation is english to spanish, without pretrained embeddings for either.

I have also used this model for a different dataset, where both src and trg are english, and i was trying to develop some relation between those, and that overfits after 1 epoch.

I dont know what is the problem, This is bugging me for the past 1 week, any help would highly appreciated.

```auto
OUTPUT_DIM = eng_vocab_size
INPUT_DIM = esp_vocab_size
ENC_EMB_DIM = 256
DEC_EMB_DIM = 256
HID_DIM = 512
ENC_DROPOUT = 0.5
DEC_DROPOUT = 0.5
enc = Encoder(INPUT_DIM, HID_DIM, ENC_EMB_DIM, ENC_DROPOUT)
dec = Decoder(DEC_EMB_DIM, HID_DIM, OUTPUT_DIM, DEC_DROPOUT)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = Seq2Seq(enc, dec, device).to(device)

```

this is the model initialization.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [March 29, 2020, 6:25am UTC](https://discuss.pytorch.org/t/simple-encoder-decoder-model-is-overfitting/74632/2 "2020-03-29T06:25:03Z")

</div>

To counter overfitting, try to add more regularization (dropout, weight decay), decrease the model capacity (smaller or less layers), add augmentation, and make sure the splits are valid (i.e. the data is equally “hard” in both datasets).

---

<div class="post-metadata">

**Author:** ![vdw](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vdw/32/10074_2.png) [@vdw](https://discuss.pytorch.org/u/vdw)\
**Post date:** [March 29, 2020, 11:35am UTC](https://discuss.pytorch.org/t/simple-encoder-decoder-model-is-overfitting/74632/3 "2020-03-29T11:35:12Z")

</div>

Apart from @ptrblck’s comment, how large is your dataset? For example, if you use a rather small dataset (maybe just a sample for testing the network), then the distribution of the training and test data could be very different. This would yield in the trends of the losses you observe.

---

<div class="post-metadata">

**Author:** ![Rohan\_Kumar](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rohan_kumar/32/10228_2.png) [@Rohan\_Kumar](https://discuss.pytorch.org/u/Rohan_Kumar)\
**Post date:** [March 31, 2020, 1:16am UTC](https://discuss.pytorch.org/t/simple-encoder-decoder-model-is-overfitting/74632/4 "2020-03-31T01:16:47Z")

</div>

My dataset contains 8000 data points, and the distribution of val set and train set are similar. I also tried with weight decay, but what is happening is the curve doesn’t look over fitting, but something weird is happening. the min loss achieved during overfitting case is almost the same as non overfitting case, and hence i am still getting poor results.

 ![image](https://discuss.pytorch.org/uploads/default/original/3X/1/3/13aceab82db2a7510fa1a811ac058c6f88597459.png)