[newbie] Fluctuations and overfitting in first epochs

1

Hey everyone,

I am training an ML model on PyTorch using the DVS Gesture dataset. However, I am facing some issues in the training process, it shows a lot of fluctuations as well as a gap (of 5~6%) between training accuracy and validation accuracy from the first epochs, and the gap keeps increasing during the training (up to 20%).

I have tried several approaches such as :

  • Normalizing (I make it worse, in the begging, the gap is reduced then it gets bigger, and the network start to overfitting early, validation accuracy doesn’t overcome 30%)

  • I am sure that the data are perfectly merged and randomly split, but I can not use a data augmentation because of the data shape (5D tensor)

  • Architecture, I have tried many from the most complex to the simplest

  • Dropout, with a high drop out the gap, disappear in the first epochs (even sometimes validation accuracy is better than training) but over epochs, it reappears as usual.

  • Regularization, just like dropout, it has a good effect in the first epochs, then as if it doesn’t exist, I even tried a weight decay of 1.

  • Batch size, I tried from 16 to 256, it reduces fluctuations but never disappears.

  • Learning rate, regardless of its value, the fluctuations don’t make sense, usually when LR is high both training and validation accuracies drop, but in my case, it is random, one drops while the other improves.

  • Scheduler (ReduceLROnPlateau) doesn’t improve anything, the gap gets bigger.

Regardless of all hyperparameters change, the fluctuations occurs with a certain pattern and couldn’t overcome it.

Here is a colab link to the model : Google Colab

2

Here is some results:

Train set accuracy: 17.58%

Epoch: 0
Test set accuracy: 7.81%
Train set accuracy: 8.59%

Epoch: 1
Test set accuracy: 10.16%
Train set accuracy: 8.59%

Epoch: 2
Test set accuracy: 17.19%
Train set accuracy: 12.11%

Epoch: 3
Test set accuracy: 14.06%
Train set accuracy: 20.51%

Epoch: 4
Test set accuracy: 7.81%
Train set accuracy: 12.30%

Epoch: 5
Test set accuracy: 13.67%
Train set accuracy: 11.13%

Epoch: 6
Test set accuracy: 17.19%
Train set accuracy: 13.28%

Epoch: 7
Test set accuracy: 15.23%
Train set accuracy: 17.97%

Epoch: 8
Test set accuracy: 14.84%
Train set accuracy: 12.50%

Epoch: 9
Test set accuracy: 16.41%
Train set accuracy: 16.99%

Epoch: 10
Test set accuracy: 14.84%
Train set accuracy: 16.99%

Epoch: 11
Test set accuracy: 21.88%
Train set accuracy: 21.48%

Epoch: 12
Test set accuracy: 26.17%
Train set accuracy: 23.63%

Epoch: 13
Test set accuracy: 25.78%
Train set accuracy: 27.93%

Epoch: 14
Test set accuracy: 37.50%
Train set accuracy: 31.84%

Epoch: 15
Test set accuracy: 32.81%
Train set accuracy: 33.79%

Epoch: 16
Test set accuracy: 19.14%
Train set accuracy: 15.04%

Epoch: 17
Test set accuracy: 14.45%
Train set accuracy: 17.19%

Epoch: 18
Test set accuracy: 28.91%
Train set accuracy: 27.54%

Epoch: 19
Test set accuracy: 31.64%
Train set accuracy: 33.20%

Epoch: 20
Test set accuracy: 36.33%
Train set accuracy: 35.94%

Epoch: 21
Test set accuracy: 45.31%
Train set accuracy: 41.21%

Epoch: 22
Test set accuracy: 35.94%
Train set accuracy: 37.89%

Epoch: 23
Test set accuracy: 39.84%
Train set accuracy: 43.16%

Epoch: 24
Test set accuracy: 45.70%
Train set accuracy: 48.83%

Epoch: 25
Test set accuracy: 45.70%
Train set accuracy: 47.07%

Epoch: 26
Test set accuracy: 45.31%
Train set accuracy: 48.05%

Epoch: 27
Test set accuracy: 49.61%
Train set accuracy: 51.17%

Epoch: 28
Test set accuracy: 46.88%
Train set accuracy: 52.54%

Epoch: 29
Test set accuracy: 42.58%
Train set accuracy: 52.93%

Epoch: 30
Test set accuracy: 44.14%
Train set accuracy: 50.39%

Epoch: 31
Test set accuracy: 50.00%
Train set accuracy: 55.47%

Epoch: 32
Test set accuracy: 50.78%
Train set accuracy: 55.66%

Epoch: 33
Test set accuracy: 51.56%
Train set accuracy: 56.64%

Epoch: 34
Test set accuracy: 52.73%
Train set accuracy: 57.81%

Epoch: 35
Test set accuracy: 50.39%
Train set accuracy: 58.40%

Epoch: 36
Test set accuracy: 48.44%
Train set accuracy: 56.84%

Epoch: 37
Test set accuracy: 52.73%
Train set accuracy: 55.86%

Epoch: 38
Test set accuracy: 52.73%
Train set accuracy: 57.03%

Epoch: 39
Test set accuracy: 50.78%
Train set accuracy: 56.84%

Epoch: 40
Test set accuracy: 50.39%
Train set accuracy: 55.08%

Epoch: 41
Test set accuracy: 50.78%
Train set accuracy: 58.20%

Epoch: 42
Test set accuracy: 49.61%
Train set accuracy: 56.25%

Epoch: 43
Test set accuracy: 48.05%
Train set accuracy: 58.98%

Epoch: 44
Test set accuracy: 51.17%
Train set accuracy: 58.01%

Epoch: 45
Test set accuracy: 51.17%
Train set accuracy: 60.35%

Epoch: 46
Test set accuracy: 51.17%
Train set accuracy: 59.57%

Epoch: 47
Test set accuracy: 50.78%
Train set accuracy: 60.74%

Epoch: 48
Test set accuracy: 48.44%
Train set accuracy: 58.40%

Epoch: 49
Test set accuracy: 49.61%
Train set accuracy: 58.79%

Epoch: 50
Test set accuracy: 50.78%
Train set accuracy: 61.52%

Epoch: 51
Test set accuracy: 51.17%
Train set accuracy: 58.98%

Epoch: 52
Test set accuracy: 51.56%
Train set accuracy: 56.05%

Epoch: 53
Test set accuracy: 51.56%
Train set accuracy: 58.59%

Epoch: 54
Test set accuracy: 51.95%
Train set accuracy: 58.98%

Epoch: 55
Test set accuracy: 53.52%
Train set accuracy: 63.48%

Epoch: 56
Test set accuracy: 54.30%
Train set accuracy: 61.33%

Epoch: 57
Test set accuracy: 54.69%
Train set accuracy: 62.89%

Epoch: 58
Test set accuracy: 54.30%
Train set accuracy: 62.11%

Epoch: 59
Test set accuracy: 52.73%