Validation set loss value increasing

I am doing 2 class classification with CNN for unbalanced data (both training and test). As go through the training, I see the change in the loss of the valid set. The accuracy of the valid set is very volatile. The best accuracy point of the valid set is stored, but the loss at the moment of being stored is rather bigger than the other. I thought it might be overfitting, so I applied dropout or batch norm, but there was no big change. loss and acc change as follows: (Above: training, below: valid)

Epoch: 0
| Loss: 0.749 | Acc: 55.214% (27607/50000)
| Loss: 0.690 | Acc: 54.150% (5415/10000)
Saving …

Epoch: 1
| Loss: 0.690 | Acc: 56.218% (28109/50000)
| Loss: 0.699 | Acc: 67.870% (6787/10000)
Saving …

Epoch: 2
| Loss: 0.690 | Acc: 56.452% (28226/50000)
| Loss: 0.688 | Acc: 50.810% (5081/10000)

Epoch: 67
| Loss: 0.547 | Acc: 72.024% (36012/50000)
| Loss: 0.796 | Acc: 62.190% (6219/10000)

Epoch: 68
| Loss: 0.537 | Acc: 72.214% (36107/50000)
| Loss: 0.828 | Acc: 68.090% (6809/10000)
Saving …

Epoch: 94
| Loss: 0.455 | Acc: 77.986% (38993/50000)
| Loss: 1.016 | Acc: 69.190% (6919/10000)
Saving …

Epoch: 95
| Loss: 0.451 | Acc: 78.082% (39041/50000)
| Loss: 0.994 | Acc: 69.010% (6901/10000)

Epoch: 96
| Loss: 0.450 | Acc: 78.114% (39057/50000)
| Loss: 0.955 | Acc: 67.390% (6739/10000)

Epoch: 97
| Loss: 0.444 | Acc: 78.386% (39193/50000)
| Loss: 0.996 | Acc: 68.520% (6852/10000)

Epoch: 98
| Loss: 0.441 | Acc: 78.844% (39422/50000)
| Loss: 1.000 | Acc: 67.530% (6753/10000)

Epoch: 99
| Loss: 0.441 | Acc: 78.618% (39309/50000)
| Loss: 0.934 | Acc: 66.080% (6608/10000)

In this case, how can I reduce the loss of the valid set? Is there a case in which training does not apply well to valid set because of data set property?

Your question is not specific to PyTorch. You’re asking about generalization, which at the core of machine learning. There is no simple answer, sadly.

Have you tried adjusting the learning rate or the amount of L2 regularization?

Also, note that about 8-10% of difference in accuracy from training to valid/test is not uncommon.

1 Like