Validation loss oscillates a lot, validation accuracy > learning accuracy, but test accuracy is high. Is my model overfitting?

1
  1. I am training a model, and using the original learning rate of the author (I use their github too), I get a validation loss that keeps oscillating a lot, it will decrease but then suddenly jump to a large value and then decrease again, but never really converges as the lowest it gets is 2 (while training loss converges to 0.0 something - much below 1)

At each epoch I get the training accuracy and at the end, the validation accuracy. Validation accuracy is always greater than the training accuracy.

When I test on real test data, I get good results, but I wonder if my model is overfitting. I expect a good model’s val loss to converge in a similar fashion with training loss, but this doesn’t happen and the fact that the val loss oscillates to very large values at times worries me.

  1. Adjusting the learning rate and scheduler etc etc, I got the val loss and training loss to a downward fashion with less oscilliation, but this time my test accuracy remains low (as well as training and validation accuracies)

I did try a couple of optimizers (adam, sgd, adagrad) with step scheduler and also the pleateu one, I played with step sizes etc. but it didn’t really help, neither did clipping gradients.

  1. Is my model overfitting?
  2. If so, how can I reduce the overfitting besides data augmentation?
  3. If not (I read some people commented it is nothing to worry about to similar cases, but I think ideally the model should converge), If it is really okay, how can I justify it? Even if I would get similar results for a k-fold experiment, would it be good enough? I don’t feel it would justify the oscillating. How should I proceed?
2

@ptrblck Your input will be appreciated

3

I think the oscillating behavior of the validation accuracy might depend on its size. I.e. if the validation dataset is “small”, it would yield noisy estimates.
However, it’s hard to quantify what a “small” dataset size would be and it would be interesting to see your training, validation, and test sizes. As a quick debugging step, you could also swap the training and validation datasets and check how the losses behave.

1 Like