Validation Log loss lower than Training

vision
1

Hi everyone,
this is my first post in the forum and I believe you can help me clarify one thing: is it acceptable to have validation log loss lowe (or much lower) than training log loss? And if so, how much this gap should be at most?

I am using Inception V3 to classify 120 classes of images with 30K training examples which I split into 21K images for training and 9k for testing using Stratified Random Sampling (30% of each class chosen at random for the validation set).

My model is pretty much the same as the pre-trained one and the only major change the number of classes which I set to 120.

Here’s my most recent callback:

Epoch 1/1499
⚠ train Loss: 6.3281 Acc: 0.0770
⚠ val Loss: 4.2735 Acc: 0.0965
(...)
Epoch 25/1499
⚠ train Loss: 4.2652 Acc: 0.3317
⚠ val Loss: 3.4767 Acc: 0.2571

I’m aware that if the gap between the two is too big and that training error is much smaller than validation error then I am over-fitting.

And then there’s also this thread: https://twitter.com/aureliengeron/status/1110839223878184960

Where Aurélien Geron,author of the book Hands-On MachineLearning with ScikitLearn, Keras and TensorFlow explains that this might be normal… but I’m not sure how credible this is.

This happens regardless of data augmentation technique (even if train/val has the same transforms using just cropsize and ToTensor()), learning rate or optimiser. More recently I’ve been exploring cyclical learning rate as well… Changed batch sizes, used different techniques for sampling… nothing seems to change this behaviour.

I also based myself mostly on: https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html
to build the scaffolding for my code. Difference is the dataset and some data augmentation changes.

So…Is this normal or is it likely that my code is badly implemented?

Thanks a lot!

1 Like
2

Aurelien’s explanations seem reasonable and it’s what I observe usually.
If you want to make sure dropout and the running estimates might be the cause for the gap, you could set your model to model.eval() and use the training dataset after the epoch (same as where you are calculating the validation loss) and compare these two values.

3 Likes
3

Thanks a lot @ptrblck - Did as suggested and, indeed, the values have changed. I am now experiment a few other pre-trained models. Cheers!