Getting low validation accuracy while finetuning inception_v3

I am using PyTorch fine-tuning tutorial for inception_v3, [ ].
I have a training set with 34 classes and a validation set with 28 classes! The training loss getting better over epochs but the validation accuracy getting low. Does anyone know where is the bug?

Do you see train loss decreasing(and train accuracy increasing) while val loss increases(and val accuracy decreasing)?
In that case, your model is overfitting.

  1. You can add more data
  2. Try out more augmentations
  3. Dropouts
  4. Use a shallow net.

@mailcorahul I see train loss decreasing and train accuracy increasing(start from ~40%) but the accuracy doesn’t change after about 100 epochs and 70% accuracy. And the validation loss increases from the second epoch.

Just to be on the same page, train loss keeps decreasing after 2nd epoch while val loss starts to increase. Then this is a case of overfitting and you can follow the above steps that I have mentioned.