Model is wrong on the dataset it was trained on


I’ve build a new model using VGG16 architecture, and trained only the fully connected layer in order to classify gender of a person. During training, the accuracy over the training set of 10k photos is reaching 94% and over the validation set around 82% .

The problem appear after the training is over, i’m doing a simple test by arbitrary choosing some photos from the training dataset, and the results just seems wrong, the accuracy i’m getting is somewhere around 50%. If i got 94% accuracy on the model, and using the same dataset the model was trained on, that means it should be wrong for only 6 out of 100 photos, but that is not the case.

I didn’t provide any code because I didn’t know what could be relavent, so if anything is needed i’ll add it.

I’ll appreciate any thoughts on why and how this could happen.

Since your training set contains 10k photos, how many photos have your sampled from it? The accuracy does not distribute uniformly, maybe some classes are difficult to classify, some are much easier. I think you can sample more photos to check it.