Test acc. slightly higher than train acc for large number of epochs

1

I have a simple fully connected NN that I tested with:
CIFAR10,
FashionMNIST,
MNIST and
Cat vs. non Cat datasets.
For all the 4 datasets I observe normal behavior in the accuracy, namely, train accuracy is higher than test acc. But, for two cases of MNIST, I get about 1 % higher accuracy for test than for train (90% train acc. and 91%test acc.). This was when running 500 epochs.
I thought to reduce the number of epochs to 300 and see what happened, and the results were then: 90% train acc. and 89% test acc.
How to interpret this?

2

One possible explanation is that the model doesn’t have enough capacity to overfit. See what happens when you make the model larger.

3

So, how would one explain that the same architecture does not face the problem with CIFAR10 that is more difficult to train? i will try your suggestion and see what happens.

4

Depending on the accuracy achieved, CIFAR10 might not be necessarily a harder dataset. E.g. classifying by prevalent color will get you to some accuracy on CIFAR, and you will almost certainly overfit on training data.

Hard to say without knowing about the results and overall setup.