Anyway, after reading this very meaningful and useful post
I tried to increase my batch_size from 12 to 128, and it works
this time, validation accuracy with model.eval() looks much more normal
Still leave this post for your reference, as I think it’s quite interesting