I have a very small private-dataset {500 samples} of a skin disease{BinaryClassificationTask}, I have tried resnet18, densenet121 and efficientNetB0 so far . Just used some basic augmentation{HorizontalFlip, RandomRotation, GaussianNoise, GaussianBlur} and trained for 50 epoch for each. The surprising thing is I am getting best test-set accuracy on resnet18 {85%} and getting around 70% on the rest of the models. Does this mean I have to fine-tune more on my bigger models? Or is it just the lack of data? Currenty, not using any learning rate scheduler and just using Adam as optimizer.