I am working on crop disease with 20k images. I have imbalance dataset, so downsampled it but when I applied the resnet model on this I am getting training accuracy initially with 86% but later epochs it is decreasing and val acc to 7% is constant with different batch_size, weight_decay. I have augmented data with brightness, rescaled, etc.
Could it be that the model overfits on your training set given that you only have 20k images and use a resnet? Does a very simple model give similar behavior?