I need help in analyzing the overfitting behaviour, I have tried a lot of possible solutions, but nothing was successful.
To explain a bit about the problem:
Basically, I have a model that is trained for the binary classification of 3D points.
Single training pair looks like this: (an image, N 3D points and N labels(binary labels)
The goal is to classify points corresponding to the input image. Model: Each 3D points is linear transformed using single layer MLP and then fed through resnet18 blocks conditioned upon images encoded using pre-trained imagenet model to decode the probability of two classes. (Using this repository for as base code)
Model is overfitting heavily(validation class 0 accuracies are stuck at around 45%)
Note: There is an imbalance in points labels(class 0 labels are 10% and class 1 labels are 90%). But while sampling points, I tried to sample both classes equally.
I tried data augmentations(random crop, flip) and decaying weights but nothing helped.
Could you give any pointers to better analyze and understand the reason for this overfitting?
Thank you in Advance!