Training Fine but 0 at validation for Segmentation

Hi All,

I have been training this pretrained resnet50 encoder unet decoder for segmentation and for some reason, it seems to be working fine for training where the dice after first epoch is 0.6 but the validation dice for segmentation is at 1e-5 instead of being atleast 0.1.
Can anyone tell me what part of code should I look at? I have already check the validation samples manually.

How does the training and validation losses look like during training?
Do you see signs of overfitting or is the validation loss never going down?

So the dice goes from 0.6 to 0.8 for training in the first 5 epochs but for validation, it sticks to 1e-4 for first two epochs followed by moving to 0.2 and then coming back to 0.1.

Although, at first, it seems like a classic case of overfitting, but I have a training set of 200000 images as compared to 30000 for validation and I have also turned on augmentation for it.

Thoughts?