Train and test loss deviates from each other

Hi there,

I’m doing a regression task with ResNet50 by feeding in 200x200 dimension images. However, when I plot out the loss, I see a really strange pattern because the train loss keeps decreasing while the test loss keeps fluctuating and sometimes increasing, which suggests my model generalizes very poorly (as shown in the figure below). Usually, I expect the test loss to at least go down a little bit and then go up, but here it just goes straight up. Did anyone observe this behavior before?

Capture