Loss explodes is equal to overfitting?

[LeviViana]

@Christian_Ses

Usually the prediction is biased towards the biggest class, which is the background.

I imagine you are using Cross-Entropy loss somewhere. You could try to balance the class importance on the loss by setting different weights. Maybe this thread could help a bit.