Hi,
no elaborate answer but just few thoughts and questions:
To understand your dataset better, how many classes do you have in your mask images?
Did you already check that your masks are loaded correctly by displaying them? Make sure the mask images from your data loader resemble the mask images from your database on your PC. Could normalization be an issue?
What loss function do you use?
Can you post a code snippet for getting the prediction mask image?