Segmentation output for sigmoid become 0.5

Hello. I have some questions about my code. I follow this code from Kaggle loss Loss Function Library - Keras & PyTorch | Kaggle

My data is highly class imblance in the image. For example, my object is 10 ~ 20 pixels from 4K images.
For this, I can not resize the image, instead of resizing, I am cropping the image with 1024 x 512 resolution.
For the input normalization, I divide by 255 about each channel. My mask is composed of 0 and 1. 1 is the target object, and my background is 0.
During the training, I apply the IoU loss. When I check that most of value of sigmoid output is 0.5. For this reason, the IoU is very low since our output is 0 and 1.
Second, the output pixel value becomes the minus value.
Third, the loss calculation become minus sometiems.
Anyone have ideo for this?
I assume that try softmax and try to solve this problem as classification problem. In that cases it does not provide good result compare to Focal tversky loss or iou loss.
I understand that sigmoid can provide the 0.5 but I believe that I should provide function like relu. instead of sigmoid. What do you think about this idea?
Thank you.