U-net loss does not decrease

Hello guys,
I am trying to implement U-net for binary semantic segmentation task. The model input is rgb image with shape=(N, 3, 224, 224), output shape is (N, 1, 224, 224). I use Adam optimizer with default lr. And my loss function is nn.BCEWithLogitsLoss(). I am going to overfit 2-5 data samples but my loss does not decrease much. What should I do? Any help is appreciated. Thank you in regard.
I hope you can help me @ptrblck