High Loss RPN Box Regression in Mask R-CNN

I’m using the Mask R-CNN resnet-50 as backbone model. The value of the “loss_rpn_box_reg” is very high (around 4/5).
There are 256 hidden layers, the value of the learning rate is low since the model converge better with low learning rate.
The bounding box are correctly given to the network, in the format (x1, y1, x2, y2) .
Any idea on how I could reduce this loss value?