FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged

So, I am running detectron 2 mask rcnn for a specific dataset. So for transfer learning I have added another pretrained weight which is trained on a different but same type of dataset. But everytime I am adding the pretrained weight, i am getting the aforementioned error. I have also added pretrained weight of Segment Anything Model (which is not similar to my dataset) I got the error. Then I am trying to use a model weight which is trained on kind of similar type of my dataset. But still getting the same error.

I have adjusted learning rate as well as batch size, still not getting any solution.

Thanks for helping me.