Nan Loss for instance segmentation with MASK -RCNN using custom dataset

Hello Community,

I followed the instance segmentation tutorial of pytorch vision using mask-RCNN inside google colab notebook
Tutorial notebook

I decided to finetune it to a different dataset.
Segmentation masks are provided in Json format.
I can convert these masks to correct format in dataloader
You can find the notebook & data in following github link:
github link

My problem is that after initial epoch i get Nan loss functions. How do i fix this issue?