Transfer weights from classification to maskrcnn

vision
1

Hi,
I am working on a Maskrcnn project for an specific a limited dataset where my number of mask are limited and the area of instance are very small. I have tried to trained the model but I cannot improve the AP to more than 0.34.

It occurred to me that I could use transfer learning from the resnet50 model I run for classification from which I got pretty good F1 score and macc (and plenty of samples).

I have no idea how to transfer the weights and I used as a base the
maskrcnn_50fpn method. Please below:

I would be very grateful if you could help with a piece of advice:

def maskrcnn_50fpn_transfer(num_classes=2, pretrained_backbone=False, **kwargs):
    """
    buils MaskRCNN_50_fnp 
    """
    resnet50 = torchvision.models.resnet50(pretrained = False)
    # 3 classes from claassification
    resnet50.fc = nn.Linear(in_features=2048, out_features= 3)
    
    checkpoint = torch.load(
        '/content/drive/MyDrive/INM373_classification/weights/resnet50/nwce/model_6.pth',
                        map_location=torch.device('cpu'))
    
    resnet50.load_state_dict(checkpoint['model'], strict = False)

    # pretrained_backbone = resnet50

    backbone = resnet_fpn_backbone('resnet50', resnet50) 
    model = MaskRCNN(backbone, num_classes, **kwargs)
    return model