Changing backbone of MaskRCNN to ResNet152 does not train

Hi, I’m doing a backbone comparison for comparing smaller backbones and their results on a dataset, however, when changing the backbone I’m probably doing something wrong, since this results in the network not being trained and having a weird output compared to the standard MaskRCNN with the ResNet50 output. Below is the code of what I’m doing, my images are 400x400, num_classes=9 and if there is a comprehensive guide on swapping backbones please point me to it.

      resnet152_backbone = torch.load('path/to/resnet152')
      model1 = torchvision.models.resnet152(pretrained=False)
      model1.load_state_dict(resnet152_backbone)
      modules = list(model1.children())[:-1]
      backbone = torch.nn.Sequential(*modules)
      backbone.out_channels = 2048

      # Make rpn
      anchor_generator = AnchorGenerator(
          sizes=((32,64,128,256,512),),
          aspect_ratios=((0.5,1.0,2.0),)
      )

      roi_pooler = torchvision.ops.MultiScaleRoIAlign(
          featmap_names=['0'],
          output_size=7,
          sampling_ratio=2,
      )
      mask_roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=['0'],
                                                            output_size=14,
                                                            sampling_ratio=2)

      # put the pieces together inside a Faster-RCNN model
      model = MaskRCNN(
          backbone,
          num_classes=num_classes,
          rpn_anchor_generator=anchor_generator,
          box_roi_pool=roi_pooler,
           mask_roi_pool=mask_roi_pooler,
      )