How to make the RegionProposalNetwork generate more proposals in FasterRCNN?

I’m trying to update the proposal losses function of MaskRCNN to increase the recall. I’m trying to do this by adding a positive weight to the BCE function

How I create my proposal losses function:

CLASS_WEIGHTS = torch.tensor([50])

def compute_loss(
        objectness: Tensor, pred_bbox_deltas: Tensor, labels: List[Tensor], regression_targets: List[Tensor]
    ) -> Tuple[Tensor, Tensor]:
        objectness (Tensor)
        pred_bbox_deltas (Tensor)
        labels (List[Tensor])
        regression_targets (List[Tensor])

        objectness_loss (Tensor)
        box_loss (Tensor)

    sampled_pos_inds, sampled_neg_inds = model.rpn.fg_bg_sampler(labels)
    sampled_pos_inds = torch.where(, dim=0))[0]
    sampled_neg_inds = torch.where(, dim=0))[0]

    sampled_inds =[sampled_pos_inds, sampled_neg_inds], dim=0)

    objectness = objectness.flatten()

    labels =, dim=0)
    regression_targets =, dim=0)

    box_loss = F.smooth_l1_loss(
        beta=1 / 9,
    ) / (sampled_inds.numel())

    objectness_loss = F.binary_cross_entropy_with_logits(objectness[sampled_inds], labels[sampled_inds],
                                                        pos_weight=CLASS_WEIGHTS # USE CLASS WEIGHT HERE
    return objectness_loss, box_loss

Then how I set the model to use this proposal losses function:

model = maskrcnn_resnet50_fpn(weights=MaskRCNN_ResNet50_FPN_Weights.DEFAULT)
model.rpn.compute_loss = compute_loss

When I train the model now:

  • the compute_loss function is called, as expected
  • the loss increases significantly (e.g. before it was 1, now it is like 50), as expected
  • BUT the recall and precision stay around the same (e.g. recall stagnates around 0.55 after training for several epochs)

Why is this the case? How do I get the recall to improve (i.e. how do I generate more proposals)?

FYI: I already tried setting the score threshold to 0, this didn’t do anything either…