Mask-RCNN with custom AnchorGenerator

I am trying to train a Mask-RCNN on a custom data set for instance segmentation of a single class. My images are 600x600 and I know that the instances of my class can always be bounded by boxes with width and height in the range 60-120. I want to take advantage of this and generate Anchor boxes only in that range. Below is my initialisation code for Mask-RCNN (model_bb is a standard Resnet50 trained on a custom dataset):

########################################################
backbone = BackboneWithFPN(model_bb, return_layers, in_channels_list, out_channels)

anchor_generator = AnchorGenerator(sizes=((60, 80, 100, 120),),
                                   aspect_ratios=((0.8, 1.0, 1.2),))

roi_pooler = MultiScaleRoIAlign(featmap_names=in_channels_list,
                                output_size=7,
                                sampling_ratio=2)

mask_roi_pooler = MultiScaleRoIAlign(featmap_names=in_channels_list,
                                     output_size=14,
                                     sampling_ratio=2)

model = MaskRCNN(backbone,
                num_classes=num_clss,
                rpn_anchor_generator=anchor_generator,
                box_roi_pool=roi_pooler,
                mask_roi_pool=mask_roi_pooler)

#######################################################

The problem is I always get the following error:
RuntimeError: shape ‘[1920000, -1]’ is invalid for input of size 10232448
The training works fine if I don’t pass the rpn_anchor, box_roi_pool and mask_roi_pool arguments.
What am I doing wrong?