FasterRCNN masking labels during training

Hi, I am using the FasterRCNN pretrained model. I have one detection model detecting object A and another detecting object B. I want to create a single model. The two models are trained on the same type of data (medical imaging) but the dataset of A does not contain labels for B and vice versa (while B object canbe present on the image containing A).

I did not see a way to mask/ignore some labels during loss computation, is this something planned or is there a way to easily achieve that (i.e.: without having to annotate dataset A for B object)?