Combine multiple faster RCNN models

vision
1

Hi,

I’m trying to train separately different components of a faster rcnn model (torchvision.models.detection.fasterrcnn_resnet50_fpn) on different datasets and combine them into a single model. My final objective is to detect tree species over a forest using overhead imagery. My training data is non exhaustive (most of the trees are unlabelled/of unknown species).
For example I first train backbone + RPN + 2 classes classifier (tree/background) on a dataset with all trees identified (regardless of the species). Then I freeze the backbone and train a new multiclass classifier on a synthetic data made of trees of known species placed on a textured background image.
Finally I combine the common backbone with rpn of first model and RoIHeads of second model.
When taken separately the two models work quite well, but combining them hasn’t worked too well so far.

  • Am I right to think different components of a model can be trained separately and later combined?
  • Where is the box_regression part of the model? Is it part of model.RoIHeads.box_predictor? Can it be separated from it?
  • Any other suggestion?

KInd regards,
Loïc

1 Like