I am trying to solve an object detection problem using FasterRCNN, where I have N detection scenarios. I am getting good results with transfer learning just the last layer (box predictor), but I end up having N models, one for each scenario. Consequently, it takes up a lot of storage space and RAM if I want all the models to be loaded simultaneously. Now, only the box predictor (last layer) is different for each network, so I want to pass my inputs in a batch, get the output from forward pass till only the last layer, and then pass the individual outputs to N different box predictors. What’s the best strategy to implement this? I can only think of swapping the last layer before running on individual inputs, which is too slow.