Add layer to Faster-RCNN model

1

Hello, I have a pretrained Faster-RCNN model that I would like to customise by adding a layer after the output of the backbone. Can anyone help me? I chose not to create a backbone and then use it as an input to FasterRCNN function because I have the faster Rcnn pretrained on its own.

2

You can do something like this

model.backbone.add_module("NAME", nn.Linear(512,3))

you can also put any layers in the second argument/

3

Hi, thank you for your message. I should have mentioned the architecture I was thinking of. So I need to use another input other than the images, that will pass through some layers and then get concatenated with the output of the backbone and continue to the FPN as normal.

4

Oh ok so you could just create a different sequential model and pass the second inputs into that. Then you would have to do the

model.backbone

and the other layers to make the input go through each individual part of the model. For example

x = model.transform(x)
x = model.backbone(x)
x1 = OTHERMODEL(x1)
x = cat(x,x1)

then pass the x into the fpn, rpn, roi, roi heads, and box predictors

5

Great! That’s exactly what I wanted. One last thing… How do I specify the multiple inputs for the faster rcnn model? I know my questions may sound silly, I am just used a beginner in pytorch.

6

What do you mean by multiple inputs do you mean one input for the backbone and one for the other sequential model. If so you can do something like this

def forward(self, x, x1):
     x = model.backbone(x)
     x1 = SECONDMODEL(x1)
...
7

Yeah, I want to modify the faster rcnn model, use another input that will pass through some layers, concatenate that with the output of the backbone and continue through the next layers of faster rcnn

8

Ya so just do what I said above. Just pass one input into the fasterrcnn backbone and then have another sequential model and pass an input into that and concatenate them together and then pass them through the next layers.

9

Hi, I have implemented the model but I am having trouble with the tensor shapes. Could you please help? The 2nd input tensor has 3 columns.

        meta_array_shape = 3 
        backbone_output_shape = (512,3)
        
        self.meta_model = nn.Sequential(
            nn.Linear(meta_array_shape, backbone_output_shape),
            nn.BatchNorm1d(backbone_output_shape),
            nn.ReLU(),
            nn.Dropout(p=0.5)
        )

        X = self.faster_rcnn.backbone(X)

        # Metadata Net
        Y = self.meta_model(meta_in)
        
        # Concat Feature Maps
        X = torch.cat(X, Y)
10

Ok well the first problem is in cat you need to do this

X = torch.cat((X,Y))

the two tensors you need to cat needs to be in their own parenthesis. Can you print out the shapes of the X and Y tensors going into the cat.