Extract features from F-RCNN/Mask-RCNN

[tomb88 (tom braude)]

I would like to extract the features after ROI Align in object detection.
Specifically for every box detected (top k boxes), I would like to extract the backbone features, either the FC layer before classification or the layer before that in the backbone.
Can anyone suggest a straightforward way of doing this or provide a short script?

Thanks

[spanev (Serge Panev)]

Hi @tomb88 ,

You can refer to this post:

How to access parameters in a certain layer of pretrained network?

Hi @pinata1337, An easy way to find out it is to print your model: >>> print(frcnn) FasterRCNN( (transform): GeneralizedRCNNTransform() (backbone): Sequential( (0): ConvBNReLU( (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU6(inplace=True) ) [...] You can always access the submodules of a module by it’s named attribute (or index when th…

So in MaskRCNN, you would define a forward hook (find more about it here):

mrcnn.roi_heads.box_roi_pool.register_forward_hook(hook_fn)

and your hook_fn would catpure the input.

[tomb88 (tom braude)]

Thanks for the prompt response.
Now I have another issue. I was able to extract the features using a forward hook , but a F-RCNN network generates 1000 proposals at this stage.
The filtering step happens later when calling postprocess_detections(). Here we chose up to k boxes and return only those selected. I would like to also return the features only for these boxes but I don’t see any place where the indexes are returned. A list named “keep” internally but is not returned.
Any suggestion as to how to achieve this?

[drapado (David Rapado)]

Hi @tomb88,

I’m also interested in extracted these features in mask rcnn per object. Did you manage to do it in the end? If so, can you say how?

[citlag (Citlalli)]

@tomb88
Hello
I have been encountering the same issue, has anyone solve it?
I am trying to get the features from the box_head but I keep getting the 1000 proposals from RPN, I know the later the RPN filters them by a threshold but I cant figure out how to call them.