I have a question about the architecture of RPN and ROIALign in Faster/Mask R-CNN (see the image).
For each anchor in RPN there are two ‘heads’: 4 bbox values and 1 confidence value that are compared to the labels (offsets for bbox and 1/0 for overlap). In the pytorch implementation though, cls_logits have 3 channels, and bbox_pred 3x4=12. I don’t quite understand, what these 3 channels mean. There should be 1 and 4, as I understand.
In RoIPooling, there are two heads, cls_score with the number of classes (91), and bbox_pred with 4*91=364. I don’t quite understand, why we need bbox/class? We have a softmax prediction, so only 1 class can exist in this anchor, so we need 1 bbox, where do the other 360 come from? And what does the label look like? Offsets for 4 correct values and 0s otherwise?
Hey there, hope you are doing well,
A little late here but as far as I understand this code, this is because the default AnchorGenerator used in faster_rcnn.py defines the aspect ratios as a tuple of length 3. Meaning around each anchor point 3 anchor boxes would be considered of the given given aspect ratio, with side corresponding: anchor_sizes = ((32,), (64,), (128,), (256,), (512,)) where 32 corresponds to fpn0, 64 to fpn1 and so on.
Therefore say your fpn0 feature map is of size 128128 (for eg.) and your orignal image was 512, 512. Then your stride between anchor points is 512/128 = 4. i.e. your anchors are (0,0), (0,4), (0,8) , …. ( 4, 0), (4, 8), …… and so on, and around each of these points 3 regions are considered of size 3216, 3232, 3264, (following default h/w aspect ratios of [0.5, 1, 2]).
Now for each of these 3 anchor regions around each of the points in your true image, represented in the fpn, the cls logits give out a objectiveness score and also a bbox_pred, which is 3 *4 .(per anchor region predict 4 corners of the bounding box)
