Faster RCNN focuses on wrong areas to make predictions

vision
1

Hello everyone,

I trained a FasterRCNN model on 400 annotated hip xrays. I want that the model detects the areas around the hip joints. After training more than 15 epochs, I am getting very good losses and IoU for both training and validation datasets. But when I use GradCam to look how the model predicts the bounding boxes, I can see that the model considers Background areas of the xrays to make predictions. Examples are the black background on the side of the hip etc. Why does this happen and what can I do such that the model makes predictions based on the hip joint areas?

I would appreciate any help!

2

The model will use any information to lower the loss. Did you check if these background areas are somehow specific to one class? E.g. in the past publications reported initially very good results also using medical data until it was realized d.g. the “unhealthy” class images all contained a medical device introduced after the diagnosis and treatment, which the model obviously learned.