How to get average precision for individual class?

Hi, I’m trying to get the individual class average precision.

Currently, I have trained object detection model using torchvision

num_classes = 3 # car, person, background
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

here, when I evaluate the model, i’m getting overall performance of the model.

IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.657
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.930
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.751
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.512
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.697
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.489
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.710
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.710
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.704
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.734

but, I’m trying to get individual class performance.
can anyone help me to get average precision for individual category?

I know, detectron2 produces the individual class performance, but for my dataset detectron2 taking much longer time to get trained. So, I’m sticking with torchvision.

Also, I have tried other approach by clubbing(torchvision + Detectron2) them together. eg,

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-Detection/faster_rcnn_R_50_FPN_1x.yaml"))
cfg.DATASETS.TRAIN = ('dataset_train',)
cfg.DATASETS.TEST = ('dataset_val',)   
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/faster_rcnn_R_50_FPN_1x.yaml")
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.LR_SCHEDULER_NAME = "WarmupCosineLR"
cfg.SOLVER.MAX_ITER = 500
cfg.SOLVER.BASE_LR = 0.005
cfg.SOLVER.MOMENTUM = 0.9
cfg.SOLVER.NESTEROV = False
cfg.SOLVER.STEPS = (400,450)
cfg.SOLVER.WEIGHT_DECAY = 0.0001
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 2
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128
cfg.TEST.EVAL_PERIOD = 50
cfg.SOLVER.IMS_PER_BATCH = 1
cfg.SEED = 42
cfg.VIS_PERIOD = 100
cfg.SOLVER.CHECKPOINT_PERIOD = 500

trainer = MyTrainer(cfg)

from detectron2.config import get_cfg
from detectron2.data import DatasetCatalog, MetadataCatalog, build_detection_test_loader
from detectron2.evaluation import COCOEvaluator, inference_on_dataset
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "./outputs/fasterrcnn_custom_test_1_loss_0.12796187750063837.pth") # this model is trained using torchvision
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.85
trainer = MyTrainer(cfg)
predictor = DefaultPredictor(cfg)
evaluator = COCOEvaluator("dataset_val", cfg, False, output_dir="./output/")
val_loader = build_detection_test_loader(cfg, "dataset_val")
inference_on_dataset(trainer.model, val_loader, evaluator)

when I performed this, I’m getting below answer,

[11/20 00:19:51 d2.evaluation.coco_evaluation]: Per-category bbox AP: 
| category   | AP    | category   | AP    |
|:-----------|:------|:-----------|:------|
| car        | 0.000 | person     | 0.000 |
OrderedDict([('bbox',
              {'AP': 0.0,
               'AP50': 0.0,
               'AP75': 0.0,
               'APs': 0.0,
               'APm': 0.0,
               'APl': 0.0,
               'AP-car': 0.0,
               'AP-person': 0.0})])

Also I noted this,

Skip loading parameter 'roi_heads.box_predictor.bbox_pred.weight' to the model due to incompatible shapes: (12, 1024) in the checkpoint but (8, 1024) in the model! You might want to double check if this is expected.
Skip loading parameter 'roi_heads.box_predictor.bbox_pred.bias' to the model due to incompatible shapes: (12,) in the checkpoint but (8,) in the model! You might want to double check if this is expected.
Some model parameters or buffers are not found in the checkpoint:

it would be really helpful you anyone can guide me here.

Note: I have tried different github repo’s to get this answer, but I’m trying this approach.
@ptrblck

Hello, did you solve this?