Hi, I’m trying to get the individual class average precision.
Currently, I have trained object detection model using torchvision
num_classes = 3 # car, person, background
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
here, when I evaluate the model, i’m getting overall performance of the model.
IoU metric: bbox
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.657
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.930
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.751
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.512
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.697
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.489
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.710
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.710
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.704
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.734
but, I’m trying to get individual class performance.
can anyone help me to get average precision for individual category?
I know, detectron2 produces the individual class performance, but for my dataset detectron2 taking much longer time to get trained. So, I’m sticking with torchvision.
Also, I have tried other approach by clubbing(torchvision + Detectron2) them together. eg,
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-Detection/faster_rcnn_R_50_FPN_1x.yaml"))
cfg.DATASETS.TRAIN = ('dataset_train',)
cfg.DATASETS.TEST = ('dataset_val',)
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/faster_rcnn_R_50_FPN_1x.yaml")
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.LR_SCHEDULER_NAME = "WarmupCosineLR"
cfg.SOLVER.MAX_ITER = 500
cfg.SOLVER.BASE_LR = 0.005
cfg.SOLVER.MOMENTUM = 0.9
cfg.SOLVER.NESTEROV = False
cfg.SOLVER.STEPS = (400,450)
cfg.SOLVER.WEIGHT_DECAY = 0.0001
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 2
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128
cfg.TEST.EVAL_PERIOD = 50
cfg.SOLVER.IMS_PER_BATCH = 1
cfg.SEED = 42
cfg.VIS_PERIOD = 100
cfg.SOLVER.CHECKPOINT_PERIOD = 500
trainer = MyTrainer(cfg)
from detectron2.config import get_cfg
from detectron2.data import DatasetCatalog, MetadataCatalog, build_detection_test_loader
from detectron2.evaluation import COCOEvaluator, inference_on_dataset
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "./outputs/fasterrcnn_custom_test_1_loss_0.12796187750063837.pth") # this model is trained using torchvision
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.85
trainer = MyTrainer(cfg)
predictor = DefaultPredictor(cfg)
evaluator = COCOEvaluator("dataset_val", cfg, False, output_dir="./output/")
val_loader = build_detection_test_loader(cfg, "dataset_val")
inference_on_dataset(trainer.model, val_loader, evaluator)
when I performed this, I’m getting below answer,
[11/20 00:19:51 d2.evaluation.coco_evaluation]: Per-category bbox AP:
| category | AP | category | AP |
|:-----------|:------|:-----------|:------|
| car | 0.000 | person | 0.000 |
OrderedDict([('bbox',
{'AP': 0.0,
'AP50': 0.0,
'AP75': 0.0,
'APs': 0.0,
'APm': 0.0,
'APl': 0.0,
'AP-car': 0.0,
'AP-person': 0.0})])
Also I noted this,
Skip loading parameter 'roi_heads.box_predictor.bbox_pred.weight' to the model due to incompatible shapes: (12, 1024) in the checkpoint but (8, 1024) in the model! You might want to double check if this is expected.
Skip loading parameter 'roi_heads.box_predictor.bbox_pred.bias' to the model due to incompatible shapes: (12,) in the checkpoint but (8,) in the model! You might want to double check if this is expected.
Some model parameters or buffers are not found in the checkpoint:
it would be really helpful you anyone can guide me here.
Note: I have tried different github repo’s to get this answer, but I’m trying this approach.
@ptrblck