Hi,
I’m new to object detection so just getting my head around a few things. I followed the tutorial provided by PyTorch on object detection where the main metric for evaluation was mean average precision (mAP).
I’ve added extra bits and will apply a non-maximum suppression to clear up the number of bounding box predictions. However, it is still unclear to me on how I can determine the overall f1-score when evaluating the full dataset.
Currently, once I have trained the model (faster RCNN) I evaluate like this:
cpu_device = torch.device("cpu")
model.eval()
with torch.no_grad():
for images, targets in valid_data_loader:
images = list(img.to(device) for img in images)
outputs = model(images)
outputs = [{k: v.to(cpu_device)for k, v in t.items()} for t in outputs]
However, I’m unsure of how I can use this to determine false positives, true positives, false negatives and true negatives to then determine the overall f1-score.
I found this one line on the GitHub repo for the tutorial:
res = {target["image_id"].item(): output for target, output in zip(targets, outputs)}
where printing out the output (batch size of 3) gives me this:
{0: {'boxes': tensor([[434.2975, 191.6537, 467.0199, 224.7192],
[ 31.4918, 354.0607, 62.8257, 386.2621],
[469.9614, 86.2549, 501.9221, 117.5876],
[215.4281, 481.5853, 247.2938, 511.3138],
[ 90.5871, 313.2582, 123.1587, 345.9512],
[331.6597, 341.3073, 364.3313, 375.1403]], device='cuda:0'), 'labels': tensor([1, 1, 1, 1, 1, 1], device='cuda:0'), 'scores': tensor([0.9648, 0.9629, 0.9624, 0.9151, 0.7567, 0.2403], device='cuda:0')}, 1: {'boxes': tensor([[1.5194e+02, 1.0456e+02, 1.8351e+02, 1.3742e+02],
[3.8975e+02, 2.0607e+02, 4.2221e+02, 2.3842e+02],
[3.0471e+02, 3.0989e+02, 3.3620e+02, 3.4213e+02],
[1.3239e+01, 1.1389e+02, 4.4452e+01, 1.4622e+02],
[2.9837e+02, 4.5101e+02, 3.3001e+02, 4.8375e+02],
[4.5444e+02, 3.5601e-02, 4.8779e+02, 2.5613e+01]], device='cuda:0'), 'labels': tensor([1, 1, 1, 1, 1, 1], device='cuda:0'), 'scores': tensor([0.9767, 0.9669, 0.9590, 0.8096, 0.7546, 0.3785], device='cuda:0')}, 2: {'boxes': tensor([[248.2404, 110.2581, 279.5806, 141.3378],
[ 29.6403, 150.2647, 64.2155, 182.6650],
[178.2941, 176.4075, 210.9212, 208.9970],
[362.3374, 93.1001, 396.6398, 125.1960],
[360.2594, 65.7590, 393.0970, 99.8704]], device='cuda:0'), 'labels': tensor([1, 1, 1, 1, 1], device='cuda:0'), 'scores': tensor([0.9661, 0.8858, 0.8422, 0.6669, 0.1381], device='cuda:0')}}
I’m not really sure how I can use this information to get each classification metric and to determine an f1 score…