Negative AP in TorchVision 0.3 Object Detection finetuning tutorial

I have been using this tutorial with a few datasets. I have found that sometimes while the AP for the bb increases the AP for the segmentation drops more than double. I have also noticed that running the Penn-Fudan dataset the precision for the small areas is negative. Is this supposed to be ok to have negative AP values❓

Also for the learning scheduler, is it ok for it to be placed before the evaluation on the test set?