I am fine-tuning a Faster R-CNN model for object detection on a custom dataset. The model is declared as follows:
model = fasterrcnn_resnet50_fpn(weights="DEFAULT")
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
Due to the nature of my data, I decided to add an augmentation to rotate the image by 180 degrees. I added the following transform:
transforms.append(T.RandomApply(torch.nn.ModuleList([T.RandomRotation(degrees=(180, 180), expand=1)]), p=0.5)
However, using a probability of 0.5 led to oscillating training performance, not reaching the baseline results on the same test set. Increasing the probability to 1 made it even worse.
I have checked the images passed through the data loader, and the bounding boxes are correct after the augmentation step.
It seems like the model is receiving incorrect data and training on random coordinates. What am I missing?