Training 'FasterRCNN' object without 'references\detection\engine.train_one_epoch' meets less decrease of loss

Hellow everyone!
I write my customized training code to train the FasterRCNN object in similar way to train the most classify-model, but it works bad with very slow cahnge of losses. I wonder whether my customized code is not suitable for training FasterRCNN, thank you.
The core code as follows(faster_rcnn is FasterRCNN object with pretrained vgg16.features as backbone):

params = [p for p in faster_rcnn.parameters() if p.requires_grad]

optimizer = optim.SGD(params, lr=0.01, momentum=0.9, weight_decay=0.0005)

lr_scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min')

for epoch in range(num_epochs):
    start = time.time()
    for img, target in data_loader:
        img = list( for image in img)
        target = [{k: for k,v in t.items()} for t in target]
        loss_dict = faster_rcnn(img, target)

        losses = sum(loss for loss in loss_dict.values())