Finetuning VGG19

Hi, I follow the transfer learning tutorial and replace the resnet18 with vgg19. However, the network is not converging in the first few epochs. The loss remains constant (~0.5) after the first epoch. Do you know what will be the issues? Thanks.

model_ft = models.vgg19(pretrained=True)
num_ftrs = model_ft.classifier[6].in_features
#num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 2)

Observe that all parameters are being optimized

optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

Decay LR by a factor of 0.1 every 7 epochs

exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)