I am solving a multiclass classification problem and I am using a pretrained resnet101 for that. I encounter a problem with the behavior during training and validation. I start by training only the fully-connected layer and then gradually ‘unfreeze’ all others. The loss on validation steadily decreases.
But when I ‘unfreeze’ the whole network, the losses on both training and validation start to increase after each epoch.
What could be a possible reason for that?