Hi together,
I just finished experimenting with an encoder-classifier architecture and found a behavior that I cannot fully explain.
The architecture consists of a transformer encoder followed by a dense layer classifier.
First I trained the model in an end-to-end supervised pipeline, meaning by backpropagation of an MSE loss through the whole model.
After training, I freezed the parameters of the encoder network and reinitialized the classifier. Then, I trained the model again (only updating the classifiers parameters) on the same task, in order to simulate a transfer learning scenario.
During both training phases, the loss does converge and everything seems sound.
What seems strange to me now, is that the results the model achieves are much better after the first trining phase. The results after transfer learning are still reasonable but worse.
I would argue, that it is probably harder for the model to learn the task in the transfer learning scenario, as the encoder is fixed, meaning only the classifiers parameters can be used for optimization and thus the complexity that the architecture can model is lower.
However, the first training is actually a proof that the classifiers complexity is high enough to achieve better results.
Can anybody give me an intuition why the results are not the same?
Thank you already!