HI all,
I would like some advice on best practice on transfer learning.
Supposing i have a dataset A, and i have done some training with that dataset to produce optimum weights W1, If i have an additional dataset B, and i decide to start training from weights W1, in an effort to produce optimum weights W2 that best maps my data. is it best to train with just the additional dataset B or should i resume training from W1 with dataset A+B.
How similar are the two datasets? Do they have the same labels?
@ptrmcl thank you for taking the time out to reply. and yes assuming dataset A and dataset B have the same labels and are relatively similar. I would like to know when applying transfer learning with a learned set of weights that was trained on dataset A. In order to get good results on dataset B, do i need to feed my model just dataset B alone or dataset A +B
So are the data in both datasets something you may see when using your model? In that case, I think I’d use both if you have the computational resources.
Would be good to get others’ opinions too.
Hey, @temi92!
I have applied transfer learning once, and did it that way:
- I used a pretrained VGG on Imagenet (your dataset A with weights W1)
- Then, I used this VGG to train on Imagenette (a similar dataset, your dataset B)
- So in my case, it was best to just continue training with dataset B, however, note that the VGG already converged on the dataset A.
Let me know if you have any questions.
BR, David
So yeah frankly in the case of Imagenet it makes sense since it is a large dataset. If both datasets are small I think it may be better to train on A+B?
Yes, might be. But depending on the architecture used, and the dataset size, I would also worry about potential overfitting. Why not try both ways, using a validation set during training and testing what works best.