Cant prevent overfitting after 2 epochs

Built a small merge-image captioning model using pretrained vgg16. But cant prevent overfitting even adding droputs. Validation loss keeps incresing after 2 epochs. So is it possible that this is my model’s maximum performance?

If you tried to train a vgg16 on a small dataset, it is possible that your model is just too powerful and will always overfit, you might want to use a simpler model.

actually im using an already trained vgg16 which is trained on ImageNet. I use vgg as an encoder . I only train my decoder and after 2 epochs overfits

I would try to use a simpler decoder still.
Also you can try different regularization techniques (batchnorm, weight decay, etc).
Another thing to check is that your training and validation sets are actually properly correlated. Make you split them with samples from each class in each, etc.

1 Like

i will check them thank you. I think my dataset is too small i use 6000 train images from Flickr8k.