Bigger dataset not helping in accuracy for BERT model

nlp
1

I am pre-training a BERT model on a custom dataset. The accuracy of the model gets stuck at ~ 20%. Since it is recommended that one should try to overfit a model on a small dataset before moving to a bigger dataset, I tried it and my model easily overfits on a small dataset (100 samples) you can see the plots below

Validation Accuracy
Validation Accuracy2528×1328 372 KB

Training Accuracy
Training Accuracy2528×1328 249 KB

Validation Loss
Validation Loss2528×1328 258 KB

Training Loss
Training Loss2528×1328 198 KB

But when I increase the size of the dataset (250K samples) the accuracy doesn’t improve after a certain point, as you can see below
Validation Accuracy Full
Validation Accuracy Full2528×1328 174 KB

Training Accuracy Full
Training Accuracy Full2528×1328 237 KB

Validation Loss Full
Validation Loss Full2528×1328 163 KB

Training Loss Full
Training Loss Full2528×1328 212 KB

I also tried with bigger-size models but still, the same problem persists. Please any suggestions in this regard would be very helpful. Also, let me know if more details are required.