How to fine tune BERT to get word embeddings on your own custom Corpus?

nlp
1

I can have pretrained word embeddings using this, https://colab.research.google.com/drive/1RhmL0BqNe52FEbdSyLpkfVuCZxE7b5ke

But Don’t know how to fine tune it on my custom dataset

2

What exactly do you want to do?
Fine tune the model or fine tune the embeddings?

3

Fine tune the embeddingd

4

If you train the model E2E (not just fine-tune the task layer), it would modify the pre-trained parameters of all the layers (including the embedding layer).
However, remember the BERT embeddings are different from the word2vec embeddings and they depend on the context. Hence, they cannot be used as it is for a different task (unlike word2vec embeddings which don’t have context).
Until absolutely necessary to fine-tune the embeddings, you can fine-tune task layers (over BERT pretrained) model and adapt it to your specific problem set.

5

Hi Irfan

were you able to solve the problem? I am in the same situation but I am working with the BioBERT model with a set of scientific articles of my own.

Greetings