Transfer learning with NLP on different context using Pytorch

nlp
1

Hi, I managed to create a model using BiDAF and Pytorch.

BiDAF: Bi-Directional Attention Flow

The model has been trained on SQuAD: The Stanford Question Answering Dataset (https://rajpurkar.github.io/SQuAD-explorer/)

Now, I would like to do transfer learning on a different context - financial regulation documents

My question is can anyone provide a ‘big picture’ view on what I should to to enable transfer learning so that the model can answer questions in relation to financial documents?

For example, do I have to create question answers pair with regards to the financial documents manually? If yes, do I have to clean up the documents (there are hundreds of them) and what software/technique should I use to clean up the data?

Any links and resources to read up?

2

It might be a good idea to use a pre-trained BERT model and apply fine-tuning on top of it. The BERT model is pretrained on a large corpus (some of which are finance related), so knowledge transfer should be possible. Also, BERT has performed really well on SuperGLUE for QnA.

You definitely need a dataset (larger the better) for transfer learning to work. Weak supervision may be helpful to create a decent quality dataset. Using crowdsourcing is another way (will cost some money). But, even with a small manually labeled dataset you may be able to achieve decent results.