I have some Question, Context and Answer triplets for training and along with that, I have sequences inside the queries that are labelled into a few categories.
Example: Suppose 2 kinds of labels are there A and B.
Query: How is the weather outside?
Label A: How is
Label B: weather outside
I wish to encode the question and context using BERT and then use these labels associated with the tokens to extract the answer span from the context.
My Approach: I thought of defining 3 different weight vectors (dimension: 768*768) or using weight vector of size (768 using elementwise multiplication with the query tokens) for the tokens labelled A, B and others. Then using the new query encoding and the BERT context encoding in an ensemble for the answer span selection task.
Any other ideas to use the sequence labels for the MRC task?