I am working on fine-tuning a Llama(decoder only) model on SQuAD Dataset, and wondering what is the right way
to structure inputs and labels.
Option1:
input: “SOS context:{context} question:{question} answer: pad …. pad”
label: “{answer} EOS pad pad … pad”
Option2:
input: “SOS context:{context} question:{question} answer: pad …. pad”
label: “pad … pad {answer} EOS pad … pad”
Option3:
input: “SOS context:{context} question:{question} answer: {answer} pad …. pad”
label: “context:{context} question:{question} answer:{answer} EOS pad …. pad”