Finetuning GPT2 for text to text generation

nlp
1

Hi!

I am trying to finetune gpt 2 for a project. I have a dataset of Reddit posts with one comment each and the task is to generate a comment for a given post. I think this is a conditional text generation task but I am not sure how to use gpt2 to do this. I was trying to use the transformers library to do so, but I am not sure how to format the data. There is not a lot online for text to text generation using gpt2. Should the model be given just the post or the post and the comment as input? Not sure how this works, can someone please give me some instruction/advice?

Thanks

2

Most text to text generation are trained on next token prediction. Along with making use of bos and eos tokens. For instance, you might give the model post + bos + comment + eos and mask after bos.

Please consider taking this free online course on Huggingface on NLP. It might fill in some of the gaps.