How to calculate F1 Score with PyTorch Lightning - T5 Model

nlp
1

Hello,

I am fine-tuning a T5 model for QA task.
Everything works fine, but the last thing that I want to implement is calculating F1 Score.
After training the model by:

trainer.fit(model, data_module)

I tested it with:

trainer.test(dataloaders=test_dl)

And got only the test_loss.

I also tried to manually calculate it, but I don;t know what to do with the outputs from the model, here is an example of the outputs I have:

outputs = trained_model.model.generate(input_ids=input_ids, attention_mask=attention_mask)
print(outputs)
print(true_labels)

And it prints:

tensor([[    0,   532, 64177,   726,     1]], device='cuda:0')
tensor([[   259,  85679,  30010,   1393,  29887,    532,  64177,  11316,  94418,
         218480,   7815,  16736,    855,   6959,      1,   -100,   -100,   -100,
           -100,   -100,   -100,   -100,   -100,   -100,   -100,   -100,   -100,
           -100,   -100,   -100,   -100,   -100]], device='cuda:0')

So I don’t know how to compare them, even after removing the padding the sizes are different.

Thanks a lot!

2

python - How to get generated tokens in T5 training_step for using user-defined metrics? - Stack Overflow
This solution works well. I met the same problem and I spent hours searching for solutions until I saw this.