Transformer for time series classification with classification costs

Hi,
I am using transformer to classify time series data into 5 categories and I have balanced data.
I would like to input my own misclassification matrix so that the misclassification costs are unequal.
Example: True Category 1 - Predicted Category 5 is worse than True Category 1 and Predicted Category 2.

Categories are ordinal in nature. So classifying 5 as 1 is worse than classifying 5 as 4.

How can I do that.
Thanks

I have no experience with this in particular but the loss function you use should provide a parameters defining the weights for all classes.