I think the discussion in Create a Variation of Cross Entropy Loss with Per Pair of Classes Weights might be useful.