How to use Soft-label for Cross-Entropy loss?

[KFrank]

Hello oasjd7!

@oasjd7

As far as I know, Cross-entropy Loss for Hard-label is:

def hard_label(input, target):
...

(Or you can just use torch.nn.CrossEntropyLoss.)

How to implement Cross-entropy Loss for soft-label?

Please see the following thread for an implementation:

Soft Cross Entropy Loss (TF has it does Pytorch have it)

softmax_cross_entropy_with_logits TF supports not needing to have hard labels for cross entropy loss: logits = [[4.0, 2.0, 1.0], [0.0, 5.0, 1.0]] labels = [[1.0, 0.0, 0.0], [0.0, 0.8, 0.2]] tf.nn.softmax_cross_entropy_with_logits(labels=labels, logits=logits) Can we do the same thing in Pytorch?

1. What kind of Softmax should I use ?
   nn.Softmax() or nn.LogSoftmax() ?

You should use LogSoftmax. You have to pass the output of Softmax
through log() anyway to calculate the cross entropy, and the
implementation of LogSoftmax is numerically more stable than (the
mathematically, but not numerically equivalent) log (Softmax).

2. How to make target labels?
   Just add random noise values to the zeros.?
   [0,0,1] -> [0.1,0.2,0.7]

If you don’t naturally have soft target labels (probabilities across the
classes), I don’t see any value in ginning up soft labels by adding
noise to your 0, 1 (one-hot) labels. Just use CrossEntropyLoss
with your hard labels.

(If your hard labels are encoded as 0, 1-style one-hot labels you will
have to convert them to integer categorical class labels, as those are
what CrossEntropyLoss requires.)

Best.

K. Frank