Label smoothing for only a subset of classes

[KFrank]

Hi macazinc!

@macazinc

Basically, two of the classes may be mislabeled as one another. I was hoping to try to use label smoothing as a way to compensate for some of the noise in these labels.

Here’s what I think you’re asking:

You have a multi-class (30 classes) classification problem. You know
that for most of your classes, your ground-truth target labels are
correct, but your labels sometimes mix up two of your classes, say 4
and 9, Let’s say that a sample labelled 4 is actually a 9 25% of the
time, and that a sample labelled 9 is actually a 4 10% of the time.

You will (most likely) want to use cross-entropy loss, but pytorch only
provides a version that takes integer categorical class labels for its
target. In your case, you want what I call soft labels, and will have
to write your own soft-label version of cross-entropy. See this post
for an implementation:

Soft Cross Entropy Loss (TF has it does Pytorch have it)

Hello Raaj! I do not believe that pytorch has a “soft” cross-entropy function built in. But you can implement it using pytorch tensor operations, so you should get the full benefit of autograd and gpu acceleration. See this (pytorch version 0.3.0) script: import torch torch.__version__ # define "soft" cross-entropy with pytorch tensor operations def softXEnt (input, target): logprobs = torch.nn.functional.log_softmax (input, dim = 1) return -(target * logprobs).sum() / input.shap…

Now let me assume that your (sometimes incorrect) target labels
are given as integer categorical labels. First use one_hot() (followed
by float()) to convert your categorical labels into soft labels (that all
happen to be zero or one). Then whenever a sample is labelled 4
(target[i, 4] == 1.0), set target[i, 4] = 0.75 and
target[i, 9] = 0.25. Similarly, when target[i, 9] == 1.0, set
target[i, 9] = 0.90, and target[i, 4] = 0.10.

You then feed the soft-label target you constructed into the soft-label
cross-entropy you implemented yourself.

Best.

K. Frank