Convert a one hot vector into smooth vector - label smoothing

Hi,
This and this didn’t really help.
I want to use two losses. First loss uses smooth labels and the second one uses the one_hot vector. For the smooth label, i converted the one_hot vector targets = torch.tensor([[1, 0, 0, 0, 0, 0, 0], [0, 1, 0, 0, 0, 0], [0, 0, 1, 0, 0, 0] ...) into a smooth vector. Since pytorch doesn’t provide a ready function for smooth target, I did this and want to confirm I’m doing things right?

outputs = torch.randn(6, 10)
targets_onehot = torch.zeros(outputs.size())
smooth_label = 1. / self.eps # 0.2
smooth_target = targets_onehot.scatter(1, targets.unsqueeze(1), smooth_label)
logfeat = F.log_softmax(outputs, dim=1)
smooth_loss = logfeat * smooth_target 
smooth_loss = torch.sum(smooth_loss, 1) / smooth_loss.size(1)

real_target = targets_onehot.scatter(1, targets.unsqueeze(1), float(1))
real_loss = logfeat * real_target 
real_loss = torch.sum(real_loss, 1) / real_loss.size(1)

loss = - real_loss * ( 1 - self.flags)  - self.flags * smooth_loss

I short, i would like to use a loss based on torch.tensor([[0.2, 0, 0, 0, 0, 0, 0], [0, 0.2, 0, 0, 0, 0], [0, 0, 0.2, 0, 0, 0] ...)`` and another one based on targets `

Looking at torch.scatter, I’m a little bit confused how the operation is done. Am I doing things right?
Thank you.

Hi Jean Paul!

I haven’t worked through your code, so I can’t comment on it.

Two side comments, though:

First, I wouldn’t call your example,
targets = torch.tensor([1, 0, 1, 0, 1, 0]),
“one hot.” You have multiple non-zero entries. (I might see this
as a “multi-label” target, where classes “0”, “2”, and “4” are active,
and classes “1”, “3”, and “5” are not active.) But this doesn’t really
change the issue.

One way to smooth a one-hot vector (or a multi-label vector, or
any binary vector made up of zeros and ones) is to run it through
torch.nn.functional.softmax (alpha * target).

(alpha is a smoothing parameter: larger alpha makes the result
sharper, and smaller alpha makes it smoother.)

Good luck.

K. Frank

1 Like

my bad… It’s actually a multi class problem. I really wanted to simply the targets but messed up… Thank you. I’ll edit the question. By the way, i managed to solve it this way, for someone interested in the future.

 targets = targets.contiguous().view(-1)
 logfeat = F.log_softmax(inputs, dim=1)
 one_hot = torch.zeros_like(inputs).scatter(1, targets.view(-1, 1), float(1))
 smooth_target = torch.zeros_like(inputs).scatter(1, targets.view(-1, 1), self.smoothing)
 smooth_loss = (smooth_target * logfeat).sum(dim=1)
 return smooth_loss.mean()