Cross entropy with logit targets

[Haziq]

I am training a model for classification where each ground truth has some uncertainty and is thus a vector of probability scores e.g. [0.1, 0, 0.7, 0, 0.2, 0, 0] instead of [0,0,1,0,0,0,0]. The cross-entropy loss in PyTorch however, accepts only an integer target so I was hoping if someone could recommend a solution or an alternative loss function that is suitable for my classification problem.

https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html

[KFrank]

Hi Haziq!

@Haziq

each ground truth has some uncertainty and is thus a vector of probability scores e.g. [0.1, 0, 0.7, 0, 0.2, 0, 0] instead of [0,0,1,0,0,0,0].

(As an aside, your title for this thread, “Cross entropy with logit
targets” should probably be “Cross entropy with probability targets,”
since you ask about “probability scores” and use probabilities in the
example you give.)

Your ground-truth target probabilities are what are sometimes called
“soft labels.” Pytorch’s CrossEntropyLoss does not support this kind
of target, but you can write a version that does. For details, please
see this post:

Soft Cross Entropy Loss (TF has it does Pytorch have it)

Hello Raaj! I do not believe that pytorch has a “soft” cross-entropy function built in. But you can implement it using pytorch tensor operations, so you should get the full benefit of autograd and gpu acceleration. See this (pytorch version 0.3.0) script: import torch torch.__version__ # define "soft" cross-entropy with pytorch tensor operations def softXEnt (input, target): logprobs = torch.nn.functional.log_softmax (input, dim = 1) return -(target * logprobs).sum() / input.shap…

Best.

K. Frank

[ylsvik]

Update: from version 1.10, Pytorch supports class probability targets in CrossEntropyLoss, so you can now simply use:

criterion = torch.nn.CrossEntropyLoss()
loss = criterion(x, y)

where x is the input, y is the target. When y has the same shape as x, it’s gonna be treated as class probabilities. Note that x is expected to contain raw, unnormalized scores for each class, while y is expected to contain probabilities for each class (typically the output of the softmax layer). You can find details in the docs.