Cross entropy with logit targets

[KFrank]

Hi Haziq!

@Haziq

each ground truth has some uncertainty and is thus a vector of probability scores e.g. [0.1, 0, 0.7, 0, 0.2, 0, 0] instead of [0,0,1,0,0,0,0].

(As an aside, your title for this thread, “Cross entropy with logit
targets” should probably be “Cross entropy with probability targets,”
since you ask about “probability scores” and use probabilities in the
example you give.)

Your ground-truth target probabilities are what are sometimes called
“soft labels.” Pytorch’s CrossEntropyLoss does not support this kind
of target, but you can write a version that does. For details, please
see this post:

Soft Cross Entropy Loss (TF has it does Pytorch have it)

Hello Raaj! I do not believe that pytorch has a “soft” cross-entropy function built in. But you can implement it using pytorch tensor operations, so you should get the full benefit of autograd and gpu acceleration. See this (pytorch version 0.3.0) script: import torch torch.__version__ # define "soft" cross-entropy with pytorch tensor operations def softXEnt (input, target): logprobs = torch.nn.functional.log_softmax (input, dim = 1) return -(target * logprobs).sum() / input.shap…

Best.

K. Frank