Does nn.CrossEntropyLoss internally build a one-hot encoding of the target?

[ptrblck]

@wjeliot

Is that true? Does this loss function simply index into the logits and work with that number?

Yes, you can directly index the logit as seen in this small example which calculates the loss manually using an explicit but slow approach.

@wjeliot

Or does it internally construct a one-hot encoding vector with a 1 in the target index position, and 0s elsewhere?

No, as it would be wasteful to create the one-hot encoded target first and multiply it with the full logit tensor. The result would be equal to indexing just the logit where the one-hot encoded target would have its 1 value. The limitation is of course that indexing won’t work with “soft-targets”, i.e. target tensors which contain probabilities and not class indices. For this use case nn.CrossEntropyLoss accepts a FloatTensor with probabilities in newer PyTorch releases.

@wjeliot

It seems to me that the latter is a much better option for computing loss because not only are you comparing the correct target to its predicted score, but you’re also comparing all the incorrect scores to 0. Surely that helps the model to learn faster? Or is my intuition incorrect here?

That’s not true, as you would multiply the logits of the uninteresting classes with a zero, so these values are not changing the loss at all.