What is k-dimensional crossentropyloss?

In the docs CrossEntropyLoss — PyTorch 2.1 documentation they discuss k-dimensional loss.

Can someone explain to me what k-dimensional loss is? Is this related to contrastive loss or symmetric loss or something like that?

If you have a problem where you have multiple nodes needing classification, then you may need k-dimensional CE loss.

A good example of this is object detection in a picture. Every pixel can be treated separately for classification. One pixel may be a “nothing” class, while another is “cat” and another “dog”, and so on.

1 Like

Another example would be sequence labeling of text data, for example:

  • Part-of-Speech Tagging or

  • Named Entity Recognition

where in both cases you want to predict a class label for each word in your sentence.

Got it thanks. Do you know of any source I can read more about the math behind this? It’s still not clear to me how it works and whether or not it’s the loss functionality i’m looking for

As far as loss is concerned, the number of k-dims is irrelevant as those will just get flattened, anyway.

It doesn’t differ from Cross Entropy Loss where we have only one classification made.

Suppose we have some photo for segmentation, that is evaluating each pixel separately for what type of object it contains. Each pixel has a corresponding index as a target, the correct class. And each pixel’s loss is evaluated independently.

Whether we keep the image 2d or flatten it doesn’t change anything since each pixel is evaluated individually.

And then you use the reduction argument to convey how to deal with those individual losses. It defaults to mean.