# Multi-Classification and CrossEntropyLoss - interplay of \*weight\* and \*ignore\_index\*?

**URL:** <https://discuss.pytorch.org/t/multi-classification-and-crossentropyloss-interplay-of-weight-and-ignore-index/68030>\
**Category:** Uncategorized\
**Created:** [January 29, 2020, 3:17pm UTC](https://discuss.pytorch.org/t/multi-classification-and-crossentropyloss-interplay-of-weight-and-ignore-index/68030 "2020-01-29T15:17:16Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bendrien](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/c67d28/32.png) [@Bendrien](https://discuss.pytorch.org/u/Bendrien)\
**Post date:** [January 29, 2020, 3:17pm UTC](https://discuss.pytorch.org/t/multi-classification-and-crossentropyloss-interplay-of-weight-and-ignore-index/68030/1 "2020-01-29T15:17:16Z")

</div>

Hello there.  
First of all, im having a great time playing around with PyTorch - thanks for that 🙂

Now to my question.  
Im building a CNN for sequential data using `CrossEntropyLoss`. I have 3 classes as output of which two are of interest and the last is used if one of the two wasnt a fitt:

- **Class 0** : no fitt
- **Class 1** : interest 1
- **Class 2** : interest 2

All three classes are exclusive and my labeld training data is highly unbalanced in this order of occurences: **Class 0** \>\> **Class 1** \> **Class 2**. To compensate for the occurences im computing weights for the _weight_ setting of the `CrossEntropyLoss` like this:

```auto
def weights(labels, classes): # classes = 3
    hist = torch.histc(labels, classes)
    m = hist.min()
    w = torch.tensor([v if v == 0 else m / v for v in hist])
    return w

```

After reading the documentation of `CrossEntropyLoss` again i stumbled over the _ignore\_index_ parameter which i havent noticed before. So i was wondering if this would help me but im not quite sure how to use this parameter in conjunction with the weights or to use it at all.

Would it make sense to set the _ignore\_index_ parameter to my **Class 0**?  
If this is the case do i have to adapt my weight computation to ignore **Class 0**?

---

<div class="post-metadata">

**Author:** ![KFrank](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/32.png) [@KFrank](https://discuss.pytorch.org/u/KFrank)\
**Post date:** [January 29, 2020, 6:35pm UTC](https://discuss.pytorch.org/t/multi-classification-and-crossentropyloss-interplay-of-weight-and-ignore-index/68030/2 "2020-01-29T18:35:01Z")

</div>

Hello Anyere!

> [@Bendrien](#):
>
> Im building a CNN for sequential data using `CrossEntropyLoss`. I have 3 classes as output of which two are of interest and the last is used if one of the two wasnt a fitt:
> 
> - **Class 0** : no fitt
> - **Class 1** : interest 1
> - **Class 2** : interest 2
> 
> …
> 
> Would it make sense to set the _ignore\_index_ parameter to my **Class 0**?

Unless there is something unusual about your classification  
problem, I think you should not use `ignore_index` for Class 0.

Suppose you feed a Class-0 sample to your model (so `target`  
is 0), and your model incorrectly classifies it as Class 2. You  
do want to tell your model not to do this, so you do want this  
incorrect classification to add a penalty to your loss function.

If you set `ignore_index = 0`, this sample (and all other Class-0  
samples) will not contribute to the loss function, and so Class-0  
misclassifications will not be penalized.

(Because your training data is unbalanced, you may indeed want  
to deweight the Class-0 samples in your training set – either by  
using `CrossEntropyLoss`’s `weight` argument, or by sampling  
fewer of your Class-0 samples. But ignoring them altogether goes  
too far and would be hiding import information from your model’s  
training.)

Good luck!

K. Frank

---

<div class="post-metadata">

**Author:** ![Bendrien](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/c67d28/32.png) [@Bendrien](https://discuss.pytorch.org/u/Bendrien)\
**Post date:** [January 30, 2020, 1:51pm UTC](https://discuss.pytorch.org/t/multi-classification-and-crossentropyloss-interplay-of-weight-and-ignore-index/68030/3 "2020-01-30T13:51:05Z")

</div>

Thank you for your answer @KFrank ! this was actually my gut feeling but wanted some other opinions.

One further question:  
I am recomputing the weights for each minibatch. Does the shown computation of the weights represent the occurences of my labels properly for the `CrossEntropyLoss`?

---

<div class="post-metadata">

**Author:** ![KFrank](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/32.png) [@KFrank](https://discuss.pytorch.org/u/KFrank)\
**Post date:** [January 31, 2020, 2:06am UTC](https://discuss.pytorch.org/t/multi-classification-and-crossentropyloss-interplay-of-weight-and-ignore-index/68030/4 "2020-01-31T02:06:34Z")

</div>

Hi Anyere!

> [@Bendrien](#):
>
> I am recomputing the weights for each minibatch. Does the shown computation of the weights represent the occurences of my labels properly for the `CrossEntropyLoss`?

With one correction, your weight computation looks correct.

Note, that in the case that `hist.min() == 0` (perfectly possible), all  
of your weights will be zero. I would eliminate `m = hist.min()`, and  
go with `1.0 / v` (in place of `m / v`).

A couple of comments:

I don’t _think_ that the overall normalization of the weights matters. I  
_think_ that if you use `reduction = 'mean'` in your `CrossEntropyLoss`  
(the default), the overall scale factor of your weights will drop out. (I.e.,  
something like `weight = 123.4 * weight` won’t have any effect.)

I would probably just calculate the weights for the entire training set,  
and not bother doing it on a per-batch basis. (I think both approaches  
are reasonable.) As you take multiple steps in your gradient-descent  
optimization algorithm, you are, in a sense, averaging over multiple  
batches, so there is nothing magic about reweighting individual batches.

Last, using `weight-of-class = 1 / number-of-samples-in-class`  
is perfectly reasonable, and I think somewhat standard. But other  
choices are okay. You might experiment with only partially reweighting  
your classes.

Good luck!

K. Frank

---

<div class="post-metadata">

**Author:** ![Bendrien](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/c67d28/32.png) [@Bendrien](https://discuss.pytorch.org/u/Bendrien)\
**Post date:** [February 4, 2020, 12:56pm UTC](https://discuss.pytorch.org/t/multi-classification-and-crossentropyloss-interplay-of-weight-and-ignore-index/68030/5 "2020-02-04T12:56:05Z")

</div>

Hey Frank

> [@KFrank](#):
>
> Note, that in the case that `hist.min() == 0` (perfectly possible), all  
> of your weights will be zero. I would eliminate `m = hist.min()` , and  
> go with `1.0 / v` (in place of `m / v` ).

I think `hist.min()` cant be zero since my labels are exclusive. There has to be always exactly one true of all three classes. And for the normalisation I just chose 1 as it is the most common value in those situations.

> [@KFrank](#):
>
> I would probably just calculate the weights for the entire training set,  
> and not bother doing it on a per-batch basis.

I was wondering about that too and decided to recalculate on a per batch basis for those cases in which my sequence does not have one class to appear at all. As far as I understand this will make the true meaning of this sequence more clear and granular for the backpropagation.

As a side note, when I’m doing my evalutation on test data I reset all weights to 1, since this would be the normal usecase of the model in the wild - this makes sense, right?

Here is one followup question 🙂  
During training I have sequences of variable length. For not having unnecessary zero-padding within my CNN I want to feed one sequence as a whole to my model. But for backpropagation I thought it would make sense to compute the losses on a fixed window to make the loss computation independent of the incoming sequence length. So i came up with this routine (its a simplified version ignoring the weight computation):

```auto
inputs, labels = sequence_data
outputs = model(inputs)

for l in range(0, len(labels), window_size):
    r = l + window_size
    loss = criterion(outputs[l:r], labels[l:r])
    optimizer.zero_grad()
    retain_graph = r < len(labels)
    loss.backward(retain_graph=retain_graph)
    optimizer.step()

```

For this to work I have had to use the `retain_graph` parameter of the backpropagation and I found the documentation on this feature a bit sparse. Do I use this correct and could someone explain to me whats happing there in more detail and what data will be retained?

---

<div class="post-metadata">

**Author:** ![Bendrien](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/c67d28/32.png) [@Bendrien](https://discuss.pytorch.org/u/Bendrien)\
**Post date:** [February 18, 2020, 1:36pm UTC](https://discuss.pytorch.org/t/multi-classification-and-crossentropyloss-interplay-of-weight-and-ignore-index/68030/7 "2020-02-18T13:36:29Z")

</div>

> [@KFrank](#):
>
> Note, that in the case that `hist.min() == 0` (perfectly possible), all  
> of your weights will be zero. I would eliminate `m = hist.min()` , and  
> go with `1.0 / v` (in place of `m / v` ).

I checked the code again and realised this situation could happen which I have not foreseen. Thanks for the headsup @KFrank But since the following is still true

> [@Bendrien](#):
>
> my labels are exclusive. There has to be always exactly one true of all three classes.

I fixed this issue by removing all zeros before taking the minimum: `m = hist[hist != 0].min()`

> [@Bendrien](#):
>
> I have had to use the `retain_graph` parameter of the backpropagation and I found the documentation on this feature a bit sparse. Do I use this correct and could someone explain to me whats happing there in more detail and what data will be retained?

This article got me covered:  
[https://towardsdatascience.com/pytorch-autograd-understanding-the-heart-of-pytorchs-magic-2686cd94ec95](https://towardsdatascience.com/pytorch-autograd-understanding-the-heart-of-pytorchs-magic-2686cd94ec95)