# Balancing a multilabel dataset

**URL:** <https://discuss.pytorch.org/t/balancing-a-multilabel-dataset/182321>\
**Category:** nlp\
**Created:** [June 17, 2023, 9:16pm UTC](https://discuss.pytorch.org/t/balancing-a-multilabel-dataset/182321 "2023-06-17T21:16:10Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Boriss\_Sidjakins](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/boriss_sidjakins/32/61379_2.png) [@Boriss\_Sidjakins](https://discuss.pytorch.org/u/Boriss_Sidjakins)\
**Post date:** [June 17, 2023, 9:16pm UTC](https://discuss.pytorch.org/t/balancing-a-multilabel-dataset/182321/1 "2023-06-17T21:16:10Z")

</div>

Hello,

I have a dataset that contains email texts and their corresponding labels. Each email can have multiple labels, making it a multilabel problem. I used multi-hot encoding for labels, so they look like [1, 0, 1, 0, 0], where 1 indicates that an email belongs to that class. However, my data is imbalanced, and some label combinations barely occur in the dataset, which makes my LSTM biased towards the majority classes. I tried specifying pos\_weight for BCEWithLogitsLoss, but that doesn’t seem to help. What are some balancing techniques that I could use?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [June 17, 2023, 10:28pm UTC](https://discuss.pytorch.org/t/balancing-a-multilabel-dataset/182321/2 "2023-06-17T22:28:20Z")

</div>

Balancing the data in a multi-label use case is not trivial and [this post](https://discuss.pytorch.org/t/unbalanced-data-in-multiple-object-detection/88104/2) describes some methods for a balanced sampling.

---

<div class="post-metadata">

**Author:** ![vdw](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vdw/32/10074_2.png) [@vdw](https://discuss.pytorch.org/u/vdw)\
**Post date:** [June 18, 2023, 1:18am UTC](https://discuss.pytorch.org/t/balancing-a-multilabel-dataset/182321/3 "2023-06-18T01:18:37Z")

</div>

How dependent are the labels? If they are (kind of) independent, would training 5 binary classifiers be an option?

---

<div class="post-metadata">

**Author:** ![Boriss\_Sidjakins](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/boriss_sidjakins/32/61379_2.png) [@Boriss\_Sidjakins](https://discuss.pytorch.org/u/Boriss_Sidjakins)\
**Post date:** [June 18, 2023, 9:29am UTC](https://discuss.pytorch.org/t/balancing-a-multilabel-dataset/182321/4 "2023-06-18T09:29:09Z")

</div>

I’m not sure how to measure their dependency, but for context, I’m classifying emails based on what architectural (software) design decisions they contain. Here is the distribution of labels in my training dataset (not-ak refers to emails that don’t contain any architectural decisions):

 ![test](https://discuss.pytorch.org/uploads/default/original/3X/e/3/e326075a14ebdf9b281b30a608af3da40c1bc6d5.jpeg)

I’m new to machine/deep learning, so I’m not sure if training 5 binary classifiers would be helpful in this case.

---

<div class="post-metadata">

**Author:** ![vdw](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vdw/32/10074_2.png) [@vdw](https://discuss.pytorch.org/u/vdw)\
**Post date:** [June 20, 2023, 6:36am UTC](https://discuss.pytorch.org/t/balancing-a-multilabel-dataset/182321/5 "2023-06-20T06:36:18Z")

</div>

Well, I would indeed consider 5 binary classifiers here, as Yes/No classifier for each of for 5 class labels.

For example if a item item1 is labeled “process, property, technology”, and an item item2 is labeled as “process, technology” it can be views as multiclass data as follow:  
(item1, process)  
(item1, property)  
(item1, technology)  
(item2, process)  
(item3, technology)

If you now train the, say, “property” classifier, you put item1 in the positive class and item2 in the negative class. For training the “process” and “technology” classifiers, of course both item1 and item2 are in the positive class

For inferencing, you pump a new item through all 5 classifiers. If the “property” classifier and the “technology” classifier say but not the others, you label the new item with “property, technology”.

---

<div class="post-metadata">

**Author:** ![J\_Johnson](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_johnson/32/55494_2.png) [@J\_Johnson](https://discuss.pytorch.org/u/J_Johnson)\
**Post date:** [June 21, 2023, 2:00am UTC](https://discuss.pytorch.org/t/balancing-a-multilabel-dataset/182321/6 "2023-06-21T02:00:50Z")

</div>

Suppose we have all of your targets(all\_targets) in a tensor of size [num\_targets, num\_classes].

First, sum along the num\_targets:

```auto
total_by_class = all_targets.sum(0)

```

Second, use this to generate a weights vector:

```auto
weights = total_by_class.mean()/total_by_class

```

Third, set empty classes back to 0.

```auto
cond = torch.isinf(weights)
weights[cond] = 0

```

Finally, pass that into your loss function:

```auto
criterion = nn.CrossEntropyLoss(weight=weights)

```

What that will do is cause the loss function to scale each class by a multiplier, by the inverse of their representation. I.e. classes with less examples get scaled more.