# Weights in BCEWithLogitsLoss **URL:** \ **Category:** Uncategorized\ **Created:** [October 17, 2018, 12:09pm UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452 "2018-10-17T12:09:20Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [May 5, 2021, 5:31am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/21 "2021-05-05T05:31:49Z")
1. Yes, that’s the default interpretation. You could of course interpret the values as you wish and redefine the positive and negative, as well as the recall, precision etc. 2. By definition `0s` would be the negative class. The `pos_weight` usage is shown in the formula in the [docs](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss) so you can re-interpret it if needed (see point 1). No, binary classification/segmentation with `nn.BCEWithLogitsLoss` expect a single output channel for the binary classes. Multi-label classification/segmentation (where each sample/pixel can belong to zero, one, or more classes) expect an output channel per class. 3. Yes, as seen here: ```python x = torch.randn(2, 1, 24, 24, requires_grad=True) y = torch.zeros(2 * 1 * 24 * 24) y[torch.randint(0, y.nelement(), (100,))] = 1. y = y.view_as(x) print('y: 0s: {}, 1s: {}, nelement: {}'.format( (y==0.).sum(), y.sum(), y.nelement())) criterion = nn.BCEWithLogitsLoss() loss = criterion(x, y) print(loss) criterion_weighted = nn.BCEWithLogitsLoss(pos_weight=(y==0.).sum()/y.sum()) loss_weighted = criterion_weighted(x, y) print(loss_weighted) ``` ---
**Author:** ![cameron\_a](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cameron_a/32/37909_2.png) [@cameron\_a](https://discuss.pytorch.org/u/cameron_a)\ **Post date:** [May 5, 2021, 5:39am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/22 "2021-05-05T05:39:29Z")
Wow! That was a quick response! Thank you! I wasn’t expecting a response so late in the night (it’s 1:30 AM in my time zone). I will inspect this code and post another reply should I have any other questions. Thank you again! ---
**Author:** ![cameron\_a](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cameron_a/32/37909_2.png) [@cameron\_a](https://discuss.pytorch.org/u/cameron_a)\ **Post date:** [May 5, 2021, 7:05am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/23 "2021-05-05T07:05:10Z")
Hello, as promised, I have some more questions lol: 1. I see that the loss value has indeed increased when applying the pos\_weight. I am struggling to understand the -Wn value and what the sigma represents in the formula [here](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss). I think the sigma is actually the sigmoid calculation 1/(1+e^(-x)). 2. Am I to use the 1st or second formula for my situation using pos\_weight? 3. The result from calling loss = criterion(x, y) yields a Tensor with 1 value in it. I would have thought this would return a tensor that was the same size as the supplied tensors so the modification can be made to each pixel location during backprop. Is this single value applied to every value in the predicted tensors? 4. Would I be able to use the weight parameter rather than the pos\_weight in this instance (to actively decrement the importance of class 0)? Thank you again. Sorry for my ignorance. I am just getting started with PyTorch ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [May 5, 2021, 7:18am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/24 "2021-05-05T07:18:58Z")
1. The `w_n` value is defined by the `weight` parameter of the criterion and the `sigma` is indeed the `sigmoid` function: > - **weight** ([_Tensor_](https://pytorch.org/docs/stable/tensors.html#torch.Tensor)_,_ _optional_) – a manual rescaling weight given to the loss of each batch element. If given, has to be a Tensor of size nbatch. 1. The second one, since `p_c` defines the `pos_weight`. 2. By default `reduction='mean'` will be used. If you want to get the unreduced loss tensor, you could use `reduction='none'`. 3. The `weight` parameter would weight each sample so I don’t think it would yield the same results, since `w_n` would be applied to both terms. ---
**Author:** ![cameron\_a](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cameron_a/32/37909_2.png) [@cameron\_a](https://discuss.pytorch.org/u/cameron_a)\ **Post date:** [May 5, 2021, 7:49am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/25 "2021-05-05T07:49:09Z")
Awesome! Thank you very much! And I didn’t realize that “reduction” was referring to either returning the mean of the entire tensor (or tensor). Thank you for informing me. ---
**Author:** ![Ravi\_Kothari](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ravi_kothari/32/40945_2.png) [@Ravi\_Kothari](https://discuss.pytorch.org/u/Ravi_Kothari)\ **Post date:** [August 4, 2021, 9:57am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/26 "2021-08-04T09:57:05Z")
I tried the weighted BCEWithLogitsLoss with following: input = (B,1,256,256) model\_output = (B,3,256,256) Grd\_truth = (B,3,256,256) initially I set the pos\_weight as a Tensor size of 3, but it was showing an error . Then I changed it to the size of (3,256,256) and it worked (better than w/o weights) ---
**Author:** ![Sampath\_Kumaran](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sampath_kumaran/32/30231_2.png) [@Sampath\_Kumaran](https://discuss.pytorch.org/u/Sampath_Kumaran)\ **Post date:** [November 9, 2021, 4:28am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/27 "2021-11-09T04:28:41Z")
For calculating pos\_weight we need to use only train\_dataset or the entire dataset for multilabel classification? ---
**Author:** ![relaxxpls](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/relaxxpls/32/48885_2.png) [@relaxxpls](https://discuss.pytorch.org/u/relaxxpls)\ **Post date:** [April 22, 2022, 4:36am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/29 "2022-04-22T04:36:23Z")
Only the train dataset. ---
**Author:** ![jia\_lee](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jia_lee/32/10385_2.png) [@jia\_lee](https://discuss.pytorch.org/u/jia_lee)\ **Post date:** [February 1, 2023, 1:30pm UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/30 "2023-02-01T13:30:57Z")
> [@ptrblck](#): > > w\_n How can we balance the binary outputs (i.e., weight all the classes)? For example, a multi-label classification [dog, cat, rabbit] and BCEWithLogitsLoss is used, how can we weight the importance among the three classes, dog, cat, rabbit? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [February 1, 2023, 7:35pm UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/31 "2023-02-01T19:35:10Z")
`pos_weight` expects a tensor with the length equal to the number of classes for multi-label use cases, so you could provide a separate weight for each class. ---
**Author:** ![jia\_lee](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jia_lee/32/10385_2.png) [@jia\_lee](https://discuss.pytorch.org/u/jia_lee)\ **Post date:** [February 2, 2023, 3:30am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/32 "2023-02-02T03:30:27Z")
Sorry for my confusion. In my practice, `weight` also expects a tensor with the length equal to the number of classes for multi-label use cases rather than the size of sample batch. In my understanding, `pos\_weight control the positive/negative balance within each class. But I’m going to control the balance between the classes. Hoping for your reply, thank you! ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [February 2, 2023, 9:31am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/33 "2023-02-02T09:31:14Z")
> [@jia\_lee](#): > > But I’m going to control the balance between the classes. In this case you might want to use an unreduced loss and apply the class weights afterwards. Since you are working on a multi-label classification I’m unsure what kind of weights you are planning to apply for samples with multiple positive class labels. ---
**Author:** ![jia\_lee](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jia_lee/32/10385_2.png) [@jia\_lee](https://discuss.pytorch.org/u/jia_lee)\ **Post date:** [February 2, 2023, 9:52am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/34 "2023-02-02T09:52:24Z")
It seems that the roles of `weight` differs between multi-label classification and multi-class single-label classification. In the first case, `weight` should have the same length as label categories. In the second case, `weight` should have the same size as nbatch. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [February 2, 2023, 10:25am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/35 "2023-02-02T10:25:59Z")
Yes, since it might not be trivial to apply weights to a multi-label classification use case. Let me give you an example. In a multi-class classification you can directly apply a class weight to the corresponding sample as seen here: ```python # multi-class classification batch_size = 10 nb_classes = 4 logits = torch.randn(batch_size, nb_classes, requires_grad=True) targets = torch.randint(0, nb_classes, (batch_size,)) weights = torch.rand(nb_classes) print(targets) # tensor([2, 3, 3, 0, 1, 2, 3, 2, 0, 1]) print(weights) # tensor([0.9253, 0.1432, 0.8336, 0.9465]) weighted_criterion = nn.CrossEntropyLoss(weight=weights, reduction="mean") loss = weighted_criterion(logits, targets) print(loss) # tensor(2.6470, grad_fn=) raw_criterion = nn.CrossEntropyLoss(reduction="none") loss_raw = raw_criterion(logits, targets) print(loss_raw) # tensor([2.3437, 3.4518, 3.3348, 2.1393, 0.9009, 5.2935, 1.5514, 0.5305, 2.6735, # 3.5571], grad_fn=) loss_weighted = (loss_raw * weights[targets] / weights[targets].sum()).sum() print(loss_weighted) # tensor(2.6470, grad_fn=) ``` Indexing the `weights` tensor with the `targets` works fine and returns the expected loss as verified in my manual comparison. However, in a multi-label classification use case each sample can belong to zero, one, or multiple classes as seen here: ```python # multi-label classification targets = torch.randint(0, 2, (batch_size, nb_classes)) print(targets) # tensor([[1, 0, 1, 1], # [1, 1, 1, 1], # [0, 0, 1, 1], # [1, 1, 1, 1], # [0, 1, 1, 0], # [1, 0, 0, 0], # [0, 0, 0, 1], # [1, 0, 0, 1], # [1, 0, 0, 1], # [1, 0, 0, 0]]) raw_criterion = nn.BCEWithLogitsLoss(reduction="none") loss_raw = raw_criterion(logits, targets.float()) print(loss_raw) # tensor([[0.7298, 0.3813, 1.4581, 0.5213], # [1.4387, 0.5741, 0.6803, 2.5363], # [0.3412, 0.3628, 0.1299, 1.4728], # [0.9664, 1.2701, 0.2159, 2.8560], # [0.6312, 0.4537, 0.6042, 0.3793], # [0.2912, 2.0887, 0.0754, 1.8689], # [1.0385, 2.0379, 0.8648, 0.3196], # [0.7231, 0.1918, 1.3323, 0.8123], # [0.8825, 0.9056, 1.9396, 0.3908], # [0.7225, 0.3262, 0.3110, 2.5515]], # grad_fn=) print(weights) # tensor([0.9253, 0.1432, 0.8336, 0.9465]) ``` It’s now unclear to me how you would like to apply the weights. E.g. take the first sample with a target of `[1, 0, 1, 1]`, which means classes 0, 2, and 3 are “active”. Would you sum the corresponding weights and multiply it directly with the unreduced loss? ---
**Author:** ![jia\_lee](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jia_lee/32/10385_2.png) [@jia\_lee](https://discuss.pytorch.org/u/jia_lee)\ **Post date:** [February 3, 2023, 6:50am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/37 "2023-02-03T06:50:23Z")
Yes, I would like to do something like 0.9253 \* 0.7298+ 0.1432 \* 0.3813 + 0.8336 \* 1.4581 + 0.9465 \* 0.5213 according to your example. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [February 3, 2023, 7:00am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/38 "2023-02-03T07:00:44Z")
In that case you can multiply the `weights` with the `loss_raw` tensor. ---
**Author:** ![jia\_lee](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jia_lee/32/10385_2.png) [@jia\_lee](https://discuss.pytorch.org/u/jia_lee)\ **Post date:** [February 3, 2023, 7:15am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/39 "2023-02-03T07:15:21Z")
Can I specify the class `weights` in nn.BCEWithLogitsLoss by its argument `weights`. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [February 3, 2023, 7:45am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/40 "2023-02-03T07:45:40Z")
Yes, in case you want to apply the same weight to each sample (which seems to be the case here). ---
**Author:** ![jia\_lee](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jia_lee/32/10385_2.png) [@jia\_lee](https://discuss.pytorch.org/u/jia_lee)\ **Post date:** [February 3, 2023, 8:39am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/41 "2023-02-03T08:39:29Z")
ps: for multi-label classification with `BCEWithLogitsLoss` 🙂 Thank you for your help~ ---
**Author:** ![Eliethesaiyan](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eliethesaiyan/32/49012_2.png) [@Eliethesaiyan](https://discuss.pytorch.org/u/Eliethesaiyan)\ **Post date:** [January 12, 2024, 3:07am UTC](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/42 "2024-01-12T03:07:05Z")
> [@ptrblck](#): > > `loss_weighted = (loss_raw * weights[targets] / weights[targets].sum()).sum()` Thank you for always giving a good explanation. Wouldn 't `loss_weighted = (loss_raw * weights[targets] / weights[targets].sum()).sum()` be `loss_weighted = (loss_raw * (1- weights[targets]) / weights[targets].sum()).sum()` So the class with more samples contributes less to the loss? [Previous page](https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452.md?page=1)