My Custom Loss doesn't do backward properly

autograd
1

Hi, I am trying to apply boosting with NN, and to do so I need to multiply the loss of each sample by a scalar weight.
I’m implementing it by slightly changing the BCELoss in the following way:

from torch.nn.modules.loss import BCELoss
import torch
from torch.nn import functional as F

from torch.nn.modules.loss import BCELoss
import torch
from torch.nn import functional as F


class WeightedBCELoss(BCELoss):
    def __init__(self, weight=None, size_average=None, reduce=None, reduction='mean', weights_boosting=None):
        super(BCELoss, self).__init__(weight, size_average, reduce, reduction)
        self.weights_boosting = weights_boosting

    def forward(self, input, target):
        total_cross_entropy = 0

        for i, sample in enumerate(input):
            y_hat = input[i]
            y = target[i]
            c_e = F.binary_cross_entropy(y_hat, y, weight=self.weight,
                                         reduction=self.reduction).item()
            c_e = c_e * self.weights_boosting[i].item()
            total_cross_entropy = total_cross_entropy + c_e

        return torch.tensor([total_cross_entropy], requires_grad=True)

But afterwards my model weights do not change at all. Do you have any idea on how to fix it?

N.B. If I try to remove that “requires_grad=True” at the last line, it goes:

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Thanks in advance

2

Hi,

See this other post that is related: Getting None on custom loss function
Does that explains your issue?

3

Nope, unfortunately… but thanks

4

You do the similar thing of rewrapping your result into a Tensor after doing computations on non differentiable elements (python numbers in your case, integer tensors in the other post).
You will have to keep everything into Tensors for the autograd to work.

5

Mmm ok I think I got the problem, so basically you suggest to work only with tensors and never go to scalar?

6

Yes. The autograd engine can only track gradients for Tensors, not other objects unfortunately.

2 Likes
7

My main problem is using the “total_cross_entropy” value as an accumulator… do you have any idea on how to do it with tensors?

8

Taking a quick look at your function, just removing the two .item() and just returning return total_cross_entropy should work no?

If you do 0 + t with t a Tensor, you will get a Tensor.

1 Like
9

Got it, thank you so much :slight_smile: