Initializing single number for re-summing in autograd?

autograd
1

Hi, I’m interested in trying to combine loss criteria, and I made a little function, to try to do that. Here’s what DOESN’T work…

def calc_loss(Y_pred, Y_true, criteria, lambdas):
    #  Y_pred: the output from the network
    #  Y_true: the target values
    #  criteria: an iterable of pytorch loss criteria, e.g. [torch.nn.MSELoss(), torch.nn.L1Loss()]
    #  lambdas:  a list of regularization parameters
    assert len(criteria) == len(lambdas), "Must have the same number of criteria as lambdas"
    loss = torch.zeros(1, requires_grad=True)       # not sure how to properly initialize a single pytorch number for autograd
    for i in range(len(criteria)):
        loss += lambdas[i] * criteria[i](Y_pred, Y_true)
    return loss

When I do that, I get the Runtime error at the “loss +=” line…

RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.

So instead, here’s the working routine that I have right now, but it’s just not ‘elegant’ IMHO:

def calc_loss(Y_pred, Y_true, criteria, lambdas):
    assert len(criteria) == len(lambdas), "Must have the same number of criteria as lambdas"

    for i in range(len(criteria)):
        if (0==i):
            loss = lambdas[i] * criteria[i](Y_pred, Y_true)
        else:
            loss += lambdas[i] * criteria[i](Y_pred, Y_true)
    return loss

For future reference how should you properly initialize a ‘zero’ loss of this type?
(I’m using Pytorch 0.4)

Thanks!

PS- Also tried a simple…

    loss = torch.dot(lambdas, criteria(Y_pred, Y_true))

…but that was just wishful thinking, because the lambdas are ‘just numbers’, so you get TypeError: 'list' object is not callable. Even if you wrap the lambdas in torch.tensor(), you get the same error.

2

Hi,

You can just initialize it with a python 0.
I haven’t run it, but the following code should work :slight_smile:

def calc_loss(Y_pred, Y_true, criteria, lambdas):
    #  Y_pred: the output from the network
    #  Y_true: the target values
    #  criteria: an iterable of pytorch loss criteria, e.g. [torch.nn.MSELoss(), torch.nn.L1Loss()]
    #  lambdas:  a list of regularization parameters
    assert len(criteria) == len(lambdas), "Must have the same number of criteria as lambdas"
    loss = 0
    for i in range(len(criteria)):
        loss += loss lambdas[i] * criteria[i](Y_pred, Y_true)
    return loss
3

Wow, that’s so simple. I guess I’d anticipated getting ‘Expected __Tensor but got ___Tensor’ errors if I tried that!
Thanks. Yes, it works.