How to combine multiple criterions to a loss function?

[BikashgG (Bikash Gyawali)]

I did this but got the following error :

add received an invalid combination of arguments - got (torch.FloatTensor), but expected one of:

• (float value)
  didn’t match because some of the arguments have invalid types: (torch.FloatTensor)
• (torch.cuda.FloatTensor other)
  didn’t match because some of the arguments have invalid types: (torch.FloatTensor)
• (torch.cuda.sparse.FloatTensor other)
  didn’t match because some of the arguments have invalid types: (torch.FloatTensor)
• (float value, torch.cuda.FloatTensor other)
• (float value, torch.cuda.sparse.FloatTensor other)

Both my loss are of the same type - nn.CrossEntropyLoss() <class ‘torch.autograd.variable.Variable’>

Any suggestion on how to resolve this please?

[tom (Thomas V)]

If you stare down the error message, you see that you have one cuda and one not.

Best regards

Thomas

[BikashgG (Bikash Gyawali)]

Yes, thank you @tom. I am very novice here.

[platero (Wei Yang)]

Is it possible to weight the losses when multiple losses are used and how? Is it correct by using the following code?

    mse_loss = nn.MSELoss(size_average=True)
    a = weight1 * mse_loss(inp, target1)
    b = weight2 * mse_loss(inp, target2)
    loss = a + b

    loss.backward()

[achaiah]

Yup, you can certainly weigh the losses however you see fit. What you’re doing in your example should be fine.

[ElleryL (EL)]

what if my second loss function requires some computed value from first loss (or even the grad of first loss?) in that case I can’t add two loss together; they must be gradients respectively; and retain_graph=True gives wrong results as well as the intermediate grads not correct

see this one

Backward() w.r.t. to different parameters with different loss function

Consider I have Variable x y = f(x) z = Q(y) # Q here is a neural net Step(1): gradient w.r.t. x. z.backward(retain_graph = True) x.grad g = x.grad.clone() x.grad.data.zero_() Step(2): have another function that take the gradients we just compute L(g) I want to take gradient of it w.r.t. to the weights of neural net Q as following var_opt = torch.optim.Adam(Q.parameters(), lr=lr) while not converge: var_opt.zero_grad() variance_loss = torch.mean(L(g)) variance_los…

[PeterXiaoGuo]

thank you very much Jordan! your answer help me to combine my custom loss function with nn.Loss.

[PeterXiaoGuo]

Hi BikashgG, I think you could use .type() to change tensor’s type from torch.FloatTensor to torch.LongTensor.

I face a similar problem when I use this CrossEntropyLoss().

Check the official document on this loss function, there is a requirement on the type of feed-in tensor.

Hope it helps,

Peter

[tarun (Tarun Kalluri)]

What if the losses are computed over different parts of the network, say loss1 is for first 3 layers and loss2 is for first 7 layers (incl. the first 3)? Wouldn’t the sum of losses method also backprop loss1 through layers 4-7? Would calling loss1.backward() and loss2.backward() separately be recommended in that case?

[sid87]

How to implement multiple loss function at diffent layer

Does torch.autograd.back([l1,l2]) mean this two loss backward for separate node? such as softmaxloss updates fc2 and layers before fc2, custom_loss updates fc1 and layers before fc1? And if I use L = l1 + l2, L.backward(), is L will update fc2 and fc1 separately?

Hope it helps!

[Gabrer]

mse_loss = nn.MSELoss(size_average=True)
a = weight1 * mse_loss(inp, target1)
b = weight2 * mse_loss(inp, target2)
loss = a + b
loss.backward()

What if I want to learn the weight1 and weight2 during the training process?
Should they be declared parameters of the two models? Or of a third one?

[justusschock]

To learn them would not make sense for me since your network will only learn to minimize the weights which will automatically minimize the partial losses and thus the total loss.

[phaypyt]

@achaiah

. What you’re doing in your example should be fine.

The weight should be int or tensor? Or both works?

[achaiah]

Int, since you’re weighing each type of loss instead of each value individually.

[PySimon]

You could add another loss that penalizes the model for learning weight1 = weight2 = 0

[Rojin (Rojin Safavi)]

@tarun

What if the losses are computed over different parts of the network, say loss1 is for first 3 layers and loss2 is for first 7 layers (incl. the first 3)? Wouldn’t the sum of losses method also backprop loss1 through layers 4-7? Would calling loss1.backward() and loss2.backward() separately be recommended in that case?

Did you learn how to do this?

[marchinidavide (Davide)]

What is the preferred way to combine loss functions between:

output = net(input)
loss1 = w1 * crit1(output, target1)
loss2 = w2 * crit2(output, target2)
loss = loss1 + loss2
loss.backward()
print(loss.item())

and

output = net(input)
loss1 = w1 * crit1(output, target1)
loss1.backward()
loss2 = w2 * crit2(output, target2)
loss2.backward()
print(loss1.item() + loss2.item())

Is there a conceptual difference and/or computational advantage in terms of speed/memory in any of the two?

Thanks in advance

[marchinidavide (Davide)]

In the DCGAN Tutorial the preferred methodology is the second when making the two forward passes with real and fake data in the discriminator … any friendly soul willing to comment and explain why this is the best method?

[gabiBettgenhaeser (Gabriele Bettgenhäuser)]

Did you find it out?

[aaks]

Hi, I was working with multi loss problem. I faced the following issue:

loss1 = criterion1() #something
loss2 = criterion2() #something
NetLoss = loss1 + loss2
running_loss1 = loss1.item()
running_loss2 = loss2.item()
net_loss += NetLoss.item()

However, for me,
net_loss > running_loss1+running_loss2

Can this be possible?