I;m building recursive neural nets, so I’ll have a variable list of criterions. I want to sum them up and backpropagate error. All the errors are single float values and of same scale. So the total loss is the sum of individual losses To understand this, I crystallized this problem. It’s for anoth…

I would not recommend separate backward() calls if there is any crossover in any of the weights covered by each loss term (i.e. if the loss terms are computed on any weights shared by both). You would be adjusting weights iteratively; the error that you backpropagate through the network with the fir…

Yup, you can certainly weigh the losses however you see fit. What you’re doing in your example should be fine.

what if my second loss function requires some computed value from first loss (or even the grad of first loss?) in that case I can’t add two loss together; they must be gradients respectively; and retain_graph=True gives wrong results as well as the intermediate grads not correct see this one …

thank you very much Jordan! your answer help me to combine my custom loss function with nn.Loss.

Hi BikashgG, I think you could use .type() to change tensor’s type from torch.FloatTensor to torch.LongTensor. I face a similar problem when I use this CrossEntropyLoss(). Check the official document on this loss function, there is a requirement on the type of feed-in tensor. Hope it helps, Pete…

What if the losses are computed over different parts of the network, say loss1 is for first 3 layers and loss2 is for first 7 layers (incl. the first 3)? Wouldn’t the sum of losses method also backprop loss1 through layers 4-7? Would calling loss1.backward() and loss2.backward() separately be recomm…

[image] How to implement multiple loss function at diffent layer Does torch.autograd.back([l1,l2]) mean this two loss backward for separate node? such as softmaxloss updates fc2 and layers before fc2, custom_loss updates fc1 and layers before fc1? And if I use L = l1 + l…

mse_loss = nn.MSELoss(size_average=True) a = weight1 * mse_loss(inp, target1) b = weight2 * mse_loss(inp, target2) loss = a + b loss.backward() What if I want to learn the weight1 and weight2 during the training process? Should they be declared parameters of the two models? Or of a third one…

To learn them would not make sense for me since your network will only learn to minimize the weights which will automatically minimize the partial losses and thus the total loss.

[image] achaiah: . What you’re doing in your example should be fine. The weight should be int or tensor? Or both works?

Int, since you’re weighing each type of loss instead of each value individually.

How to combine multiple criterions to a loss function?

sid87 (Sid87) October 7, 2018, 8:25pm 20

Hope it helps!