BikashgG
(Bikash Gyawali)
July 28, 2017, 5:04pm
11
I did this but got the following error :
add received an invalid combination of arguments - got (torch.FloatTensor), but expected one of:
(float value)
didn’t match because some of the arguments have invalid types: (torch.FloatTensor)
(torch.cuda.FloatTensor other)
didn’t match because some of the arguments have invalid types: (torch.FloatTensor)
(torch.cuda.sparse.FloatTensor other)
didn’t match because some of the arguments have invalid types: (torch.FloatTensor)
(float value, torch.cuda.FloatTensor other)
(float value, torch.cuda.sparse.FloatTensor other)
Both my loss are of the same type - nn.CrossEntropyLoss() <class ‘torch.autograd.variable.Variable’>
Any suggestion on how to resolve this please?
tom
(Thomas V)
July 28, 2017, 6:08pm
12
If you stare down the error message, you see that you have one cuda and one not.
Best regards
Thomas
6 Likes
BikashgG
(Bikash Gyawali)
July 28, 2017, 6:10pm
13
Yes, thank you @tom . I am very novice here.
platero
(Wei Yang)
October 5, 2017, 10:33am
14
Is it possible to weight the losses when multiple losses are used and how? Is it correct by using the following code?
mse_loss = nn.MSELoss(size_average=True)
a = weight1 * mse_loss(inp, target1)
b = weight2 * mse_loss(inp, target2)
loss = a + b
loss.backward()
5 Likes
Yup, you can certainly weigh the losses however you see fit. What you’re doing in your example should be fine.
3 Likes
ElleryL
(EL)
May 21, 2018, 4:57am
16
what if my second loss function requires some computed value from first loss (or even the grad of first loss?) in that case I can’t add two loss together; they must be gradients respectively; and retain_graph=True gives wrong results as well as the intermediate grads not correct
see this one
Consider
I have Variable x
y = f(x)
z = Q(y) # Q here is a neural net
Step(1): gradient w.r.t. x.
z.backward(retain_graph = True)
x.grad
g = x.grad.clone()
x.grad.data.zero_()
Step(2): have another function that take the gradients we just compute
L(g)
I want to take gradient of it w.r.t. to the weights of neural net Q
as following
var_opt = torch.optim.Adam(Q.parameters(), lr=lr)
while not converge:
var_opt.zero_grad()
variance_loss = torch.mean(L(g))
variance_los…
1 Like
thank you very much Jordan! your answer help me to combine my custom loss function with nn.Loss.
Hi BikashgG, I think you could use .type() to change tensor’s type from torch.FloatTensor to torch.LongTensor.
I face a similar problem when I use this CrossEntropyLoss().
Check the official document on this loss function, there is a requirement on the type of feed-in tensor.
Hope it helps,
Peter
tarun
(Tarun Kalluri)
August 8, 2018, 7:51am
19
What if the losses are computed over different parts of the network, say loss1 is for first 3 layers and loss2 is for first 7 layers (incl. the first 3)? Wouldn’t the sum of losses method also backprop loss1 through layers 4-7? Would calling loss1.backward() and loss2.backward() separately be recommended in that case?
2 Likes
Gabrer
October 25, 2018, 6:57pm
21
mse_loss = nn.MSELoss(size_average=True)
a = weight1 * mse_loss(inp, target1)
b = weight2 * mse_loss(inp, target2)
loss = a + b
loss.backward()
What if I want to learn the weight1 and weight2 during the training process?
Should they be declared parameters of the two models? Or of a third one?
1 Like
To learn them would not make sense for me since your network will only learn to minimize the weights which will automatically minimize the partial losses and thus the total loss.
1 Like
The weight should be int
or tensor
? Or both works?
Int, since you’re weighing each type of loss instead of each value individually.
You could add another loss that penalizes the model for learning weight1 = weight2 = 0
Rojin
(Rojin Safavi)
June 19, 2019, 10:20pm
26
tarun:
What if the losses are computed over different parts of the network, say loss1 is for first 3 layers and loss2 is for first 7 layers (incl. the first 3)? Wouldn’t the sum of losses method also backprop loss1 through layers 4-7? Would calling loss1.backward() and loss2.backward() separately be recommended in that case?
Did you learn how to do this?
What is the preferred way to combine loss functions between:
output = net(input)
loss1 = w1 * crit1(output, target1)
loss2 = w2 * crit2(output, target2)
loss = loss1 + loss2
loss.backward()
print(loss.item())
and
output = net(input)
loss1 = w1 * crit1(output, target1)
loss1.backward()
loss2 = w2 * crit2(output, target2)
loss2.backward()
print(loss1.item() + loss2.item())
Is there a conceptual difference and/or computational advantage in terms of speed/memory in any of the two?
Thanks in advance
2 Likes
In the DCGAN Tutorial the preferred methodology is the second when making the two forward passes with real and fake data in the discriminator … any friendly soul willing to comment and explain why this is the best method?
3 Likes
aaks
April 1, 2020, 9:04pm
30
Hi, I was working with multi loss problem. I faced the following issue:
loss1 = criterion1() #something
loss2 = criterion2() #something
NetLoss = loss1 + loss2
running_loss1 = loss1.item()
running_loss2 = loss2.item()
net_loss += NetLoss.item()
However, for me,
net_loss > running_loss1+running_loss2
Can this be possible?