How can i introduce weights to my final loss

hello,
how can i introduce weights to my final loss and how can i learn the best weight?

def criterion(triplet_y, softmax_y, labels):
    losses = [embedding_criterion(output, labels)[0] for output in triplet_y] + \
                 [xent_criterion(output, labels) for output in softmax_y] 
    loss = sum(losses)
    return loss

If you use trainable weights on your last loss, the training would push them to a negative value, as it’s yielding the minimal loss, wouldn’t it?

Anyway, you could define a new parameter as:

weight = nn.Parameter(torch.tensor(1))

and push it to the optimizer with all other parameters:

optimizer = torch.optim.SGD(list(model.parameters()) + [weight], lr=1e-3)

maybe i did not explain well what i want to do
im mean that i wanna have a weight for every loss like this :

 losses = W1 *  [embedding_criterion(output, labels)[0] for output in triplet_y] + \
                W2 *  [xent_criterion(output, labels) for output in softmax_y] 

but this is not working, it generates an error

What kind of error is it generating?

i did

0.2 *  [xent_criterion(output, labels) for output in softmax_y] + \
[xent_criterion(output, labels) for output in softmax_y] 

the error is
[xent_criterion(output, labels) for output in softmax_y] +
TypeError: can’t multiply sequence by non-int of type ‘float’

but i think that the problem is solved, i must do[ 0.2 * xent_criterion(output, labels) for output in softmax_y] + \

but still dont know how to learn the best valur of the weight, i just take 0.2 randomly

Ah OK, yes the error is thrown as you cannot multiply a scalar with a list directly.

I’m still unsure, if “learning” the weight would work, since it would just push it to a negative value.
However, if you want to use the weight as a trainable parameter, refer to my code snippet from my previous post.

1 Like

thank you for your help