How can i introduce weights to my final loss

1

hello,
how can i introduce weights to my final loss and how can i learn the best weight?

def criterion(triplet_y, softmax_y, labels):
    losses = [embedding_criterion(output, labels)[0] for output in triplet_y] + \
                 [xent_criterion(output, labels) for output in softmax_y] 
    loss = sum(losses)
    return loss
2

If you use trainable weights on your last loss, the training would push them to a negative value, as itā€™s yielding the minimal loss, wouldnā€™t it?

Anyway, you could define a new parameter as:

weight = nn.Parameter(torch.tensor(1))

and push it to the optimizer with all other parameters:

optimizer = torch.optim.SGD(list(model.parameters()) + [weight], lr=1e-3)
3

maybe i did not explain well what i want to do
im mean that i wanna have a weight for every loss like this :

 losses = W1 *  [embedding_criterion(output, labels)[0] for output in triplet_y] + \
                W2 *  [xent_criterion(output, labels) for output in softmax_y]

but this is not working, it generates an error

4

What kind of error is it generating?

5

i did

0.2 *  [xent_criterion(output, labels) for output in softmax_y] + \
[xent_criterion(output, labels) for output in softmax_y]

the error is
[xent_criterion(output, labels) for output in softmax_y] +
TypeError: canā€™t multiply sequence by non-int of type ā€˜floatā€™

but i think that the problem is solved, i must do[ 0.2 * xent_criterion(output, labels) for output in softmax_y] + \

but still dont know how to learn the best valur of the weight, i just take 0.2 randomly

6

Ah OK, yes the error is thrown as you cannot multiply a scalar with a list directly.

Iā€™m still unsure, if ā€œlearningā€ the weight would work, since it would just push it to a negative value.
However, if you want to use the weight as a trainable parameter, refer to my code snippet from my previous post.

1 Like
7

thank you for your help