Loss normalize(or unify units) when using multiple loss

vision
1

Hi guys,

If I want to use two combined losses, (L = loss1 + loss2)
and suppose the decimal unit between two losses are different,
for example, loss 1 is around 0.01~0.09 and loss2 is around 1~9,
Do I need to unify the units of two losses?
The loss is MSE.

Please give me any advice.

2

As a general rule of thumb, it is advisable to have the two losses on the same scale. Else, the model responsible for the higher loss will have all parameter updates thus rendering the other model almost close to useless. In your case, dividing loss2 by 100 should bring them to the same scale.

3

Thank you for your answer.

also, Is it quite related to weighted loss?

4

Loss weighting is often done to normalize the scales.