Two Loss Functions with weight factor

autograd
1

Good evening,

I’m using at the moment for my approach two loss functions and weight them with a factor:

loss_1= loss_1_function(output, args)
loss_2= loss_2_function(output, args)



overall_loss_tensorboard = args.alpha_loss*loss_1.item() + (1-args.alpha_loss)*loss_2.item()
(args.alpha_loss*loss_1+ (1-args.alpha_loss)*loss_2).backward()

Is this the right way, how one would weight two losses and combine them?

Thanks and best regards
Jonas

2

Hi Jonas, I am also in the process of tuning multiple loss functions. Your approach seems somehow manual tuning. I suggest you to find a coarse optimization for alpha. After, I suggest you to start from this article https://towardsdatascience.com/self-paced-multitask-learning-76c26e9532d0 and try some automatic tuning method.

Best,
Stefano

3

Hi Stefano,

Yep that’s right and the worst thing is that this alpha factor is just one of 5 parameters… So thanks a lot for the link! I also wanted to look into the field of Bayesian Hyperparameter Optimization.

However just to be sure, the approach how I factor the two losses is correct or?

Thanks and best regrads
Jonas

4

The way you are weighing the two losses is correct.

1 Like