Another question about the sum of two losses.
What if they are number of different scales, like on is 10^(-5) and the other is 0.5 ? How could I choose smartly the weights ?
My problem is that depending on the weights my model produces Nan or succeed to run with a fixed handly found ratio between the losses (in my case 0.9 and 0.1), any suggestions @ptrblck?
Thank you in advance.
Your approach of a “fixed” scaling sounds valid. You could alternatively try to dynamically scale the losses so that they would have approx. the same magnitude, but I would assume that you would have to remove this scaling at one point once the losses decrease (otherwise you would artificially blow them up throughout the training).
You may want to take a look at this paper: https://openreview.net/pdf?id=HyxY6JHKwr . The idea of learning different loss weights reminded me of their work.
I have a related question, where the model is a multitask model. I’m trying to conditionally sum the losses for each task if the labels for the task are present.
loss = None
if task1_labels is not None:
if loss is None:
loss = task1_loss
else:
loss += task1_loss
if task2_labels is not None:
if loss is None:
loss = task2_loss
else:
loss += task2_loss
if task3_labels is not None:
if loss is None:
loss = task3_loss
else:
loss += task3_loss
What I noticed though that in the case where all 3 tasks have labelled, the task1_loss gets updated as the other losses are summed (i.e. ie the first assignment makes task1_loss and loss references to the same tensor).
It’s not clear to me that this results in the correct graph?
That is because loss will always be assigned one of the losses; if loss is None: will always be triggered once.
So loss will always be a reference to the first taskX_loss it is assigned.
Maybe instead you could do:
loss = None
if task1_labels is not None:
if loss is None:
loss = task1_loss.clone()
else:
loss += task1_loss
if task2_labels is not None:
if loss is None:
loss = task2_loss.clone()
else:
loss += task2_loss
if task3_labels is not None:
if loss is None:
loss = task3_loss.clone()
else:
loss += task3_loss
i.e. clone each loss in the first conditional assignment. idk how good that is in regards to memory or gradient backprop though.
An alternative (better way?) could be to always sum the losses no matter what:
loss = 0
if task1_labels is not None:
loss += task1_loss
if task2_labels is not None:
loss += task2_loss
if task3_labels is not None:
loss += task3_loss
I wanted to test if loss is None afterwards - I’m using allennlp and have to either return the loss in the output dict or not depending on whether I’m training or testing. I could use your code but add a standard bool flag which gets true in each condition though.
What I’ve done for now is
loss = None
if task1_labels is not None:
if loss is None:
loss = 0.0
loss += task1_loss
if task2_labels is not None:
if loss is None:
loss = 0.0
loss += task2_loss
if task3_labels is not None:
if loss is None:
loss = 0.0
loss += task3_loss
My model is still getting ready (my dataloader needs optimising) so I’m interested to see if this works.
Are there instances where no taskX_labels are defined? Otherwise (if there is always one or more taskX_labels that are not None), then the two code snippets are equivalent.
At the moment no, but when it’s working there will be.
When you say equivalent do you mean my new example and yours, or my original and new example?
My last snippet and your last snippet.
I would not recommend separate backward() calls if there is any crossover in any of the weights covered by each loss term (i.e. if the loss terms are computed on any weights shared by both). You would be adjusting weights iteratively; the error that you backpropagate through the network with the first loss term L_1 will change your initial weight w_init to a new weight w_L1. If you then backpropagate with your second loss term, you will be changing w_L1 with respect to loss computed on w_init which may destabilize training.
If you backpropagate isolated areas of your network, calling backward() twice may not only be ok but necessary. By correcting weights once per training time point, we preserve the relationship between the two interacting components of the network without increasing loss with respect to each others performance, which should result in more stable loss curves for both loss terms. This is true as long as the two loss terms are computed using the weights for a given point in time during training; this is the same logic that is applied to adversarial networks like GANs. There are exceptions to this in some applications. Keep in mind adversarial training schemes are a bit of “black sheep” when it comes to this topic due to their unique stability issues.
To put it more simply, if part of the network’s error computed using L1 feeds into another sequential part of the network’s error computed using L2, call backward() once on each term so long as it is in the same point in time during training (e.g. same batch), with the exception of some specific applications that attempt to stabilize interactions between these two parts of the network. If they don’t interact with each other at all (i.e. if there is no communication between them at all), you can also make two separate backward() calls, but know that these are two independent networks and may as well be treated as independent training runs in the context of what we are doing 
Edits: Grammar and clarified concepts