How combined loss impact on separated network

1

Assume that i have three separated classifiers connected to the same nodes (see the attached image). How will the .backward(), .step() impact the training in the following scenarios?

scenario 1:
loss_1.backward()
loss_1.step()
loss_2.backward()
loss_2.step()
loss_3.backward()
loss_3.step()

scenario 2:
loss_1.backward()
loss_2.backward()
loss_3.backward()
total_loss = loss_1 + loss_2 + loss_3
total_loss.step()

scenario 3:
total_loss = loss_1 + loss_2 + loss_3
total_loss.backward()
total_loss.step()

I’m wondering if I sum up the total loss from two separated classifiers, and backpropogate, will these separated classifiers only be influenced by their own part of the loss within the total_loss? Will pytorch automatically use each of these part within the total_loss, or just backpro based on the total_loss ?

Many thanks in advanced!

Screenshot_pytorch
Screenshot_pytorch346×313 5.06 KB

2

loss.step() is not a valid call since you would need to call optimizer.step(), which contains the parameters it will update.

The total_loss is a sum of the sub-losses and will thus create a new computation node in your graph. Calling total_loss.backward() will then backpropagate through each classifier computing the gradients of their used parameters w.r.t. its loss.

3

Thanks for the reply!