I have a model with multiple layers/components, these layers have various losses associated with them. Right now I am summing these losses and then running loss.backwards(loss1 + loss2 + ...), but if my understanding is correct, this loss is applied equally to all components even if they should not be assigned certain losses. How can I make sure only certain parts of my model have certain losses applied to them?
.backward has an inputs argument where you can specify which parameters get gradients accumulated into.