How to calculate gradient of a component of the loss

Hi,

I have a loss function with two parts: loss = loss1+loss2 and backpropagate.

I am looking for a way to calculate the gradient of each part with respect to the model parameter (weight &bias). How should I handle this?

Currently, I can only get the gradient from the summation loss:

loss.backward()
print(self.Network[numlayer].weight.grad)

How do print out the for example gradient of loss1 wrt weight?

Thank you!

Bests,

I think the naive solution will be the following :
First, you don’t sum the losses, but instead do this

optimizer.zero_grad()
loss1.backward()
### do something with these gradients, like
## print(self.Network[numlayer].weight.grad)
optimizer.zero_grad()
loss2.backward()
## do something with these gradients
## ...

Since we do backpropagation 2 times in the same step, it can slow down the step, but I’m not sure about that since we compute gradients separately, like, in out case d(loss)/dW = d(loss_1 + loss_2)/dW = d(loss_1)/dW + d(loss_2)/dW => autograd engine will compute these gradients separately too and the only overhead we’ll get is optimizer.zero_grad()?

Though, I’m not sure how to handle the case when you need to have access to both gradients at once
: thinking :

Thank you for your advice! But I am wondering if it is correct to have optimizer.zero_grad() in between two backward(). Would zero_grad() clear out all the grad of loss1 and then optimizer.step() only update the gradient from loss2?

Yes, sure if you want to update weights using gradients from the backward pass of loss1 you should put optimizer.step() after loss1.backward()

But again, I’m not sure about influence on speed of this approach, you can try to measure it by comparing these snippets

import time
start_time = time.time()
loss = loss1 + loss2

optimizer.zero_grad()
loss.backward() 
optimizer.step()

end_time = time.time() - start_time
print("%s seconds" % (end_time))

and this one

import time
start_time = time.time()

optimizer.zero_grad()
loss1.backward() 
optimizer.step()

optimizer.zero_grad()
loss2.backward()
optimizer.step()

end_time = time.time() - start_time
print("%s seconds" % (end_time))