Change gradient values before optimizer.backward()

autograd
1

Hey,

What i’m trying to do is after loss.backward() i want to assign the gradients to another variable then update these in another function then replace the grads in the model.

loss.backward()
new_gradient = change_grad_values_function(old_gradient_values)
model.gradients = new_gradients
optimizer.step()

How do i go about doing this??

2

you can modify the gradients via

for name,param in model.named_parameters():
    gradient = param.grad
    new_gradient = func(gradient)
    param.grad = new_gradient
optimizer.step()
1 Like
3

Thanks for the answer, just a follow up ques, suppose i want to add gradients of one model to another before calling optimizer.step() how would i do this??

4

Well, you can manipulate those tensors in any way you want. Therefore, you can sum anything you want.

If you are asking what is the proper way of doing so (including a theoretical PoV), that’s a really hard question which depends on maaany factors and bit out of scope :slight_smile:

But for example, if you have a model which works with rgb images and another identical model which works with depth images, then the gradients will be exactly the same shape and you can sum them (I’m not addressing whether this makes sense or not).