What i’m trying to do is after loss.backward() i want to assign the gradients to another variable then update these in another function then replace the grads in the model.
new_gradient = change_grad_values_function(old_gradient_values)
model.gradients = new_gradients
How do i go about doing this??
you can modify the gradients via
for name,param in model.named_parameters():
gradient = param.grad
new_gradient = func(gradient)
param.grad = new_gradient
Thanks for the answer, just a follow up ques, suppose i want to add gradients of one model to another before calling optimizer.step() how would i do this??
Well, you can manipulate those tensors in any way you want. Therefore, you can sum anything you want.
If you are asking what is the proper way of doing so (including a theoretical PoV), that’s a really hard question which depends on maaany factors and bit out of scope
But for example, if you have a model which works with rgb images and another identical model which works with depth images, then the gradients will be exactly the same shape and you can sum them (I’m not addressing whether this makes sense or not).