Modifing gradients between backeard and optimization step procedure

[hamedB (Hamed Behzadi)]

Hello

I have seen a couple of previous topics regarding observing or modifying the gradients of non-leaf tensors. some of them are for the year 2018 or 2019 which suggest function .grad. However this function returns None. And through the autograd doc. I have seen much explanation about the required_grad function.
My question is that is it possible to see the values of the gradients of the tensor in computational graph and is it possible to modify the gradients after loss.backward() and before optimizer.step() ?

Thanks.

[anantguptadbl (Anant Gupta)]

@hamedB you need to register hooks on the intermediate variables to get the grad

How to calculate / get intermediate results of backpropagation

I’ve found this one example which illustrates my problem: import torch from torch.autograd import grad import torch.nn as nn # Create some dummy data. x = torch.ones(2, 2, requires_grad=True) gt = torch.ones_like(x) * 16 - 0.5 # "ground-truths" # We will use MSELoss as an example. loss_fn = nn.MSELoss() # Do some computations. v = x + 2 y = v ** 2 # Compute loss. loss = loss_fn(y, gt) print(f'Loss: {loss}') # Now compute gradients: d_loss_dx = grad(outputs=loss, inputs=x) d_loss_dv = g…

https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html

[hamedB (Hamed Behzadi)]

Thanks @anantguptadbl for your reply. I am familiar with the hook. But I am not sure how to apply the modifications on the actual tensor. Hook can records the values on an external list. However I want after computing the gradients by loss.backward(), add some values to each gradients computed and then call optimizer.step()
if you have any idea, I appreciate it.

Thanks