How to manually set gradients for the entire network?

[kyy]

Hi, I am trying to manually set the gradients of the entire network during training. Specifically I am interested to do the following:

optimizer.zero_grad()
# set gradients for all components of the network
optimizer.step()

I know that for a single module we can use following:

How to set gradients manually and update weights using that?

1.Are you 100% sure that there is no difference in param after the step?, maybe you use an adaptive optimizer and the learning rate is almost zero? 2.Are you 100% sure that the optimizer is responsible for this param update? I wrote this code to test this: (I am using v0.4.0): import torch target = torch.randn(1, 10) input = torch.randn(1, 20) network = torch.nn.Linear(20, 10) optimizer = torch.optim.SGD(network.parameters(), lr=0.01) optimizer.zero_grad() loss = ((network(input)-target)**2)…

But it would be complicated to set each module of the entire network in this way since the network may contain dozens of different modules. I am wondering if there is a better way to do so? I am thinking of possibilities of using something similar to state_dict when copying weights for the entire network, just wonder if there are similar things for gradients?

Thank you!