How to update the weight by an old grad?

Hi!:blush: I am new in PyTorch. I want to update the weight by a old grad:
Here is a pseudocode example:

output = model(input)
loss1 = criterion1(output, target)
loss2 = criterion2(output, target)
grad1 = loss1.compute_grad()
grad2 = loss2.compute_grad()
... # other operation will be done here.

The key point is that the grad2 is calculated w.r.t the old weight, before the parameters.update(grad1).
I have no ideas to do this in an efficient way. Can create_graph=True or other method in Pytorch can help me? An easy example is better. Thank you! :blush:

It’s probably not the most efficient way, but you could clone all .grad attributes, zero them out in the model, perform your custom operations, copy them back into the model parameters, and call optimizer.step() when you are done.

1 Like

Thank you! It helps!