Strange Gradients / Optimizers

Hello,

I have a model, for which I am doing a backward call twice. However (by mistake) I forgot to to do the optimizer step after the second backward call.
I would think that my second backward call should not affect anything. But I was hoping to make sure that this is the case?
The thing is that my results change quite a lot (compared to when I don’t do the second backward call). Is something going on with the optimizer (Adam)? which is setup as optimizer = optim.Adam(model.parameters(),lr)

pseudo code:

model.zero_grad()
opt = model(data)
loss = cross_entropy(targets, opt)
loss.backward(retain_variables=True)
optimizer.step()
loss.backward()

Thanks