This would yield the same results, if you are using an optimizer without any running estimates.
E.g. this example returns the same updated values for SGD, but will fail for e.g. Adam:
torch.manual_seed(2809)
model = nn.Linear(10, 1, bias=False)
w0 = model.weight.clone()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
criterion = nn.MSELoss()
data = torch.randn(1, 10)
target = torch.randn(1, 1)
# 1)
output = model(data)
loss1 = criterion(output, target)
loss1.backward(retain_graph=True)
loss2 = criterion(output, target)
loss2.backward()
optimizer.step()
print(w0 - model.weight)
optimizer.zero_grad()
# 2)
torch.manual_seed(2809)
model = nn.Linear(10, 1, bias=False)
# make sure weight is equal
print((w0 == model.weight).all())
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
output = model(data)
loss1 = criterion(output, target)
loss1.backward(retain_graph=True)
optimizer.step()
optimizer.zero_grad()
loss2 = criterion(output, target)
loss2.backward()
optimizer.step()
optimizer.zero_grad()
print(w0 - model.weight)
Have a look at this post for some possible work flows.
What error are you getting while trying to install apex?