Backprop in PyTorch?

[alwynmathew]

Do we need to zero_grad the network too? As net isn’t wrapped by the optimizer.

optimizer = torch.optim.Adam([x], lr=1)
net.eval()
.
.
optimizer.zero_grad()
net.zero_grad()
loss.backward()
optimizer.step()

according to below link:

Add one tensor from dataloader to optimiser

In fact, you should zero grad both, model and tensor. Realize that you are doing: while(loss > threshold): output = model(input) loss = loss_fn(output) optimizer.zero_grad() loss.backward() optimizer.step() That way gradients are being accumulated on model. As model isn’t wrapped by the optimizer, its gradients are never zeroed. (In pytorch if you don’t make gradients zero they accumulate to the previous ones)