In fact, you should zero grad both, model and tensor. Realize that you are doing:
while(loss > threshold):
output = model(input)
loss = loss_fn(output)
optimizer.zero_grad()
loss.backward()
optimizer.step()
That way gradients are being accumulated on model. As model isn’t wrapped by the optimizer, its gradients are never zeroed. (In pytorch if you don’t make gradients zero they accumulate to the previous ones)