# Why do we need to set the gradients manually to zero in pytorch?

**URL:** <https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903>\
**Category:** Uncategorized\
**Created:** [July 14, 2017, 5:31pm UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903 "2017-07-14T17:31:53Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [July 14, 2017, 5:31pm UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/1 "2017-07-14T17:31:53Z")

</div>

Why do we need to set the gradients manually to zero in pytorch? e.g:

`w1.grad.data.zero_()`

why do we need that? What happens if we don’t use that?

It feels that needing to ask this question means there is something conceptual/fundamental about the design of pytorch that I dont understand.

---

<div class="post-metadata">

**Author:** ![ruotianluo](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ruotianluo/32/383_2.png) [@ruotianluo](https://discuss.pytorch.org/u/ruotianluo)\
**Post date:** [July 14, 2017, 5:50pm UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/2 "2017-07-14T17:50:44Z")

</div>

Every time a variable is back propogated through, the gradient will be accumulated instead of being replaced. (This makes it easier for rnn, because each module will be back propogated through several times.)

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [July 14, 2017, 5:56pm UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/3 "2017-07-14T17:56:29Z")

</div>

@ruotianluo I still don’t understand. Though, is there a link I can just read to understand this?

Is there no link to understand how pytorch works and so I can form a mental model of it?

Like something like this seems very strange to someone coming from tensorflow.

---

<div class="post-metadata">

**Author:** ![samarth-robo](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samarth-robo/32/3487_2.png) [@samarth-robo](https://discuss.pytorch.org/u/samarth-robo)\
**Post date:** [July 14, 2017, 6:05pm UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/4 "2017-07-14T18:05:25Z")

</div>

Since the `backward()` function accumulates gradients, and you don’t want to mix up gradients between minibatches, you have to zero them out at the start of a new minibatch. This is exactly like how a general (additive) accumulator variable is initialized to 0 in code.

By the way, the best practice is to use the [zero\_grad()](http://pytorch.org/docs/master/optim.html#torch.optim.Optimizer.zero_grad) function on the optimizer.

---

<div class="post-metadata">

**Author:** ![tom](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/32/3162_2.png) [@tom](https://discuss.pytorch.org/u/tom)\
**Post date:** [July 14, 2017, 6:16pm UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/5 "2017-07-14T18:16:21Z")

</div>

A more explicit example in a similar direction as @ruotianluo is the ability to add gradients from several forward passes, for example in GANs:

> [@How to use the backward functions for multiple losses?](https://discuss.pytorch.org/t/how-to-use-the-backward-functions-for-multiple-losses/1826/5):
>
> Hello @Nabarun_Goswami, to try to clear this up, in the DCGAN example, you have (think about mathematical functions here, I left out everything not relevant). loss = criterion(netD(real, params))+criterion(netD(fake, params)) Spelling out the chain rule for the gradient of the loss w.r.t. the params: ∇params loss = ∇params netD(real, params)\* ∇netD loss(netD (real,params)) + ∇params netD(fake, params)\* ∇netD loss(netD (fake,params)), note how ∇params netD is evaluated at two different poin…

If you wanted, you could also achieve minibatches that are larger than fit in your memory by combining several sub-minibatches into one gradient step, but I have not really seen that done.

Best regards

Thomas

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [July 14, 2017, 8:14pm UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/6 "2017-07-14T20:14:37Z")

</div>

I think I don’t even know what “accumulating gradients” even means though. So I’m not sure what we are even talking about.

---

<div class="post-metadata">

**Author:** ![hughperkins](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hughperkins/32/915_2.png) [@hughperkins](https://discuss.pytorch.org/u/hughperkins)\
**Post date:** [July 15, 2017, 12:17am UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/7 "2017-07-15T00:17:38Z")

</div>

You mean like, why doesnt `backward` just zero out the gradients before doing the back propagation? I somewhat agree on this point actually… I would have to really struggle to think of a time when I’ve called `backward` without first zeroing out the gradients.

---

<div class="post-metadata">

**Author:** ![hughperkins](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hughperkins/32/915_2.png) [@hughperkins](https://discuss.pytorch.org/u/hughperkins)\
**Post date:** [July 15, 2017, 12:19am UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/8 "2017-07-15T00:19:02Z")

</div>

(like, it seems like we could have an option in backward to not zero out the gradients, like `backward(preserve_grads=True)`, but by default, seems like zeroing out the gradients could be the default action)

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [July 16, 2017, 10:12am UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/9 "2017-07-16T10:12:31Z")

</div>

Hi,

I think the big difference with tensorflow is the following.  
Since you use a static graph, you define exactly what should be done to make one gradient computation/update. And then you just tell it to do it using a given input/target.

In pytorch, it is significantly more flexible as the autograd engine will just “remember” how to compute the gradient for a given variable while you are performing computations with this Variable. This means that you can get the gradients wrt a variable, then perform computation with it again, then recompute gradients corresponding to these new operations.  
In this scheme, there is a not a single point where you stop performing “forward” operations and you know that the only thing that is left to be done is compute the gradients. So it is trickier to automatically set the gradients to 0 because you don’t know when a computation end, and when a new starts.

An example where the gradient accumulation is useful is for example if you share some part of a network for two different tasks:

```auto
input = Variable(data)
# Get the features
features = feature_extractor(input)

# Compute first loss and get the gradients for it
loss1 = task1(features)
loss1.backward(retain_graph=True)
# This add the gradients wrt loss1 in both the "task1" net and the "feature_extractor" net
# So each parameter "w" in "feature_extractor" has it gradient d(loss1)/dw

# Perform the second task and get the gradients for it as well
loss2 = task2(features)
loss2.backward()
# This will add gradients in "task2" and accumulate in "feature_extractor"
# Now each parameter in "feature_extractor" contains d(loss1)/dw + d(loss2)/dw

```

So the fact that the gradients are accumulated allows you to get the correct gradient for all the computations that you do with a given Variable even if you use it at multiple places in convoluted ways.  
The drawback here is that you have to manually reset the values to 0 so that the gradients computed previously do not interfere with the ones you are currently computing.

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [July 18, 2017, 6:12pm UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/10 "2017-07-18T18:12:06Z")

</div>

thats interesting. I would have thought that the “sharing” would have been multiplicative since RNNs a composition of functions and not a mere addition (as in ur transfer learning example). I think I am more confused. ☹

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [July 21, 2017, 8:18am UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/11 "2017-07-21T08:18:52Z")

</div>

The addition comes from the rules of differentiation. Iif `f = f1 + f2`, then the gradients for a parameter in both branches is the sum of the contributions of each branch.

---

<div class="post-metadata">

**Author:** ![Jiawei\_Zhuang](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/b77776/32.png) [@Jiawei\_Zhuang](https://discuss.pytorch.org/u/Jiawei_Zhuang)\
**Post date:** [October 10, 2017, 1:41am UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/12 "2017-10-10T01:41:43Z")

</div>

`y.backward()` doesn’t just assign the value of y’(x) to `x.grad` (say `y` depends on `x`). It actually adds y’(x) to the current value of `x.grad` (think it as `x.grad += true_gradient`).

In the following example, `y.backward()` is called 5 times, so the final value of `x.grad` will be 5\*cos(0)=5.

```
import torch
from torch.autograd import Variable

x = Variable(torch.Tensor([[0]]), requires_grad=True)

for t in range(5):
    y = x.sin() 
    y.backward()
    
print(x.grad) # shows 5

```

Calling `x.grad.data.zero_()` before `y.backward()` can make sure `x.grad` is exactly the same as current y’(x), not a sum of y’(x) in all previous iterations.

```
x = Variable(torch.Tensor([[0]]), requires_grad=True) 

for t in range(5):
    if x.grad is not None:
        x.grad.data.zero_()
    y = x.sin() 
    y.backward()

print(x.grad) # shows 1

```

I also got confused by this “zeroing gradient” when first learning pytorch. The doc of `torch.autograd.backward` does mention that

> This function accumulates gradients in the leaves - you might need to zero them before calling it.

But this is quite hard to find and pretty confusing for (say) tensorflow users.

Official tutorials like [60 Minute Blitz](http://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html#) or [PyTorch with Examples](http://pytorch.org/tutorials/beginner/pytorch_with_examples.html) both say nothing about why one needs to call `grad.data.zero_()` during training. I think it would be useful to explain this a little more in beginner-level tutorials. RNN is a good example for why accumulating gradient (instead of refreshing) is useful, but I guess new users wouldn’t even know that `backward()` is accumulating gradient 😅

---

<div class="post-metadata">

**Author:** ![jdhao](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jdhao/32/2558_2.png) [@jdhao](https://discuss.pytorch.org/u/jdhao)\
**Post date:** [November 6, 2017, 7:25am UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/13 "2017-11-06T07:25:11Z")

</div>

@tom, since it is possible to accumulate the loss of several minibatch and do one parameter update. For example I want to update the parameter every 64 minibatch, I have the following code

```auto
total_loss = Variable(torch.zeros(1), requires_grad=True)

for idx, (data, target) in train_loader:

    data, target = Variable(data), Variable(target)
    output = model(data)
    loss = criterion(output, target)
    total_loss = total_loss + loss

    if (idx+1)%64 == 0:
        total_loss = total_loss/(64*batchsize)
        total_loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        total_loss = Variable(torch.zeros(1), requires_grad=True)

```

Is the above code correct to achieve the desired effect?

---

<div class="post-metadata">

**Author:** ![tom](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/32/3162_2.png) [@tom](https://discuss.pytorch.org/u/tom)\
**Post date:** [November 6, 2017, 8:51am UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/14 "2017-11-06T08:51:10Z")

</div>

I’d do the backward on the (reweighted) loss in each run and not do total loss.

Best regards

Thomas

---

<div class="post-metadata">

**Author:** ![jdhao](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jdhao/32/2558_2.png) [@jdhao](https://discuss.pytorch.org/u/jdhao)\
**Post date:** [November 6, 2017, 1:25pm UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/15 "2017-11-06T13:25:33Z")

</div>

I find that doing the backward for each reweighted loss is much slower than accumulate the loss and then backward.

---

<div class="post-metadata">

**Author:** ![tom](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/32/3162_2.png) [@tom](https://discuss.pytorch.org/u/tom)\
**Post date:** [November 6, 2017, 4:23pm UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/16 "2017-11-06T16:23:42Z")

</div>

Thanks for sharing this insight. I would have thought that you get similar speed at much less memory with separate backward calls. But clearly, the experiment proves my modest intuition to be wrong.

Best regards

Thomas

---

<div class="post-metadata">

**Author:** ![jdhao](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jdhao/32/2558_2.png) [@jdhao](https://discuss.pytorch.org/u/jdhao)\
**Post date:** [November 12, 2017, 7:50am UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/17 "2017-11-12T07:50:25Z")

</div>

@albanD, I have some doubt about what the computation graph looks like in the case that we accumulate the loss manually.

Assume there are total 256 batches for the dataset. For the normal case (case1), for each batch in an epoch, a new computation graph is created and after the backward pass, the graph is freed. So 256 computation graphs are created and freed during one epoch.

In this case(case2), since we only do backward on every 64 batches. Does that mean only 4 graphs are created? The graph are created by composing 64 smaller graphs in case1 and the root node in the bigger graph is `total_loss`. The 64 smaller graph all have the same set of learnable parameters. If that is the case, the bigger graph will consume a lot of memory since it have 64 copies of the small graph.

Is that right? Do you have any ideas?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [November 13, 2017, 11:27am UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/18 "2017-11-13T11:27:27Z")

</div>

Hi,

Indeed, in one case, you will create 256 graphs that work with one input.  
In the second case, you will create only 4 graphs. but each of these 4 graphs is actually composed of 64 times the graph above and some `Add` operations at the end that sum the loss.

Indeed, in the second case you will use much more memory. Indeed, for the 64 iterations, you will create a single graph that just keep growing, and so you will use more and more memory.

---

<div class="post-metadata">

**Author:** ![jdhao](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jdhao/32/2558_2.png) [@jdhao](https://discuss.pytorch.org/u/jdhao)\
**Post date:** [November 13, 2017, 12:05pm UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/19 "2017-11-13T12:05:44Z")

</div>

So We have to make sure that batchsize is not too large, or we will run out of memory.

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [November 14, 2017, 11:17am UTC](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20 "2017-11-14T11:17:42Z")

</div>

Here are three equivalent code, with different runtime/memory comsumption.  
Assume that you want to run sgd with a batch size of 100.  
(I didn’t run the code below there might be some typos, sorry in advance)

1: single batch of 100 (least runtime, more memory)

```auto
# some code
# Initialize dataset with batch size 100
for input, target in dataset:
    pred = net(input)
    loss = crit(pred, target)
    # one graph is created here
    opt.zero_grad()
    loss.backward()
    # graph is cleared here
    opt.step()

```

2: multiple small batches of 10 (more runtime, least memory)

```auto
# some code
# Initialize dataset with batch size 10
opt.zero_grad()
for i, (input, target) in enumerate(dataset):
    pred = net(input)
    loss = crit(pred, target)
    # one graph is created here
    loss.backward()
    # graph is cleared here
    if (i+1)%10 == 0:
        # every 10 iterations of batches of size 10
        opt.step()
        opt.zero_grad()

```

3: accumulate loss for multiple batches (more runtime, more memory)

```auto
# some code
# Initialize dataset with batch size 10
loss = 0
for i, (input, target) in enumerate(dataset):
    pred = net(input)
    current_loss = crit(pred, target)
    # current graph is appended to existing graph
    loss = loss + current_loss
    if (i+1)%10 == 0:
        # every 10 iterations of batches of size 10
        opt.zero_grad()
        loss.backward()
        # huge graph is cleared here
        opt.step()

```

It should be clear that case 3 is not what you want.  
The choice between case 1 and 2 is a tradeoff between memory and speed so that depends on what you want to do.  
Note that if you can fit a batch size of 50 in your memory, you can do a variation of case 2 with batch size of 50 and update every 2 iterations.

[Next page](https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903.md?page=2)