# How to accumulate gradients with Opacus

**URL:** <https://discuss.pytorch.org/t/how-to-accumulate-gradients-with-opacus/134916>\
**Category:** Opacus\
**Created:** [October 23, 2021, 4:38am UTC](https://discuss.pytorch.org/t/how-to-accumulate-gradients-with-opacus/134916 "2021-10-23T04:38:49Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![deponce](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/ee7513/32.png) [@deponce](https://discuss.pytorch.org/u/deponce)\
**Post date:** [October 23, 2021, 4:38am UTC](https://discuss.pytorch.org/t/how-to-accumulate-gradients-with-opacus/134916/1 "2021-10-23T04:38:49Z")

</div>

Hey everyone, I attempt to accumulate gradients in training to save GPU memory.

The training loop works quite well without privacy\_engine

```auto
opt.zero_grad()
for i, (input, target) in enumerate(dataset):
    pred = net(input)
    loss = crit(pred, target)
    # one graph is created here
    loss.backward()
    # graph is cleared here
    if (i+1)%10 == 0:
        # every 10 iterations of batches of size 10
        opt.step()
        opt.zero_grad()

```

However, when I attach the privacy\_engine to the optimizer, CUDA is out of memory.  
Do anyone know how to solve this problem? Thanks in advance.

---

<div class="post-metadata">

**Author:** ![ffuuugor](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ffuuugor/32/39394_2.png) [@ffuuugor](https://discuss.pytorch.org/u/ffuuugor)\
**Post date:** [November 2, 2021, 7:43pm UTC](https://discuss.pytorch.org/t/how-to-accumulate-gradients-with-opacus/134916/2 "2021-11-02T19:43:01Z")

</div>

Hi!  
It is expected that Opacus has a certain memory overhear. At the very least, we have to store per-sample gradients for all model parameters - that alone increases the memory required to store gradient by the factor of batch size.

In order to address that I suggest using `virtual_step()` method in the optimizer.  
It does gradient clipping and accumulation (thus saving memory), but doesn’t do the actual optimizer step.  
Your code would look smth like this:

```auto
opt.zero_grad()
for i, (input, target) in enumerate(dataset):
    pred = net(input)
    loss = crit(pred, target)
    # one graph is created here
    loss.backward()
    # graph is cleared here
    if (i+1)%10 == 0:
        # every 10 iterations of batches of size 10
        opt.step()
        opt.zero_grad()
    else:
        opt.virtual_step()

```

For more examples using virtual\_step see our Cifar10 tutorial: [opacus/cifar10.py at main · pytorch/opacus · GitHub](https://github.com/pytorch/opacus/blob/main/examples/cifar10.py)