# How can we release GPU memory cache?

**URL:** <https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530>\
**Category:** Uncategorized\
**Created:** [March 7, 2018, 10:10am UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530 "2018-03-07T10:10:36Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![lyakaap](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lyakaap/32/3070_2.png) [@lyakaap](https://discuss.pytorch.org/u/lyakaap)\
**Post date:** [March 7, 2018, 10:10am UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/1 "2018-03-07T10:10:36Z")

</div>

I would like to do a hyper-parameter search so I trained and evaluated with all of the combinations of parameters.  
But watching nvidia-smi memory-usage, I found that GPU-memory usage value slightly increased each after a hyper-parameter trial and after several times of trials, finally I got `out of memory error`. I think it is due to cuda memory caching in no longer use Tensor. I know `torch.cuda.empty_cache` but it needs do del valuable beforehand. In my case, I couldn’t locate memory consuming variable.  
What is the best way to release the GPU memory cache?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [March 7, 2018, 10:17am UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/2 "2018-03-07T10:17:56Z")

</div>

Hi,

`torch.cuda.empty_cache()` (EDITED: fixed function name) will release all the GPU memory cache that can be freed.  
If after calling it, you still have some memory that is used, that means that you have a python variable (either torch Tensor or torch Variable) that reference it, and so it cannot be safely released as you can still access it.

You should make sure that you are not holding onto some objects in your code that just grow bigger and bigger with each loop in your search.

---

<div class="post-metadata">

**Author:** ![lyakaap](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lyakaap/32/3070_2.png) [@lyakaap](https://discuss.pytorch.org/u/lyakaap)\
**Post date:** [March 7, 2018, 10:40am UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/3 "2018-03-07T10:40:56Z")

</div>

Thanks, @albanD !  
So, the variables no longer referenced will be freed by using `torch.cuda.empty_cache()` right?

And I started to locate the memory consuming objects, but I couldn’t locate what variables gave me a bad effect.  
As for variables related to cuda, I use same variable name (e.g. model, criterion) in different trials.

Anyway, I suspected below evaluation loop (sorry, it’s version 0.4, maybe version causes the problem?),

```auto
for i, (X, y) in tqdm(enumerate(val_loader), total=len(val_loader)):
    X = Variable(X.cuda())
    y = Variable(y.squeeze().cuda(non_blocking=False))

    with torch.no_grad():
        outputs = model(X)
        loss = criterion(outputs, y)

    prec1, prec5 = accuracy(outputs.data, target, top_k=(1, 5))
    losses.update(loss.data[0], X.size(0))
    top1.update(prec1[0], X.size(0))
    top5.update(prec5[0], X.size(0))

```

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [March 7, 2018, 11:25am UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/4 "2018-03-07T11:25:09Z")

</div>

So any variable that is no longer reference is freed in the sense that its memory can be used to create new tensors, but this memory is not released to the os (so will still look like it’s used using nvidia-smi).  
`empty_cache` forces the allocator that pytorch uses to release to the os any memory that it kept to allocate new tensors, so it will make a visible change while looking at nvidia-smi, but in reality, this memory was already available to allocate new tensors.

Your code look good, I would double check that things that you send to your logger are not `Variables` but just python numbers using `.item()` as necessary.

Also, if you’re using 0.4 (I assume current master), then you should remove `Variable` and `.data` from your code and replace `loss.data[0]` by `loss.item()` 😉

---

<div class="post-metadata">

**Author:** ![lyakaap](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lyakaap/32/3070_2.png) [@lyakaap](https://discuss.pytorch.org/u/lyakaap)\
**Post date:** [March 7, 2018, 11:38am UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/5 "2018-03-07T11:38:24Z")

</div>

Thank you for detailed answering! I tried other things such as `del model, loss, ...` but no one helps…  
Actually, I am using a little bit older version from the current master branch. But I have thought I should remove Variable and refactor code someday, so your instruction is so helpful, Thanks!

---

<div class="post-metadata">

**Author:** ![chuong\_nguyen](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/cdc98d/32.png) [@chuong\_nguyen](https://discuss.pytorch.org/u/chuong_nguyen)\
**Post date:** [July 13, 2018, 4:24am UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/6 "2018-07-13T04:24:41Z")

</div>

> [@albanD](#):
>
> torch.cuda.empty()

AttributeError: module ‘torch.cuda’ has no attribute ‘empty’

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [July 17, 2018, 9:37am UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/7 "2018-07-17T09:37:15Z")

</div>

Yes that was a typo I fixed it thanks.

---

<div class="post-metadata">

**Author:** ![vip](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/49beb7/32.png) [@vip](https://discuss.pytorch.org/u/vip)\
**Post date:** [March 5, 2019, 8:58pm UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/8 "2019-03-05T20:58:39Z")

</div>

The variables prec1[0] and prec5[0] still hold reference to tensors. These should be replaced with prec1[0].item() and prec5[0].item() respectively. This is due to the fact that the accuracy method part of the pytorch imagenet training example code returns a tensor and can cause memory leak.

> [@lyakaap](#):
>
> top1.update(prec1[0], X.size(0))  
> top5.update(prec5[0], X.size(0))

@albanD Could you please clarify the difference between detach() and item() when called on a tensor. Are these effectively the same?

---

<div class="post-metadata">

**Author:** ![Weifeng](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/weifeng/32/7899_2.png) [@Weifeng](https://discuss.pytorch.org/u/Weifeng)\
**Post date:** [March 6, 2019, 2:25am UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/9 "2019-03-06T02:25:53Z")

</div>

`detach()` return a tensor that shares storage (and the same device) with origin tensor, while `item()` return a converted python object.

---

<div class="post-metadata">

**Author:** ![yxchng](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/7bcc69/32.png) [@yxchng](https://discuss.pytorch.org/u/yxchng)\
**Post date:** [March 27, 2019, 12:07pm UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/10 "2019-03-27T12:07:25Z")

</div>

I also face memory problem and using this command works.

Why torch doesn’t empty cache automatically though? Does emptying cache have much overhead?

---

<div class="post-metadata">

**Author:** ![Ran](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ran/32/7989_2.png) [@Ran](https://discuss.pytorch.org/u/Ran)\
**Post date:** [May 24, 2019, 3:30am UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/11 "2019-05-24T03:30:09Z")

</div>

I have a question that is it safety to use torch.cuda.empty\_cache() before each iteration during training?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [May 24, 2019, 11:59am UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/12 "2019-05-24T11:59:49Z")

</div>

This should be safe, but might negatively affect the performance, since PyTorch might need to reallocate this memory again.  
What is your use case that you would like to call it for each iteration?

---

<div class="post-metadata">

**Author:** ![Ran](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ran/32/7989_2.png) [@Ran](https://discuss.pytorch.org/u/Ran)\
**Post date:** [May 26, 2019, 2:34pm UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/13 "2019-05-26T14:34:13Z")

</div>

What is the negative effect? My code runs more and more slowly and sometimes it may interrupt due to memory error, so I have thought to use it before each iteration. I don’t know does it have an effect on the speed?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [May 27, 2019, 5:08am UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/14 "2019-05-27T05:08:30Z")

</div>

If you see increasing memory usage, you might accidentally store some tensors with the an attached computation graph. E.g. if you store the `loss` for printing or debugging purposes, you should save `loss.item()` instead.

This issue won’t be solved, if you clear the cache repeatedly. As I said this might just trigger unnecessary allocations which will take some time thus potentially slowing down your code.

---

<div class="post-metadata">

**Author:** ![Ran](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ran/32/7989_2.png) [@Ran](https://discuss.pytorch.org/u/Ran)\
**Post date:** [May 27, 2019, 6:15am UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/15 "2019-05-27T06:15:48Z")

</div>

I see.  
Thanks very much.

---

<div class="post-metadata">

**Author:** ![mortonjt](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mortonjt/32/17120_2.png) [@mortonjt](https://discuss.pytorch.org/u/mortonjt)\
**Post date:** [November 5, 2019, 7:42pm UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/16 "2019-11-05T19:42:39Z")

</div>

Just to clarify, `item()` deallocates `loss`? Its not clear to me what exactly `item` is doing.

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [November 5, 2019, 8:13pm UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/17 "2019-11-05T20:13:51Z")

</div>

`.item()` converts a Tensor containing a single element into a python number.  
It does not “deallocate” loss. But it won’t keep it alive.

---

<div class="post-metadata">

**Author:** ![debvrat](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/debvrat/32/23126_2.png) [@debvrat](https://discuss.pytorch.org/u/debvrat)\
**Post date:** [April 29, 2020, 3:24am UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/18 "2020-04-29T03:24:52Z")

</div>

This works only some of the times.

Even when I clear out all the variables, restart the kernel, and execute `torch.cuda.empty_cache()` as the first line in my code, I still get a ‘CUDA out of memory’ error.

---

<div class="post-metadata">

**Author:** ![dimimal](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dimimal/32/31128_2.png) [@dimimal](https://discuss.pytorch.org/u/dimimal)\
**Post date:** [April 29, 2020, 4:08pm UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/19 "2020-04-29T16:08:53Z")

</div>

I have the same issue as well. Has anyone found why does this happen?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [April 29, 2020, 4:34pm UTC](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/20 "2020-04-29T16:34:43Z")

</div>

Hi,

Running empty\_cache at the beginning of your process is not useful as nothing is allocated yet.  
When you restart the kernel, you force all memory to be deallocated.  
So if you still run out of memory it is simply because your program requires more than what you have. You will most likely have to reduce the batch size or the size of your model.

[Next page](https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530.md?page=2)