# Clearing the GPU is a headache

**URL:** <https://discuss.pytorch.org/t/clearing-the-gpu-is-a-headache/84762>\
**Category:** vision\
**Created:** [June 9, 2020, 12:49pm UTC](https://discuss.pytorch.org/t/clearing-the-gpu-is-a-headache/84762 "2020-06-09T12:49:16Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![alx](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alx/32/17306_2.png) [@alx](https://discuss.pytorch.org/u/alx)\
**Post date:** [June 9, 2020, 12:49pm UTC](https://discuss.pytorch.org/t/clearing-the-gpu-is-a-headache/84762/1 "2020-06-09T12:49:16Z")

</div>

Hi all, before adding my model to the gpu I added the following code:

```auto
def empty_cached():
  gc.collect()
  torch.cuda.empty_cache()

```

The idea buying that it will clear out to GPU of the previous model I was playing with.

Here’s a scenario, I start training with a resnet18 and after a few epochs I notice the results are not that good so I interrupt training, change the model, run the function above.  
When I do it that way I get a:

```auto
RuntimeError: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 11.17 GiB total capacity; 10.56 GiB already allocated; 9.81 MiB free; 10.85 GiB reserved in total by PyTorch)

```

However, if I interupt training, restart the kernel and run the same model that wouldn’t work before, it now works.

It seems like nothing works as good as restarting the kernel. What would come the closest to it?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [June 10, 2020, 7:30am UTC](https://discuss.pytorch.org/t/clearing-the-gpu-is-a-headache/84762/2 "2020-06-10T07:30:27Z")

</div>

You would have to delete all references to the tensors you would like to free.  
`gc.collect()` and `empty_cache()` will not free tensors, which are still used and referenced.

---

<div class="post-metadata">

**Author:** ![alx](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alx/32/17306_2.png) [@alx](https://discuss.pytorch.org/u/alx)\
**Post date:** [June 10, 2020, 7:35pm UTC](https://discuss.pytorch.org/t/clearing-the-gpu-is-a-headache/84762/3 "2020-06-10T19:35:23Z")

</div>

I see!

How can I delete references to the tensors?

I tried with` del(model, optimizer, criterion)` but when I query the GPU memory before and after it seems like its the same.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [June 11, 2020, 4:35am UTC](https://discuss.pytorch.org/t/clearing-the-gpu-is-a-headache/84762/4 "2020-06-11T04:35:31Z")

</div>

`del tensor` should work. To release the cached memory, you would need to call `torch.cuda.empty_cache()` afterwards.

Here is a small example:

```python
print(torch.cuda.memory_allocated()/1024**2)
print(torch.cuda.memory_cached()/1024**2)

x = torch.randn(1024*1024).cuda()

# 4MB allocation and potentially larger cache
print(torch.cuda.memory_allocated()/1024**2)
print(torch.cuda.memory_cached()/1024**2)

y = torch.randn(8*1024*1024).cuda()

# 4+32=36MB allocation and potentially larger cache
print(torch.cuda.memory_allocated()/1024**2)
print(torch.cuda.memory_cached()/1024**2)

del x

# 32MB allocation, cache should stay the same
print(torch.cuda.memory_allocated()/1024**2)
print(torch.cuda.memory_cached()/1024**2)

torch.cuda.empty_cache()

# 32MB allocation and cache
print(torch.cuda.memory_allocated()/1024**2)
print(torch.cuda.memory_cached()/1024**2)

del y

# 0MB allocation, 32MB cache
print(torch.cuda.memory_allocated()/1024**2)
print(torch.cuda.memory_cached()/1024**2)

torch.cuda.empty_cache()

# 0MB allocation and cache
print(torch.cuda.memory_allocated()/1024**2)
print(torch.cuda.memory_cached()/1024**2)

```

---

<div class="post-metadata">

**Author:** ![alx](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alx/32/17306_2.png) [@alx](https://discuss.pytorch.org/u/alx)\
**Post date:** [June 11, 2020, 4:35pm UTC](https://discuss.pytorch.org/t/clearing-the-gpu-is-a-headache/84762/6 "2020-06-11T16:35:24Z")

</div>

Amazing, it works great!

I was using the `nvidia-smi` to determine the memory allocation on the GPU. What would be the difference between that and `torch.cuda.memory_allocated`.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [June 12, 2020, 6:00am UTC](https://discuss.pytorch.org/t/clearing-the-gpu-is-a-headache/84762/7 "2020-06-12T06:00:48Z")

</div>

`nvidia-smi` shows the all processes which use memory on the device.  
So besides PyTorch or course other processes might use memory.  
Also, note that PyTorch loads the CUDA kernels, cudnn, CUDA runtime etc. after the first CUDA operation, which will also allocate memory (and cannot be freed until the script exits).  
Depending on the device, CUDA version etc. this CUDA context might take ~700MB.

---

<div class="post-metadata">

**Author:** ![Saifeddine\_Barkia](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saifeddine_barkia/32/27005_2.png) [@Saifeddine\_Barkia](https://discuss.pytorch.org/u/Saifeddine_Barkia)\
**Post date:** [August 5, 2020, 8:00am UTC](https://discuss.pytorch.org/t/clearing-the-gpu-is-a-headache/84762/8 "2020-08-05T08:00:22Z")

</div>

@ptrblck sir, I have the same problem here, can you help me determine where I put the cuda. empty\_cache()?

> [@RuntimeError: CUDA out of memory?](https://discuss.pytorch.org/t/runtimeerror-cuda-out-of-memory/91385/4):
>
> Check the attention\_maps and raw\_features, as well. If they are not scalers, and they do not participate in the training (contribute to it, and you are just using them as pure outputs) considert doing a detach() on them as well.

---

<div class="post-metadata">

**Author:** ![ljj7975](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ljj7975/32/12561_2.png) [@ljj7975](https://discuss.pytorch.org/u/ljj7975)\
**Post date:** [July 8, 2021, 3:04pm UTC](https://discuss.pytorch.org/t/clearing-the-gpu-is-a-headache/84762/9 "2021-07-08T15:04:42Z")

</div>

is there a way to programmatically clear CUDA context?

I would like to release the memory allocated by CUDA context after model training because my process need to do other operations after that

> [@Model.to("cpu") does not release GPU memory allocated by registered buffer](https://discuss.pytorch.org/t/model-to-cpu-does-not-release-gpu-memory-allocated-by-registered-buffer/126102):
>
> I have figured that registered\_buffer does not release GPU memory when the model is moved back to CPU. Here is the minimal code for reproducing the observation import torch from torch import nn from subprocess import Popen, PIPE class TestNet(nn.Module): def \_\_init\_\_( self ): super().\_\_init\_\_() self.register\_buffer("test", torch.Tensor([1,2,3]).view(-1, 1, 1), False) def log\_nvidia\_smi(): print("--- nvidia-smi ---") for line in Popen('nvidia-smi', shel…

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [July 8, 2021, 6:45pm UTC](https://discuss.pytorch.org/t/clearing-the-gpu-is-a-headache/84762/10 "2021-07-08T18:45:17Z")

</div>

No, you cannot delete the CUDA context while the PyTorch process is still running and would have to shutdown the current process and use a new one for the downstream application.

---

<div class="post-metadata">

**Author:** ![wiiiktor](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wiiiktor/32/41325_2.png) [@wiiiktor](https://discuss.pytorch.org/u/wiiiktor)\
**Post date:** [August 29, 2021, 1:37pm UTC](https://discuss.pytorch.org/t/clearing-the-gpu-is-a-headache/84762/11 "2021-08-29T13:37:24Z")

</div>

Is there a conveninent method for deleting all the model tensors in one go?