# CUDA memory leak while training

**URL:** <https://discuss.pytorch.org/t/cuda-memory-leak-while-training/82855>\
**Category:** Uncategorized\
**Created:** [May 25, 2020, 7:18pm UTC](https://discuss.pytorch.org/t/cuda-memory-leak-while-training/82855 "2020-05-25T19:18:42Z")\
**Posts on this page:** 1\
**Showing post:** 5

<div class="post-metadata">

**Author:** ![Martynas\_Jurkus](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/martynas_jurkus/32/24756_2.png) [@Martynas\_Jurkus](https://discuss.pytorch.org/u/Martynas_Jurkus)\
**Post date:** [May 27, 2020, 6:39am UTC](https://discuss.pytorch.org/t/cuda-memory-leak-while-training/82855/5 "2020-05-27T06:39:54Z")

</div>

Here’s the memory usage without `torch.cuda.empty_cache()`

 ![gpu_0 (2)](https://discuss.pytorch.org/uploads/default/original/3X/4/e/4ee07cf849bc9ac097ef412d803bfba85e0ab63f.jpeg)

It doesn’t say much.  
I also set up memory profiling found in this topic [How to debug causes of GPU memory leaks? - #18 by Jean\_Da\_Rolt](https://discuss.pytorch.org/t/how-to-debug-causes-of-gpu-memory-leaks/6741/18)

Memory usage fluctuates a bit but stays around 12800Mb after step ~220

What I noticed is that it ALWAYS crashes on step 507/3957. Not sure if that indicates anything at all.

> RuntimeError: CUDA out of memory. Tried to allocate 1.14 GiB (GPU 0; 14.76 GiB total capacity; 12.09 GiB already allocated; 483.44 MiB free; 13.48 GiB reserved in total by PyTorch)

I also tried to remove comet logger and all logging, but result was the same.  
I could try to remove PyTorch Lightning and do training manually, I don’t think that it would help.

---

_[View the full topic](https://discuss.pytorch.org/t/cuda-memory-leak-while-training/82855)._