# CUDA out of memory. No solution works **URL:** \ **Category:** Uncategorized\ **Created:** [December 7, 2021, 2:53pm UTC](https://discuss.pytorch.org/t/cuda-out-of-memory-no-solution-works/138767 "2021-12-07T14:53:01Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![msuliman](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/f9ae1b/32.png) [@msuliman](https://discuss.pytorch.org/u/msuliman)\ **Post date:** [December 7, 2021, 2:53pm UTC](https://discuss.pytorch.org/t/cuda-out-of-memory-no-solution-works/138767/1 "2021-12-07T14:53:01Z")
I am sure that this topic might have been discussed many times, but none of the proposed solutions work for me. I was running my model, and then I had to interrupt the execution to do some modifications and rerun it. However, when I do so, I get the following error message: RuntimeError: CUDA out of memory. Tried to allocate 2.67 GiB (GPU 0; 23.65 GiB total capacity; 6.30 GiB already allocated; 363.19 MiB free; 6.46 GiB reserved in total by PyTorch). My model is small, so I am sure this is not a problem of fitting a large model into a smaller GPU memory. In fact, it was running before I interrupted its execution I have tried all of the following, but none of them are working: 1- Restarting the kernal. 2- using torch.cuda.empty\_cache(). before/after restarting the kernal. 3- Cheking the allocated meoery by: print(torch.cuda.memory\_allocated()) and getting that it is zero. 4 - The “nvidia-smi” shows that 67% of the GPU memory is allocated, but doesn’t show what allocates it. In fact, when you sum the individual GPU memory usage of the processes, they don’t add up to that total memory usage. The individuals processes use 1% of the total GPU memory. The only thing that will work is if I restart my PC. But even after that, if I run the model and then terminate it, the out-of-memory issue will appear again. Edit (1): Below is my GPU status ![Screenshot 2021-12-07 at 3.05.08 pm](https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf04cd0ae4f4febb81310ca0a77c4fb61799bd06.png) Edit (2): Found this discussion [PyTorch doesn't free GPU's memory of it gets aborted due to out-of-memory error](https://discuss.pytorch.org/t/pytorch-doesnt-free-gpus-memory-of-it-gets-aborted-due-to-out-of-memory-error/13775) by @my3bikaht and used killall python through the terminal which solved my issue. But I still find it weird as I am not using multithreading ( and don’t have child processes) in my model. In fact, I was terminating exactly the same code and restarting it normally before, and then suddenly this issue started to happen. ---
**Author:** ![my3bikaht](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/50afbb/32.png) [@my3bikaht](https://discuss.pytorch.org/u/my3bikaht)\ **Post date:** [December 7, 2021, 3:35pm UTC](https://discuss.pytorch.org/t/cuda-out-of-memory-no-solution-works/138767/2 "2021-12-07T15:35:53Z")
I guess you were looking for this topic: > [@PyTorch doesn't free GPU's memory of it gets aborted due to out-of-memory error](https://discuss.pytorch.org/t/pytorch-doesnt-free-gpus-memory-of-it-gets-aborted-due-to-out-of-memory-error/13775): > > I noticed that 99% of the GPU RAM is still being used after and no process is listed by nvidia-smi after PyTorch aborts with an RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch\_1518244421288/work/torch/lib/THC/generic/THCStorage.see you:58 One could run sudo nvidia-smi --gpu-reset -i 0 as a remedy, however, it would be nice if PyTorch could free the memory after such a “crash” – not sure if this is only a problem in my particular case, so I wanted to ask if… ---
**Author:** ![msuliman](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/f9ae1b/32.png) [@msuliman](https://discuss.pytorch.org/u/msuliman)\ **Post date:** [December 7, 2021, 3:56pm UTC](https://discuss.pytorch.org/t/cuda-out-of-memory-no-solution-works/138767/3 "2021-12-07T15:56:42Z")
Thanks for the reply. I was going through that topic, and, killall python solved the issue But I still find it weird as I am not using multithreading (I don’t have child processes) in my model. In fact, I was terminating exactly the same code and restarting it normally before, and then suddenly this issue started to happen ---
**Author:** ![Raymond\_Cool](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raymond_cool/32/71976_2.png) [@Raymond\_Cool](https://discuss.pytorch.org/u/Raymond_Cool)\ **Post date:** [August 7, 2024, 11:29am UTC](https://discuss.pytorch.org/t/cuda-out-of-memory-no-solution-works/138767/4 "2024-08-07T11:29:31Z")
And here it is 2024 and I’m seeing exactly this - what’s going on? Seems like torch is problematic. I’m open to trying fixes, but this just started happening out of nowhere. And none of the boilerplate stuff makes any difference.