My free memory is MUCH LARGER than allocating memory and I still get out of memory error. [RuntimeError: CUDA out of memory]

[C_J]

I have this weird issue,

RuntimeError: CUDA out of memory. Tried to allocate 3.62 GiB (GPU 0; 47.99 GiB total capacity; 24.39 GiB already allocated; 20.41 GiB free; 24.71 GiB reserved in total by PyTorch)

My free memory (20.41GiB) is LARGER than the memory that it tries to allocate (3.62GiB).
Actually it’s 16.79GiB larger than what it needs.

I tried to find similar case and found one,

Cuda Out of Memory, even when I have enough free [SOLVED]

EDIT: SOLVED - it was a number of workers problems, solved it by lowering them I am using a 24GB Titan RTX and I am using it for an image segmentation Unet with Pytorch, it is always throwing Cuda out of Memory at different batch sizes, plus I have more free memory than it states that I need, and by lowering batch sizes, it INCREASES the memory it tries to allocate which doesn’t make any sense. here is what I tried: Image size = 448, batch size = 8 “RuntimeError: CUDA error: out of memory” …

In the post they say “it was a number of workers problems, solved it by lowering them”
But I don’t think it applies to my case.

Can I get some help please?

Thanks for reading!

[JuanFMontesinos (Juan Montesinos)]

It’s not like it’s free. It’s reserved, which counts in practice as used. So you have 45 already compromised and want to allocate 4.

[C_J]

Thanks for reply JuanFMontesinos,
In same code with slightly different setting (added 1 more gpu), I have this:

RuntimeError: CUDA out of memory. Tried to allocate 3.62 GiB (GPU 2; 47.99 GiB total capacity; 13.14 GiB already allocated; 31.59 GiB free; 13.53 GiB reserved in total by PyTorch)

In this case if we add 13.14+13.53=26.67GiB, we still have enough memory to allocate 3.62GiB.
And it doesn’t match your explanation.

[JuanFMontesinos (Juan Montesinos)]

Can you verify via nvidia-smi command that you are not consuming resources with any other task before running this?

Also, how are you running this multi-gpu scenario? Note that modules such as Dataparallel usually have a main gpu which requires more memory.

[C_J]

I am monitoring using nvidia-smi, but the memory usage never reaches close to maximum.
In some other post, I saw that someone is calculating free memory as:
free memory = reserved memory - allocated memory
In this case, it makes sense.
But then I’m not sure how to reserve more memory, since it looks like it’s done automatically.

[learner12345 (Ali)]

I’m also getting a similar type of error.
I have finetuned Yolov5 on custom dataset, it went very well, after getting best weights. I tried to refine those best weights by training for more 30 epochs. But I get cuda out of memory error.

I tried to empty cache memory as Suggested by stack overflow but still can’t get out of this error. Althought memory is not being used by the GPU. I have checked using Nvidia-smi

Any help will be appreciated much,
many thanks

[ptrblck]

Double post from here.

[C_J]

It says you have only 252.69MiB Cuda memory and you are trying to allocate 250.00MiB, which is pretty close. This makes sense to me that you don’t have enough GPU memory available.
In my case it tries to allocate 3.62GiB but there are 20.41GiB free GPU memory, which I think is weird.