CUDA out of memory(specific question)

[June90]

I have GTX 2080 ti 11G and I’m training EfficientDet on Window 10.
I don’t fully understand why I have only 320.00 KiB free? Is anyone can explain this issue?
I think to be working properly, Tried to allocate 16.00 MiB should be less than 320.00 KiB free.
I should know why…

RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 11.00 GiB total capacity; 8.87 GiB already allocated; 320.00 KiB free; 9.03 GiB reserved in total by PyTorch)

[June90]

@stas Could help me with these issues?

[stas (Stas Bekman)]

Sometimes this has to do with memory fragmentation. Sometimes you are just out of memory. Here clearly it’s the latter since 16MB > 320KB.

Typically users are puzzled about the memory fragmentation scenario where the OOM happens trying to allocate say 16MB, while reporting 200MB free, which makes no sense immediately, until you understand that in those 200MB there is not a single contiguous chunk of free memory that is larger than 16MB.

You can find the memory fragmentation discussed in multiple threads here on this forum, e.g. here is one of them where I was trying to figure it out:

GPU RAM fragmentation diagnostics

Following up on Unable to allocate cuda memory, when there is enough of cached memory, while there is no way to defrag nvidia GPU RAM, is there a way to get the memory allocation map? I’m asking in the simple context of just having one process using the GPU exclusively. Using free memory info from nvml can be very misleading due to fragmentation, so it would be useful to be able to have some sort of estimation of fragmentation, e.g. 1GB free, 0.3G of which is in chunks less than 50MB. And it w…