Why is there a discrepency between nvidia-smi reported vram vs the one from torch.cuda.mem_get_info()?

Hello everyone,
Hope youa re having a great day.
I was playing a bit with memory stats in pytorch today that I noticed the numbers reported by pytorch for the gpu vram doesnt match that of nvidia-smi.

I have a RTX3080 10G which shows up as 10240MiB in nvidia-smi but when I try torch.cuda.mem_get_info(0) to get vram amount, I get 10,492,772,352 bytes which translates to 10,006MiB !
10240 MiB is 10,737,418,240 bytes which means 244,645,888 bytes are missing/not accounted for. This is around 233.3125MiB!

where does this discrepancy come from? wheres the other 244,645,888 bytes?

I think you’re looking for this. Basically, there’s some memory set aside to speed up future allocations/garbage collection. See torch.cuda.max_memory_reserved for a value that will be closer to the nvidia-smi output.

1 Like

Thanks but it doesn’t make sense, this is not allocated memory, this is the total amount of memory. Whether its being used up by my processes or some of cuda’s or something else is another story.
Why shouldnt the total amount be different?