Non-invasive GPU memory profiling

[Andrei_Pokrovsky]

Howdy!

Does anyone have any recommendations on how to profile GPU memory in a non-invasive fashion?

Some options seem to be nvidia-smi with memory monitoring (sampling based, so it seems to miss peak usage among other shortcomings), nvprof with memory trace (seems too slow), nvprof with api trace (doesn’t report allocation amounts and doesn’t account for fragmentation) or a python-level source-invasive solution using gc, previously suggested in a separate thread here:

How to debug causes of GPU memory leaks?

Hi, I have been trying to figure out why my code crashes after several batches because of cuda memory error. I understand that probably there is some variable(s) that is not freed because I keep it in the graph. The question is how to debug that kind of thing? I have wrote a simple line profiler that examines amount of GPU memory on each step, and it seems like on each loss.backwards() step there was massive memory allocation and not all of it frees, it that piles up for a couple of absolutely…

One simple solution seems to instrument the code around every cudaMalloc allocation in pytorch source and check available memory but it seems pytorch currently doesn’t expose a callback for tracing these calls.

[albanD]

You can get some information from these functions that give you the current state of the allocated memory and the caching allocator and the max usage: max_memory_allocated, max_memory_cached, memory_allocated and memory_cached.
Or did you wanted more precise profiling?

[Andrei_Pokrovsky]

Thank you! I think this should work.