What is difference torch.cuda.memory_allocated() vs. max_memory_allocated

Memory Format
1

I want to know how my GPU is used for my gradients and optimizer state;
which function should I use?
And I wonder where I should place this function. Is there any good demo?

2

Now I know memory allocated is the current state.
But I still don’t know when I should use max_memory_allocated.

3

max_memory_allocated will show the peak memory usage for allocated memory since the beginning of the script execution or since you have explicitly reset the peak via reset_peak_memory_stats().
This can be useful to check e.g. how high the peak usage is during the backward pass, as it might be tricky to add memory_allocated() calls into the backward pass directly e.g. via hooks.

2 Likes