Why max_memory_allocated() after reset doesn't return 0?

1

I’m trying to measure the memory usage of each layer by torch.cuda.max_memory_allocated() and torch.cuda.reset_max_memory_allocated():

for layer in sequential_module:
    torch.cuda.reset_max_memory_allocated(0)
    x = layer(x)
    size = torch.cuda.max_memory_allocated(0)
    sizes.append(size)

I understood torch.cuda.reset_max_memory_allocated() can be used to reset the starting point to track the maximum memory allocations. So I’ve expected that torch.cuda.max_memory_allocated() after reset always returns 0. But it doesn’t.

In [1]: import torch

In [2]: torch.cuda.max_memory_allocated(0)
Out[2]: 0

In [3]: x = torch.rand(100, 100, 100, device=0)

In [4]: torch.cuda.max_memory_allocated(0)
Out[4]: 4865536

In [5]: torch.cuda.reset_max_memory_allocated(0)

In [6]: torch.cuda.max_memory_allocated(0)
Out[6]: 4865536

How do I understand this behavior?

2

Since you are not freeing any memory, torch.cuda.max_memory_allocated will still return the currently used memory, as it’s still the peak.
Have a look at this code snippet:

# Check for empty
torch.cuda.max_memory_allocated(0)

# Create one tensor
x = torch.rand(100, 100, 100, device='cuda:0')
# should yield the same value
torch.cuda.memory_allocated(0)
torch.cuda.max_memory_allocated(0)

# Create other tensor
y = torch.rand(100, 100, 100, device='cuda:0')
# Should be same, but higher
torch.cuda.memory_allocated(0)
torch.cuda.max_memory_allocated(0)

# Delete one tensor
del y
# max_memory_allocated should keep it's old value
torch.cuda.memory_allocated(0)
torch.cuda.max_memory_allocated(0)

# Reset to track new peak memory usage
torch.cuda.reset_max_memory_allocated(0)
torch.cuda.memory_allocated(0)
torch.cuda.max_memory_allocated(0)
1 Like
3

That’s clear explanation and code snippet. Thank you so much!

1 Like