Leftover in memory

I get confused when I try to profile the memory consumption of my simple program.

import torch
import torch.nn as nn
from memory_profiler import profile

def simple_func():
    x = torch.rand(3000, 2000, requires_grad=True)
    f = nn.Linear(2000, 1000)
    y = f(x)
    dd = torch.autograd.grad(y, (x,)+tuple(f.parameters()), grad_outputs=torch.rand(3000, 1000))

@profile
def compound_func():
    simple_func()
    simple_func()
    simple_func()
    simple_func()

compound_func()

The program output

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    48    306.8 MiB    306.8 MiB           1   @profile
    49                                         def compound_func():
    50    319.4 MiB     12.6 MiB           1       simple_func()
    51    403.2 MiB     83.8 MiB           1       simple_func()
    52    403.2 MiB      0.0 MiB           1       simple_func()
    53    403.2 MiB      0.0 MiB           1       simple_func()

I don’t understand why the first two simple_func calls give increment in memory usage. It seems there is often leftover in memory even if all objects are local and should be deleted after the call.

I have tried an even simpler function

def simple_func():
    x = torch.rand(3000, 2000)

But there is still leftover

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    48    306.8 MiB    306.8 MiB           1   @profile
    49                                         def compound_func():
    50    308.5 MiB      1.7 MiB           1       simple_func()
    51    331.2 MiB     22.7 MiB           1       simple_func()
    52    331.2 MiB      0.0 MiB           1       simple_func()
    53    331.2 MiB      0.0 MiB           1       simple_func()

Do we need to clean the memory manually? I have tried “del” before the end of the simple_func, but it doesn’t help.
Or is it just the measurement issue by memory_profiler?