I found out that calling torch.empty() does not allocate memory to fit future data. Is it an intended behavior? I believed that torch.empty() works like any other construction methods, but without filling in any values.
torch.empty does allocate memory, but does not initialize this memory space.
I.e. the values in your tensor will just be interpreted as the bits which are currently in the allocated memory space.
Yes, otherwise you couldn’t index the tensor or work in some other way.
I’m not sure, how the memory profiler works, but using the process_memory_info() data you’ll get the expected result:
import torch
import os
import psutil
process = psutil.Process(os.getpid())
mem_t0 = process.memory_info().data
print(mem_t0)
x = torch.empty(1000, 1000, dtype=torch.float32)
mem_t1 = process.memory_info().data
mem_expected = x.numel() * 4 /1024 # in kBytes
print('Expected {} kBytes'.format(mem_expected))
print('Actual {} kBytes'.format((mem_t1 - mem_t0) / 1024))
> Expected 3906.25 kBytes
> Actual 3908.0 kBytes