Question about pytorch tensor memory management

1

I did an experiment of PyTorch tensor. The code is shown below.

import torch
t = torch.tensor([
    [1,2],
    [3,4]
])
t2 = torch.tensor([
    [5,6],
    [7,8]
])
print ('id (t[0][0]):', id (t[0][0]))
print ('id (t2[0][0]):', id (t2[0][0]))
t[0][0]=t2[0][0]
print ('t:', t)
print ('t2:', t2)
print ('id (t[0][0]):', id (t[0][0]))
print ('id (t2[0][0]):', id (t2[0][0]))

Here is the result of that.

id (t[0][0]): 1663486076824
id (t2[0][0]): 1663486077112
t: tensor([[5, 2],
        [3, 4]])
t2: tensor([[5, 6],
        [7, 8]])
id (t[0][0]): 1663486077184
id (t2[0][0]): 1663486077184

Why both memory addresses of t and t2 change?

And here is the comparative experiment from list.

t = [
    [1,2],
    [3,4]
]
t2 = [
    [5,6],
    [7,8]
]
print ('id (t[0][0]):', id (t[0][0]))
print ('id (t2[0][0]):', id (t2[0][0]))
t[0][0]=t2[0][0]
print ('t:', t)
print ('t2:', t2)
print ('id (t[0][0]):', id (t[0][0]))
print ('id (t2[0][0]):', id (t2[0][0]))

The result of list:

id (t[0][0]): 1522970848
id (t2[0][0]): 1522970976
t: [[5, 2], [3, 4]]
t2: [[5, 6], [7, 8]]
id (t[0][0]): 1522970976
id (t2[0][0]): 1522970976

You can see from that the memory management of list is different from tensor. I want to know how PyTorch manages memory (e.g. reference, memory change, and assign).

2

And another question is:

import torch
a = torch.tensor([[1, 2], [3, 4]])
print(id(a[0][0]))
print(id(a[0][1]))
print(id(a[1][0]))
print(id(a[1][1]))

#-----------------------------------------------------------------------
#output
2557723599424
2557723600288
2557723599424
2557723600288

Why a[0][0] has the same address with a[1][0]?

3

@KFrank already explained the culprits of using id() here. If you want to check the actual memory pointer, use .data_ptr():

a[0][0].data_ptr()
a[0][1].data_ptr()
a[1][0].data_ptr()
a[1][1].data_ptr()
4

@ptrblck

What’s the difference between id and data_ptr (e.g. corresponding to the following picture)?

Does the memory management of PyTorch tensor like list?

image
image1524×618 80.3 KB

5

id() returns the identity of an object while data_ptr returns the address of the first element of the tensor.
Depending how you are indexing the tensor, the actual object might not be the same as seen in e.g. in this example and also explained in the linked topic:

a = torch.randn(1, requires_grad=True)

for _ in range(10):
    print(id(a))
# 140002184801472
# 140002184801472
# 140002184801472
# 140002184801472
# 140002184801472
# 140002184801472
# 140002184801472
# 140002184801472
# 140002184801472
# 140002184801472


for _ in range(10):
    print(id(a[0]))
# 140002184800512
# 140002184799552
# 140002184831440
# 140002184831440
# 140002184831440
# 140002184831440
# 140002184831440
# 140002184831440
# 140002184831440
# 140002184831440

In particular indexing a is a differentiable operation and will thus create a new object.