Behaviour of id with detached tensors

1

I noticed something strange, when you run:

a = torch.tensor(0)
b = torch.tensor(1)

print(id(a) == id(b))
print(id(a.detach()) == id(b.detach()))

The output of the first print is False whereas it’s True for the second - the detached tensors get the same python id (i.e. memory address). Is this behavior expected?

What’s worse, this seems to be nonteterministic:

for i in range(20):
    print(id(a.detach()) == id(b.detach()))

if you run this loop a few times in a row, you should notice that the detached tensors occasionally get different memory addresses. What’s going on here?

2

I don’t think this is how it works.

So a.detach() creates a new (Python) Tensor object using the same underlying memory to store values (a “view” in autograd lingo). Now id gets the id (the address of the Python object). But after id returns, the refcount of the Python Tensor returned by a.detach() goes to 0 and Python may decide to deallocate it. Then b.detach() creates a new Tensor that may be at the same memory location if Python collected the unused a.detach().

So as a summary:

  • a.detach() always creates a new Tensor, albeit one sharing the underlying memory blob for the values with a.
  • what you are seeing is more a behaviour of Python w.r.t. anonymous temporary objects you create.

Best regards

Thomas

1 Like