Question about pytorch inplace operation of diectly accessing data

1

All inplace operations have a trailing underscore (e.g. tensor.add_, tensor.mul_), are using the “inplace syntax” (e.g. tensor += 1), or are directly accessing the data (e.g. tensor[0] = 1.).

I know all inplace operations can be divided into three categories. I’m still not clear about the situation of accessing data. So I list some examples to help me to understand. You can just tell me whether it is an in-place operation or not.

  1. a=tensor[0], a=1
  2. a=1, tensor[0]=a
  3. tensor[0]=tensor[0]+1
  4. a=torch.tensor(0), a=torch.tensor(1)
  5. a=tensor[0]+1
2

Hi ZiQing!

Please see the following script and the comments it contains:

import torch
print (torch.__version__)

# set up tensor and look at its value (contents) and identity / location

tensor = torch.arange (5)
print ('tensor:', tensor)
print ('id (tensor):', id (tensor))

# 1. a=tensor[0], a=1
#    this is not inplace -- we change what the python variable a refers to

print ('example 1:')

a = tensor[0]
print ('a:', a)
a = 1
print ('a:', a)
# neither the value of tensor nor the actual object to which it refers changes
print ('tensor:', tensor)
print ('id (tensor):', id (tensor))

# 2. a=1, tensor[0]=a
#    inplace -- we change the value of tensor by indexing into it

print ('example 2:')

a = 1
tensor[0] = a
# tensor still refers to the same object instance, but its value has changed
print ('tensor:', tensor)
print ('id (tensor):', id (tensor))

# 3. tensor[0]=tensor[0]+1
#    also inplace -- we change the value of tensor by indexing into it

print ('example 3:')

tensor[0] = tensor[0] + 1
print ('tensor:', tensor)
print ('id (tensor):', id (tensor))

# 4. a=torch.tensor(0), a=torch.tensor(1)
#    not inplace -- first a refers to one object, then it refers to another
#    the original object is not changed
#    (but nothing refers to it anymore so it becomes a candidate for garbage collection)

print ('example 4:')

a = torch.tensor (0)
print ('a:', a)
print ('id (a):', id (a))
a = torch.tensor (1)
print ('a:', a)
print ('id (a):', id (a))

# 5. a=tensor[0]+1
#    not inplace -- the expression tensor[0] + 1 creates a new tensor to which a now refers

print ('example 5:')

a = tensor[0] + 1
print ('a:', a)
print ('id (a):', id (a))

#  6. let's add one more example
#     a is a view into tensor, so when we modify the contents of tensor, a sees the change
#     but tensor still refers to the original tensor object (whose contents has changed)
#     and a still refers to the same view object (whose viewed value has changed)

print ('example 6:')

a = tensor[0]
print ('tensor:', tensor)
print ('id (tensor):', id (tensor))
print ('a:', a)
print ('id (a):', id (a))
tensor[0] = 99
print ('tensor:', tensor)
print ('id (tensor):', id (tensor))
print ('a:', a)
print ('id (a):', id (a))

Note that when an object is “freed,” that is, when the (last) reference
that refers to it is set to something else, that object’s memory can be
reused. Therefore that object’s id() (location in memory) can show
up as the id() of a new object.

With that in mind, here is the above script’s output:

1.10.2
tensor: tensor([0, 1, 2, 3, 4])
id (tensor): 2206358926664
example 1:
a: tensor(0)
a: 1
tensor: tensor([0, 1, 2, 3, 4])
id (tensor): 2206358926664
example 2:
tensor: tensor([1, 1, 2, 3, 4])
id (tensor): 2206358926664
example 3:
tensor: tensor([2, 1, 2, 3, 4])
id (tensor): 2206358926664
example 4:
a: tensor(0)
id (a): 2206358926904
a: tensor(1)
id (a): 2206362259272
example 5:
a: tensor(3)
id (a): 2206358926904
example 6:
tensor: tensor([2, 1, 2, 3, 4])
id (tensor): 2206358926664
a: tensor(2)
id (a): 2206362259352
tensor: tensor([99,  1,  2,  3,  4])
id (tensor): 2206358926664
a: tensor(99)
id (a): 2206362259352

Best.

K. Frank

3

@KFrank

Thanks! The answer is quite detailed.

As I know, Python separates data type into Mutable data type (i.e. When the value changes, the memory address does not change) and Immutable data type (i.e. When the value changes, the memory address changes, that is, the id changes). According to the results that you have showed, I think torch.tensor is similar with list in memory management.

So I did the following experiments as the example 6. But I found that the results were a little different.

Here is the script of tensor:

# example 6 of torch.tensor
import torch
tensor = torch.arange (5)
a = tensor[0]
print ('tensor:', tensor)
print ('id (tensor[0]):', id (tensor[0]))
print ('a:', a)
print ('id (a):', id (a))
tensor[0] = 99
print ('tensor:', tensor)
print ('id (tensor[0]):', id (tensor[0]))
print ('a:', a)
print ('id (a):', id (a))

Here is the output of the script of tensor:

tensor: tensor([0, 1, 2, 3, 4])
id (tensor[0]): 1663486027312
a: tensor(0)
id (a): 1663486027096
tensor: tensor([99,  1,  2,  3,  4])
id (tensor[0]): 1663504471168
a: tensor(99)
id (a): 1663486027096

Here is the script of list:

tensor = [0, 1, 2, 3, 4]
a = tensor[0]
print ('tensor:', tensor)
print ('id (tensor[0]):', id (tensor[0]))
print ('a:', a)
print ('id (a):', id (a))
tensor[0] = 99
print ('tensor:', tensor)
print ('id (tensor[0]):', id (tensor[0]))
print ('a:', a)
print ('id (a):', id (a))

Here is the output of the script of list:

tensor: [0, 1, 2, 3, 4]
id (tensor[0]): 1522970816
a: 0
id (a): 1522970816
tensor: [99, 1, 2, 3, 4]
id (tensor[0]): 1522973984
a: 0
id (a): 1522970816

The value of a is different in two scripts. So it seems that memory management of tensor is not exactly the same with list?

And I don’t quite understand why the value of a changes in the script of torch.tensor. In my opinion, the whole process should be that variable a refers to the original tensor[0] address, and tensor[0] refers to the new address of 99. But why the value of original tensor[0] changes?