In-place vs. the same variables' name

1

Hi,

From this question RuntimeError when using a += b but not when doing a= a + b

I got this answer:

The difference between a += b and a = a + b is that in the first case, b is added in a inplace (so the content of a is changed to now contain a+b). In the second case, a brand new tensor is created that contains a+b and then you assign this new tensor to the name a.
To be able to compute gradients, you sometimes need to keep the original value of a, and so we prevent inplace operation from being done because otherwise, we wonā€™t be able to compute gradients

from @albanD

However, I have some other questions:

  1. For a = a+b, a brand new tensor is created that contains a+b and then you assign this new tensor to the name a . So, where is the tensor of the original a after assigning this new to tensor to the name a ?
    Can Pytorch ā€˜rememberā€™ the original tensor of a for back propagation?

  2. if I do this:

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))   --->stageā‘ 
        x = self.pool(F.relu(self.conv2(x)))   --->stageā‘”
        x = x.view(-1, 16 * 5 * 5)             --->stageā‘¢
        x = F.relu(self.fc1(x))                --->stageā‘£
        x = F.relu(self.fc2(x))                --->stageā‘¤
        x = self.fc3(x)
        return x

though I use the variableā€™s name x at all stages, the pytorch can remember all x at every stageļ¼Œ in order to calculate gradient during back propagation. Am I right?

thank you in advance!

2

  1. If no other names point to the original tensor named a, then the original tensor will be lost to the garbage collector. On the other hand if a was a Variable, then the original variable a is stored in the computation graph for the new Variable named a. When you operate on Variables, PyTorch automatically stores everything it needs for backpropagation.

  2. Your code is fine. PyTorch will store all of the intermediate values that it needs.

1 Like
3

Thank you very much.

One more question, since Pytorch will store all of the intermediate values that it needs, why use the in-place operation will get some error :sweat_smile:

4

Inplace operations modify the original data, which often makes it tricky/impossible to store the intermediate values needed for backpropagation.

1 Like
5

Thank you very much! I see : :yum:

1 Like