In place functions by reference

1

Is there a way to do in place operations when you are accessing a tensor by reference? As an example I’d like to be able to change the original temp calling sigmoid (and possibly other operations more complex than += ) on temp2.

>>> import torch
>>> temp = {0:torch.zeros(4), 1:torch.zeros(4)}
>>> temp2 = temp[0]
>>> temp2+=1
>>> temp
{0: tensor([1., 1., 1., 1.]), 1: tensor([0., 0., 0., 0.])}
>>> temp2 = temp2.sigmoid()
>>> temp2
tensor([0.7311, 0.7311, 0.7311, 0.7311])
>>> temp
{0: tensor([1., 1., 1., 1.]), 1: tensor([0., 0., 0., 0.])}
2

Hi Pytorcher!

Yes, you can use temp2.copy_ (temp2.sigmoid()).

Best.

K. Frank

3

That did it, thanks!

4

Actually, it looks like that worked for simple cases but now that I’m running a more complex model it is giving me a:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

I ran with with torch.autograd.set_detect_anomaly(True) and it does look like its the copy operations, specifically I am running a batch_norm as you can see in this block of the stack.

  File "/home/my_layer.py", line 431, in forward
    mainOut.copy_( self.batchNormLayer(mainOut))
  File "/home/me/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/me/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 81, in forward
    exponential_average_factor, self.eps)
  File "/home/me/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/nn/functional.py", line 1670, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled

Traceback (most recent call last):
  File "main.py", line 617, in <module>
    main()
  File "main.py", line 144, in main
    main_worker(args.gpu, ngpus_per_node, args)
  File "main.py", line 319, in main_worker
    train(train_loader, model, criterion, optimizer, scheduler, epoch, args)
  File "main.py", line 431, in train
    loss.backward()
  File "/home/me/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/tensor.py", line 166, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/me/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64, 512, 7, 7]], which is output 0 of CudnnConvolutionBackward, is at version 3; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
5

Hi Pytorcher!

Inplace operations can “break the computation graph” and therefore
prevent proper backpropagation. (The autograd engine flags this for
you, I guess because it can lead to hard-to-debug errors.)

But you originally asked:

Why is it that you want to perform an inplace operation, and how
would you want this operation to interact with backpropagation?

Would it make sense not to use an inplace operation and just let
autograd do its normal thing?

Best.

K. Frank

6

Yes, I do want autograd to just do its normal thing.

Basically the situation I have is that sometimes I want my module to get a single tensor, and sometimes a dictionary of tensors like in the example. At the top of my call I want to do something like:

forward(input)
if type(input) is dictionary
    mainOut = input[0]
else:
    mainOut = input
...
do stuff with mainOut including either
mainOut = self.batchNormLayer(mainOut)
or
mainOut.copy_( self.batchNormLayer(mainOut))
...
return input

Maybe just at the end I should do another dictionary check and set input[0] to be mainOut again, i.e. you can’t do an actual reference like c++ with python like I was thinking here because of the way the in place operations work like that.

7

Hi Pytorcher!

This is exactly right (if I understand your use case). Thus, something
like:

def forward (input):
    if type (input) is dict:
        some_tensor = input[0]
    else:
        some_tensor = input
    
    # do stuff with some_tensor
    # then do
    main_out = self.batchNormLayer (some_tensor)
    
    if type (input) is dict:
        input[0] = main_out
        return input
    else:
        return main_out

Best.

K. Frank