Leaf variable was used in an inplace operation

(Oswin) #1

I code a function which implements some operations including torch.mm, torch.index_select and torch.cc. However, there comes out an AssertionError, leaf variable was used in an inplace operation.

In the source code of Variable.py(line 199), I found the assertion, assert self.__version == 0. But it’s not clear to say what is going wrong here. Could anyone help me on this?

(colesbury) #2

A “leaf variable” is a variable you create directly, not as the result of some other operation.

For example,

x = torch.autograd.Variable(torch.Tensor([1, 2, 3, 4]))  # leaf variable
y = x + 1  # not a leaf variable

An in-place operation is something which modifies the data of a variable. For example,

x += 1  # in-place
y = x + 1 # not in place

PyTorch doesn’t allow in-place operations on variables you create directly (such as parameters of your model) because that usually isn’t correct. You can work around it by either using an operations that’s not in-place or by cloning the variable.

x2 = x.clone()  # clone the variable
x2 += 1  # in-place operation

Leaf variable has been moved into the graph interior
RuntimeError: volatile can only be set on leaf variables

Is it bad practice to get around pytorch disallowing in-place operations by assigning to Variable().data?

(Thomas V) #4

That depends on the situation.
For example to initialize or update parameters, assigning to .data is the way to go. Usually, you cannot backprop when changing Variables’ .data in the middle of a forward pass…
I’d summerise my experience as “don’t do it unless you have reason to believe its the rigght thing”.

Best regards


Tensor and tensor.data

Yes, I remember now, messing with backprop and autograd was why I was running into problems with in-place assignment before. Using .data as I am currently for initialising word embeddings seems ok then.

(Yoni Keren) #6


Trying to read the code for optim (I want to implement something a bit differently) and your previous example/explanation of what is a leaf Variable doesn’t seem to be valid anymore.

In particular,
you wrote

y = x + 1  # not a leaf variable

Well, here’s output from my termial for the code which you have mentioned:

>>> x = torch.autograd.Variable(torch.Tensor([1, 2, 3, 4]))
>>> x.is_leaf
>>> y = x + 1
>>> y.is_leaf
>>> y
Variable containing:
[torch.FloatTensor of size (4,)]

So, can someone please explain what is a leaf Variable, and what is not a leaf variable? Clearly a non-leaf-variable cannot be optimized, but what is it?

(Ed Beeching) #7
Came across a similar issue. Reason is requires grad.

x = torch.autograd.Variable(torch.Tensor([1, 2, 3, 4]), requires_grad=True)
y = x + 1

(RainbowSecret) #8

Hi, I used to create leaf variable like:

y = torch.autograd.Variable(torch.zeros([batch_size, c, h, w]), requires_grad=True)

Then I want to assign value to indexed parts of y like below,(y_local is a Variable computed based on other variables and I want to assign the value of y_local to part of the y and ensure that the gradients from y can flow to the y_local.)

y.data[:,:,local_x[i]:local_x[i+1],local_y[i]:local_y[i+1]] = y_local.data

I am wondering such operation supports the normal gradient backward for the y_local varible?

(unk_user) #9

I am also facing similar issue. Please let me know how were you able to resolve it

(dreamyun) #10

leaf variable, in essence, is a variable, or a tensor with requires_grad=True. So, if a tensor with requires_grad=False, it does not belong to the variable, let alone leaf variable.


But y is a variable in this case, it’s not a leaf node.


Could you please explain why it’s not correct?

(Daniel) #13

That usually isn’t corret… well… what if it IS correct? What should I do?
I’m pretty sure of what I’m doing, but can’t do it on Pytorch.

(Nima Rafiee) #14

I had a similar case. I used:

y = torch.zeros([batch_size, c, h, w]), requires_grad=False)

then I update the value of y according to the value of the network output and then apply a loss function on y and it worked for me.