In the source code of Variable.py(line 199), I found the assertion, assert self.__version == 0. But it’s not clear to say what is going wrong here. Could anyone help me on this?
A “leaf variable” is a variable you create directly, not as the result of some other operation.
x = torch.autograd.Variable(torch.Tensor([1, 2, 3, 4])) # leaf variable y = x + 1 # not a leaf variable
An in-place operation is something which modifies the data of a variable. For example,
x += 1 # in-place y = x + 1 # not in place
PyTorch doesn’t allow in-place operations on variables you create directly (such as parameters of your model) because that usually isn’t correct. You can work around it by either using an operations that’s not in-place or by cloning the variable.
x2 = x.clone() # clone the variable x2 += 1 # in-place operation
Leaf variable has been moved into the graph interior
RuntimeError: volatile can only be set on leaf variables
Is it bad practice to get around pytorch disallowing in-place operations by assigning to Variable().data?
That depends on the situation.
For example to initialize or update parameters, assigning to
.data is the way to go. Usually, you cannot backprop when changing Variables’
.data in the middle of a forward pass…
I’d summerise my experience as “don’t do it unless you have reason to believe its the rigght thing”.
Tensor and tensor.data
Yes, I remember now, messing with backprop and autograd was why I was running into problems with in-place assignment before. Using .data as I am currently for initialising word embeddings seems ok then.
Trying to read the code for optim (I want to implement something a bit differently) and your previous example/explanation of what is a leaf Variable doesn’t seem to be valid anymore.
y = x + 1 # not a leaf variable
Well, here’s output from my termial for the code which you have mentioned:
>>> x = torch.autograd.Variable(torch.Tensor([1, 2, 3, 4])) >>> x.is_leaf True >>> y = x + 1 >>> y.is_leaf True >>> y Variable containing: 2 3 4 5 [torch.FloatTensor of size (4,)]
So, can someone please explain what is a leaf Variable, and what is not a leaf variable? Clearly a non-leaf-variable cannot be optimized, but what is it?
Came across a similar issue. Reason is requires grad. x = torch.autograd.Variable(torch.Tensor([1, 2, 3, 4]), requires_grad=True) x.is_leaf #True y = x + 1 y.is_leaf #False
Hi, I used to create leaf variable like:
y = torch.autograd.Variable(torch.zeros([batch_size, c, h, w]), requires_grad=True)
Then I want to assign value to indexed parts of y like below,(y_local is a Variable computed based on other variables and I want to assign the value of y_local to part of the y and ensure that the gradients from y can flow to the y_local.)
y.data[:,:,local_x[i]:local_x[i+1],local_y[i]:local_y[i+1]] = y_local.data
I am wondering such operation supports the normal gradient backward for the y_local varible?
I am also facing similar issue. Please let me know how were you able to resolve it
leaf variable, in essence, is a variable, or a tensor with requires_grad=True. So, if a tensor with requires_grad=False, it does not belong to the variable, let alone leaf variable.
Could you please explain why it’s not correct?
That usually isn’t corret… well… what if it IS correct? What should I do?
I’m pretty sure of what I’m doing, but can’t do it on Pytorch.
I had a similar case. I used:
y = torch.zeros([batch_size, c, h, w]), requires_grad=False)
then I update the value of y according to the value of the network output and then apply a loss function on y and it worked for me.