I think I found a mistake on Automatic differentiation package - torch.autograd — PyTorch 1.12 documentation.

The explanations of `torch.Tensor.is_leaf`

& `torch.Tensor.detach_`

imply that a `Tensor`

with `requires_grad = False`

is a leaf, but exactly the opposite is the case, since the gradient is calculated w.r.t. all leaves. Am I wrong?

Kind Regards

Hi Milan,

A simple explanation -

A leaf tensor is any tensor on which no operations *(that autograd can track)* have been performed.

When a tensor is first initialized, it becomes a leaf node/tensor.

Basically, all inputs and weights of a neural network are leaf tensors and hence leaf nodes in the computational graph.

Whenever *any* operation (that autograd can track - this is important) is performed on a leaf tensor, it does not remain a leaf anymore.

See:

```
x = torch.tensor([1.0, 2.0])
print(x.is_leaf) # True
print(x.requires_grad) # False
y = (x*2).sum()
print(y.is_leaf) # True
print(y.requires_grad) # False
```

Although y is the result of some operations on x, y is still a leaf as **autograd isn’t tracking anything.**

Not necessarily. It has less to do with the requires_grad attribute and more to do with what autograd tracks.

As you saw above, *exactly the opposite* is not always the case. A tensor with `requires_grad = False`

can be a leaf.

`detach`

simple detaches a tensor from its current computation graph and returns a new tensor which is a leaf and has its `requires_grad = False`

.

`detach_`

is just that but does it in-place without returning a new tensor.

Does this answer your question?

2 Likes

Thank you for clarification!

I have a follow-up question:

why is it, that

```
a = torch.tensor([1., 2.], requires_grad=True)
b = 2 * a
c = a.sum()
d = (2*a).sum()
print(a.is_leaf) # True
print(b.is_leaf) # False
print(c.is_leaf) # False
print(d.is_leaf) # False
# ,but
a = torch.tensor([1., 2.], requires_grad=False)
b = 2 * a
c = a.sum()
d = (2*a).sum()
print(a.is_leaf) # True
print(b.is_leaf) # True
print(c.is_leaf) # True
print(d.is_leaf) # True
```

In the first snippet, since `a`

has `requires_grad=True`

, the autograd engine is now tracking every operation involving `a`

.

Hence, all calculations involving `b, c, d`

are being tracked which is why they aren’t leaves any more. See:

For the second code, a’s `requires_grad=False`

and autograd isn’t tracking, so all `b, c, d`

are leaves - this is conceptually exactly what the code sample in my previous reply is.

Let me know if you have further questions,

Srishti

1 Like

Thanks, you helped me a lot!