# Aren't weights of models non\_leaves. So how do they have their grad values?

**URL:** <https://discuss.pytorch.org/t/arent-weights-of-models-non-leaves-so-how-do-they-have-their-grad-values/100410>\
**Category:** autograd\
**Created:** [October 23, 2020, 2:46pm UTC](https://discuss.pytorch.org/t/arent-weights-of-models-non-leaves-so-how-do-they-have-their-grad-values/100410 "2020-10-23T14:46:50Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![SiddharthSingi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/siddharthsingi/32/26161_2.png) [@SiddharthSingi](https://discuss.pytorch.org/u/SiddharthSingi)\
**Post date:** [October 23, 2020, 2:46pm UTC](https://discuss.pytorch.org/t/arent-weights-of-models-non-leaves-so-how-do-they-have-their-grad-values/100410/1 "2020-10-23T14:46:50Z")

</div>

I know its obvious why weights need to have gradient values, but I have a particular question about leaves and non leaves. Please read on, this question is not as long as it seems

So I understand that the grad is not available for any intermediate node, only available for leaf nodes (However the reasoning is still not perfectly clear to me like [here](https://discuss.pytorch.org/t/what-is-the-purpose-of-is-leaf/87000/6))

Example of no grad value for intermediate node: (feel free to skip directly to the question below)

```auto
a = torch.rand(10, requires_grad=True)
b = torch.rand(10)
c = a + b

d = torch.tensor(5.0)
e = c*d

e.mean().backward()
print(a.grad)
print(c.grad)

```

Here a.grad should give me output since it is a leaf node, however c.grad is an intermediate node and shouldnt give me any output, which is exactly what we get:

Output:

```auto
tensor([0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000,
        0.5000])
C:\Anaconda\envs\robot\lib\site-packages\torch\tensor.py:746: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by 
mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
  warnings.warn("The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad "
None

```

.

**MY QUESTION** is: In any model that we create, with multiple multiple layers, arent the weights of each layers non\_leaves, hence how come they have grad values but other non\_leaves do not.

For example:

```auto
class ActorNet(nn.Module):
    def __init__ (self, obs_space, action_space):
        super(). __init__ ()
        self.fc1 = nn.Linear(obs_space, 32)
        self.fc2 = nn.Linear(32, action_space)        

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return x

```

Here isnt the layer fc2 a non leaf. However its weights will still definitely have a grad value.

---

<div class="post-metadata">

**Author:** ![KFrank](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/32.png) [@KFrank](https://discuss.pytorch.org/u/KFrank)\
**Post date:** [October 23, 2020, 6:14pm UTC](https://discuss.pytorch.org/t/arent-weights-of-models-non-leaves-so-how-do-they-have-their-grad-values/100410/2 "2020-10-23T18:14:36Z")

</div>

Hi Siddharth!

> [@SiddharthSingi](#):
>
> ```auto
> def forward(self, x):
> x = F.relu(self.fc1(x))
> x = F.relu(self.fc2(x))
> return x
> 
> ```
> 
> Here isnt the layer fc2 a non leaf.

On the contrary, `fc2.weight` (and `fc2.bias`) are leaf tensors.

In the forward pass the values of `fc2.weight` do not depend on  
the values of `x` (nor on the values in `fc1`). They only change (in  
the conventional use case) when you call `optimizer.step()`.

Are you imagining that only the first layer is a leaf, that is that in:

```python
    def forward(self, x):
        # x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return x

```

`fc2` becomes a leaf _because_ we commented out `fc1`?

To reiterate, the values of `x` of course depend upon whether `fc1`  
was called, but the values of `fc2.weight` do not.

Best.

K. Frank

---

<div class="post-metadata">

**Author:** ![SiddharthSingi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/siddharthsingi/32/26161_2.png) [@SiddharthSingi](https://discuss.pytorch.org/u/SiddharthSingi)\
**Post date:** [October 25, 2020, 7:24am UTC](https://discuss.pytorch.org/t/arent-weights-of-models-non-leaves-so-how-do-they-have-their-grad-values/100410/3 "2020-10-25T07:24:14Z")

</div>

@KFrank  
Thanks for the reply. It’s now such an obvious reason as to why the weights are leaf tensors and not non leaves. I was imagining the graph completely wrong in my head. Thanks for helping out.