# How to get the gradients for both the input and intermediate variables?

**URL:** <https://discuss.pytorch.org/t/how-to-get-the-gradients-for-both-the-input-and-intermediate-variables/105730>\
**Category:** autograd\
**Created:** [December 10, 2020, 6:20am UTC](https://discuss.pytorch.org/t/how-to-get-the-gradients-for-both-the-input-and-intermediate-variables/105730 "2020-12-10T06:20:42Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![oat](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oat/32/34409_2.png) [@oat](https://discuss.pytorch.org/u/oat)\
**Post date:** [December 10, 2020, 6:20am UTC](https://discuss.pytorch.org/t/how-to-get-the-gradients-for-both-the-input-and-intermediate-variables/105730/1 "2020-12-10T06:20:42Z")

</div>

Assuming the following forward pass: x → y → z, in which  
x is a scalar  
y = x \* x  
z = 2 \* y

I want to track the following derivatives/gradients, including the gradient for the intermediate function y:

- dz/dy, gradient of z with respect to y, which should be “2”
- dy/dx, gradient of y with respect to x, which should be “2x”
- dz/dx, gradient of z with respect to x, which should be “4x”

So, I initiated both x and y with the **`requires_grad=True`** argument. However, I can only get **`y.grad`** , which is “2”, and **`x.grad`** returned “None”, as shown below.

May I ask:

- Why I’m unable to get x.grad (dz/dx) in this case?
- How to get the gradients for both the input and intermediate variables via .backward()?

 ![Screenshot 2020-12-10 at 13.58.04](https://discuss.pytorch.org/uploads/default/original/3X/9/d/9dcb130dc2b59c5a59496077ec12a009d73e1482.png)

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [December 10, 2020, 6:40am UTC](https://discuss.pytorch.org/t/how-to-get-the-gradients-for-both-the-input-and-intermediate-variables/105730/2 "2020-12-10T06:40:32Z")

</div>

> [@oat](#):
>
> Why I’m unable to get x.grad (dz/dx) in this case?

You are detaching the computation graph by recreating a tensor in:

```python
 y= torch.tensor(x * x)

```

use `y = x*x` instead.

PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier 😉

---

<div class="post-metadata">

**Author:** ![oat](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oat/32/34409_2.png) [@oat](https://discuss.pytorch.org/u/oat)\
**Post date:** [December 10, 2020, 6:56am UTC](https://discuss.pytorch.org/t/how-to-get-the-gradients-for-both-the-input-and-intermediate-variables/105730/3 "2020-12-10T06:56:04Z")

</div>

Thanks, @ptrblck.

However, it seems **`y.grad`** is not tracked using `y = x * x`:

```auto
x = torch.tensor(0.3, requires_grad=True)
print(x)
# [output] tensor(0.3000, requires_grad=True)

y = x * x
print(y)
# [output] tensor(0.0900, grad_fn=<MulBackward0>)

z = 2 * y
print(z)
# [output] tensor(0.1800, grad_fn=<MulBackward0>)

z.backward()

print(y.grad)
# [output] None

print(x.grad)
# [output] tensor(1.2000)

```

Anyway, I found [this post](https://discuss.pytorch.org/t/why-cant-i-see-grad-of-an-intermediate-variable/94) that’s relevant to my question, and I’ll digest it first.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [December 10, 2020, 6:58am UTC](https://discuss.pytorch.org/t/how-to-get-the-gradients-for-both-the-input-and-intermediate-variables/105730/4 "2020-12-10T06:58:09Z")

</div>

Your code should raise a warning and explain the reason and workaround for it:

```python
UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
  print(y.grad)

```

After adding `y.retain_grad()` you’ll get the gradient value for `y`.

---

<div class="post-metadata">

**Author:** ![oat](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oat/32/34409_2.png) [@oat](https://discuss.pytorch.org/u/oat)\
**Post date:** [December 10, 2020, 7:05am UTC](https://discuss.pytorch.org/t/how-to-get-the-gradients-for-both-the-input-and-intermediate-variables/105730/5 "2020-12-10T07:05:42Z")

</div>

Thanks, @ptrblck.

I’m doing the test in a Jupyter notebook via vscode. I’m not sure why I don’t get the warning message you pointed out.

Anyway, adding **`y.retain_grad()`** does work in this case as you suggested:

```auto
x = torch.tensor(0.3, requires_grad=True)
print(x)
# [output] tensor(0.3000, requires_grad=True)

y = x * x
print(y)
# [output] tensor(0.0900, grad_fn=<MulBackward0>)

y.retain_grad()

z = 2 * y
print(z)
# [output] tensor(0.1800, grad_fn=<MulBackward0>)

z.backward()

print(y.grad)
# [output] tensor(2.)

print(x.grad)
# [output] tensor(1.2000)

```