# Detach a intermediate variable without recomputing whole graph

**URL:** <https://discuss.pytorch.org/t/detach-a-intermediate-variable-without-recomputing-whole-graph/75551>\
**Category:** autograd\
**Created:** [April 6, 2020, 1:16pm UTC](https://discuss.pytorch.org/t/detach-a-intermediate-variable-without-recomputing-whole-graph/75551 "2020-04-06T13:16:03Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![sbaio](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e495f1/32.png) [@sbaio](https://discuss.pytorch.org/u/sbaio)\
**Post date:** [April 6, 2020, 1:16pm UTC](https://discuss.pytorch.org/t/detach-a-intermediate-variable-without-recomputing-whole-graph/75551/1 "2020-04-06T13:16:03Z")

</div>

Hi,

I have a situation that looks like this:

```auto
model = resnet50()
classifier = nn.Linear(1000, 20)

x = model(input)
out1 = classifier(x)
out2 = classifier(x.detach()) # Computation to avoid

```

where the classifier is expensive and would like to avoid recomputing out2\_d = classifier(x.detach())

Can we use out1 to backward with x detached ?

Thank you

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [April 6, 2020, 3:28pm UTC](https://discuss.pytorch.org/t/detach-a-intermediate-variable-without-recomputing-whole-graph/75551/2 "2020-04-06T15:28:59Z")

</div>

Hi,

You mean that you want to get the gradients for the elements in the classifier but not the ones in model?  
If s, you can use `autograd.grad` to specify what you want the gradients for: `grads = autograd.grad(loss, classifier.parameters())`.

---

<div class="post-metadata">

**Author:** ![sbaio](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e495f1/32.png) [@sbaio](https://discuss.pytorch.org/u/sbaio)\
**Post date:** [April 6, 2020, 4:01pm UTC](https://discuss.pytorch.org/t/detach-a-intermediate-variable-without-recomputing-whole-graph/75551/3 "2020-04-06T16:01:19Z")

</div>

Thank you for the quick response, actually my example was simplifying my case too much. Let me provide a better example.

I’m not trying to compute the gradient of the loss w.r.t to my classifier parameters only.

```auto
x1, x2 = model(x)
out = heavy_module(x1, x2)
loss = Loss1(out)
loss.backward()

out_d = heavy_module(x1.detach(), x2)
loss_d = Loss2(out_d)
loss_d.backward()

```

Can we avoid the second heavy\_module computation ? while having access to both out and out\_d for different losses.

Thank you

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [April 6, 2020, 4:04pm UTC](https://discuss.pytorch.org/t/detach-a-intermediate-variable-without-recomputing-whole-graph/75551/4 "2020-04-06T16:04:36Z")

</div>

You won’t be able to have two Tensors if you do a single forward 😃  
Why can’t you reuse `out` to compute `loss_d`?

---

<div class="post-metadata">

**Author:** ![sbaio](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e495f1/32.png) [@sbaio](https://discuss.pytorch.org/u/sbaio)\
**Post date:** [April 6, 2020, 4:15pm UTC](https://discuss.pytorch.org/t/detach-a-intermediate-variable-without-recomputing-whole-graph/75551/5 "2020-04-06T16:15:22Z")

</div>

If I use `out` to compute `loss_d`, then x1 won’t be detached and Loss2 will have gradients at the parameters that produced x1 (which I want to avoid).

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [April 6, 2020, 5:58pm UTC](https://discuss.pytorch.org/t/detach-a-intermediate-variable-without-recomputing-whole-graph/75551/6 "2020-04-06T17:58:10Z")

</div>

Hi,

You would need two completely different graphs here I’m afraid. So you won’t be able to do that without re-doing the forward.