# No\_grad vs detach

**URL:** <https://discuss.pytorch.org/t/no-grad-vs-detach/56890>\
**Category:** Uncategorized\
**Created:** [September 26, 2019, 11:07am UTC](https://discuss.pytorch.org/t/no-grad-vs-detach/56890 "2019-09-26T11:07:25Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![palimboa](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/palimboa/32/5883_2.png) [@palimboa](https://discuss.pytorch.org/u/palimboa)\
**Post date:** [September 26, 2019, 11:07am UTC](https://discuss.pytorch.org/t/no-grad-vs-detach/56890/1 "2019-09-26T11:07:25Z")

</div>

Hi,

I have read this thread here carefully: [question](https://discuss.pytorch.org/t/detach-no-grad-and-requires-grad/16915/18) but I am still confused about it.

Let’s say we have 3 operations:

```auto
x1 = input
x2 = op1(x1)
x3 = op2(x2)
x4 = op3(x3)
loss = lossfunction(x4)

```

What is the difference between:

```auto

x1 = input
x2 = op1(x1)
with torch.no_grad()
    x3 = op2(x2)
x4 = op3(x3)
loss = lossfunction(x4)

```

and (using detach)

```auto
x1 = input
x2 = op1(x1)
x3 = op2(x2).detach()
x4 = op3(x3)
loss = lossfunction(x4)

```

Assuming all operations have learnable parameters, how does the gradient propagation, memory management and activations change?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [September 26, 2019, 2:01pm UTC](https://discuss.pytorch.org/t/no-grad-vs-detach/56890/2 "2019-09-26T14:01:44Z")

</div>

Hi,

In this particular case, they will be quite similar.  
The only difference I guess is that in the first case, you will never even instantiate the state in op2. While you will in the second case and discard them later.  
If you do inplace operations in op2 though, there will be some difference.