# How to perform backward pass only on sample with minimum loss and improve performance?

**URL:** <https://discuss.pytorch.org/t/how-to-perform-backward-pass-only-on-sample-with-minimum-loss-and-improve-performance/35960>\
**Category:** autograd\
**Created:** [January 30, 2019, 7:06am UTC](https://discuss.pytorch.org/t/how-to-perform-backward-pass-only-on-sample-with-minimum-loss-and-improve-performance/35960 "2019-01-30T07:06:25Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dendorferpatrick](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/ea5d25/32.png) [@dendorferpatrick](https://discuss.pytorch.org/u/dendorferpatrick)\
**Post date:** [January 30, 2019, 7:06am UTC](https://discuss.pytorch.org/t/how-to-perform-backward-pass-only-on-sample-with-minimum-loss-and-improve-performance/35960/1 "2019-01-30T07:06:25Z")

</div>

Hi all,  
I am training a GAN for time series prediction. During training, I sample `n_samples` predictions and backpropagate the L2-loss for the best sample.

```
for _ in range(n_samples):
    out = net(x)    
    loss = lossfunc(x,y)
    l2_loss = l2_loss.append(loss)

l2_loss.stack( l2_loss ) 
min_loss = torch.min( l2_loss )
min_loss.backward()

```

When I call `min_loss.backward()`, the time for calculating the backward pass grows linear with `n_samples`. However, the gradient for the non minimum losses should be 0 and not considered for optimisation. I only want to backpropagate the L2-loss for the best sample.  
Therefore, I do not understand why the calculation time increases and does not stay constant? Does `.backward()` also calculate all gradients even if they are not used? Is there a way to make my code more efficient and detach all non minimum losses?  
Thank you for your answer:)

---

<div class="post-metadata">

**Author:** ![rasbt](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rasbt/32/60_2.png) [@rasbt](https://discuss.pytorch.org/u/rasbt)\
**Post date:** [January 30, 2019, 7:04pm UTC](https://discuss.pytorch.org/t/how-to-perform-backward-pass-only-on-sample-with-minimum-loss-and-improve-performance/35960/2 "2019-01-30T19:04:42Z")

</div>

As far as I know backward computes the gradients all the way back to the leave nodes. I mean, there’s no way for `backward` knowing which particular gradients you are interested in and when. Maybe the simplest way for computing the gradients you are interested in is doing it semi-automatically via the `grad` function:

```
from torch.autograd import grad

d_a_b = grad(a, b, retain_graph=True)
d_a_c = grad(a, c, retain_graph=True)
...

```

Then use `d_a_b` etc. to update the weights

---

<div class="post-metadata">

**Author:** ![lmnt](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/7ab992/32.png) [@lmnt](https://discuss.pytorch.org/u/lmnt)\
**Post date:** [January 30, 2019, 9:08pm UTC](https://discuss.pytorch.org/t/how-to-perform-backward-pass-only-on-sample-with-minimum-loss-and-improve-performance/35960/3 "2019-01-30T21:08:06Z")

</div>

This thread [Disconnected Gradient in Pytorch](https://discuss.pytorch.org/t/disconnected-gradient-in-pytorch/6341/5) might be helpful. It discusses why you would do what @rasbt described and/or detach each time in the loop.

---

<div class="post-metadata">

**Author:** ![vmirly1](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vmirly1/32/3309_2.png) [@vmirly1](https://discuss.pytorch.org/u/vmirly1)\
**Post date:** [January 30, 2019, 10:06pm UTC](https://discuss.pytorch.org/t/how-to-perform-backward-pass-only-on-sample-with-minimum-loss-and-improve-performance/35960/4 "2019-01-30T22:06:12Z")

</div>

How about instead of using the function `torch.min` to obtain `min_loss`, compare the value of the loss and keep the min\_loss in the loop. Then, at the end call `.backward()`on `min_loss` :

```python
min_loss_value= 99.00
for _ in range(n_samples):
    out = net(x)    
    loss = lossfunc(x,y)
    if oss.detach() < min_loss_value:
       min_loss = loss
       min_loss_value = loss.detach()

min_loss.backward()

```

I think this way, you definitely have only one loss to compute its gradients. But I guess when you use `torch.min()` the gradients for others in the list may still be computed. So, basically, autograd computes the gradients on `torch.max` and `torch.min`. For example, if we have a tensor with elements `[1, 2, 3]`, then `torch.max()` will retrieve the max value, but there is a tensor of gradients which `[0, 0, 1]`. These gradient values are like the effect of each element on the output of `torch.max()`.