# Gradient descent back propagating for intermediate loss function

**URL:** <https://discuss.pytorch.org/t/gradient-descent-back-propagating-for-intermediate-loss-function/8624>\
**Category:** autograd\
**Created:** [October 12, 2017, 4:02pm UTC](https://discuss.pytorch.org/t/gradient-descent-back-propagating-for-intermediate-loss-function/8624 "2017-10-12T16:02:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![zack.zcy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zack.zcy/32/2267_2.png) [@zack.zcy](https://discuss.pytorch.org/u/zack.zcy)\
**Post date:** [October 12, 2017, 4:02pm UTC](https://discuss.pytorch.org/t/gradient-descent-back-propagating-for-intermediate-loss-function/8624/1 "2017-10-12T16:02:55Z")

</div>

Hi there, recently, I’m writing a customized loss function, I’m not sure about whether propagating gradient through the whole network or just part of it.  
Specifically, I have an autoEncoder model, my loss function consists of 2 separate loss, where 1st part is taking the output of autoEncoder as input, whereas 2nd part is taking the intermediate output of the encoder network(without the decoder network) as its input. My question is that should I backprop gradient of 2nd loss through the decoder network?

```
class autoEncoder(nn.Module):
    def __init__ (self):
        super(autoEncoder,self). __init__ ()
    def forward(self,x):
        h = encoder(x)
        x = decoder(h)
        return x

```

For simplicity, encoder and decoder are 2 classes implementing the encoder network and decoder network respectively. with that defined, my loss function is formalized as follows:

```
customized_loss = loss_1(x) + loss_2(h) # loss_1 and loss_2 are torch.autograd.variable.Variable 

```

Should I write gradient descent as follows ?

```
grad_params = torch.autograd.grad(loss_1,autoEncoder.parameters(),create_graph=True)
grad_params[-2].add_(torch.autograd.grad(loss_2,encoder.parameters(),create_graph=True))
torch.optim.SGD(grad_params).step()

```

or computes loss2 w.r.t he decoder network and update the whole network

```
customized_loss.backward()
torch.optim.SGD().step()
```

---

<div class="post-metadata">

**Author:** ![zack.zcy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zack.zcy/32/2267_2.png) [@zack.zcy](https://discuss.pytorch.org/u/zack.zcy)\
**Post date:** [October 13, 2017, 2:27pm UTC](https://discuss.pytorch.org/t/gradient-descent-back-propagating-for-intermediate-loss-function/8624/2 "2017-10-13T14:27:13Z")

</div>

Will pytorch automatically build a graph that traces the loss2 variable?

---

<div class="post-metadata">

**Author:** ![smth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/32/13_2.png) [@smth](https://discuss.pytorch.org/u/smth)\
**Post date:** [October 14, 2017, 8:34am UTC](https://discuss.pytorch.org/t/gradient-descent-back-propagating-for-intermediate-loss-function/8624/3 "2017-10-14T08:34:09Z")

</div>

you can simply do `customized_loss.backward()` and gradients wrt both loss1 and loss2 are propagated back to the parameters.

---

<div class="post-metadata">

**Author:** ![zack.zcy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zack.zcy/32/2267_2.png) [@zack.zcy](https://discuss.pytorch.org/u/zack.zcy)\
**Post date:** [October 18, 2017, 8:36am UTC](https://discuss.pytorch.org/t/gradient-descent-back-propagating-for-intermediate-loss-function/8624/4 "2017-10-18T08:36:07Z")

</div>

thanks dude, wonderful framework!

---

<div class="post-metadata">

**Author:** ![Shisho\_Sama](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shisho_sama/32/6926_2.png) [@Shisho\_Sama](https://discuss.pytorch.org/u/Shisho_Sama)\
**Post date:** [September 5, 2019, 4:21am UTC](https://discuss.pytorch.org/t/gradient-descent-back-propagating-for-intermediate-loss-function/8624/6 "2019-09-05T04:21:48Z")

</div>

Did you get your answer?