# Tracking Loss through Parallel Paths

**URL:** <https://discuss.pytorch.org/t/tracking-loss-through-parallel-paths/64018>\
**Category:** Uncategorized\
**Created:** [December 14, 2019, 12:54am UTC](https://discuss.pytorch.org/t/tracking-loss-through-parallel-paths/64018 "2019-12-14T00:54:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joshua\_Clancy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joshua_clancy/32/16023_2.png) [@Joshua\_Clancy](https://discuss.pytorch.org/u/Joshua_Clancy)\
**Post date:** [December 14, 2019, 12:54am UTC](https://discuss.pytorch.org/t/tracking-loss-through-parallel-paths/64018/1 "2019-12-14T00:54:43Z")

</div>

Hello!

I am a bit of a noob here so forgive me if this has a simple answer. For context, I successively implemented a tied weight VAE a few months ago and now I am testing out some other ideas to do with autoencoders. My current model has three parallel paths going through it in an attempt to make three specialized and separate latent spaces.

 ![image](https://discuss.pytorch.org/uploads/default/original/3X/4/2/42acfdcc3635b169852fdd1dd09d6d6d13fad484.png)

I am experimenting with how to specialize the latent spaces, but that is not my concern here. This question is specifically about the loss. Right now I have a list of losses (one for each path) and I loop through the losses performing backpropagation for each.

```auto
for loss in local_loss_list:
                lossy = loss
                # backpropagation
                optimizer.zero_grad()
                lossy.backward(retain_graph=True)
                # one step of the optmizer (using the gradients from backpropagation)               
                optimizer.step()
                total_loss = total_loss + lossy

```

I did this automatically without much thought, but now I am concerned. Will the loss from each parallel path backpropagate properly down the correct path? If so how? It seems to work, but I do not understand how the loss is being associated with the correct path of weights.

Thanks!!!

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [December 14, 2019, 1:14am UTC](https://discuss.pytorch.org/t/tracking-loss-through-parallel-paths/64018/2 "2019-12-14T01:14:33Z")

</div>

Hi,

The autograd computes the gradient for what you give it. So if you backprop for a loss computed with only one branch, it will backprop only that branch.

Note that if you just want to get the sum of the gradients and do a single update, you can do:

```auto
lossy = sum(local_loss_list)
# backpropagation
optimizer.zero_grad()
lossy.backward()
# one step of the optmizer (using the gradients from backpropagation)               
optimizer.step()
total_loss = total_loss + lossy

```

---

<div class="post-metadata">

**Author:** ![Joshua\_Clancy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joshua_clancy/32/16023_2.png) [@Joshua\_Clancy](https://discuss.pytorch.org/u/Joshua_Clancy)\
**Post date:** [December 14, 2019, 6:05am UTC](https://discuss.pytorch.org/t/tracking-loss-through-parallel-paths/64018/3 "2019-12-14T06:05:17Z")

</div>

> [@albanD](#):
>
> hat if you just want to get the sum of the gradients and do a single update

Yeah now that you mention it I think I have read that before. Thank you. So to be clear, keeping the parallel path’s losses separate is not actually doing anything because autograd’s computational graph keeps track of which weights are effecting the loss. So I can then just add them up and update them based on that? Is there any scenario where I should be worried about autograd not assigning the gradient to the correct weightings that are effecting the loss? Or does it just do all of that for you?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [December 16, 2019, 10:43am UTC](https://discuss.pytorch.org/t/tracking-loss-through-parallel-paths/64018/4 "2019-12-16T10:43:38Z")

</div>

The autograd is a tool that just computes gradients. So it will set the gradients only for what is used for each loss.