# Average loss in DP and DDP **URL:** \ **Category:** distributed\ **Created:** [August 19, 2020, 2:55am UTC](https://discuss.pytorch.org/t/average-loss-in-dp-and-ddp/93306 "2020-08-19T02:55:17Z")\ **Posts on this page:** 1\ **Showing post:** 4
**Author:** ![mrshenli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/32/12220_2.png) [@mrshenli](https://discuss.pytorch.org/u/mrshenli)\ **Post date:** [August 20, 2020, 2:15pm UTC](https://discuss.pytorch.org/t/average-loss-in-dp-and-ddp/93306/4 "2020-08-20T14:15:16Z")
> [@TT\_YY](#): > > In my understanding, the gradient is a vector that points a direction where the loss increases the most. I learned from your explanation that we don’t have the “total” loss until we “gather”, “all\_gather”, or “all\_reduce” the loss computed in each GPU. If we use a loss in each process instead of total loss to compute each gradient and average all the gradients, will it be a correct “total” gradient of the total loss? Good question. Instead of communicating loss, DDP communicates gradients. So the loss is local to every process, but after the backward pass, the gradient is globally averaged, so that all processes will see the same gradient. [This](https://pytorch.org/docs/stable/notes/ddp.html) is brief explanation, and [this](https://arxiv.org/pdf/2006.15704.pdf) is a full paper describing the algorithm. > If it is not correct, I think it means that we need to do all\_reduce of the loss before we do loss.backward in order to hand total loss information to each process for computing correct gradients. Is my thinking correct? The reason we didn’t communicating loss is because that’s not sufficient. When computing gradients, we need both loss and activation, and the activation depends on local inputs. So we need to either communicate loss + activation or gradients. DDP does the later. --- _[View the full topic](https://discuss.pytorch.org/t/average-loss-in-dp-and-ddp/93306)._