Average loss in DP and DDP

[mrshenli (Shen Li)]

@TT_YY

In my understanding, the gradient is a vector that points a direction where the loss increases the most. I learned from your explanation that we don’t have the “total” loss until we “gather”, “all_gather”, or “all_reduce” the loss computed in each GPU. If we use a loss in each process instead of total loss to compute each gradient and average all the gradients, will it be a correct “total” gradient of the total loss?

Good question. Instead of communicating loss, DDP communicates gradients. So the loss is local to every process, but after the backward pass, the gradient is globally averaged, so that all processes will see the same gradient. This is brief explanation, and this is a full paper describing the algorithm.

If it is not correct, I think it means that we need to do all_reduce of the loss before we do loss.backward in order to hand total loss information to each process for computing correct gradients. Is my thinking correct?

The reason we didn’t communicating loss is because that’s not sufficient. When computing gradients, we need both loss and activation, and the activation depends on local inputs. So we need to either communicate loss + activation or gradients. DDP does the later.