Distributed Data Parallel slower than Data Parallel

[mrshenli (Shen Li)]

The code snippet in this comment can serve as an example. Search for torch.cuda.Event.