Training fast with small dataset, slow with large dataset

[mrshenli (Shen Li)]

Hey @vibe2

I have some questions regarding the perf measurements.

1. Have you profiled which step is the bottleneck in every iteration? Is it data loading, or forward, or backward, or optimizer?
2. For GPU training, time.time() might not give the accurate measure, as CUDA ops return immediately after it is added to the stream. To get more accurate numbers, elapsed_time is a better option. The following post can serve as an example:

Distributed Data Parallel slower than Data Parallel

Hey @TT_YY, I took a closer look at the code and noticed that you converted BatchNorm to SyncBatchNorm for DDP, which might be the source of the slowness. If you look at SyncBatchNorm's implementation (see below), it launches its own communication, which is not handled by DDP. This additional comm leads to ~10% slowdown in your program when running on 2 GPUs. When I use BatchNorm instead of SyncBatchNorm, DDP is faster than DP. In general, when comparing DDP and DP speed, we need to make sure th…