I’m training a conv model using DataParallel (DP) and DistributedDataParallel (DDP) modes. For DDP, I only use it on a single node and each process is one GPU.
My model has many BatchNorm2d layers. Given all other things the same, I observe that DP trains better than DDP (in classification accuracy). Even if I add SyncBN from pytorch 1.1, I still observe that DP > DDP+SyncBN > DDP without SyncBN in test accuracy.
If I understand correctly, DP doesn’t do SyncBN, so DP should in theory achieve the same test accuracy as DDP (given small batch size per GPU)? If we assume larger effective batch size leads to better result, I should expect the following test performance ranking:
DDP+SyncBN > DP == DDP
but in practice, I observe: DP > DDP+SyncBN > DDP
Because DDP+SyncBN is 30% faster than DP, I really hope to solve the training gap so that I can take advantage of DDP’s superior speed. Thanks for any help!
Hi Jim,
From docs DistributedDataParallel can be used in the following two ways:
(1) Single-Process Multi-GPU
(2) Multi-Process Single-GPU
Second method the highly recommended way to use DistributedDataParallel, with
multiple processes, each of which operates on a single GPU. This is
currently the fastest approach to do data parallel training using PyTorch
and applies to both single-node(multi-GPU) and multi-node data
parallel training.
By “performance”, I mean the classification accuracy. Somehow DDP+SyncBN achieves worse test accuracy than DP, so there must be some problematic differences in the numeric algorithm. The speed isn’t the issue here. Thanks!
I can only comment on the differences between DP and DDP w.r.t. batch normalization. With DP your module is replicated before each call to forward, which means that only the BN stats from the first replica are kept around. With DDP each process keeps their own version of BN. And with SyncBN you’ll end up with stats that are “more averaged” than the stats kept when using DP, because they only include stats for a batch in a single replica, instead of all replicas.
I found the problem in my code, it’s because of the cudnn batch norm. According to this github issue, the solution is to edit the batchnorm part in torch/nn/functional.py or set torch.backends.cudnn.enabled = False.
Could edit the batchnorm part in torch/nn/functional.py work for sync bn?
The batch norm in torch.nn.functional is used just for evaluation. I think editing this would do nothing to sync batch norm. How do you edit the file to make sync bn work normally?
You are right, although the performance improves after disable cudnn, the gap still remains. I can’t figure out the problem and now I have to use nn.DataParallel .
@Mr.Z Do you find the problem? I also get a very worse accuracy when use SyncBN + DDP for batchsize=16( 4 GPUs on one node, 4 images for each GPU), and when I use DataParallel + SyncBN, evrything is OK.
Same here. Performance of DDP model is weaker than one trained on a single GPU. Playing with lr/bs does not help. As number of GPUs in DPP training grows - performance degrades.
Has anyone found the solution ?
UPDT: the reason was found for my case. When training DDP model we need to use DistributedSampler which is passed to Dataloader. We need to train_dataloader.sampler.set_epoch(epoch) on every epoch start.
Our experience with Kinetics 400 using PyTorch 1.3 on a node with two GPUs as follows:
Single GPU > DP (-0.2%) > DP w/ sync BN (-0.3%)
Single GPU serves as the baseline for DP and DP w/ sync BN.
The tradeoff with distributed training is understandable but sync BN causing worse accuracy is not trivial to ignore.
My setting is same with you, just testing in HMDB51. I also get the results as follows:
DP>DP w/sync BN. Now, Do you find the solution to deal with this issue?