# Multiprocessing for multiple gpus **URL:** \ **Category:** Uncategorized\ **Created:** [April 5, 2018, 5:35am UTC](https://discuss.pytorch.org/t/multiprocessing-for-multiple-gpus/15985 "2018-04-05T05:35:30Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![rahul](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/f14d63/32.png) [@rahul](https://discuss.pytorch.org/u/rahul)\ **Post date:** [April 5, 2018, 5:35am UTC](https://discuss.pytorch.org/t/multiprocessing-for-multiple-gpus/15985/1 "2018-04-05T05:35:30Z")
I was wondering why is it not advised to use multiple GPUs using muliprocesing? As an example, [http://pytorch.org/docs/master/notes/cuda.html](http://pytorch.org/docs/master/notes/cuda.html) towards the end you have advise “Use nn.DataParallel instead of multiprocessing” While there is an example to use multiple GPUs using multiprocessing [http://pytorch.org/tutorials/intermediate/dist\_tuto.html](http://pytorch.org/tutorials/intermediate/dist_tuto.html) . I am confused as to which is the correct rule? That seems contradictory? ---
**Author:** ![haolibai](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/f4b2a3/32.png) [@haolibai](https://discuss.pytorch.org/u/haolibai)\ **Post date:** [April 21, 2019, 10:21am UTC](https://discuss.pytorch.org/t/multiprocessing-for-multiple-gpus/15985/2 "2019-04-21T10:21:40Z")
I have the same question with you. ---
**Author:** ![rwightman](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/b9e5f3/32.png) [@rwightman](https://discuss.pytorch.org/u/rwightman)\ **Post date:** [April 21, 2019, 11:36pm UTC](https://discuss.pytorch.org/t/multiprocessing-for-multiple-gpus/15985/3 "2019-04-21T23:36:57Z")
I can’t speak to the specifics of the guidelines here. In my own usage, DataParallel is the quick and easy way to get going with multiple GPUs on a single machine. However, if you want to push the performance, I’ve found that using the NVIDIA apex implementation of DistributedDataParallel with one GPU per process and a few of their other opts better saturates the GPUs on a single machine and usually results in approx 10-15% higher throughput. I can’t speak to how apex DDP compares to the PyTorch native impl, I switched to apex because I’m also using AMP mixed precision from time to time. Their example is a good sample: [https://github.com/NVIDIA/apex/blob/master/examples/imagenet/main\_amp.py](https://github.com/NVIDIA/apex/blob/master/examples/imagenet/main_amp.py) ---
**Author:** ![rwightman](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/b9e5f3/32.png) [@rwightman](https://discuss.pytorch.org/u/rwightman)\ **Post date:** [April 22, 2019, 9:51pm UTC](https://discuss.pytorch.org/t/multiprocessing-for-multiple-gpus/15985/4 "2019-04-22T21:51:41Z")
For curiosity’s sake, I ran a quick test on a machine that I recently bumped up to 3 pascal GPU. Previous comparison was made with 2 x RTX cards. For imagenet style training @ 224x224, smaller model (something like a mnasnet/mobilenetv2), 8 physical core CPU: 830 img/sec avg - single training process, 3 GPU, torch.nn.DataParallel, 8 (or 9 for fairness) worker processes 1015 img/sec avg - 3 training process, 1 GPU per-proc, apex.DistributedDataParallel, 3 workers per training process Everything else in those two runs is the same, same preprocessing, using the same ‘fast’ preload + collation routines from Nvidia’s examples. So looks like throwing in another GPU increases the impact to \> 20%.