# Debugging DataParallel, no speedup and uneven memory allocation

**URL:** <https://discuss.pytorch.org/t/debugging-dataparallel-no-speedup-and-uneven-memory-allocation/1100>\
**Category:** Uncategorized\
**Created:** [March 16, 2017, 3:50am UTC](https://discuss.pytorch.org/t/debugging-dataparallel-no-speedup-and-uneven-memory-allocation/1100 "2017-03-16T03:50:08Z")\
**Posts on this page:** 1\
**Showing post:** 14

<div class="post-metadata">

**Author:** ![smth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/32/13_2.png) [@smth](https://discuss.pytorch.org/u/smth)\
**Post date:** [March 16, 2017, 7:55pm UTC](https://discuss.pytorch.org/t/debugging-dataparallel-no-speedup-and-uneven-memory-allocation/1100/14 "2017-03-16T19:55:57Z")

</div>

in the backward pass of DataParallel, we reduce the weights from GPU2 onto GPU1.

Our DataParallel algorithm is roughly like this:

in forward:

- scatter mini-batch to GPU1, GPU2
- replicate model on GPU2 (it is already on GPU1)
- model\_gpu1(input\_gpu1), model\_gpu2(input\_gpu2) (this step is parallel\_apply)
- gather output mini-batch from GPU1, GPU2 onto GPU1

in backward:

- scatter grad\_output and input
- parallel\_apply model’s backward pass
- reduce GPU2 replica’s gradients onto GPU1 model
- 
## Now there is only a single model again with accumulated gradients from GPU1 and GPU2
- gather the grad\_input

Hence, unlike in Chainer, you do not actually have to have a separate trainer that is aware of DataParallel.  
Hope this makes it clear.

wrt why your model is slower via DataParallel, you have 61 million parameters. So, I presume you have some `Linear` layers at the end (i.e. fully connected layers). Put them outside the purview of `DataParallel` to avoid having to distribute / reduce those parameter weights and gradients. Here is an example of doing that:

`https://github.com/pytorch/examples/blob/master/imagenet/main.py#L68`  
`https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py`

When training AlexNet or VGG, we only put `model.features` in DataParallel, and not the whole model itself, because AlexNet and VGG have large `Linear` layers at the end of the network.

Maybe your situation is similar?

---

_[View the full topic](https://discuss.pytorch.org/t/debugging-dataparallel-no-speedup-and-uneven-memory-allocation/1100)._