# Best practice for uneven dataset sizes with DistributedDataParallel

**URL:** <https://discuss.pytorch.org/t/best-practice-for-uneven-dataset-sizes-with-distributeddataparallel/67308>\
**Category:** distributed\
**Created:** [January 21, 2020, 9:04pm UTC](https://discuss.pytorch.org/t/best-practice-for-uneven-dataset-sizes-with-distributeddataparallel/67308 "2020-01-21T21:04:21Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![mrshenli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/32/12220_2.png) [@mrshenli](https://discuss.pytorch.org/u/mrshenli)\
**Post date:** [January 27, 2020, 7:17pm UTC](https://discuss.pytorch.org/t/best-practice-for-uneven-dataset-sizes-with-distributeddataparallel/67308/2 "2020-01-27T19:17:51Z")

</div>

I saw people doing option 1.

People reporting this issue was usually because applications do not know how many batches each process will take prior to training. It seems in your case, you deterministically know what processes will take one more batch? In that case, I think we might be able to do better. For example,

option 1. randomly skipping one batch in each of the processes that takes one more input batch  
option 2. using no\_sync on the **first** batch in each of the processes that takes one more input batch. no\_sync won’t lead to parameter disparities, it will just accumulate the grad in param.grad. As long as you don’t run `optimizer.step()` in that iteration, it should be fine. The next forward-backward pass out of the no\_sync context will accumulate more grad to param.grad and consume them together.

---

_[View the full topic](https://discuss.pytorch.org/t/best-practice-for-uneven-dataset-sizes-with-distributeddataparallel/67308)._