# Multiprocessing for-loop on CPU

**URL:** <https://discuss.pytorch.org/t/multiprocessing-for-loop-on-cpu/59836>\
**Category:** autograd\
**Created:** [November 2, 2019, 2:11pm UTC](https://discuss.pytorch.org/t/multiprocessing-for-loop-on-cpu/59836 "2019-11-02T14:11:53Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Knievel](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/9d8465/32.png) [@Knievel](https://discuss.pytorch.org/u/Knievel)\
**Post date:** [November 2, 2019, 2:11pm UTC](https://discuss.pytorch.org/t/multiprocessing-for-loop-on-cpu/59836/1 "2019-11-02T14:11:53Z")

</div>

I try to speed up a multidirectional RNN with `torch.multiprocessing` as I don’t get it to efficiently run on the GPU, but I have access to a lot of CPUs on a cluster. The one point where I want to apply multiprocessing is the for-loop over the different directions, as they are computed completely independent and only after all iterations are finished, the results are combined in a final layer.

So instead of

```auto
results = []
for d in directions:
    results.append(self.forward_direction(batch, param1, param2))

```

i tried doing

```auto
pool = mp.Pool(processes=self.num_directions)
results = pool.starmap(self.forward_direction, (batch, param1, param2))

```

This results in  
`MaybeEncodingError: Error sending result: '[tensor([...], grad_fn=<StackBackward>)]'. Reason: 'RuntimeError('Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries. If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).',)'`

Did not find a solution yet, is it even possible to multiprocess the for-loop without breaking autograd?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [November 2, 2019, 5:08pm UTC](https://discuss.pytorch.org/t/multiprocessing-for-loop-on-cpu/59836/2 "2019-11-02T17:08:04Z")

</div>

Hi,

You can look at the DistributedDataParallel module to perform distributed training.  
Otherwise, we very recently added to master support for autograd with the distributed package. (not the multiprocessing package, the distributed package).

---

<div class="post-metadata">

**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\
**Post date:** [February 16, 2021, 3:28am UTC](https://discuss.pytorch.org/t/multiprocessing-for-loop-on-cpu/59836/3 "2021-02-16T03:28:00Z")

</div>

is that the answer even to the example I posted here: [How to parallelize a loop over the samples of a batch - #7 by Brando\_Miranda](https://discuss.pytorch.org/t/how-to-parallelize-a-loop-over-the-samples-of-a-batch/32698/7) ? @albanD

* * *

related links from research:

- [Multiprocessing for-loop on CPU](https://discuss.pytorch.org/t/multiprocessing-for-loop-on-cpu/59836)
- [How to use multiprocessing in PyTorch? - Stack Overflow](https://stackoverflow.com/questions/56174874/how-to-use-multiprocessing-in-pytorch)
- [How to parallelize a loop over the samples of a batch - #7 by Brando\_Miranda](https://discuss.pytorch.org/t/how-to-parallelize-a-loop-over-the-samples-of-a-batch/32698/7)
- [machine learning - How to parallelize a training loop ever samples of a batch when CPU is only available in pytorch? - Stack Overflow](https://stackoverflow.com/questions/66226135/how-to-parallelize-a-training-loop-ever-samples-of-a-batch-when-cpu-is-only-avai)