# Multiprocessing inside forward on single GPU

**URL:** <https://discuss.pytorch.org/t/multiprocessing-inside-forward-on-single-gpu/88093>\
**Category:** Uncategorized\
**Created:** [July 6, 2020, 10:50am UTC](https://discuss.pytorch.org/t/multiprocessing-inside-forward-on-single-gpu/88093 "2020-07-06T10:50:12Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gaoyuan\_Yang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gaoyuan_yang/32/13209_2.png) [@Gaoyuan\_Yang](https://discuss.pytorch.org/u/Gaoyuan_Yang)\
**Post date:** [July 6, 2020, 10:50am UTC](https://discuss.pytorch.org/t/multiprocessing-inside-forward-on-single-gpu/88093/1 "2020-07-06T10:50:12Z")

</div>

Hi, I want to run two lines in parallel inside forward function on single GPU.  
The two sub-processes are independent from each other. I want to run self.l1 and self.l2 simultaneously here. The minimum code is as follows:

```python
import torch
import torch.nn as nn
import torch.multiprocessing as mp
import time

mp = mp.get_context('spawn')

class Net(nn.Module):
    
    def __init__ (self):
        super(Net, self). __init__ ()
        self.l1 = nn.Linear(784, 100)
        self.l2 = nn.Linear(784, 100)

    def forward(self, x):

        processes = []
        num_processes = 2

        for i in range(num_processes):
            if i == 0:
                p = mp.Process(target=self.l1(x))
            else:
                p = mp.Process(target=self.l2(x))
            p.start()
            processes.append(p)
        for p in processes:
            p.join()

       

  
if __name__ == ' __main__':
    BATCH_SIZE = 64
    x = torch.randn(BATCH_SIZE, 784).cuda()
    mynet = Net().cuda() # ; torch.cuda.synchronize()
    mynet(x)

```

I found similar question: [for loop inside single GPU](https://discuss.pytorch.org/t/parallelize-simple-for-loop-for-single-gpu/33701) and [for loop inside forward](https://discuss.pytorch.org/t/how-to-write-parallel-for-loop-in-models-forward-method/51652).  
There are two relevant questions : [parallel over samples](https://discuss.pytorch.org/t/how-to-parallelize-a-loop-over-the-samples-of-a-batch/32698) and [parallel execution](https://discuss.pytorch.org/t/parallel-execution-of-modules-in-nn-modulelist/43940), the answers to these two questions are reformulated version of the original problem.  
Besides, I tried the nn.Parallel(block1, block2) inside this post [two blocks in parallel](https://discuss.pytorch.org/t/how-can-l-run-two-blocks-in-parallel/61618) however it did not work.  
I know distribute model to multiple GPU is well documented, however I have only one GPU here.  
multiprocessing over model level is documented here [multiprocessing over model level](https://pytorch.org/docs/stable/notes/multiprocessing.html)

Any help would be appreciated, thanks a lot!

---

<div class="post-metadata">

**Author:** ![YOGESH\_SANKU](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yogesh_sanku/32/42261_2.png) [@YOGESH\_SANKU](https://discuss.pytorch.org/u/YOGESH_SANKU)\
**Post date:** [January 9, 2022, 12:23pm UTC](https://discuss.pytorch.org/t/multiprocessing-inside-forward-on-single-gpu/88093/2 "2022-01-09T12:23:34Z")

</div>

Hello, Did u solve the problem

---

<div class="post-metadata">

**Author:** ![smell\_well](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smell_well/32/49922_2.png) [@smell\_well](https://discuss.pytorch.org/u/smell_well)\
**Post date:** [May 25, 2022, 2:59am UTC](https://discuss.pytorch.org/t/multiprocessing-inside-forward-on-single-gpu/88093/3 "2022-05-25T02:59:55Z")

</div>

hi, I have same problem, did you find a method to deal with it?