Access data of a GPU process from main

mrshenli · September 15, 2020, 3:18am

This is true, because Python global vars are per-process concept.

Does everybody using DDP plot the loss charts in a spawned process on GPU?

This can be done using torch.multiprocessing.SimpleQueue. E.g., let the main process create the queue, pass it to the child process, and then let the child process put the loss object to the queue. Then, the main process should be able to see that.

The test below can serve as an example:

github.com

pytorch/pytorch/blob/2c4b4aa81bc8dba8272e9c7190edcaa3e114ec15/test/test_multiprocessing.py#L580-L600


      
          def test_event_multiprocess(self):
              event = torch.cuda.Event(enable_timing=False, interprocess=True)
              self.assertTrue(event.query())
          
              ctx = mp.get_context('spawn')
              p2c = ctx.SimpleQueue()
              c2p = ctx.SimpleQueue()
              p = ctx.Process(
                  target=TestMultiprocessing._test_event_multiprocess_child,
                  args=(event, p2c, c2p))
              p.start()
          
              c2p.get()  # wait for until child process is ready
              torch.cuda._sleep(50000000)  # spin for about 50 ms
              event.record()
              p2c.put(0)  # notify child event is recorded
          
              self.assertFalse(event.query())
              c2p.get()  # wait for synchronization in child
              self.assertTrue(event.query())
              p.join()