Shared Memory with mpi-backend

distributed
1

Is it possible to create a shared memory / tensors among openmpi spawned process group? I know this can be done with the processes created by torch.multiprocessing package. But in my case I am unable to make this package work with openmpi processes.

1 Like
2

I’d like to know the answer for this too.

3

One solution might be

  1. create a shared memory buffer using MPI
  2. create a numpy ndarray from that shared memory (example)
  3. create a PyTorch tensor from that numpy ndarray (example)

The numpy ndarray and the tensor should then share the same storage, which is on the shared memory.

1 Like
4

I didn’t say it, but I meant sharing cuda tensors.
(e.g: doing something like Hogwild example- but having the model on GPU instead of CPU and share its parameters with 2 MPI process on the same node)

(For CPU tensors, your answer is correct, I read once a blog post demonstrating it, and all we need to modify is using shared MPI mem instead).
However, my 2 cents on this: it works out of the box only for host shared memory. Somehow couldn’t create shared+pinned host memory with pytorch easily. (I managed to do it with ugly hack: os.fork. I didn’t bother to make it work for MPI too)