Is it possible to create a shared memory / tensors among openmpi spawned process group? I know this can be done with the processes created by torch.multiprocessing package. But in my case I am unable to make this package work with openmpi processes.
I’d like to know the answer for this too.
One solution might be
- create a shared memory buffer using MPI
- create a numpy
ndarrayfrom that shared memory (example)
- create a PyTorch tensor from that numpy
The numpy ndarray and the tensor should then share the same storage, which is on the shared memory.
I didn’t say it, but I meant sharing cuda tensors.
(e.g: doing something like Hogwild example- but having the model on GPU instead of CPU and share its parameters with 2 MPI process on the same node)
(For CPU tensors, your answer is correct, I read once a blog post demonstrating it, and all we need to modify is using shared MPI mem instead).
However, my 2 cents on this: it works out of the box only for host shared memory. Somehow couldn’t create shared+pinned host memory with pytorch easily. (I managed to do it with ugly hack: os.fork. I didn’t bother to make it work for MPI too)