When I execute the code example of mnist_hogwild, I find that multiple processes are running parallelly on one gpu.
Question: Can multiple processes be executed in parallel on multiple GPUs?
Yes, that should be possible. Just move/copy the model to different devices. Did you encounter any error when doing that?