Since gpu is asynchronous in Pytorch. Can I regard it as a process parallel?
depends on how you use it.
you can generate different process and each run on a GPU
or you can run all the GPUs on one process. but different GPU has different steam. They are parallel with each other. by default, each GPU has one default stream.