Model parallelism in Multi-GPUs: forward/backward graph

autograd
6

Yes exactly. At least that’s what I’ve used it for.
The transfer between both GPUs is done via P2P so that no host communication is needed as far as I know.

2 Likes