Hi @Ryobot, for distributed model parallel across different machines, it requires inter-node communication, right now we don’t have good support for it like DistributedDataParallel wrapper.
For the single node multiple GPU model parallel( like DataParallel wrapper), yes parallel_apply should work in your case.