How to use DataParallel in Pytorch while using parts of model?

[royboy (Roy Li)]

I’m unsure what exactly you’re trying to do, are you trying to run different parts of your model on different GPU? If so, @ailzhang’s answer here might be relevant.

How to implement distributed model parallel

Hi @Ryobot, for distributed model parallel across different machines, it requires inter-node communication, right now we don’t have good support for it like DistributedDataParallel wrapper. For the single node multiple GPU model parallel( like DataParallel wrapper), yes parallel_apply should work in your case.

If not, can you describe what you’re trying to do in more detail?