Split single model in multiple gpus

[Aleks1]

Hi Peter,

I am also trying to share my model on different GPUs. But I am not sure if it’s really efficient. Since I am using over 6000 custom layers I execute them in a for loop. All layers are independent from each other and can run in parallel. Here is my original forum question. Maybe you can help?

Sending Layers of nn.ModuleList() and tensors to GPU

Hi, I am trying to run over 6000 custom layers initialized with nn.ModuleList() on GPUs. I can use 80-120 GPUs on cluster. What is the best way to distribute the layers on the GPU? Should I do it in init method or in forward? Do the layers run asynchronously when I execute them in a for loop? I don’t know how to execute over 6000 layers more efficiently with pytorch. Thank you very much for your help