Split single model in multiple gpus

[CindGao]

Got it, thank you!

BTW, I facing the similar problem as this discussion, the GPU memory on the fisrt GPU cost much more than it should be.

Model parallelism in pytorch for large(r than 1 GPU) models?

@apaszke Hi, very thanks for your examples. I notice that when I split the whole model in 4 gpus and do forward/backward, the GPU memory on the fisrt GPU cost much more than it should be. For example, if the whole model cost 12GB on a single GPU, when split it to four GPUs, the first GPU cost 11GB and the sum of others cost about 11GB. Is there an explaination for how does the GPU memory be malloced when using multiple GPUs for model parallelism. Another question, when forward with the mode…

I can’t figure out what wrong with it. Do you know where may be the problem?
Thank you!