You can manually keep half of your model on gpu 1 and half model in gpu 0. You’ll also correspondingly have to update gpu id of tensors corresponding to part of model it is going into.
Ideally you would like to divide model reoughly sequentially into halfs and set them on multiple gpus, and during forward change data from gpuid 0 to 1 or vice versa depending on how your model is located.
This is called model parallelism and some better explaining could be found here .