Training on Multiple GPUs

1

Every time I used DataParallel to run my code on multiple GPUs, I feel it so difficult to make it work. Recently I came across this error:

all tensor must be on devices[0]

I tried hours to make it work but I couldn’t, please help me out.

What I did:

  • os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
  • os.environ["CUDA_VISIBLE_DEVICES"] = "3, 6"
  • I moved all input_tensor and model to cuda:0 with input_tensor.to("cuda:0") and model.to("cuda:0")
  • model = torch.nn.DataParallel(model)
  • loss.mean().backward()

I tried:

  • os.environ["CUDA_VISIBLE_DEVICES"] = "0, 6"
  • input_tensor.to("cuda:6") and model.to("cuda:6")…nothing worked.

What am I doing wrong?

As my server cuda:0 is always busy, I like to run run my code on other gpus like cuda:3 and cuda:6 efficiently. Any help highly appreciated.