Copying a Variable to a different GPU

autograd
1

I’m probably missing something obvious, but how can I copy a Variable from one GPU to another?

As a concrete example, suppose I have a Variable x on GPU 0 and I want to copy it to a new Variable xx on GPU 1 such that gradients will properly flow from xx back to x. I’ve tried the following (on PyTorch 0.2.0_1 installed from binary to a virtualenv):

Attempt #1:

x = Variable(torch.randn(3, 4).cuda())
xx = x.cuda(device=1)
# TypeError: cuda() got an unexpected keyword argument 'device'

Attempt #2:

x = Variable(torch.randn(3, 4).cuda())
with torch.cuda.device(1):
  xx = x.clone()
# xx.data.get_device() is 0

Attempt #3:

x = Variable(torch.randn(3, 4).cuda())
xx = Variable(x.data.cuda(device=1))
# xx is on GPU 1, but the computational graph is broken
# and gradients don't flow from xx to x

Is there some simple way to do this that I’ve overlooked?

1 Like
2

Looking at torch.autograd.variable, you might try device_id in Attempt #1:

x = Variable(torch.randn(3, 4).cuda())
xx = x.cuda(device_id=1)
# TypeError: cuda() got an unexpected keyword argument 'device'

Best regards

Thomas

P.S.: Thank you for the instructive material you publish, I have benefited greatly from it.

3

Thanks for opening the issue.
As pointed out by @tom, there seems to be an incompatibility between Tensor and Variable, and this is not a desired behavior. I’ve opened an issue in https://github.com/pytorch/pytorch/issues/2685.
Sorry for the trouble.

4

Perfect, thanks @tom and @fmassa!