Copying a tensor from one device to another

Hi,
Is there a way to copy the values (data and grad) of a tensor to another in a different device?

x = torch.ones(2, 2, device='cuda:0')
y = torch.zeros(2, 2, device='cuda:1')

# something like 
x.copy_to(y)

And what is the implication of this on this on the grad_fn?

Hi,

You can move a Tensor to a specific device by doing x_cuda1 = x.to("cuda:1").
This is a differentiable operation. So if x requires gradient, then any op on x_cuda1 will contribute to the gradients toward x.
For the .grad field, you can do x_cuda1.grad = x.grad.to("cuda:1").

2 Likes

@albanD say I want to use y on a different compute graph. And I only want to copy the values. Can that be accommodated?

You can use .detach() to break the dependency between the two.
So x_cuda1 = x.detach().to("cuda:1").