Is there a way to copy the values (data and grad) of a tensor to another in a different device?
x = torch.ones(2, 2, device='cuda:0')
y = torch.zeros(2, 2, device='cuda:1')
# something like
And what is the implication of this on this on the grad_fn?
You can move a Tensor to a specific device by doing
x_cuda1 = x.to("cuda:1").
This is a differentiable operation. So if
x requires gradient, then any op on
x_cuda1 will contribute to the gradients toward
For the .grad field, you can do
x_cuda1.grad = x.grad.to("cuda:1").
@albanD say I want to use y on a different compute graph. And I only want to copy the values. Can that be accommodated?
You can use
.detach() to break the dependency between the two.
x_cuda1 = x.detach().to("cuda:1").