You can move a Tensor to a specific device by doing x_cuda1 = x.to("cuda:1").
This is a differentiable operation. So if x requires gradient, then any op on x_cuda1 will contribute to the gradients toward x.
For the .grad field, you can do x_cuda1.grad = x.grad.to("cuda:1").