How to change the default device of GPU? device_ids[0]

That controls what devices CUDA exposes and PyTorch can’t do nothing in this regards.