ProcessGroupNCCL can not find GPUs

1

Dear All,

I run into
RuntimeError: ProcessGroupNCCL is only supported with GPUs, no GPUs found!
But as I check pytorch, it shows I could found my two GPUs. Please advice. Thank you

>>> import torch
>>> print(torch.__version__)
2.0.0+cu117
>>> print(torch.cuda.device_count())
2
2

Are you setting CUDA_VISIBLE_DEVICES to an invalid value, which would mask the available devices?

1 Like