For some reason, I’m getting this warning:
[W ProcessGroupNCCL.cpp:1569] Rank 6 using best-guess GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.
This is triggered on
The problem is that the code sometimes really hang!
I was wondering if there is an explanation to what’s going on and optimally a solution to this issue.
Thank you very much.