I’ve tried upgraded NCCL in the system, but not sure how to let the Pytorch use the latest version of the NCCL.
You could use the system NCCL installation via:
NCCL_INCLUDE_DIR="/usr/include/" NCCL_LIB_DIR="/usr/lib/" USE_SYSTEM_NCCL=1 python setup.py install
where the env vars point to the folders containing the corresponding NCCL files.