Pytorch, CUDA, and NCCL

1

I’d like to upgrade NCCL on my system to 2.10.3; it supports bfloat16, which I’d like to use. I don’t know the dependency relationships among Pytorch, CUDA, and NCCL. Does Pytorch have bindings to particular versions of NCCL, as suggested by this issue? Can I choose to use a newer version of NCCL without upgrading either Pytorch or CUDA?

2

The PyTorch binaries ship with a statically linked NCCL using the NCCL submodule. The current CUDA11.3 nightly binary uses NCCL 2.10.3 already, so you could use it.
On the other hand, if you want to use a specific NCCL version, which isn’t shipped in a binary release, you could build from source and use your locally installed NCCL via:

NCCL_INCLUDE_DIR="/usr/include/" \
    NCCL_LIB_DIR="/usr/lib/" \
    USE_SYSTEM_NCCL=1 \
    python setup.py install
3

Thanks. Do you know when the current CUDA 11.3 nightly will become official?

4

We are currently targeting PyTorch 1.10.0 as the stable release using the CUDA11.3 runtime.

5

Thank you, ptrblock.