I am trying to build pytorch from source to enable support for MPI in pytorch DDP. I have cloned the source code and switched to the v2.0.0 branch, after which I have updated the submodules.
GCC version: 8.3.0
cuda version is 11.0
I am using intel MPI in my system.
The installation is in a conda environment for a HPC cluster.
I have also attached screenshots of nvidia-smi.
Upon executing all steps as mentioned in the readme, I executed python setup.py develop after which I get the following errors.
/scratch/ishans/pytorchgpu/pytorch/aten/src/ATen/cuda/CublasHandlePool.cpp:112:24: error: ‘cublasSetWorkspace’ was not declared in this scope
TORCH_CUDABLAS_CHECK(cublasSetWorkspace(handle, workspace_it->second.get(), getChosenWorkspaceSize()));
^~~~~~~~~~~~~~~~~~
I am unable to debug this issue. Any idea on how I can fix this ?