Issue compiling based on order of -isystem include dirs in conda environment

torch.compile
1

I am trying to compile pytorch inside a conda environment using my system version headers of cuda/cuda-toolkit located at /usr/local/cuda-12/include.

I get all sorts of compilation issues since there are headers in my environment include, /home/coyote/.conda/envs/deepchem/include that mess up compiling with system nvcc and system gcc when I have this order with environment includes being AUTOMATICALLY listed first when running python setup.py install

/usr/local/cuda-12/bin/nvcc ... -isystem /home/coyote/.conda/envs/deepchem/include <more -isysyem paths> -isystem /usr/local/cuda-12/include ... -c pytorch/aten/src/ATen/native/cuda/CumminmaxKernel.cu -o caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/CumminmaxKernel.cu.o

errors such as below without moving /home/coyote/.conda/envs/deepchem/include/nv to /home/coyote/.conda/envs/deepchem/include/_nv

/usr/local/cuda-12/include/cuda_bf16.hpp(383): error: identifier "__NV_IS_DEVICE" is undefined

or later after moving /home/coyote/.conda/envs/deepchem/include/nv to /home/coyote/.conda/envs/deepchem/include/_nv then getting issues with the cub folder in the environment include

/home/coyote/.conda/envs/deepchem/include/cub/iterator/tex_ref_input_iterator.cuh(99): error: texture is not a template

/home/coyote/.conda/envs/deepchem/include/cub/iterator/tex_ref_input_iterator.cuh(111): error: identifier "cudaBindTexture" is undefined

But I don’t get these compile issues if I manually put system cuda -isystem /usr/local/cuda-12/include somewhere before the environment include -isystem /home/coyote/.conda/envs/deepchem/include

So my question is how to make this order automatically when running setup.py install. How can I make /usr/local/cuda-12/include the first of all -isystem includes?

2

Could you explain which files are in deepchem/include as I cannot find these references in the repository?

3

Well there are a lot since I have included a lot of packages for chemical machine learning here including tensorflow with cuda. There are 59 *.h header files that intersect both system include (/usr/local/cuda-12/include) and environment includes (/home/coyote/.conda/envs/deepchem/include) and 9 directories below that are in both as well.

$ grep -f <(ls -d /usr/local/cuda-12/include/*/ | cut -d/ -f6) <(ls -d /home/coyote/.conda/envs/deepchem/include/*/ | cut -d/ -f8)
CL
crt
cub
cuda
nv
nvtx3
Openacc
Openmp
thrust

Regardless with deepchem since they implement both tensorflow and pytorch it would be nice to install tensorflow with cuda and pytorch from source with cuda12.1, i.e. for instance adding a manual -isystem include before the rest with some code resembling setup.py with -isystem includes · GitHub but having the “extra_compile_args” of this manual -isystem after all the CFLAGS included “-I…” but before the rest of the -isystem includes. In pytorch’s “extra_compile_args” these all come after the -isystem includes" so it won’t be helpful to add it there.

4

So far updating CMake variables such as CUDNN_INCLUDE_PATH, CUDNN_LIBRARY, CUDNN_LIBRARY_PATH, CUB_INCLUDE_DIR and temporarily moving /home/coyote/.conda/envs/deepchem/include/nv to /home/coyote/.conda/envs/deepchem/include/_nv works for compiling some caffe2 sources.