Issue compiling based on order of -isystem include dirs in conda environment

I am trying to compile pytorch inside a conda environment using my system version headers of cuda/cuda-toolkit located at /usr/local/cuda-12/include.

I get all sorts of compilation issues since there are headers in my environment include, /home/coyote/.conda/envs/deepchem/include that mess up compiling with system nvcc and system gcc when I have this order with environment includes being AUTOMATICALLY listed first when running python install

/usr/local/cuda-12/bin/nvcc ... -isystem /home/coyote/.conda/envs/deepchem/include <more -isysyem paths> -isystem /usr/local/cuda-12/include ... -c pytorch/aten/src/ATen/native/cuda/ -o caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/

errors such as below without moving /home/coyote/.conda/envs/deepchem/include/nv to /home/coyote/.conda/envs/deepchem/include/_nv

/usr/local/cuda-12/include/cuda_bf16.hpp(383): error: identifier "__NV_IS_DEVICE" is undefined

or later after moving /home/coyote/.conda/envs/deepchem/include/nv to /home/coyote/.conda/envs/deepchem/include/_nv then getting issues with the cub folder in the environment include

/home/coyote/.conda/envs/deepchem/include/cub/iterator/tex_ref_input_iterator.cuh(99): error: texture is not a template

/home/coyote/.conda/envs/deepchem/include/cub/iterator/tex_ref_input_iterator.cuh(111): error: identifier "cudaBindTexture" is undefined

But I don’t get these compile issues if I manually put system cuda -isystem /usr/local/cuda-12/include somewhere before the environment include -isystem /home/coyote/.conda/envs/deepchem/include

So my question is how to make this order automatically when running install. How can I make /usr/local/cuda-12/include the first of all -isystem includes?

Could you explain which files are in deepchem/include as I cannot find these references in the repository?

Well there are a lot since I have included a lot of packages for chemical machine learning here including tensorflow with cuda. There are 59 *.h header files that intersect both system include (/usr/local/cuda-12/include) and environment includes (/home/coyote/.conda/envs/deepchem/include) and 9 directories below that are in both as well.

$ grep -f <(ls -d /usr/local/cuda-12/include/*/ | cut -d/ -f6) <(ls -d /home/coyote/.conda/envs/deepchem/include/*/ | cut -d/ -f8)

Regardless with deepchem since they implement both tensorflow and pytorch it would be nice to install tensorflow with cuda and pytorch from source with cuda12.1, i.e. for instance adding a manual -isystem include before the rest with some code resembling with -isystem includes · GitHub but having the “extra_compile_args” of this manual -isystem after all the CFLAGS included “-I…” but before the rest of the -isystem includes. In pytorch’s “extra_compile_args” these all come after the -isystem includes" so it won’t be helpful to add it there.

So far updating CMake variables such as CUDNN_INCLUDE_PATH, CUDNN_LIBRARY, CUDNN_LIBRARY_PATH, CUB_INCLUDE_DIR and temporarily moving /home/coyote/.conda/envs/deepchem/include/nv to /home/coyote/.conda/envs/deepchem/include/_nv works for compiling some caffe2 sources.