Failed to find nvToolsExt

Hello,

libtorch 2.0.1 with CUDA 11.8 and cudNN 8.5 works perfectly fine for me in Visual Studio Community 2022.

However, i now wanted to upgrade to libtorch Preview (nightly) with CUDA 12.1 and cudNN 8.9, but now i get an error message when trying to build with cmake:

[CMake] …
[CMake] – Found CUDAToolkit: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1/include (found version “12.1.105”)
[CMake] CMake Error at C:/…/libtorch/share/cmake/Caffe2/public/cuda.cmake:74 (message):
[CMake] Failed to find nvToolsExt.

I saw in NVTX github that the most recent NVTX version (nvtx3) is using a header only file instead of a library.
Could that be the reason for the error i get?
Is this a known bug or am i doing something wrong?

Best Regards,
Max

1 Like

Hello Max,

The error message you are encountering indicates that CMake is unable to find the nvToolsExt library, which is typically provided by the NVIDIA CUDA Toolkit. This library is used for GPU profiling and debugging.

The issue could be related to the version mismatch between your CUDA installation and the libtorch Preview version you are trying to use. It’s possible that the libtorch Preview version you have is expecting a different version of nvToolsExt than the one provided by CUDA 12.1.

To address this issue, you can try the following steps:

  1. Confirm that the nvToolsExt library is indeed present in your CUDA 12.1 installation. Check the C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1/lib64 directory for the library file (nvToolsExt64_1.lib on Windows).

  2. Ensure that the CUDA 12.1 installation directory is correctly set in your system’s environment variables (PATH variable) or in the CMake configuration. This ensures that CMake can locate the necessary CUDA files.

  3. If the library is present and the environment variables are correctly set, you can try manually adding the path to the nvToolsExt library in your CMakeLists.txt file. Add the following line before the find_package(Torch REQUIRED) line:

    link_directories("C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1/lib64")
    

    Replace the path with the correct location of the lib64 directory on your system.

  4. If the above steps do not resolve the issue, it’s possible that there is an incompatibility between the libtorch Preview version and CUDA 12.1. In that case, you might need to wait for a newer libtorch Preview version that is compatible with CUDA 12.1 or consider downgrading your CUDA installation to a version that is compatible with the libtorch Preview version you want to use.

I hope this helps! Let me know if you have any further questions.

1 Like

Thank you for the extensive answer. I am running into this problem too, but I cannot find a nvToolsExt lib anywhere under C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1. I ran the NVIDIA CUDA installer as specified on the download page and installed all possible packages.

How can I best install the nvToolsExt lib? It does not seem to come with the default installer

2 Likes

I had a similar problem and what I did was install NVTX with CUDA 11.8 installer. With CUDA 12 installed, run CUDA 11.8 installer and select custom installation. Then select Nsight NVTX alone and leave everything else unchecked.

image

4 Likes

@Te93 This worked for me! Now have PyTorch working on CUDA 12.2

Thanks :slight_smile:

1 Like

I’ve just sign into this forum to say thank you. I was about to create an issue on pytorch’s github saying that link libraries paths to cuda didn’t match the installation of Nvidia’s new 12.3 toolkit, but your direction to install NVTX from an older version of the toolkit saved the day. I think Nvidia should include NVTX as well on newer versions.

Thank you.

1 Like