Importing CUDA extension, undefined symbol: nvrtcGetProgramLogSize

Hello,

I’ve been modifying a CUDA extension from the official LatticeNet repo (my fork link is coming, from which you can also find the original), so I could use it without installing all the other extra infrastructure packages I don’t need. I’ve managed to get it to the stage, where I can compile the extension and attempt to import it.

When I do import it after torch, I get the following error:

/usr/local/lib/python3.9/dist-packages/latticenet.cpython-39-x86_64-linux-gnu.so: undefined symbol: nvrtcGetProgramLogSize

From my searches, I found that this is a CUDA symbol, but I do not know how to interpret it being not found. CUDA is available on both machines I tried this on.

Here’s a link to my fork:

Here’s my setup.py script

import pathlib
from setuptools import setup, Extension
from setuptools import setup
from distutils.sysconfig import get_python_inc

from torch.utils.cpp_extension import BuildExtension, CUDAExtension

PROJ_DIR = pathlib.Path(__file__).parent.resolve()
EIGEN_INCLUDE = '/usr/include/eigen3'
OPENCV_INCLUDE = '/usr/include/opencv4'

setup(
    ...
    zip_safe=False,
    ext_modules=[
        CUDAExtension(
            name='latticenet',
            sources=['src/Lattice.cu',
                     'src/HashTable.cu',
                     'src/PyBridge.cxx'],
            include_dirs=[str(PROJ_DIR / 'include'),
                          str(PROJ_DIR / 'deps/include'),
                          str(PROJ_DIR / 'deps'),
                          OPENCV_INCLUDE,
                          EIGEN_INCLUDE],
            extra_compile_args=[f'-D PROJECT_SOURCE_DIR="{PROJ_DIR}"',
                                f'-D CMAKE_SOURCE_DIR="{PROJ_DIR}"'],
        )
    ],
    cmdclass={
        'build_ext': BuildExtension
    }
)

Tested in environments:
Local:
python version: 3.10.9
pytorch 1.13.1 py3.10_cuda11.7_cudnn8.5.0_0 pytorch
pytorch-cuda 11.7 h67b0de4_1 pytorch
GCC version: 11.3.0

Google Colab:
python version: 3.9.16
pytorch 1.13.1+cu116
CUDA version: 12.0
GCC version: 9.4.0

Here’s a notebook for compiling this extension in Colab

The original work provides a Docker file, in which PyTorch is downloaded and compiled from source, and a bunch of other actions. I haven’t been successful in compiling the latest (or the older one used in the Dockerfile) version from source locally, or on Colab, so I tried to use pre-compiled binaries. Could this be the main problem?

I don’t see any calls into nvrtc in your repository and thus don’t know why your lib would want to load this symbol. I might miss something as the GitHub search did not find anything useful.
In any case, PyTorch itself ships with libnvrtc and I would assume importing torch before your extension could solve the issue.

Thanks for checking my problem out!

It seems that nvrtcGetProgramLogSize is called in the jitify.hpp header file, but just like you said, I can find libnvrtc in my PyTorch folder. Maybe I need to somehow specify its location directly, for it to be linked?

But it doesn’t seem to be the first nvrtc reference in that file, so it should probably stop earlier. The nvrtcGetProgramLogSize call itself is inside a function of return type nvrtcResult, I think it should trip up on this before even getting to nvrtcGetProgramLogSize if the library wasn’t linked in the first place. But that’s just a guess, I don’t have the most detailed understanding.

I do import torch first, because otherwise I get

ImportError: libc10.so: cannot open shared object file: No such file or directory