[SOLVED] PyTorch no longer supports this GPU because it is too old

I see.
I will look into the instructions.
Thanks a lot.

In my case, even after compiling from the source I still get this error:

Found GPU0 GeForce GT 650M which is of cuda capability 3.0.
PyTorch no longer supports this GPU because it is too old.

Any ideas?

@apako Did you managed to get it running? I own a similar card, GeForce GTX 650, also tried to compile it myself (inside a Docker image with nvidia-docker, based on nvidia/cuda:9.0-cudnn7-devel)

Yes, it runs well despite the warning message.

During the build form sources I still get the following error:

...
nvcc fatal   : Unsupported gpu architecture 'compute_30'
...

I thought that during build from sources PyTorch will detect GPU architecture and it builds for it package …

How to solve this issue ?

You can specify the compute capability via TORCH_CUDA_ARCH_LIST=3.0 and try to rebuild it. However, based on the error message, I guess you might be using CUDA>=11.0, which dropped the compute capability 3.0, so you might need to use an older CUDA toolkit.

1 Like

Oh, thanks, I will try :wink:
Yes, I am using CUDA>=11.0 … maybe this is the case …

Yeah, I downgrade the CUDA version to 10.2 and building starts but failed with strange error:

FAILED: confu-deps/NNPACK/src/x86_64-fma/2d-fourier-8x8.py.o 
cd /home/{User}/Projects/AI/pytorch/build/confu-deps/NNPACK && PYTHONPATH=/home/{User}/Projects/AI/pytorch/third_party/python-six:/home/{User}/Projects/AI/pytorch/third_party/python-peachpy /usr/local/bin/python3 -m peachpy.x86_64 -mabi=sysv -g4 -mimage-format=elf -I/home/{User}/Projects/AI/pytorch/third_party/NNPACK/src -I/home/{User}/Projects/AI/pytorch/third_party/NNPACK/src/x86_64-fma -I/home/{User}/Projects/AI/pytorch/third_party/FP16/include -o /home/{User}/Projects/AI/pytorch/build/confu-deps/NNPACK/src/x86_64-fma/2d-fourier-8x8.py.o /home/{User}/Projects/AI/pytorch/third_party/NNPACK/src/x86_64-fma/2d-fourier-8x8.py
/home/{User}/Projects/AI/pytorch/third_party/python-peachpy/peachpy/writer.py:114: ResourceWarning: unclosed file <_io.FileIO name='/home/{User}/Projects/AI/pytorch/build/confu-deps/NNPACK/src/x86_64-fma/2d-fourier-8x8.py.o' mode='wb' closefd=True>
  self.output_file = None
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
  File "/opt/python-3.8.2/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/python-3.8.2/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/{User}/Projects/AI/pytorch/third_party/python-peachpy/peachpy/x86_64/__main__.py", line 280, in <module>
    main()
  File "/home/{User}/Projects/AI/pytorch/third_party/python-peachpy/peachpy/x86_64/__main__.py", line 267, in main
    execute_script(writers, options.input[0])
  File "/home/{User}/Projects/AI/pytorch/third_party/python-peachpy/peachpy/x86_64/__main__.py", line 198, in execute_script
    execute_script(writers, source_filename)
  File "/home/{User}/Projects/AI/pytorch/third_party/python-peachpy/peachpy/x86_64/__main__.py", line 198, in execute_script
    execute_script(writers, source_filename)
  File "/home/{User}/Projects/AI/pytorch/third_party/python-peachpy/peachpy/x86_64/__main__.py", line 202, in execute_script
    exec(code, globals())
  File "/home/{User}/Projects/AI/pytorch/third_party/NNPACK/src/x86_64-fma/2d-fourier-8x8.py", line 5, in <module>
    import fft.complex_soa
  File "/home/{User}/Projects/AI/pytorch/third_party/NNPACK/src/x86_64-fma/fft/complex_soa.py", line 7, in <module>
    from common import cos_npi_over_8, sin_npi_over_8, cos_npi_over_4, sin_npi_over_4
ImportError: cannot import name 'cos_npi_over_8' from 'common' (/home/{User}/.local/lib/python3.8/site-packages/common/__init__.py)
[25/5198] Building C object confu-deps/XNNPACK/CMakeFiles/XNNPACK.dir/src/f32-vlrelu/gen/vlrelu-scalar-x4.c.o
In file included from ../third_party/XNNPACK/include/xnnpack.h:15:0,
                 from ../third_party/XNNPACK/src/xnnpack/params.h:15,
                 from ../third_party/XNNPACK/src/xnnpack/vunary.h:11,
                 from ../third_party/XNNPACK/src/f32-vlrelu/gen/vlrelu-scalar-x4.c:13:
../third_party/pthreadpool/include/pthreadpool.h:997:2: warning: ‘pthreadpool_function_1d_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_1d_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1003:2: warning: ‘pthreadpool_function_1d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_1d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1010:2: warning: ‘pthreadpool_function_2d_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_2d_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1017:2: warning: ‘pthreadpool_function_2d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_2d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1026:2: warning: ‘pthreadpool_function_3d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_3d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1037:2: warning: ‘pthreadpool_function_4d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_4d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[26/5198] Building C object confu-deps/XNNPACK/CMakeFiles/XNNPACK.dir/src/f32-vrnd/gen/vrndd-scalar-libm-x1.c.o
In file included from ../third_party/XNNPACK/include/xnnpack.h:15:0,
                 from ../third_party/XNNPACK/src/xnnpack/params.h:15,
                 from ../third_party/XNNPACK/src/xnnpack/vunary.h:11,
                 from ../third_party/XNNPACK/src/f32-vrnd/gen/vrndd-scalar-libm-x1.c:15:
../third_party/pthreadpool/include/pthreadpool.h:997:2: warning: ‘pthreadpool_function_1d_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_1d_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1003:2: warning: ‘pthreadpool_function_1d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_1d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1010:2: warning: ‘pthreadpool_function_2d_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_2d_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1017:2: warning: ‘pthreadpool_function_2d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_2d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1026:2: warning: ‘pthreadpool_function_3d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_3d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1037:2: warning: ‘pthreadpool_function_4d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_4d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[27/5198] Building C object confu-deps/XNNPACK/CMakeFiles/XNNPACK.dir/src/f32-velu/gen/velu-scalar-rr2-p6-x6.c.o
In file included from ../third_party/XNNPACK/include/xnnpack.h:15:0,
                 from ../third_party/XNNPACK/src/xnnpack/params.h:15,
                 from ../third_party/XNNPACK/src/xnnpack/vunary.h:11,
                 from ../third_party/XNNPACK/src/f32-velu/gen/velu-scalar-rr2-p6-x6.c:14:
../third_party/pthreadpool/include/pthreadpool.h:997:2: warning: ‘pthreadpool_function_1d_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_1d_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1003:2: warning: ‘pthreadpool_function_1d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_1d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1010:2: warning: ‘pthreadpool_function_2d_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_2d_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1017:2: warning: ‘pthreadpool_function_2d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_2d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1026:2: warning: ‘pthreadpool_function_3d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_3d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1037:2: warning: ‘pthreadpool_function_4d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_4d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[28/5198] Building C object confu-deps/XNNPACK/CMakeFiles/XNNPACK.dir/src/f32-vmulcaddc/gen/c2-minmax-scalar-2x.c.o
In file included from ../third_party/XNNPACK/include/xnnpack.h:15:0,
                 from ../third_party/XNNPACK/src/xnnpack/params.h:15,
                 from ../third_party/XNNPACK/src/xnnpack/vmulcaddc.h:11,
                 from ../third_party/XNNPACK/src/f32-vmulcaddc/gen/c2-minmax-scalar-2x.c:13:
../third_party/pthreadpool/include/pthreadpool.h:997:2: warning: ‘pthreadpool_function_1d_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_1d_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1003:2: warning: ‘pthreadpool_function_1d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_1d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1010:2: warning: ‘pthreadpool_function_2d_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_2d_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1017:2: warning: ‘pthreadpool_function_2d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_2d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1026:2: warning: ‘pthreadpool_function_3d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_3d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1037:2: warning: ‘pthreadpool_function_4d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_4d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[29/5198] Building C object confu-deps/XNNPACK/CMakeFiles/XNNPACK.dir/src/f32-vrnd/gen/vrndd-scalar-libm-x2.c.o
In file included from ../third_party/XNNPACK/include/xnnpack.h:15:0,
                 from ../third_party/XNNPACK/src/xnnpack/params.h:15,
                 from ../third_party/XNNPACK/src/xnnpack/vunary.h:11,
                 from ../third_party/XNNPACK/src/f32-vrnd/gen/vrndd-scalar-libm-x2.c:15:
../third_party/pthreadpool/include/pthreadpool.h:997:2: warning: ‘pthreadpool_function_1d_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_1d_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1003:2: warning: ‘pthreadpool_function_1d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_1d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1010:2: warning: ‘pthreadpool_function_2d_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_2d_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1017:2: warning: ‘pthreadpool_function_2d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_2d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1026:2: warning: ‘pthreadpool_function_3d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_3d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1037:2: warning: ‘pthreadpool_function_4d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_4d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[30/5198] Building C object confu-deps/XNNPACK/CMakeFiles/XNNPACK.dir/src/f32-vmulcaddc/gen/c4-minmax-scalar-2x.c.o
In file included from ../third_party/XNNPACK/include/xnnpack.h:15:0,
                 from ../third_party/XNNPACK/src/xnnpack/params.h:15,
                 from ../third_party/XNNPACK/src/xnnpack/vmulcaddc.h:11,
                 from ../third_party/XNNPACK/src/f32-vmulcaddc/gen/c4-minmax-scalar-2x.c:13:
../third_party/pthreadpool/include/pthreadpool.h:997:2: warning: ‘pthreadpool_function_1d_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_1d_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1003:2: warning: ‘pthreadpool_function_1d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_1d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1010:2: warning: ‘pthreadpool_function_2d_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_2d_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1017:2: warning: ‘pthreadpool_function_2d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_2d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1026:2: warning: ‘pthreadpool_function_3d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_3d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1037:2: warning: ‘pthreadpool_function_4d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_4d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[31/5198] Building C object confu-deps/XNNPACK/CMakeFiles/XNNPACK.dir/src/f32-vrnd/gen/vrndd-scalar-libm-x4.c.o
In file included from ../third_party/XNNPACK/include/xnnpack.h:15:0,
                 from ../third_party/XNNPACK/src/xnnpack/params.h:15,
                 from ../third_party/XNNPACK/src/xnnpack/vunary.h:11,
                 from ../third_party/XNNPACK/src/f32-vrnd/gen/vrndd-scalar-libm-x4.c:15:
../third_party/pthreadpool/include/pthreadpool.h:997:2: warning: ‘pthreadpool_function_1d_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_1d_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1003:2: warning: ‘pthreadpool_function_1d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_1d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1010:2: warning: ‘pthreadpool_function_2d_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_2d_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1017:2: warning: ‘pthreadpool_function_2d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_2d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1026:2: warning: ‘pthreadpool_function_3d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_3d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../third_party/pthreadpool/include/pthreadpool.h:1037:2: warning: ‘pthreadpool_function_4d_tiled_t’ is deprecated [-Wdeprecated-declarations]
  pthreadpool_function_4d_tiled_t function,
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "setup.py", line 773, in <module>
    build_deps()
  File "setup.py", line 315, in build_deps
    build_caffe2(version=version,
  File "/home/{User}/Projects/AI/pytorch/tools/build_pytorch_libs.py", line 58, in build_caffe2
    cmake.build(my_env)
  File "/home/{User}/Projects/AI/pytorch/tools/setup_helpers/cmake.py", line 346, in build
    self.run(build_args, my_env)
  File "/home/{User}/Projects/AI/pytorch/tools/setup_helpers/cmake.py", line 141, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/opt/python-3.8.2/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '8']' returned non-zero exit status 1.

Error is:

ImportError: cannot import name 'cos_npi_over_8' from 'common'

But how it’s possible, all submodules updated properly …

This error seems to be raised by NNPACK and while I’m not deeply familiar with it, I would recommend to uninstall all previously installed PyTorch versions, clear the build via python setup.py clean, checkout all sumbodules, and try to rebuild it.

1 Like

Another strange issue:

-- Using third party subdirectory Eigen.
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.8.5", minimum required is "3.0") 
-- Found PythonLibs: /usr/lib/libpython3.8.so.1.0 (Required is at least version "3.0") 
CMake Error at cmake/Dependencies.cmake:959 (if):
  if given arguments:

    "VERSION_LESS" "3"

  Unknown arguments specified
Call Stack (most recent call first):
  CMakeLists.txt:549 (include)


-- Configuring incomplete, errors occurred!

I just hard-coded version of the Python that was detected wrong to 3.8

And then build starts but faced with another issue:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/redra/Projects/AI/pytorch/third_party/python-peachpy/peachpy/x86_64/__main__.py", line 280, in <module>
    main()
  File "/home/redra/Projects/AI/pytorch/third_party/python-peachpy/peachpy/x86_64/__main__.py", line 267, in main
[ 57%] Built target XNNPACK
    execute_script(writers, options.input[0])
  File "/home/redra/Projects/AI/pytorch/third_party/python-peachpy/peachpy/x86_64/__main__.py", line 198, in execute_script
    execute_script(writers, source_filename)
  File "/home/redra/Projects/AI/pytorch/third_party/python-peachpy/peachpy/x86_64/__main__.py", line 198, in execute_script
    execute_script(writers, source_filename)
  File "/home/redra/Projects/AI/pytorch/third_party/python-peachpy/peachpy/x86_64/__main__.py", line 202, in execute_script
    exec(code, globals())
  File "/home/redra/Projects/AI/pytorch/third_party/NNPACK/src/x86_64-fma/2d-fourier-8x8.py", line 5, in <module>
    import fft.complex_soa
  File "/home/redra/Projects/AI/pytorch/third_party/NNPACK/src/x86_64-fma/fft/complex_soa.py", line 7, in <module>
    from common import cos_npi_over_8, sin_npi_over_8, cos_npi_over_4, sin_npi_over_4
ImportError: cannot import name 'cos_npi_over_8' from 'common' (/home/redra/.local/lib/python3.8/site-packages/common/__init__.py)
[ 57%] Building C object sleef/src/libm/CMakeFiles/sleefavx512fnofma.dir/sleefsimddp.c.o
[ 57%] Building CXX object caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/caffe2.pb.cc.o
make[2]: *** [confu-deps/NNPACK/CMakeFiles/nnpack.dir/build.make:81: confu-deps/NNPACK/src/x86_64-fma/2d-fourier-8x8.py.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:3282: confu-deps/NNPACK/CMakeFiles/nnpack.dir/all] Error 2

Seems like this is related to sudo make install -- ImportError: cannot import name cos_npi_over_8 · Issue #2296 · facebookarchive/caffe2 · GitHub

Even build from docker failed:

#23 325.0 [  8%] Building CXX object third_party/ideep/mkl-dnn/src/cpu/x64/CMakeFiles/dnnl_cpu_x64.dir/shuffle/jit_uni_shuffle.cpp.o
#23 337.2 [  8%] Built target dnnl_cpu_x64
#23 337.2 make: *** [all] Error 2
#23 337.2 Makefile:140: recipe for target 'all' failed
#23 337.2 Building wheel torch-1.8.0a0
#23 337.2 -- Building version 1.8.0a0
#23 337.2 cmake -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/opt/pytorch/torch -DCMAKE_PREFIX_PATH=/opt/conda/bin/../ -DNUMPY_INCLUDE_DIR=/opt/conda/lib/python3.8/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/opt/conda/bin/python -DPYTHON_INCLUDE_DIR=/opt/conda/include/python3.8 -DPYTHON_LIBRARY=/opt/conda/lib/libpython3.8.so.1.0 -DTORCH_BUILD_VERSION=1.8.0a0 -DUSE_NUMPY=True /opt/pytorch
#23 337.2 cmake --build . --target install --config Release -- -j 8
#23 337.2 Traceback (most recent call last):
#23 337.2   File "setup.py", line 773, in <module>
#23 337.2     build_deps()
#23 337.2   File "setup.py", line 315, in build_deps
#23 337.2     build_caffe2(version=version,
#23 337.2   File "/opt/pytorch/tools/build_pytorch_libs.py", line 58, in build_caffe2
#23 337.2     cmake.build(my_env)
#23 337.2   File "/opt/pytorch/tools/setup_helpers/cmake.py", line 345, in build
#23 337.2     self.run(build_args, my_env)
#23 337.2   File "/opt/pytorch/tools/setup_helpers/cmake.py", line 140, in run
#23 337.2     check_call(command, cwd=self.build_dir, env=env)
#23 337.2   File "/opt/conda/lib/python3.8/subprocess.py", line 364, in check_call
#23 337.2     raise CalledProcessError(retcode, cmd)
#23 337.2 subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '8']' returned non-zero exit status 2.
------
error committing ki40ko690hn6nxbbd83w6smqr: invalid mutable ref 0xc001c50fa0: invalid: executor failed running [/bin/sh -c TORCH_CUDA_ARCH_LIST="3.0 3.5 5.2 6.0 6.1 7.0+PTX 8.0" TORCH_NVCC_FLAGS="-Xfatbin -compress-all"     CMAKE_PREFIX_PATH="$(dirname $(which conda))/../"     python setup.py install]: exit code: 1
make: *** [docker.Makefile:47: devel-image] Error 1

I would recommend to create a new conda or pip environment with the wanted Python version and try to reinstall it in this clean environment.

1 Like

I also tried to build on WIndows:

  Stringify NVFUSER runtime source file
  Building Custom Rule C:/pytorch/caffe2/CMakeLists.txt
  Stringify NVFUSER runtime source file
  Building Custom Rule C:/pytorch/caffe2/CMakeLists.txt
  Building NVCC (Device) object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/THC/Release/torch_cuda_generated_THCReduce
  ApplyUtils.cu.obj
  nvcc fatal   : Unsupported gpu architecture 'compute_30'
  CMake Error at torch_cuda_generated_THCReduceApplyUtils.cu.obj.Release.cmake:221 (message):
    Error generating
    C:/pytorch/build/caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/THC/Release/torch_cuda_generated_THCReduceApplyUtils.
  cu.obj

compute_30, but I have set TORCH_CUDA_ARCH_LIST=3.0 … seems like another variable should be set for caffe2

This error is raised, since the CUDA compiler nvcc doesn’t recognize the GPU architecture. As explained before, CUDA>=11.0 dropped the compute capability 3.0, so you would have to use an older CUDA version.

I’ve found the root cause for this issue:

@ptrblck What do you mean by >=5?
Can’t we run this on compute capability of 3.5 as PyTorch officially support 3.5 or higher compute capability right?

Thanks,

Hi @jpeg729 - Pytorch supports compute capability 3.5 or higher right?
I am wondering why it is not working with my GT 730 4GB card. Anything you feel you could suggest to me?

The current PyTorch binaries support compute capability 3.7 and higher. If you are using a GPU with compute capability 3.5, you would have to build form source.

I did that. I followed this document - https://github.com/pytorch/pytorch/#from-source and build it.

I will share you the specs.
I have Nvidia driver 470.x.x with Cuda 11.4.x installed for my UBUNTU 20.04 LTS system.
I didn’t find conda install -c pytorch magma-cuda114 so went with conda install -c pytorch magma-cuda112

Rest all the installation was smooth.
Still, if I try to run something, I get the below-quoted errors:

RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
----------------------------


--------------------------------------------------------------------------------------------

home/assertst/.local/lib/python3.8/site-packages/torch/cuda/__init__.py:83: UserWarning: 
    Found GPU%d %s which is of cuda capability %d.%d.
    PyTorch no longer supports this GPU because it is too old.
    The minimum cuda capability supported by this library is %d.%d.
    
  warnings.warn(old_gpu_warn.format(d, name, major, minor, min_arch // 10, min_arch % 10))
Device Used:  NVIDIA GeForce GT 730
Capability:  (3, 5)
<class 'darknet.Darknet'>
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/assertst/anaconda3/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/assertst/anaconda3/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "nephro.py", line 88, in processResult
    output = self.model(Variable(img), self.CUDA)
  File "/home/assertst/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/assertst/nephroplus/darknet.py", line 319, in forward
    x = self.module_list[i](x)
  File "/home/assertst/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/assertst/.local/lib/python3.8/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/home/assertst/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/assertst/.local/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 756, in forward
    return F.leaky_relu(input, self.negative_slope, self.inplace)
  File "/home/assertst/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 1472, in leaky_relu
    result = torch._C._nn.leaky_relu_(input, negative_slope)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
----------------------------
Killing processResult Thread
----------------------------

Do you understand the situation @ptrblck ?