I cannot post more than two links here as a beginner, and which was definitely needed. That is why the whole issue is at https://github.com/pytorch/pytorch/issues/43210 but shall be answered here.
The error is caused by our poor support for MSVC OpenMP in detectron. Please build with MKL so Intel OpenMP will be used. See https://pytorch.org/docs/stable/notes/windows.html#include-optional-components.
1 Like
As for (23), please open a new issue at the corresponding repo. It is not related to PyTorch.
I was not sure which point you meant with 23, since I had 2 things in that (my fault). I have changed that, could you please write again which number should be a new issue: the lost packages (23) or the new issue with dtaidistance (now 24)?
@peterjc123
I do not get along with your suggested link. Why do I have to install mkl like that, if I can simply install it with conda?
It seems to be better to follow GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration and use the dependency installer there (and where mkl is included anyway?):
conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing_extensions future six requests
####
Aside from that, there is a mistake in your stated link.
REM Download MKL files
curl https://s3.amazonaws.com/ossci-windows/mkl_2020.0.166.7z -k -O 7z x -aoa mkl_2018.2.185.7z -omkl
It works with
7z x -aoa mkl_2020.0.166.7z -omkl
since that is what the curl command is downloading as a 7z.
Anyway, I have now succeeded in using the two commands, having now C:\Users\Admin\mkl with the necessary files in it, but I do not need the directory at all, since I can simply install mkl with conda, see above.
####
I do not know now how to install from source with MKL. Looking at your link, under Windows FAQ — PyTorch 2.1 documentation, I find:
conda install -c peterjc123 vc vs2017_runtime
conda install mkl_fft intel_openmp numpy mkl
But that does not seem to help me, first, I have vs2019, then I want to install pytorch with mkl, and I find a wide range of commands that do not seem to help.
Because pip install intel-openmp does not work, I use from your link:
conda install -c defaults intel-openmp -f
Which seems to be needed for installing pytorch with mkl, as you say: “Please build with MKL so Intel OpenMP will be used.”
Please give me a hint what I shall do next to get pytorch built with MKL, thank you.
The conda one is a runtime package, not a source package.
Will fix.
## Just after doing `curl`
set "CMAKE_INCLUDE_PATH=%cd%\\mkl\\include"
set "LIB=%cd%\\mkl\\lib;%LIB%"
I mean the dtaidistance one.
Now I get the error:
DistutilsPlatformError: Microsoft Visual C++ 14.0 is required. Get it with “Build Tools for Visual Studio”
Although “MSVC v142 – VS 2019 C+±x64/x86-Buildtools” is already installed:
(dl) C:\Users\Admin\Downloads\Pytorch\pytorch>set "CMAKE_INCLUDE_PATH=C:\Users\Admin\Downloads\Pytorch\mkl\include"
(dl) C:\Users\Admin\Downloads\Pytorch\pytorch>set "LIB=C:\Users\Admin\Downloads\Pytorch\mkl\lib;%LIB%"
(dl) C:\Users\Admin\Downloads\Pytorch\pytorch>set CMAKE_GENERATOR=Ninja
(dl) C:\Users\Admin\Downloads\Pytorch\pytorch>python setup.py install --cmake
Building wheel torch-1.7.0a0+d4c5f56
-- Building version 1.7.0a0+d4c5f56
Traceback (most recent call last):
File "C:\Users\Admin\anaconda3\envs\dl\lib\site-packages\setuptools\msvc.py", line 273, in _msvc14_get_vc_env
out = subprocess.check_output(
File "C:\Users\Admin\anaconda3\envs\dl\lib\subprocess.py", line 411, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "C:\Users\Admin\anaconda3\envs\dl\lib\subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'cmd /u /c "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Auxiliary\Build\vcvarsall.bat" x64 && set' returned non-zero exit status 255.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "setup.py", line 737, in <module>
build_deps()
File "setup.py", line 316, in build_deps
build_caffe2(version=version,
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\build_pytorch_libs.py", line 52, in build_caffe2
my_env = _create_build_env()
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\build_pytorch_libs.py", line 45, in _create_build_env
my_env = _overlay_windows_vcvars(my_env)
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\build_pytorch_libs.py", line 14, in _overlay_windows_vcvars
vc_env = _get_vc_env(vc_arch)
File "C:\Users\Admin\anaconda3\envs\dl\lib\site-packages\setuptools\msvc.py", line 314, in msvc14_get_vc_env
return _msvc14_get_vc_env(plat_spec)
File "C:\Users\Admin\anaconda3\envs\dl\lib\site-packages\setuptools\msvc.py", line 278, in _msvc14_get_vc_env
raise distutils.errors.DistutilsPlatformError(
distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/
Now I am running it with settings:
set "CMAKE_INCLUDE_PATH=C:\Users\Admin\Downloads\Pytorch\mkl\include"
set "LIB=C:\Users\Admin\Downloads\Pytorch\mkl\lib;%LIB%"
set USE_NINJA=OFF
set CMAKE_GENERATOR=Visual Studio 16 2019
starting again with:
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>python setup.py install --cmake
starts with:
(dl) C:\Users\Admin\Downloads\Pytorch\pytorch>python setup.py install --cmake
Building wheel torch-1.7.0a0+d4c5f56
-- Building version 1.7.0a0+d4c5f56
cmake -GVisual Studio 16 2019 -Ax64 -Thost=x64 -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_GENERATOR=Visual Studio 16 2019 -DCMAKE_INCLUDE_PATH=C:\Users\Admin\Downloads\Pytorch\mkl\include -DCMAKE_INSTALL_PREFIX=C:\Users\Admin\Downloads\Pytorch\pytorch\torch -DCMAKE_PREFIX_PATH=C:\Users\Admin\anaconda3\envs\dl\Lib\site-packages -DJAVA_HOME=C:\Users\Admin\AppData\Local\Programs\AdoptOpenJDK\ -DNUMPY_INCLUDE_DIR=C:\Users\Admin\anaconda3\envs\dl\lib\site-packages\numpy\core\include -DPYTHON_EXECUTABLE=C:\Users\Admin\anaconda3\envs\dl\python.exe -DPYTHON_INCLUDE_DIR=C:\Users\Admin\anaconda3\envs\dl\include -DPYTHON_LIBRARY=C:\Users\Admin\anaconda3\envs\dl/libs/python38.lib -DTORCH_BUILD_VERSION=1.7.0a0+d4c5f56 -DUSE_NINJA=OFF -DUSE_NUMPY=True C:\Users\Admin\Downloads\Pytorch\pytorch
-- The CXX compiler identification is MSVC 19.27.29111.0
-- The C compiler identification is MSVC 19.27.29111.0
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/cl.exe - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/cl.exe - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Not forcing any particular BLAS to be found
-- Performing Test COMPILER_WORKS
-- Performing Test COMPILER_WORKS - Success
-- Performing Test SUPPORT_GLIBCXX_USE_C99
-- Performing Test SUPPORT_GLIBCXX_USE_C99 - Success
-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED
-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED - Success
-- std::exception_ptr is supported.
-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING
-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING - Failed
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS - Success
-- Current compiler supports avx2 extension. Will build perfkernels.
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS - Success
-- Current compiler supports avx512f extension. Will build fbgemm.
-- Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY - Failed
-- Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY - Failed
-- Performing Test COMPILER_SUPPORTS_RDYNAMIC
-- Performing Test COMPILER_SUPPORTS_RDYNAMIC - Failed
-- Building using own protobuf under third_party per request.
-- Use custom protobuf build.
--
-- 3.11.4.0
-- Looking for pthread.h
-- Looking for pthread.h - not found
-- Found Threads: TRUE
-- Caffe2 protobuf include directory: $<BUILD_INTERFACE:C:/Users/Admin/Downloads/Pytorch/pytorch/third_party/protobuf/src>$<INSTALL_INTERFACE:include>
-- Trying to find preferred BLAS backend of choice: MKL
-- MKL_THREADING = OMP
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of void*
-- Check size of void* - done
-- Looking for cblas_sgemm
-- Looking for cblas_sgemm - found
-- MKL libraries: C:/Users/Admin/Downloads/Pytorch/mkl/lib/mkl_intel_lp64.lib;C:/Users/Admin/Downloads/Pytorch/mkl/lib/mkl_intel_thread.lib;C:/Users/Admin/Downloads/Pytorch/mkl/lib/mkl_core.lib;C:/Users/Admin/Downloads/Pytorch/mkl/lib/libiomp5md.lib
-- MKL include directory: C:/Users/Admin/Downloads/Pytorch/mkl/include
-- MKL OpenMP type: Intel
-- MKL OpenMP library: C:/Users/Admin/Downloads/Pytorch/mkl/lib/libiomp5md.lib
CMake Warning at cmake/Dependencies.cmake:226 (message):
Target platform "Windows" is not supported in {Q/X}NNPACK. Supported
platforms are Android, iOS, Linux, and macOS. Turn this warning off by
USE_{Q/X}NNPACK=OFF.
Call Stack (most recent call first):
CMakeLists.txt:472 (include)
Almost at the bottom of the previous output we see that MKL is used. That shows me that can compile with the compiler MSVC and use MKL it seems. I hope that this is the way you asked me to use MKL.
This run has now led to the same error as reported in an earlier post. It seems that I have to remove the pytorch/caffe2/CMakeLists.txt before start, here the error message:
Generating quantization/server/sigmoid_dnnlowp_op_test.py
Generating quantization/server/spatial_batch_norm_dnnlowp_op_test.py
Generating quantization/server/tanh_dnnlowp_op_test.py
Generating quantization/server/utils.py
Building Custom Rule C:/Users/Admin/Downloads/Pytorch/pytorch/caffe2/CMakeLists.txt
Traceback (most recent call last):
File "setup.py", line 737, in <module>
build_deps()
File "setup.py", line 316, in build_deps
build_caffe2(version=version,
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\build_pytorch_libs.py", line 62, in build_caffe2
cmake.build(my_env)
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\setup_helpers\cmake.py", line 345, in build
self.run(build_args, my_env)
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\setup_helpers\cmake.py", line 141, in run
check_call(command, cwd=self.build_dir, env=env)
File "C:\Users\Admin\anaconda3\envs\dl\lib\subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '/p:CL_MPCount=4']' returned non-zero exit status 1.
I let it run again after removing the pytorch/caffe2/CMakeLists.txt, which caused error:
-- Configuring incomplete, errors occurred!
See also "C:/Users/Admin/Downloads/Pytorch/pytorch/build/CMakeFiles/CMakeOutput.log".
See also "C:/Users/Admin/Downloads/Pytorch/pytorch/build/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
File "setup.py", line 737, in <module>
build_deps()
File "setup.py", line 316, in build_deps
build_caffe2(version=version,
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\build_pytorch_libs.py", line 54, in build_caffe2
cmake.generate(version,
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\setup_helpers\cmake.py", line 329, in generate
self.run(args, env=my_env)
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\setup_helpers\cmake.py", line 141, in run
check_call(command, cwd=self.build_dir, env=env)
File "C:\Users\Admin\anaconda3\envs\dl\lib\subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '-GVisual Studio 16 2019', '-Ax64', '-Thost=x64', '-DBUILD_PYTHON=True', '-DBUILD_TEST=True', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_GENERATOR=Visual Studio 16 2019', '-DCMAKE_INCLUDE_PATH=C:\\Users\\Admin\\Downloads\\Pytorch\\mkl\\include', '-DCMAKE_INSTALL_PREFIX=C:\\Users\\Admin\\Downloads\\Pytorch\\pytorch\\torch', '-DCMAKE_PREFIX_PATH=C:\\Users\\Admin\\anaconda3\\envs\\dl\\Lib\\site-packages', '-DJAVA_HOME=C:\\Users\\Admin\\AppData\\Local\\Programs\\AdoptOpenJDK\\', '-DNUMPY_INCLUDE_DIR=C:\\Users\\Admin\\anaconda3\\envs\\dl\\lib\\site-packages\\numpy\\core\\include', '-DPYTHON_EXECUTABLE=C:\\Users\\Admin\\anaconda3\\envs\\dl\\python.exe', '-DPYTHON_INCLUDE_DIR=C:\\Users\\Admin\\anaconda3\\envs\\dl\\include', '-DPYTHON_LIBRARY=C:\\Users\\Admin\\anaconda3\\envs\\dl/libs/python38.lib', '-DTORCH_BUILD_VERSION=1.7.0a0+d4c5f56', '-DUSE_NINJA=OFF', '-DUSE_NUMPY=True', 'C:\\Users\\Admin\\Downloads\\Pytorch\\pytorch']' returned non-zero exit status 1.
Now I let it run without the directory caffe2_builders which I had added after the cloning from git only due to a probably outdated Youtube tutorial. The directory came from pytorch-scripts project, which is outdated. Let’s see.
Result:
LINK : fatal error LNK1356: Die zur vollständigen Archivierung angegebene Bibliothek "\C:/Users/Admin/Downloads/Pytorch/pytorch/build/lib/Release/caffe2_protos.lib\.lib" wurde nicht gefunden. [C:\Users\Admi
n\Downloads\Pytorch\pytorch\build\caffe2\torch_cpu.vcxproj]
Traceback (most recent call last):
File "setup.py", line 737, in <module>
build_deps()
File "setup.py", line 316, in build_deps
build_caffe2(version=version,
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\build_pytorch_libs.py", line 62, in build_caffe2
cmake.build(my_env)
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\setup_helpers\cmake.py", line 345, in build
self.run(build_args, my_env)
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\setup_helpers\cmake.py", line 141, in run
check_call(command, cwd=self.build_dir, env=env)
File "C:\Users\Admin\anaconda3\envs\dl\lib\subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '/p:CL_MPCount=4']' returned non-zero exit status 1.
Run again with the directory caffe2_builders causes again the same error:
LINK : fatal error LNK1356: Die zur vollständigen Archivierung angegebene Bibliothek "\C:/Users/Admin/Downloads/Pytorch/pytorch/build/lib/Release/caffe2_protos.lib\.lib" wurde nicht gefunden. [C:\Users\Admi
n\Downloads\Pytorch\pytorch\build\caffe2\torch_cpu.vcxproj]
Traceback (most recent call last):
File "setup.py", line 737, in <module>
build_deps()
File "setup.py", line 316, in build_deps
build_caffe2(version=version,
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\build_pytorch_libs.py", line 62, in build_caffe2
cmake.build(my_env)
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\setup_helpers\cmake.py", line 345, in build
self.run(build_args, my_env)
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\setup_helpers\cmake.py", line 141, in run
check_call(command, cwd=self.build_dir, env=env)
File "C:\Users\Admin\anaconda3\envs\dl\lib\subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '/p:CL_MPCount=4']' returned non-zero exit status 1.
After this, I have done everything from the scratch again. Perhaps there were conflicts with the outdated Youtube approach that included files of the pytorch-scrypts project. It seems promising up to now, so perhaps there was simply a problem with the added outdated pytorch-scrypts files.
(myenv) C:\Users\Admin\Downloads\Pytorch>git clone https://github.com/pytorch/pytorch
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>git submodule update --init --recursive
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set "CMAKE_INCLUDE_PATH=C:\Users\Admin\Downloads\Pytorch\mkl\include"
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set "LIB=C:\Users\Admin\Downloads\Pytorch\mkl\lib;%LIB%"
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set CMAKE_GENERATOR=Visual Studio 16 2019
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set USE_NINJA=OFF
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>python setup.py install --cmake
But I get the same error:
Building Custom Rule C:/Users/Admin/Downloads/Pytorch/pytorch/caffe2/CMakeLists.txt
Traceback (most recent call last):
File "setup.py", line 737, in <module>
build_deps()
File "setup.py", line 316, in build_deps
build_caffe2(version=version,
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\build_pytorch_libs.py", line 62, in build_caffe2
cmake.build(my_env)
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\setup_helpers\cmake.py", line 345, in build
self.run(build_args, my_env)
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\setup_helpers\cmake.py", line 141, in run
check_call(command, cwd=self.build_dir, env=env)
File "C:\Users\Admin\anaconda3\envs\dl\lib\subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '/p:CL_MPCount=4']' returned non-zero exit status 1.
I do not run any tests any more, each run takes about 7 hours. I am waiting for an idea what I should do. Thank you.
1 Like
hello,i am build pytorch in windos c++ too.
i aslo meet this problem as same as you.
LINK : fatal error LNK1356: Die zur vollständigen Archivierung angegebene Bibliothek “\C:/Users/Admin/Downloads/Pytorch/pytorch/build/lib/Release/caffe2_protos.lib.lib” wurde nicht gefunden. [C:\Users\Admi
n\Downloads\Pytorch\pytorch\build\caffe2\torch_cpu.vcxproj]
if you solve this problem,i hope you can tell me your idea.
Thanks
I have stopped any tests on this after putting some days into this. I am waiting for a helping reply, good that you join me in waiting. Generally, I guess that the error is linked with the same problem that causes ninja not to work. I do not know why I cannot use ninja. Setting off ninja is just a workaround, yet there should be a better way I think, because ninja will make the install much faster. I do not want to let it run for 7 hours again and again only to have the same errors thrown.
If you want to try something that seems promising, have a look at PyTorch build from source on Windows and further down, especially
set CUDAHOSTCXX=
thanks for your reply.i will keep to test in ohther way.
i already to see that,but it’s no work for me.
thanks and keep in touch.
1 Like
I succeed in my win10 with build ninja ,so i tell you my way but it’s may must work for you.
first,i update my source code ,and git submodule update --init --recursive to update my third_part.
and then,i set (USE_MKLDNN=OFF)may no need to you.
python setup.py build --cmake
python setup.py install
cuda also can use.
i hope you can succeed to build it.
good luck
1 Like
Glad you made it, that means that it is really still possible and that I have system / environment / card specific problems.
For example, I need to use set USE_NINJA=OFF, else I get this error:
File "C:\Users\Admin\anaconda3\envs\dl\lib\site-packages\setuptools\msvc.py", line 314, in msvc14_get_vc_env
return _msvc14_get_vc_env(plat_spec)
File "C:\Users\Admin\anaconda3\envs\dl\lib\site-packages\setuptools\msvc.py", line 278, in _msvc14_get_vc_env
raise distutils.errors.DistutilsPlatformError(
distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/
Which is one clear difference with your settings. And in addition, I must use MKL, since else, I get “Error loading caffe2_detectron_ops_gpu.dll”, see the header of this thread - even if pytorch gets fully installed then.
At the moment, it crashes just a minute after the start, so something else has gone wrong with the command or with the cloned pytorch directory.
subprocess.CalledProcessError: Command '['cmake', '-GVisual Studio 16 2019', '-Ax64', '-Thost=x64', '-DBUILD_PYTHON=True', '-DBUILD_TEST=True', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_GENERATOR=Visual Studio 16 2019', '-DCMAKE_INCLUDE_PATH=C:\\Users\\Admin\\Downloads\\Pytorch\\mkl\\include', '-DCMAKE_INSTALL_PREFIX=C:\\Users\\Admin\\Downloads\\Pytorch\\pytorch\\torch', '-DCMAKE_PREFIX_PATH=C:\\Users\\Admin\\anaconda3\\envs\\dl\\Lib\\site-packages', '-DJAVA_HOME=C:\\Users\\Admin\\AppData\\Local\\Programs\\AdoptOpenJDK\\', '-DNUMPY_INCLUDE_DIR=C:\\Users\\Admin\\anaconda3\\envs\\dl\\lib\\site-packages\\numpy\\core\\include', '-DPYTHON_EXECUTABLE=C:\\Users\\Admin\\anaconda3\\envs\\dl\\python.exe', '-DPYTHON_INCLUDE_DIR=C:\\Users\\Admin\\anaconda3\\envs\\dl\\include', '-DPYTHON_LIBRARY=C:\\Users\\Admin\\anaconda3\\envs\\dl/libs/python38.lib', '-DTORCH_BUILD_VERSION=1.7.0a0+4e964f3', '-DUSE_MKLDNN=OFF', '-DUSE_NINJA=OFF', '-DUSE_NUMPY=True', 'C:\\Users\\Admin\\Downloads\\Pytorch\\pytorch']' returned non-zero exit status 1.
I will probably get further when I can finally use ninja, as you can do it. That should be possible. I will start some new tries as soon as I find the time.
I think your problem is easy to solve.because i alread install VS2019 ,so i have msvc tool to build it.
i use Native tools command prompt for vs 2019 to run python setup.py build --cmake
just keep to try again.good luck friend
1 Like
Thanks for the motivation, it is more important than the solution after so many tries. I am surely backing your friend here in this weird challenge of my computer’s nerves :))).
That is understood, I also have MSVC2019 installed, and setting ninja off is using that MSVC successfully. The MSVC error appears as soon as I use ninja, which is made to improve installation speed (at the moment, with ninja set OFF, I guess it is about seven hours running time for a successful run and about three for an erroneous run). Up to now, I have installed from anaconda prompt, I will try your recommended Native tools command prompt for vs 2019 as soon as I find the time, probably in a few weeks only.
I tried it with Native tools command prompt for vs 2019. I started the process with
C:\Windows\System32>cd C:\Users\Admin\Downloads\Pytorch\pytorch
C:\Users\Admin\Downloads\Pytorch\pytorch>set "CMAKE_INCLUDE_PATH=C:\Users\Admin\Downloads\Pytorch\mkl\include"
C:\Users\Admin\Downloads\Pytorch\pytorch>set "LIB=C:\Users\Admin\Downloads\Pytorch\mkl\lib;%LIB%"
C:\Users\Admin\Downloads\Pytorch\pytorch>set USE_NINJA=ON
C:\Users\Admin\Downloads\Pytorch\pytorch>set CMAKE_GENERATOR=Ninja
set USE_MKLDNN=ON
C:\Users\Admin\Downloads\Pytorch\pytorch>"C:\Users\Admin\anaconda3\python.exe" "setup.py" install --cmake
After less than a minute, I get:
C:\Users\Admin\anaconda3\lib\site-packages\numpy\__init__.py:138: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service
from . import _distributor_init
Building wheel torch-1.8.0a0
-- Building version 1.8.0a0
cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_GENERATOR=Ninja -DCMAKE_INCLUDE_PATH=C:\Users\Admin\Downloads\Pytorch\mkl\include -DCMAKE_INSTALL_PREFIX=C:\Users\Admin\Downloads\Pytorch\pytorch\torch -DCMAKE_PREFIX_PATH=C:\Users\Admin\anaconda3\Lib\site-packages -DJAVA_HOME=C:\Users\Admin\AppData\Local\Programs\AdoptOpenJDK\ -DPYTHON_EXECUTABLE=C:\Users\Admin\anaconda3\python.exe -DPYTHON_INCLUDE_DIR=C:\Users\Admin\anaconda3\include -DPYTHON_LIBRARY=C:\Users\Admin\anaconda3/libs/python38.lib -DTORCH_BUILD_VERSION=1.8.0a0 -DUSE_NINJA=ON -DUSE_NUMPY=True C:\Users\Admin\Downloads\Pytorch\pytorch
-- The CXX compiler identification is MSVC 19.27.29111.0
-- The C compiler identification is MSVC 19.27.29111.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Not forcing any particular BLAS to be found
CMake Warning at CMakeLists.txt:234 (message):
TensorPipe cannot be used on Windows. Set it to OFF
CMake Error at CMakeLists.txt:237 (find_library):
Could not find libuv_tmp_LIBRARY using the following names: uv, libuv
-- Configuring incomplete, errors occurred!
See also "C:/Users/Admin/Downloads/Pytorch/pytorch/build/CMakeFiles/CMakeOutput.log".
Traceback (most recent call last):
File "setup.py", line 727, in <module>
build_deps()
File "setup.py", line 314, in build_deps
build_caffe2(version=version,
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\build_pytorch_libs.py", line 54, in build_caffe2
cmake.generate(version,
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\setup_helpers\cmake.py", line 330, in generate
self.run(args, env=my_env)
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\setup_helpers\cmake.py", line 141, in run
check_call(command, cwd=self.build_dir, env=env)
File "C:\Users\Admin\anaconda3\lib\subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '-GNinja', '-DBUILD_PYTHON=True', '-DBUILD_TEST=True', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_GENERATOR=Ninja', '-DCMAKE_INCLUDE_PATH=C:\\Users\\Admin\\Downloads\\Pytorch\\mkl\\include', '-DCMAKE_INSTALL_PREFIX=C:\\Users\\Admin\\Downloads\\Pytorch\\pytorch\\torch', '-DCMAKE_PREFIX_PATH=C:\\Users\\Admin\\anaconda3\\Lib\\site-packages', '-DJAVA_HOME=C:\\Users\\Admin\\AppData\\Local\\Programs\\AdoptOpenJDK\\', '-DPYTHON_EXECUTABLE=C:\\Users\\Admin\\anaconda3\\python.exe', '-DPYTHON_INCLUDE_DIR=C:\\Users\\Admin\\anaconda3\\include', '-DPYTHON_LIBRARY=C:\\Users\\Admin\\anaconda3/libs/python38.lib', '-DTORCH_BUILD_VERSION=1.8.0a0', '-DUSE_NINJA=ON', '-DUSE_NUMPY=True', 'C:\\Users\\Admin\\Downloads\\Pytorch\\pytorch']' returned non-zero exit status 1.
So, how to fix the following?
- UserWarning: mkl-service package failed to import, therefore Intel® MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service
from . import _distributor_init - TensorPipe cannot be used on Windows. Set it to OFF
- “Could not find libuv_tmp_LIBRARY using the following names: uv, libuv”
The previous post is a start from Native tools command prompt for vs 2019 which does not work for me, as it is documented above. That is why I drop this approach and switch back to anaconda prompt.
Using Anaconda Prompt:
(myenv) C:\WINDOWS\system32>cd C:\Users\Admin\Downloads\Pytorch\pytorch
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set “CMAKE_INCLUDE_PATH=C:\Users\Admin\Downloads\Pytorch\mkl\include”
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set “LIB=C:\Users\Admin\Downloads\Pytorch\mkl\lib;%LIB%”
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set USE_NINJA=ON
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set CMAKE_GENERATOR=Ninja
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set USE_MKLDNN=ON
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set “CUDAHOSTCXX=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\bin\Hostx64\x64\cl.exe”
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>python setup.py install --cmake
Which makes:
cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_GENERATOR=Ninja -DCMAKE_INCLUDE_PATH=C:\Users\Admin\Downloads\Pytorch\mkl\include -DCMAKE_INSTALL_PREFIX=C:\Users\Admin\Downloads\Pytorch\pytorch\torch -DCMAKE_PREFIX_PATH=C:\Users\Admin\anaconda3\envs\dl\Lib\site-packages -DJAVA_HOME=C:\Users\Admin\AppData\Local\Programs\AdoptOpenJDK\ -DNUMPY_INCLUDE_DIR=C:\Users\Admin\anaconda3\envs\dl\lib\site-packages\numpy\core\include -DPYTHON_EXECUTABLE=C:\Users\Admin\anaconda3\envs\dl\python.exe -DPYTHON_INCLUDE_DIR=C:\Users\Admin\anaconda3\envs\dl\include -DPYTHON_LIBRARY=C:\Users\Admin\anaconda3\envs\dl/libs/python38.lib -DTORCH_BUILD_VERSION=1.8.0a0 -DUSE_MKLDNN=ON -DUSE_NINJA=ON -DUSE_NUMPY=True C:\Users\Admin\Downloads\Pytorch\pytorch
And this causes again an error, but at least after many hours and not just in the first minute:
CMake Error at torch_cuda_generated_SpectralOps.cu.obj.Release.cmake:281 (message):
Error generating file
C:/Users/Admin/Downloads/Pytorch/pytorch/build/caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/./torch_cuda_generated_SpectralOps.cu.obj
[2365/3377] Building NVCC (Device) object caffe2/CMakeFile...en/native/cuda/torch_cuda_generated_SortingKthValue.cu.obj
SortingKthValue.cu
SortingKthValue.cu
[2366/3377] Building NVCC (Device) object caffe2/CMakeFile...n/src/ATen/native/cuda/torch_cuda_generated_SoftMax.cu.obj
SoftMax.cu
SoftMax.cu
[2367/3377] Building NVCC (Device) object caffe2/CMakeFile...c/ATen/native/cuda/torch_cuda_generated_ScanKernels.cu.obj
ScanKernels.cu
ScanKernels.cu
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "setup.py", line 727, in <module>
build_deps()
File "setup.py", line 314, in build_deps
build_caffe2(version=version,
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\build_pytorch_libs.py", line 62, in build_caffe2
cmake.build(my_env)
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\setup_helpers\cmake.py", line 346, in build
self.run(build_args, my_env)
File "C:\Users\Admin\Downloads\Pytorch\pytorch\tools\setup_helpers\cmake.py", line 141, in run
check_call(command, cwd=self.build_dir, env=env)
File "C:\Users\Admin\anaconda3\envs\dl\lib\subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '4']' returned non-zero exit status 1.
After so many tries, the following has worked for me.
I had to set ninja off. Ninja is in order to speed up the process, too bad that I cannot use it. Without ninja, it ran through the whole night for about 9.5 hours. I also needed to download the source code of MKL, and then, together with the mkl installation and other tricks (which are all documented above), it works:
(myenv) C:\WINDOWS\system32>cd C:\Users\Admin\Downloads\Pytorch\pytorch
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set “CMAKE_INCLUDE_PATH=C:\Users\Admin\Downloads\Pytorch\mkl\include”
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set “LIB=C:\Users\Admin\Downloads\Pytorch\mkl\lib;%LIB%”
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set USE_NINJA=OFF
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set CMAKE_GENERATOR=Visual Studio 16 2019
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set USE_MKLDNN=ON
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set “CUDAHOSTCXX=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\bin\Hostx64\x64\cl.exe”
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>python setup.py install --cmake
This has run through, yet the spyder was corrupted by the cuda install:
(myenv) C:\WINDOWS\system32>spyder
_cffi_ext.c
C:\Users\Admin\anaconda3\lib\site-packages\zmq\backend\cffi\__pycache__\_cffi_ext.c(268): fatal error C1083: Datei (Include) kann nicht geöffnet werden: "zmq.h": No such file or directory
Traceback (most recent call last):
File "C:\Users\Admin\anaconda3\Scripts\spyder-script.py", line 6, in <module>
from spyder.app.start import main
File "C:\Users\Admin\anaconda3\lib\site-packages\spyder\app\start.py", line 22, in <module>
import zmq
File "C:\Users\Admin\anaconda3\lib\site-packages\zmq\__init__.py", line 50, in <module>
from zmq import backend
File "C:\Users\Admin\anaconda3\lib\site-packages\zmq\backend\__init__.py", line 40, in <module>
reraise(*exc_info)
File "C:\Users\Admin\anaconda3\lib\site-packages\zmq\utils\sixcerpt.py", line 34, in reraise
raise value
File "C:\Users\Admin\anaconda3\lib\site-packages\zmq\backend\__init__.py", line 27, in <module>
_ns = select_backend(first)
File "C:\Users\Admin\anaconda3\lib\site-packages\zmq\backend\select.py", line 28, in select_backend
mod = __import__(name, fromlist=public_api)
File "C:\Users\Admin\anaconda3\lib\site-packages\zmq\backend\cython\__init__.py", line 6, in <module>
from . import (constants, error, message, context,
ImportError: DLL load failed while importing error: Das angegebene Modul wurde nicht gefunden.
Installing it over the existing installation again:
(myenv) C:\WINDOWS\system32>conda install spyder
Opening spyder:
(myenv) C:\WINDOWS\system32>spyder
Test according to this:
import torch
torch.__version__
Out[3]: '1.8.0a0+2ab74a4'
torch.cuda.current_device()
Out[4]: 0
torch.cuda.device(0)
Out[5]: <torch.cuda.device at 0x24e6b98a400>
torch.cuda.device_count()
Out[6]: 1
torch.cuda.get_device_name(0)
Out[7]: 'GeForce GT 710'
torch.cuda.is_available()
Out[8]: True
Finally 
++ A compressed summary of all steps can be found on Stack Exchange at python - How to install pytorch FROM SOURCE (with cuda enabled for a deprecated CUDA cc 3.5 of an old gpu) using anaconda prompt on Windows 10? - Super User ++
Hello guys,
I’m facing some problems, I really need pytorch with cuda activated for a univerity assignment. My laptop has a Nvidia geforce 920M so I managed to successfully (with no errors) install pytorch from source following your guide. Anyway, when I import torch the system gives me this error:
PS C:\Users\AZand_000> python
Python 3.8.5 (default, Sep 3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Warning:
This Python interpreter is in a conda environment, but the environment has
not been activated. Libraries may fail to load. To activate this environment
please see https://conda.io/activation
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\__init__.py", line 128, in <module>
raise err
OSError: [WinError 126] Impossibile trovare il modulo specificato. Error loading "C:\ProgramData\Anaconda3\lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll" or one of its dependencies.
I have tried to build torch again trying to use ninja and it has built succesfully again. Anyway, it continues to give me this error. I’ve searched everywhere without success and I don’t want to build it again because it takes more then 24 hours on my laptop.
Please tell me that someone has a easy solution.
Hey that installation is really aweful, 24 hours is a good benchmark. Could it be that you need to use conda activate base in order to start a conda environment at all? See Python is in conda env, but has not been activated in Windows virtual env - Stack Overflow. And after that start the environment where your pytorch has been installed to with activate yourpytorchenvironment? Just to start with.