PyTorch compiled from source for Windows is failing when importing torch

vision
1

Hello, I am getting this error when compiling PyTorch from source for Windows 10.
Since my GPU (GTX Titan Black) has compute capability version 3.5 and PyTorch current binaries are compatible with >= 3.7 I guess I have to compile it from source for getting support for my device.
I will try to describe the process I have followed:
My environment:
Windows 10
Visual Studio 2017
Anaconda3 2020.02 (Python 3.7.6 64-bit)
CUDA 10.1 with cuDNN 7.6.5

I have followed the instruction from PyTorch github repository for compiling from source for Windows:

As I use Ninja for generating CMake, after clonning the repository I just launch:
python setup.py install

It seems that all ends without error but when I check it in Python, after “import torch” I am getting this error:
Python 3.7.6 (default, Jan 8 2020, 20:23:39) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type “help”, “copyright”, “credits” or “license” for more information.

import torch
Traceback (most recent call last):
File “”, line 1, in
File “C:\dev\anaconda\lib\site-packages\torch_init_.py”, line 76, in
ctypes.CDLL(dll)
File “C:\dev\anaconda\lib\ctypes_init_.py”, line 364, in init
self._handle = _dlopen(self._name, mode)
OSError: [WinError 1114] A dynamic link library (DLL) initialization routine failed

If I modify init.py manually at line 76 and add a print to the for loop for getting the dll names loading:
for dll in dlls:
print (dll)
ctypes.CDLL(dll)

The output I get is:
Python 3.7.6 (default, Jan 8 2020, 20:23:39) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type “help”, “copyright”, “credits” or “license” for more information.

import torch
C:\dev\anaconda\lib\site-packages\torch\lib\asmjit.dll
C:\dev\anaconda\lib\site-packages\torch\lib\c10.dll
C:\dev\anaconda\lib\site-packages\torch\lib\c10_cuda.dll
C:\dev\anaconda\lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll
Traceback (most recent call last):
File “”, line 1, in
File “C:\dev\anaconda\lib\site-packages\torch_init_.py”, line 77, in
ctypes.CDLL(dll)
File “C:\dev\anaconda\lib\ctypes_init_.py”, line 364, in init
self._handle = _dlopen(self._name, mode)
OSError: [WinError 1114] A dynamic link library (DLL) initialization routine failed

Could anyone help me with this issue?
Thanks in advance for your time and your attention.

Regards

2

Usually, it is because you have some conflicting DLLs in the system directory, including libiomp5md.dll.

3

@peterjc123 Thanks for your response. I am checking for libiomp5md.dll and it seems it is in intel-openmp package and $(anaconda_install)\Library\bin. I have uninstalled intel-openmp but the problem is still there.
After your comment that some “conflicting DLLs…” I have checked the dependencies of the library that produces the error caffe2_detectron_ops_gpu.dll, and based on its output, I don´t see dependency with the openmp library and other dependencies exist as well. Is there a way that ctypes.CDLL shows which dll dependency fails? Or other solution that I could try?

Thanks again

4

Please capture the logs during import torch using Process Monitor and share with me.

5

Ok, this is the complete log for the import torch process…

Just before exiting the process, after loading “caffe2_detectron_ops_gpu.dll”, the invalid results are from … which I am not sure what problem could be…

“8:31:43,7081797”,“python.exe”,“24076”,“QueryNormalizedNameInformationFile”,“C:\dev\anaconda\Lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll”,“SUCCESS”,“”
“8:31:43,7094063”,“python.exe”,“24076”,“CreateFile”,“C:\Users\TestUser<stdin>”,“NAME INVALID”,“Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: n/a”
“8:31:43,7097122”,“python.exe”,“24076”,“CreateFile”,“C:\Users\TestUser<stdin>”,“NAME INVALID”,“Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: n/a”
“8:31:43,7098654”,“python.exe”,“24076”,“CreateFile”,“C:\dev\anaconda\python37.zip<stdin>”,“PATH NOT FOUND”,“Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: n/a”
“8:31:43,7100966”,“python.exe”,“24076”,“CreateFile”,“C:\dev\anaconda\DLLs<stdin>”,“NAME INVALID”,“Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: n/a”
“8:31:43,7104754”,“python.exe”,“24076”,“CreateFile”,“C:\dev\anaconda\Lib<stdin>”,“NAME INVALID”,“Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: n/a”
“8:31:43,7108174”,“python.exe”,“24076”,“CreateFile”,“C:\dev\anaconda<stdin>”,“NAME INVALID”,“Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: n/a”
“8:31:43,7114373”,“python.exe”,“24076”,“CreateFile”,“C:\dev\anaconda\Lib\site-packages<stdin>”,“NAME INVALID”,“Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: n/a”
“8:31:43,7116744”,“python.exe”,“24076”,“CreateFile”,“C:\dev\anaconda\Lib\site-packages\win32<stdin>”,“NAME INVALID”,“Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: n/a”
“8:31:43,7119107”,“python.exe”,“24076”,“CreateFile”,“C:\dev\anaconda\Lib\site-packages\win32\lib<stdin>”,“NAME INVALID”,“Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: n/a”
“8:31:43,7121061”,“python.exe”,“24076”,“CreateFile”,“C:\dev\anaconda\Lib\site-packages\pythonwin<stdin>”,“NAME INVALID”,“Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: n/a”
“8:31:43,7123956”,“python.exe”,“24076”,“CreateFile”,“C:\dev\anaconda\Lib\site-packages\torch_init_.py”,“SUCCESS”,“Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: n/a, OpenResult: Opened”
“8:31:43,7125289”,“python.exe”,“24076”,“QueryEAFile”,“C:\dev\anaconda\Lib\site-packages\torch_init_.py”,“SUCCESS”,“”
“8:31:43,7221640”,“python.exe”,“24076”,“QueryInformationVolume”,“C:\dev\anaconda\Lib\site-packages\torch_init_.py”,“SUCCESS”,“VolumeCreationTime: 24/04/2019 21:45:47, VolumeSerialNumber: AA50-113C, SupportsObjects: True, VolumeLabel: "
“8:31:43,7222032”,“python.exe”,“24076”,“QueryAllInformationFile”,“C:\dev\anaconda\Lib\site-packages\torch_init_.py”,“BUFFER OVERFLOW”,“CreationTime: 06/04/2020 15:29:55, LastAccessTime: 07/04/2020 9:12:01, LastWriteTime: 07/04/2020 9:12:01, ChangeTime: 07/04/2020 9:12:01, FileAttributes: A, AllocationSize: 16.384, EndOfFile: 13.409, NumberOfLinks: 1, DeletePending: False, Directory: False, IndexNumber: 0x2000000157d2f, EaSize: 0, Access: Generic Read, Position: 0, Mode: Synchronous IO Non-Alert, AlignmentRequirement: Long”
“8:31:43,7223340”,“python.exe”,“24076”,“ReadFile”,“C:\dev\anaconda\Lib\site-packages\torch_init_.py”,“SUCCESS”,“Offset: 0, Length: 4.096, Priority: Normal”
“8:31:43,7225964”,“python.exe”,“24076”,“ReadFile”,“C:\dev\anaconda\Lib\site-packages\torch_init_.py”,“SUCCESS”,“Offset: 0, Length: 8.192”
“8:31:43,7227418”,“python.exe”,“24076”,“CloseFile”,“C:\dev\anaconda\Lib\site-packages\torch_init_.py”,“SUCCESS”,”"
“8:31:43,7230954”,“python.exe”,“24076”,“CreateFile”,“C:\dev\anaconda\Lib\ctypes_init_.py”,“SUCCESS”,“Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: n/a, OpenResult: Opened”
“8:31:43,7232210”,“python.exe”,“24076”,“QueryEAFile”,“C:\dev\anaconda\Lib\ctypes_init_.py”,“SUCCESS”,“”
“8:31:43,7404920”,“python.exe”,“24076”,“QueryInformationVolume”,“C:\dev\anaconda\Lib\ctypes_init_.py”,“SUCCESS”,“VolumeCreationTime: 24/04/2019 21:45:47, VolumeSerialNumber: AA50-113C, SupportsObjects: True, VolumeLabel: "
“8:31:43,7405325”,“python.exe”,“24076”,“QueryAllInformationFile”,“C:\dev\anaconda\Lib\ctypes_init_.py”,“BUFFER OVERFLOW”,“CreationTime: 06/04/2020 11:48:22, LastAccessTime: 06/04/2020 11:48:22, LastWriteTime: 18/12/2019 20:48:49, ChangeTime: 06/04/2020 11:48:55, FileAttributes: A, AllocationSize: 20.480, EndOfFile: 17.574, NumberOfLinks: 2, DeletePending: False, Directory: False, IndexNumber: 0x200000014f30c, EaSize: 0, Access: Generic Read, Position: 0, Mode: Synchronous IO Non-Alert, AlignmentRequirement: Long”
“8:31:43,7406588”,“python.exe”,“24076”,“ReadFile”,“C:\dev\anaconda\Lib\ctypes_init_.py”,“SUCCESS”,“Offset: 0, Length: 4.096, Priority: Normal”
“8:31:43,7407871”,“python.exe”,“24076”,“ReadFile”,“C:\dev\anaconda\Lib\ctypes_init_.py”,“SUCCESS”,“Offset: 0, Length: 8.192”
“8:31:43,7409187”,“python.exe”,“24076”,“ReadFile”,“C:\dev\anaconda\Lib\ctypes_init_.py”,“SUCCESS”,“Offset: 8.192, Length: 8.192”
“8:31:43,7410093”,“python.exe”,“24076”,“CloseFile”,“C:\dev\anaconda\Lib\ctypes_init_.py”,“SUCCESS”,”"
“8:32:25,6834947”,“python.exe”,“24076”,“Thread Exit”,“”,“SUCCESS”,“Thread ID: 21572, User Time: 0.0000000, Kernel Time: 0.0156250”
“8:32:25,6834965”,“python.exe”,“24076”,“Thread Exit”,“”,“SUCCESS”,“Thread ID: 1832, User Time: 0.0000000, Kernel Time: 0.0625000”
“8:32:25,6835112”,“python.exe”,“24076”,“Thread Exit”,“”,“SUCCESS”,“Thread ID: 22704, User Time: 0.0000000, Kernel Time: 0.0625000”
“8:33:09,6798533”,“python.exe”,“24076”,“Thread Exit”,“”,“SUCCESS”,“Thread ID: 776, User Time: 0.0000000, Kernel Time: 0.0000000”
“8:33:09,6799656”,“python.exe”,“24076”,“Thread Exit”,“”,“SUCCESS”,“Thread ID: 20244, User Time: 0.0000000, Kernel Time: 0.0000000”

6

The file you provided could not be downloaded. What’s more, I can hardly see anything because I couldn’t adjust the size of the column in that form.

7

Sorry it seems the default option in my account did not allow downloads. Now you shoud be able to download it. Thanks and sorry for that

8

The following are the loaded DLLs:

C:\dev\anaconda\python3.dll
C:\dev\anaconda\python3.dll
C:\dev\anaconda\python3.dll
C:\Windows\System32\kernel32.dll
C:\Windows\System32\kernel32.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\asmjit.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\asmjit.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\asmjit.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\c10.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\c10.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\c10.dll
C:\dev\anaconda\msvcp140.dll
C:\dev\anaconda\msvcp140.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\c10_cuda.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\c10_cuda.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\c10_cuda.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll
C:\dev\anaconda\vcomp140.dll
C:\dev\anaconda\vcomp140.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\torch_cuda.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\torch_cpu.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\torch_cuda.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\torch_cpu.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\torch_cuda.dll
C:\Program Files\NVIDIA Corporation\NvToolsExt\bin\x64\nvToolsExt64_1.dll
C:\Program Files\NVIDIA Corporation\NvToolsExt\bin\x64\nvToolsExt64_1.dll
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cusparse64_100.dll
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cusparse64_100.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\torch_cpu.dll
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\curand64_100.dll
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\curand64_100.dll
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cudnn64_7.dll
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cudnn64_7.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\torch_cpu.dll
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cufft64_100.dll
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cufft64_100.dll
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cublas64_100.dll
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cublas64_100.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\fbgemm.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\fbgemm.dll
C:\dev\anaconda\Lib\site-packages\torch\lib\fbgemm.dll

The interesting fact is that it tries to load the DLLs from CUDA 10.0 but you say that you are trying to build against CUDA 10.1. Maybe you can make sure you are not mixing the two versions of CUDA during build by executing the commands before build.

set CUDA_PATH=%CUDA_PATH_V10_1%
set "PATH=%CUDA_PATH%\bin;%CUDA_PATH%\libnvvp;%PATH%"
9

Hi again, sorry for the delay.

I have re-built PyTorch making sure CUDA_PATH points to 10.1, but the problem is still there. I have checked the libraries loaded and now they come from CUDA\v10.1… I do not know where the problem comes from. Maybe I need to build disabling some features?
Thanks again for your help

12

I guess the cause is somehow related to OpenMP. Would you please add MKL to build to verify my hypothesis? As for the installation script, you can refer to https://github.com/pytorch/builder/blob/master/windows/build_pytorch.bat#L68-L73.

Update:
A fast way to do that verification is to run the following code before importing torch:

import ctypes
ctypes.CDLL(r"C:\Windows\System32\vcomp140.dll")

If this fixes your problem, then the DLL from Anaconda C:\dev\anaconda\vcomp140.dll may be incompatible.

13

You were correct that somehow was related with OpenMP. I did not find which problem exactly was since I had installed MKL and built using it. However your suggestion of using “builder” git repository has been the solution for me.
I have built it (pytorh .whl and libtorch package) using that repository (with settings env variables) and the result is good for me. I guess using setup.py, I were missing something that did not resolve the MKL dependency as it needs it. Now I have it with compute capabitly 3.5.

Thanks a lot for your time and your help

14

I have the very same error (OSError: [WinError 1114]) with the binary distributed in official windows pip nightlies. Any clue which dll is incompatible or missing?

15

Could you please attach the VS debugger and post the stacktrace and the loaded DLLs?

16

I’m working with pycharm and the official python binaries. How can I list the loaded dlls?

17

For the loaded DLLs, you could use the following code:

import psutil, os
p = psutil.Process( os.getpid() )
for dll in p.memory_maps():
  print(dll.path)
18

Ok I ran this

import psutil, os
p = psutil.Process( os.getpid() )
for dll in p.memory_maps():
  print(dll.path)

import torch

and got that

C:\Windows\System32\l_intl.nls
C:\Windows\System32\C_1252.NLS
C:\Windows\System32\C_850.NLS
C:\Windows\System32\locale.nls
C:\Windows\Globalization\Sorting\SortDefault.nls
C:\Program Files\WindowsApps\Microsoft.LanguageExperiencePackfr-FR_19611.1.1.0_neutral__8wekyb3d8bbwe\Windows\System32\fr-FR\KernelBase.dll.mui
C:\Program Files\WindowsApps\Microsoft.LanguageExperiencePackfr-FR_19611.1.1.0_neutral__8wekyb3d8bbwe\Windows\System32\fr-FR\kernel32.dll.mui
C:\Users\julie\AppData\Local\Programs\Python\Python37\python.exe
C:\Users\julie\AppData\Local\Programs\Python\Python37\python37.dll
C:\Windows\System32\pdh.dll
E:\venv\Lib\site-packages\psutil\_psutil_windows.cp37-win_amd64.pyd
C:\Users\julie\AppData\Local\Programs\Python\Python37\DLLs\_lzma.pyd
C:\Users\julie\AppData\Local\Programs\Python\Python37\DLLs\_bz2.pyd
C:\Users\julie\AppData\Local\Programs\Python\Python37\DLLs\_socket.pyd
C:\Users\julie\AppData\Local\Programs\Python\Python37\DLLs\select.pyd
C:\Windows\System32\version.dll
C:\Windows\System32\wtsapi32.dll
C:\Users\julie\AppData\Local\Programs\Python\Python37\vcruntime140.dll
C:\Users\julie\AppData\Local\Programs\Python\Python37\python3.dll
C:\Windows\System32\apphelp.dll
C:\Windows\System32\rsaenh.dll
C:\Windows\System32\IPHLPAPI.DLL
C:\Windows\System32\cryptbase.dll
C:\Windows\System32\cryptsp.dll
C:\Windows\System32\winsta.dll
C:\Windows\System32\umpdc.dll
C:\Windows\System32\powrprof.dll
C:\Windows\System32\bcrypt.dll
C:\Windows\System32\gdi32full.dll
C:\Windows\System32\win32u.dll
C:\Windows\System32\msvcp_win.dll
C:\Windows\System32\bcryptprimitives.dll
C:\Windows\System32\KernelBase.dll
C:\Windows\System32\ucrtbase.dll
C:\Windows\System32\psapi.dll
C:\Windows\System32\msvcrt.dll
C:\Windows\System32\kernel32.dll
C:\Windows\System32\gdi32.dll
C:\Windows\System32\ws2_32.dll
C:\Windows\System32\shlwapi.dll
C:\Windows\System32\user32.dll
C:\Windows\System32\imm32.dll
C:\Windows\System32\shell32.dll
C:\Windows\System32\rpcrt4.dll
C:\Windows\System32\sechost.dll
C:\Windows\System32\advapi32.dll
C:\Windows\System32\ntdll.dll
Traceback (most recent call last):
  File "E:\test.py", line 6, in <module>
    import torch
  File "E:\venv\lib\site-packages\torch\__init__.py", line 80, in <module>
    ctypes.CDLL(dll)
  File "C:\Users\julie\AppData\Local\Programs\Python\Python37\lib\ctypes\__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: [WinError 1114] Une routine d’initialisation d’une bibliothèque de liens dynamiques (DLL) a échoué
19

The probably reason is that you are using French which triggered some bug that is related to locale. Would you please build a RelWithDebInfo build of PyTorch and attach the JIT debugger of Visual Studio? If you couldn’t do that, I can help you to build it.

20

I don’t feel like setting up a c++ environment and recompiling torch with all dependencies. Is there a simpler way to narrow this down?

21

No, since this is a C++ problem and we don’t know where it is. So I guess you need to do that.

22

Ok, what’s the easiest way to compile it (without cuda, torchvision, etc?)