I reinstalled Cuda and Cuda dnn before reinstalling Pytorch.
I installed Cuda 11.8 and Cuda dnn 9.5.1 using the GUI.
I am running python in an Anaconda enviornment - there are instructions for installing the Cuda toolkit using Anaconda, is this method required when using an Anaconda environment?
========================================================================
When I run:
torch.cuda.is_available() → True
torch.cuda.is_initialized() → True
torch.cuda.get_device_name() → 'NVIDIA GeForce GTX 1050 Ti'
torch.cuda.get_device_properties(0) → _CudaDeviceProperties(name='NVIDIA GeForce GTX 1050 Ti', major=6, minor=1, total_memory=4095MB, multi_processor_count=6, uuid=08972f5b-ebc1-9f67-6e64-a5b775549112, L2_cache_size=1MB)
When I send a tensor to my device there is no error, and no error in matrix multiplication for float32 or float64.
x = torch.rand((5,5)).to(device = 0)
x @ x works fine.
x = x.to(torch.float64)
x @ x also works fine.
x = x.to(device = 'cpu')
x @ x works fine because it is float64.
x = x.to(torch.float32)
x @ x crashes
To me this indicates that my Cuda functionallity is working.
========================================================================
When the crash occurs there is no traceback reported, using Visual Studio 2022 gives me the following Call Stack:
torch_cpu.dll!00007ff8adc6440d()
torch_cpu.dll!00007ff8adc63688()
torch_cpu.dll!00007ff8adc6395a()
torch_cpu.dll!00007ff8adf30e1b()
torch_cpu.dll!00007ff8adf311b8()
torch_cpu.dll!00007ff8adf31124()
torch_cpu.dll!00007ff8adf2fd29()
torch_cpu.dll!00007ff8adf2fade()
torch_cpu.dll!00007ff8a99e6fdd()
torch_cpu.dll!00007ff8a9f97ab2()
torch_cpu.dll!00007ff8a9f9c03e()
torch_cpu.dll!00007ff8a9a5bd5f()
torch_cpu.dll!00007ff8a9b328fa()
torch_cpu.dll!00007ff8a9b37d0d()
torch_cpu.dll!00007ff8a9b3dca1()
torch_cpu.dll!00007ff8aa817d21()
torch_cpu.dll!00007ff8aa7cf256()
torch_cpu.dll!00007ff8aa5b5055()
torch_cpu.dll!00007ff8abe283d3()
torch_cpu.dll!00007ff8abdf421c()
torch_cpu.dll!00007ff8aa548c01()
torch_cpu.dll!00007ff8a9b35068()
torch_cpu.dll!00007ff8a9b46ef7()
torch_cpu.dll!00007ff8aaaf054e()
torch_cpu.dll!00007ff8aaad1346()
torch_cpu.dll!00007ff8aa64f6c1()
torch_cpu.dll!00007ff8a9731f74()
torch_python.dll!00007ff906501216()
torch_python.dll!00007ff906583043()
torch_python.dll!00007ff9064f139b()
python312.dll!method_vectorcall_VARARGS_KEYWORDS(_object * func, _object * const * args, unsigned __int64 nargsf, _object * kwnames) Line 367
python312.dll!_PyObject_VectorcallTstate(_ts * tstate, _object * callable, _object * const * args, unsigned __int64 nargsf, _object *) Line 92
[Inline Frame] python312.dll!vectorcall_unbound(_ts *) Line 2230
python312.dll!vectorcall_maybe(_ts * tstate, _object * name, _object * const * args, __int64) Line 2283
python312.dll!slot_nb_matrix_multiply(_object * self, _object * other) Line 8570
python312.dll!binary_op1(_object * v, _object * w, const int op_slot) Line 882
[Inline Frame] python312.dll!binary_op(_object *) Line 921
python312.dll!PyNumber_MatrixMultiply(_object * v, _object * w) Line 1119
python312.dll!_PyEval_EvalFrameDefault(_ts * tstate, _PyInterpreterFrame * frame, int throwflag) Line 843
[Inline Frame] python312.dll!_PyEval_EvalFrame(_ts *) Line 89
[Inline Frame] python312.dll!_PyEval_Vector(_ts *) Line 1683
python312.dll!PyEval_EvalCode(_object * co, _object * globals, _object * locals) Line 578
python312.dll!run_eval_code_obj(_ts * tstate, PyCodeObject * co, _object * globals, _object * locals) Line 1723
python312.dll!run_mod(_mod * mod, _object * filename, _object * globals, _object * locals, PyCompilerFlags * flags, _arena * arena) Line 1744
python312.dll!PyRun_InteractiveOneObjectEx(_iobuf * fp, _object * filename, PyCompilerFlags * flags) Line 261
python312.dll!_PyRun_InteractiveLoopObject(_iobuf * fp, _object * filename, PyCompilerFlags * flags) Line 137
[Inline Frame] python312.dll!_PyRun_AnyFileObject(_iobuf *) Line 72
[Inline Frame] python312.dll!PyRun_AnyFileExFlags(_iobuf * fp, const char *) Line 104
python312.dll!pymain_run_stdin(PyConfig * config) Line 520
python312.dll!pymain_run_python(int * exitcode) Line 635
python312.dll!Py_RunMain() Line 711
[Inline Frame] python312.dll!pymain_main(_PyArgv *) Line 739
python312.dll!Py_Main(int argc, wchar_t * * argv) Line 752
========================================================================
The Output section of Visual Studio 2022 provides:
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\python.exe’. Symbols loaded.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\ntdll.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\kernel32.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\KernelBase.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\ucrtbase.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\python312.dll’. Symbols loaded.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\ws2_32.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\rpcrt4.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\bcrypt.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\advapi32.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\msvcrt.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\sechost.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\version.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\vcruntime140.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\zlib.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\bcryptprimitives.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\python3.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\DLLs_ctypes.pyd’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\ole32.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\combase.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\gdi32.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\win32u.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\gdi32full.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\msvcp_win.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\user32.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\oleaut32.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Library\bin\ffi-8.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\imm32.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\DLLs_wmi.pyd’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\propsys.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\vcruntime140_1.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\DLLs_bz2.pyd’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Library\bin\libbz2.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\DLLs_lzma.pyd’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Library\bin\liblzma.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\DLLs_socket.pyd’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\IPHLPAPI.DLL’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\DLLs\select.pyd’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\msvcp140.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\asmjit.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\c10.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\dbghelp.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\c10_cuda.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin\cudart64_110.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\cryptbase.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\caffe2_nvrtc.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin\nvrtc64_112_0.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\shell32.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\cudnn64_9.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\cudnn_adv64_9.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\cudnn_ops64_9.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\cudnn_graph64_9.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\cudnn_cnn64_9.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\cudnn_engines_precompiled64_9.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\nvcuda.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\cudnn_engines_runtime_compiled64_9.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\cudnn_heuristic64_9.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\cupti64_2022.3.0.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\fbgemm.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Library\bin\libiomp5md.dll’. Symbols loaded.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\nvToolsExt64_1.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\shm.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\torch_cuda.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\torch_cpu.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin\cusparse64_11.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin\cufft64_10.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin\cusolver64_11.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin\cublas64_11.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin\cublasLt64_11.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Library\bin\mkl_intel_thread.2.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Library\bin\uv.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\userenv.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\dbgcore.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\torch.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\torch_global_deps.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\torch_python.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch\lib\zlibwapi.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\kernel.appcore.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\uxtheme.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\clbcatq.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\amsi.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\profapi.dll’.
‘python.exe’ (Win32): Loaded ‘C:\ProgramData\Microsoft\Windows Defender\Platform\4.18.24090.11-0\MpOAV.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\torch_C.cp312-win_amd64.pyd’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Library\bin\mkl_core.2.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Library\bin\tbbmalloc.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\mkl_mklinit.cp312-win_amd64.pyd’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Library\bin\mkl_rt.2.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\mkl_py_mkl_service.cp312-win_amd64.pyd’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\numpy\core_multiarray_umath.cp312-win_amd64.pyd’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\numpy\core_multiarray_tests.cp312-win_amd64.pyd’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\numpy\linalg_umath_linalg.cp312-win_amd64.pyd’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\numpy\fft_pocketfft_internal.cp312-win_amd64.pyd’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\numpy\random\mtrand.cp312-win_amd64.pyd’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\numpy\random\bit_generator.cp312-win_amd64.pyd’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\numpy\random_common.cp312-win_amd64.pyd’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\DLLs_hashlib.pyd’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Library\bin\libcrypto-3-x64.dll’. Symbols loaded.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\crypt32.dll’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\numpy\random_bounded_integers.cp312-win_amd64.pyd’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\numpy\random_mt19937.cp312-win_amd64.pyd’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\numpy\random_philox.cp312-win_amd64.pyd’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\numpy\random_pcg64.cp312-win_amd64.pyd’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\numpy\random_sfc64.cp312-win_amd64.pyd’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Lib\site-packages\numpy\random_generator.cp312-win_amd64.pyd’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Library\bin\mkl_def.2.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Library\bin\mkl_vml_def.2.dll’. Module was built without symbols.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\DLLs_queue.pyd’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\DLLs_uuid.pyd’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\DLLs_ssl.pyd’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\Library\bin\libssl-3-x64.dll’. Symbols loaded.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\DLLs\unicodedata.pyd’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\DLLs_multiprocessing.pyd’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\DLLs_asyncio.pyd’.
‘python.exe’ (Win32): Loaded ‘C:\Users\Keith\anaconda3\envs\PT\DLLs_overlapped.pyd’.
‘python.exe’ (Win32): Loaded ‘C:\Windows\System32\mswsock.dll’.
The thread 6752 has exited with code 0 (0x0).
The thread 9244 has exited with code 0 (0x0).
Unhandled exception at 0x00007FF8ADC6440D (torch_cpu.dll) in python.exe: 0xC0000094: Integer division by zero.
========================================================================
Again, the crash only occurs using float32 on cpu. Any help would be greatly appreciated.