I recieve this error after few epochs during training on torch. I am using windows, cuda-toolkit version 12.2. it happens randomly during training.
Windows fatal exception: access violation
Current thread 0x000045e0 (most recent call first):
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\pkg_resources_vendor\packaging\version.py”, line 519 in _cmpkey
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\pkg_resources_vendor\packaging\version.py”, line 213 in init
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\utils\module.py”, line 562 in version_leq
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\utils\misc.py”, line 585 in is_module_ver_at_least
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\utils_pytorch_numpy_unification.py”, line 203 in floor_divide
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\utils_pytorch_numpy_unification.py”, line 223 in unravel_index
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\utils.py”, line 578 in generate_pos_neg_label_crop_centers
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\croppad\array.py”, line 1158 in randomize
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\croppad\dictionary.py”, line 982 in randomize
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\croppad\dictionary.py”, line 1000 in call
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\transform.py”, line 98 in _apply_transform
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\transform.py”, line 141 in apply_transform
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\compose.py”, line 111 in execute_compose
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\compose.py”, line 335 in call
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\data\dataset.py”, line 934 in _transform
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\data\dataset.py”, line 112 in getitem
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\torch\utils\data_utils\fetch.py”, line 51 in
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\torch\utils\data_utils\fetch.py”, line 51 in fetch
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\torch\utils\data\dataloader.py”, line 674 in _next_data
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\torch\utils\data\dataloader.py”, line 630 in next
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\data\thread_buffer.py”, line 49 in enqueue_values
File “C:\ProgramData\anaconda3\envs\monai\lib\threading.py”, line 917 in run
File “C:\ProgramData\anaconda3\envs\monai\lib\threading.py”, line 980 in _bootstrap_inner
File “C:\ProgramData\anaconda3\envs\monai\lib\threading.py”, line 937 in _bootstrap
Thread 0x00003858 (most recent call first):
Thread 0x0000512c (most recent call first):
File “C:\ProgramData\anaconda3\envs\monai\lib\threading.py”, line 316 in wait
File “C:\ProgramData\anaconda3\envs\monai\lib\threading.py”, line 581 in wait
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\tqdm_monitor.py”, line 60 in run
File “C:\ProgramData\anaconda3\envs\monai\lib\threading.py”, line 980 in _bootstrap_inner
File “C:\ProgramData\anaconda3\envs\monai\lib\threading.py”, line 937 in _bootstrap
Main thread:
Thread 0x00006a48 (most recent call first):
File “C:\ProgramData\anaconda3\envs\monai\lib\threading.py”, line 316 in wait
File “C:\ProgramData\anaconda3\envs\monai\lib\queue.py”, line 180 in get
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\data\thread_buffer.py”, line 76 in iter
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\data\thread_buffer.py”, line 99 in buffer_iterator
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\data\thread_buffer.py”, line 199 in iter
File “D:\pyhton_files\source_codes_python\new-july-2023\yazdan.py”, line 4217 in train_regression_flexible
File “D:\pyhton_files\source_codes_python\new-july-2023\yazdan.py”, line 4807 in easy_flexible_regression
File “d:\pyhton_files\projects\lowdose_ct_recon\step#4—traintheregressionmodel.py”, line 87 in
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\spyder_kernels\py3compat.py”, line 356 in compat_exec
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 473 in exec_code
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 615 in _exec_file
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 528 in runfile
File “C:\Users\Yazdan\AppData\Local\Temp\ipykernel_27192\612619163.py”, line 1 in
Restarting kernel…
strong text
Thank you for your help