Error "RuntimeError: CUDA error: device-side assert triggered"

Hello! I have a problem with training a neural network on yolov8. Here is the error “RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.”
The error appeared unexpectedly, I started training and everything was fine, but on the next launch this error appeared. I launch it in the console (Windows 11) using the command “yolo task=detect mode=train epochs=20 data=C:\AI\Realtime\data_custom.yaml imgsz=640”. Video card nvidia 4070, Cuda installed (cuda_12.2.2_537.13_windows), video card drivers are the newest, tried reinstalling torch, did not help. torch.version ‘2.0.1+cu118’.

Rerun your code with blocking launches as explained in the error message and narrow down the failing line of code.

I run it not through code, but through the console. How can I set the value here? If it is not possible through a command in the console, then how will it look in code?

You can export env variables in your terminal.