CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1

I train Deformable DETR on crowd human dataset and got this error

Any help …?!

Based on the stacktace and error message it seems that an indexing operation is failing. Rerun the code via CUDA_LAUNCH_BLOCKING=1 and check which operation exactly fails. Then check the min. and max. values of the index tensor as well as the shape of the tensor which should be indexed.

1 Like

I run the code through command line not a jupyter and this is the training command
python main.py --dataset_file coco --batch_size 2 --coco_path …\crowd2 --output_dir output --epochs 25

How I could add the CUDA_LAUNCH_BLOCKING=1 flag

Add this env variable in front of the terminal command as:

CUDA_LAUNCH_BLOCKING=1 python main.py --dataset_file coco --batch_size 2 --coco_path …\crowd2 --output_dir output --epochs 25

or export it in your current terminal:

export CUDA_LAUNCH_BLOCKING=1
python main.py --dataset_file coco --batch_size 2 --coco_path …\crowd2 --output_dir output --epochs 25

I am running the code on windows platform (anaconda), so i guess this bash command isn’t supported.
is there any alternative ?

maybe you can use this code

os.environ['CUDA_LAUNCH_BLOCKING'] = "1"