I train Deformable DETR on crowd human dataset and got this error
Any help …?!
Based on the stacktace and error message it seems that an indexing operation is failing. Rerun the code via CUDA_LAUNCH_BLOCKING=1
and check which operation exactly fails. Then check the min. and max. values of the index tensor as well as the shape of the tensor which should be indexed.
I run the code through command line not a jupyter and this is the training command
python main.py --dataset_file coco --batch_size 2 --coco_path …\crowd2 --output_dir output --epochs 25
How I could add the CUDA_LAUNCH_BLOCKING=1 flag
Add this env variable in front of the terminal command as:
CUDA_LAUNCH_BLOCKING=1 python main.py --dataset_file coco --batch_size 2 --coco_path …\crowd2 --output_dir output --epochs 25
or export
it in your current terminal:
export CUDA_LAUNCH_BLOCKING=1
python main.py --dataset_file coco --batch_size 2 --coco_path …\crowd2 --output_dir output --epochs 25
I am running the code on windows platform (anaconda), so i guess this bash command isn’t supported.
is there any alternative ?
maybe you can use this code
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"