When I use amp for accelarate the model, i met the problem“RuntimeError: CUDA error: device-side assert triggered”?

mixed-precision
1

I wangt to use amp for mixed-precision training. But when i set amp.autocast is true:

image
image889×427 42.3 KB

I met the problem as follows:
image
image950×89 7.54 KB

What puzzles me is i can train the model normally, when i set the amp.autocast is false. So how can i to solve the problem?

2

Could you rerun the script via CUDA_LAUNCH_BLOCKING=1 python script.py args and post the stadcktrace here by wrapping it into three backticks ```, please?

3

The same error.

image
image1152×827 102 KB

4

Thanks for the update! The stacktrace points to an invalid indexing operation in:

points[batch_indices, idx, :]

so check the shape of points as well as the min. and max. values of batch_indices and idx and make sure they are valid.

5

But i can train the model normally, when i set the amp.autocast is false. So i don’t think it is the points’value that is the key to the problem.