How to resolve this error

1

2023-08-18 18:45:28 Exception message: CUDA error: an illegal memory access was encountered
2023-08-18 18:45:28 CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
2023-08-18 18:45:28 For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2023-08-18 18:45:28 Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Strangely, the above error was resolved by increasing batch size. Why?
I dont want to increase the batch size and solve this error. Is there any other way to resolve this

2

Rerun your code with compute-sanitizer python script.py args and post the output here.

3

CUDA_LAUNCH_BLOCKING=1 tried running with this, there is no error.

4

Did you also run it with compute-sanitizer as asked?