Cuda error on a NLP transformer

I created an NLP transformer. But I don’t know why I got this error randomly. It’s not related to a specific batch. It raise randomly.

C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\ block: [131,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed.

I have an NVIDIA RTX 2060 OC 6GB

Based on the error message an indexing operation fails due to invalid values in the index tensor.
Rerun your code via CUDA_LAUNCH_BLOCKING=1 python args and check the stacktrace to see which line of code fails.