Pytorch is giving an error saying that it had found a bug and telling me to report it to pytorch

i have the latest version of cuda ==12.2 but pytorch doesnt support a stable version if cuda==12.2 so i had downloaded pytorch==2.1.0 with a pytorch-cuda=12.1 but when iam running the torch code it is giving me an error saying that found a bug please report to pytorch.

Traceback (most recent call last):
File “/home/ubuntu/miniconda3/envs/akhil_env/bin/fairseq-train”, line 8, in
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq_cli/”, line 574, in cli_main
distributed_utils.call_main(cfg, main)
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/distributed/”, line 404, in call_main
main(cfg, **kwargs)
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq_cli/”, line 205, in main
valid_losses, should_stop = train(cfg, trainer, task, epoch_itr)
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/”, line 81, in inner
return func(*args, **kwds)
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq_cli/”, line 331, in train
log_output = trainer.train_step(samples)
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/”, line 81, in inner
return func(*args, **kwds)
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/”, line 1029, in train_step
raise e
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/”, line 974, in train_step
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/tasks/”, line 546, in optimizer_step
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/optim/”, line 223, in step
self.fp32_optimizer.step(closure, groups=groups)
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/optim/”, line 135, in step
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/torch/optim/”, line 373, in wrapper
out = func(*args, **kwargs)
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/optim/”, line 195, in step
state[“exp_avg_sq”] = torch.zeros_like(p_data_fp32)
RuntimeError: handle_0 INTERNAL ASSERT FAILED at “/opt/conda/conda-bld/pytorch_1695392035891/work/c10/cuda/driver_api.cpp”:15, please report a bug to PyTorch.

Could you post a minimal and executable code snippet reproducing the error by wrapping it into three backticks ```, please?

so basically i am using fairseq model in my code and when iam trying to train it it is giving me this error in this file of the model /fairseq/optim/ (adam optimizer) and this is the line where iam getting the error :

            if len(state) == 0:
                state["step"] = 0
                # Exponential moving average of gradient values
                state["exp_avg"] = torch.zeros_like(p_data_fp32)
                # Exponential moving average of squared gradient values
                state["exp_avg_sq"] = torch.zeros_like(p_data_fp32)

Thank you