Pytorch is giving an error saying that it had found a bug and telling me to report it to pytorch

1

i have the latest version of cuda ==12.2 but pytorch doesnt support a stable version if cuda==12.2 so i had downloaded pytorch==2.1.0 with a pytorch-cuda=12.1 but when iam running the torch code it is giving me an error saying that found a bug please report to pytorch.

Traceback (most recent call last):
File “/home/ubuntu/miniconda3/envs/akhil_env/bin/fairseq-train”, line 8, in
sys.exit(cli_main())
^^^^^^^^^^
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq_cli/train.py”, line 574, in cli_main
distributed_utils.call_main(cfg, main)
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/distributed/utils.py”, line 404, in call_main
main(cfg, **kwargs)
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq_cli/train.py”, line 205, in main
valid_losses, should_stop = train(cfg, trainer, task, epoch_itr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/contextlib.py”, line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq_cli/train.py”, line 331, in train
log_output = trainer.train_step(samples)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/contextlib.py”, line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/trainer.py”, line 1029, in train_step
raise e
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/trainer.py”, line 974, in train_step
self.task.optimizer_step(
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/tasks/fairseq_task.py”, line 546, in optimizer_step
optimizer.step()
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/optim/fp16_optimizer.py”, line 223, in step
self.fp32_optimizer.step(closure, groups=groups)
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/optim/fairseq_optimizer.py”, line 135, in step
self.optimizer.step(closure)
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/torch/optim/optimizer.py”, line 373, in wrapper
out = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/optim/adam.py”, line 195, in step
state[“exp_avg_sq”] = torch.zeros_like(p_data_fp32)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: handle_0 INTERNAL ASSERT FAILED at “/opt/conda/conda-bld/pytorch_1695392035891/work/c10/cuda/driver_api.cpp”:15, please report a bug to PyTorch.

Screenshot 2023-10-19 at 2.59.56 PM
Screenshot 2023-10-19 at 2.59.56 PM2522×310 33.4 KB

2

Could you post a minimal and executable code snippet reproducing the error by wrapping it into three backticks ```, please?

3

so basically i am using fairseq model in my code and when iam trying to train it it is giving me this error in this file of the model /fairseq/optim/adam.py (adam optimizer) and this is the line where iam getting the error :

            if len(state) == 0:
                state["step"] = 0
                # Exponential moving average of gradient values
                state["exp_avg"] = torch.zeros_like(p_data_fp32)
                # Exponential moving average of squared gradient values
                state["exp_avg_sq"] = torch.zeros_like(p_data_fp32)

Thank you