i have the latest version of cuda ==12.2 but pytorch doesnt support a stable version if cuda==12.2 so i had downloaded pytorch==2.1.0 with a pytorch-cuda=12.1 but when iam running the torch code it is giving me an error saying that found a bug please report to pytorch.
Traceback (most recent call last):
File “/home/ubuntu/miniconda3/envs/akhil_env/bin/fairseq-train”, line 8, in
sys.exit(cli_main())
^^^^^^^^^^
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq_cli/train.py”, line 574, in cli_main
distributed_utils.call_main(cfg, main)
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/distributed/utils.py”, line 404, in call_main
main(cfg, **kwargs)
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq_cli/train.py”, line 205, in main
valid_losses, should_stop = train(cfg, trainer, task, epoch_itr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/contextlib.py”, line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq_cli/train.py”, line 331, in train
log_output = trainer.train_step(samples)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/contextlib.py”, line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/trainer.py”, line 1029, in train_step
raise e
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/trainer.py”, line 974, in train_step
self.task.optimizer_step(
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/tasks/fairseq_task.py”, line 546, in optimizer_step
optimizer.step()
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/optim/fp16_optimizer.py”, line 223, in step
self.fp32_optimizer.step(closure, groups=groups)
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/optim/fairseq_optimizer.py”, line 135, in step
self.optimizer.step(closure)
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/torch/optim/optimizer.py”, line 373, in wrapper
out = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/home/ubuntu/miniconda3/envs/akhil_env/lib/python3.11/site-packages/fairseq/optim/adam.py”, line 195, in step
state[“exp_avg_sq”] = torch.zeros_like(p_data_fp32)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: handle_0 INTERNAL ASSERT FAILED at “/opt/conda/conda-bld/pytorch_1695392035891/work/c10/cuda/driver_api.cpp”:15, please report a bug to PyTorch.