Cuda version conundrum

That’s not true, since sm_89 is binary compatible to sm_86 and sm_80 and will thus work with any PyTorch binary using CUDA >= 11.1.

I don’t know where the error message is raised from, but your 4090 is also not a Hopper GPU.

Again, I doubt it’s not detected by CUDA, since the error message is also pointing to the wrong architecture name, so I guess it’s a 3rd party library,
Run a pure PyTorch workload and check that the device is properly recognized and working. A simple smoke test would be python -c "import torch; print(torch.cuda.get_device_properties(0); print(torch.randn(1).cuda())".

PyTorch supports all recent CUDA versions and the nightly binaries ship with 11.8 and 12.1.
Both releases support GPU architectures up to sm_90 (Hopper).