If I understand you correctly, you are not building from source but are installing the pip wheels.
Note that in this case the local CUDA toolkit won’t be used unless you are building a custom CUDA extension (or PyTorch from source of course).
Given the error message you have most likely installed PyTorch with the CUDA 10.2 runtime, which is too old for your Ampere GPU.
Use the CUDA 11 runtime via:
pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113
and it should work (make sure to uninstall the old binaries before installing the new one).