Your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension. You would need to properly install a current NVIDIA driver and PyTorch via the provided install instructions from here.
I have had the same problem. The easiest way to work around it (for me), was install cuda toolkit with conda (I assume you are using a conda installation):