The PyTorch binaries ship with their own CUDA runtime (in the mentioned case with CUDA11.1) and your local CUDA toolkit will only be used if you build PyTorch from source or compile custom CUDA extensions. To run PyTorch on the GPU, you would need to have a compatible driver, which should be the case in your setup.
I’m not sure I understand this concern, since you would have to download the binaries in any case.