Your install steps focus heavily on installing the CUDA toolkit, which will only be used if you build PyTorch from source or a custom CUDA extension. The pip wheels and conda binaries ship with their own CUDA runtime that you are selecting during the install process.
You would only need to install a proper NVIDIA driver and should be able to use the binaries (unless you want to compile CUDA code and build PyTorch with it or an extension).