The conda binaries and pip wheels are not yet built with cudatoolkit=11.2 and you would have to use 9.2, 10.1, 10.2, or 11.0 as given in the install instructions.
Since these binaries ship with their own CUDA runtime, you would only need a local NVIDIA driver corresponding to the CUDA runtime you are selecting.
In case you want to build PyTorch from source or any custom CUDA extensions, you should install a matching local CUDA toolkit.