The locally installed CUDA toolkit won’t be used if you install the PyTorch binaries as they ship with their own CUDA dependencies. Downgrading won’t make installing PyTorch easier or harder.
Which Python version are you using and is it still a supported one?