Solution
Since CuDNN minor releases are backward compatible, I could simply remove nvidia-cudnn-cu11==8.7.0.84 shipped with pytorch for now as my system-wide CuDNN is 8.9.
Then, pytorch uses the system-wide cudnn.
pip uninstall -y nvidia-cudnn-cu11
Not sure if this will be the final solution after the merge of [BE]: Update cudnn to 9.1.0.70 by eqy · Pull Request #123475 · pytorch/pytorch · GitHub