This might not be possible if you install the pre-built binaries as it depends how each framework uses the CUDA runtime.
E.g. the PyTorch binaries ship with their own CUDA runtime, which you are selecting during the install process. If you are installing the pip wheels, the PyTorch lib folder will ship with all ibraries while the conda binaries will install the cudatoolkit or cuda conda package. If TF provides conda binaries with the same dependency, both frameworks might be able to share these libs.
However, I also don’t know if TF actually ships with the CUDA runtime or if your system libraries will be used.
If you really depend on the same CUDA toolkit/runtime for both frameworks, the best approach might be to install a CUDA toolkit locally (you wouldn’t need to do it to run PyTorch-only) and build both frameworks from source.