Installing C++ extensions in pip's sandbox

When using the recent PEPs for pyproject.toml, pip will install and compile projects in a sandbox (pyproject.toml - pip documentation v22.3). Given that PyTorch cares deeply about which variant is installed (CUDA/CPU/…), any C++ extensions being compiled/linked in that environment might target the wrong PyTorch version.
Are there ways to ensure that the PyTorch outside the environment is used during pip installations? Do you have a favorite/recommended method to get around this sandboxing to avoid double installations?

One solution that sprung to mind is to pass the --extra-index-url config to pip during install. But that still means re-installing PyTorch (along with all the other dependencies) and 2) putting a burden on the user to add the (verbose) flag.