Install with pdm problem

I want to install pytorch 2.2.0 by pdm with cache on because I have many projects using the same version of pytorch. I tried many python package manager such as pip, conda, venv, pipenv, poetry. But If I want to isolate each project with a environment, pytorch will be installed again and again. And I found PDM which can install package in a central cache and each project use symlink to share that package.
I can use this method to install other package such as click. But when I install pytorch 2.2.0, it failed. I know it’s a problem related to PDM, but I can’t find solution there. So I post a question here. More information can be found here.