PyTorch build in conda environment with poetry

I’m trying to build PyTorch from source so that it uses the system CUDA and cuDNN installs. I am using PyTorch, ONNX and Tensorflow on the same docker image so it makes sense for me for PyTorch to use the same libraries while minimising image size (I could use nightly builds but this defeats the objective of minimising size).
It seems to building fine into a conda env. My problems come when trying to use Poetry to install some packages that depend on torch. No matter whether I build the latest 1.13 or 1.12.1, poetry insists on ‘upgrading’ my built package to the 1.12.1 stable version. I have a feeling that there is something clever I need to do with poetry to make it recognise the built package as satisfying requirements (I’ve checked the dependencies and they only specify torch>=1.4). Also, it might be to do with the fact that the built version number is 1.12.0a0+git664058f which might not be parsed correctly by poetry.
Has anyone got better experience with poetry than me that could help?