How to compile PyTorch with custom CMake options?

I want to compile PyTorch by custom CMake options. More specifically, I want to set paths for Python libraries. It looks like setup.py does not use any environmental variables for setting the Python libraries for compilation. I also do not want to change CMakeLists.txt manually. I wonder if there is an easier way of doing this?