Use pytorch CUDA binraries in a machine without GPU

I have build pytorch in a cuda-enabled machine and I want to use the same binaries in a machine without cuda, but I get a build error:

CMake Error at /libtorch/share/cmake/Caffe2/Caffe2Config.cmake:90 (message):
Your installed Caffe2 version uses CUDA but I cannot find the CUDA
libraries. Please set the proper CUDA prefixes and / or install CUDA.
Call Stack (most recent call first):
/bigdisk/lax/rl/pytorch_prebuilt/cuda/libtorch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)
CMakeLists.txt:31 (find_package)

Is it possible to use cuda binaries for cpu-only machines? One approach seems to be linking manually without using
find_package(Torch REQUIRED)

Does anyone have any suggestions?