How to configure cmake while building pytorch from source code?

studyself · November 23, 2022, 12:42pm

HI,
I need to use FBEGMM in pytorch, and I’m building pytorch from source code.
The errors ask me to set cmake configuration as
-DENABLE_PRECOMPILED_HEADERS=OFF.

I modify cmake.py and try to add -DENABLE_PRECOMPILED_HEADERS=OFF into build_options as

However, the building process prompts:
CMake Warning:
Manually-specified variables were not used by the project:

CMAKE_CFLAGS_ISYSTEM
ENABLE_PRECOMPILED_HEADERS

/Below is Errors from log********** ***********/

[3991/6869] Building CXX object third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/helpers.cc.o
FAILED: third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/helpers.cc.o
/usr/bin/c++ -I/qssd/ptSRC/cmake/…/third_party/benchmark/include -I/qssd/ptSRC/cmake/…/third_party/cudnn_frontend/include -I/qssd/ptSRC/third_party/tensorpipe -I/qssd/ptSRC/build/third_party/tensorpipe -I/qssd/ptSRC/third_party/tensorpipe/third_party/libnop/include -I/qssd/ptSRC/third_party/tensorpipe/third_party/libuv/include -isystem /qssd/ptSRC/cmake/…/third_party/googletest/googlemock/include -isystem /qssd/ptSRC/cmake/…/third_party/googletest/googletest/include -isystem /qssd/ptSRC/third_party/protobuf/src -isystem /opt/intel/mkl/include -isystem /qssd/ptSRC/third_party/gemmlowp -isystem /qssd/ptSRC/third_party/neon2sse -isystem /qssd/ptSRC/third_party/XNNPACK/include -isystem /usr/include -isystem /qssd/ptSRC/third_party/ittapi/include -isystem /qssd/ptSRC/cmake/…/third_party/eigen -I/usr/local/cuda/include -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -O3 -DNDEBUG -fPIC -std=gnu++14 -MD -MT third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/helpers.cc.o -MF third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/helpers.cc.o.d -o third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/helpers.cc.o -c /qssd/ptSRC/third_party/tensorpipe/tensorpipe/channel/helpers.cc
In file included from /usr/include/c++/7/ext/string_conversions.h:41:0,
from /usr/include/c++/7/bits/basic_string.h:6361,
from /usr/include/c++/7/string:52,
from /qssd/ptSRC/third_party/tensorpipe/tensorpipe/channel/helpers.h:13,
from /qssd/ptSRC/third_party/tensorpipe/tensorpipe/channel/helpers.cc:9:
/usr/include/c++/7/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
#include_next <stdlib.h>
^~~~~~~~~~
compilation terminated.
[3993/6869] Building CXX object third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/error.cc.o
FAILED: third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/error.cc.o
/usr/bin/c++ -I/qssd/ptSRC/cmake/…/third_party/benchmark/include -I/qssd/ptSRC/cmake/…/third_party/cudnn_frontend/include -I/qssd/ptSRC/third_party/tensorpipe -I/qssd/ptSRC/build/third_party/tensorpipe -I/qssd/ptSRC/third_party/tensorpipe/third_party/libnop/include -I/qssd/ptSRC/third_party/tensorpipe/third_party/libuv/include -isystem /qssd/ptSRC/cmake/…/third_party/googletest/googlemock/include -isystem /qssd/ptSRC/cmake/…/third_party/googletest/googletest/include -isystem /qssd/ptSRC/third_party/protobuf/src -isystem /opt/intel/mkl/include -isystem /qssd/ptSRC/third_party/gemmlowp -isystem /qssd/ptSRC/third_party/neon2sse -isystem /qssd/ptSRC/third_party/XNNPACK/include -isystem /usr/include -isystem /qssd/ptSRC/third_party/ittapi/include -isystem /qssd/ptSRC/cmake/…/third_party/eigen -I/usr/local/cuda/include -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -O3 -DNDEBUG -fPIC -std=gnu++14 -MD -MT third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/error.cc.o -MF third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/error.cc.o.d -o third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/error.cc.o -c /qssd/ptSRC/third_party/tensorpipe/tensorpipe/channel/error.cc
In file included from /usr/include/c++/7/ext/string_conversions.h:41:0,
from /usr/include/c++/7/bits/basic_string.h:6361,
from /usr/include/c++/7/string:52,
from /qssd/ptSRC/third_party/tensorpipe/tensorpipe/channel/error.h:11,
from /qssd/ptSRC/third_party/tensorpipe/tensorpipe/channel/error.cc:9:
/usr/include/c++/7/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
#include_next <stdlib.h>
^~~~~~~~~~

studyself · November 23, 2022, 12:44pm

I am new for pytorch.
Please help me.
Thank you very much.

khushi-411 · November 23, 2022, 2:07pm

Hi, Could you check the output of the path of CMAKE? I think it is not configured correctly. You can use echo $CMAKE_PREFIX_PATH to check the path.

You can use this command to set up the path and then develop your environment.

export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}

Thanks!

studyself · November 23, 2022, 2:56pm

Hi， Thank you for the reply!
Is $CMAKE_PREFIX_PATH correct?

khushi-411 · November 23, 2022, 3:11pm

Yes, looks correct to me. I think you need to clear the cache first then rebuild. Please use: python setup.py clean. Then follow the build process. Also, I suggest setting flags to build your PyTorch as it takes a long time. You can use this command:

USE_DISTRIBUTED=0 USE_GLOO=0 BUILD_TEST=0 BUILD_CAFFE2=0 USE_CUDA=1 USE_CUDNN=0 USE_ASAN=0 USE_MKLDNN=0 USE_KINETO=0 DEBUG=0  MAX_JOBS=12 USE_XNNPACK=0 USE_FBGEMM=0 USE_NNPACK=0 USE_QNNPACK=0  python setup.py develop

Set USE_FBGEMM=1 since you want to use that.

I wrote a blog ago nearly a year ago on it, here: Installing PyTorch from Source. You can refer to it if you need any help!

studyself · November 23, 2022, 3:21pm

I really clear the cache each time before building.
The error is there all the way:
/usr/include/c++/7/cstdlib:75:15: fatal error: stdlib.h: No such file or directory

studyself · November 23, 2022, 3:44pm

Thank you for your message.
BUT Errors hit me again

I think cmake configuration or gcc has problem.
But I don’t know how to fix it.

[286/2044] Building CXX object third_party/fmt/CMakeFiles/fmt.dir/src/format.cc.o
FAILED: third_party/fmt/CMakeFiles/fmt.dir/src/format.cc.o
/usr/bin/c++ -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DUSE_EXTERNAL_MZCRC -D_FILE_OFFSET_BITS=64 -I/qssd/ptSRC/third_party/onnx -I/qssd/ptSRC/build/third_party/onnx -I/qssd/ptSRC/third_party/foxi -I/qssd/ptSRC/build/third_party/foxi -I/qssd/ptSRC/third_party/fmt/include -isystem /qssd/ptSRC/third_party/protobuf/src -isystem /home/quinnqiu/.conda/envs/wenetruntime/include -isystem /usr/include -isystem /qssd/ptSRC/third_party/ittapi/include -isystem /qssd/ptSRC/cmake/../third_party/eigen -I/usr/local/cuda/include -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -O3 -DNDEBUG -DNDEBUG -fPIC -DTH_HAVE_THREAD -std=gnu++14 -MD -MT third_party/fmt/CMakeFiles/fmt.dir/src/format.cc.o -MF third_party/fmt/CMakeFiles/fmt.dir/src/format.cc.o.d -o third_party/fmt/CMakeFiles/fmt.dir/src/format.cc.o -c /qssd/ptSRC/third_party/fmt/src/format.cc
In file included from /qssd/ptSRC/third_party/fmt/include/fmt/format-inl.h:14:0,
                 from /qssd/ptSRC/third_party/fmt/src/format.cc:8:
/usr/include/c++/7/cmath:45:15: fatal error: math.h: No such file or directory
 #include_next <math.h>
               ^~~~~~~~
compilation terminated.

khushi-411 · November 23, 2022, 4:31pm

Hmm, can you run and paste the output here?

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
python collect_env.py

Mine is: (it works fine!)

Collecting environment information...
PyTorch version: 1.14.0a0+git428d647
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.22.1
Libc version: glibc-2.31

Python version: 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-53-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1650 Ti
Nvidia driver version: 470.141.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False

Versions of relevant libraries:
[pip3] mypy==0.960
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.6
[pip3] torch==1.14.0a0+gitc1cf006
[conda] mkl                       2022.1.0           hc2b9512_224  
[conda] mkl-include               2022.1.0           h06a4308_224  
[conda] numpy                     1.21.6                   pypi_0    pypi
[conda] torch                     1.14.0a0+git0efd4e9          pypi_0    pypi

studyself · November 24, 2022, 12:18am

Should I update my enviroments?

Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.22.1
Libc version: glibc-2.27

Python version: 3.9.15 (main, Nov  4 2022, 16:13:54)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-4.15.0-91-generic-x86_64-with-glibc2.27
Is CUDA available: N/A
CUDA runtime version: 11.2.67
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration:
GPU 0: GeForce GTX 1070
GPU 1: GeForce GTX 1070

Nvidia driver version: 460.91.03
cuDNN version: Probably one of the following:
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7.6.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.7
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

Versions of relevant libraries:
[pip3] numpy==1.23.4
[conda] blas                      1.0                         mkl
[conda] magma-cuda112             2.5.2                         1    pytorch
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-include               2022.1.0           h06a4308_224
[conda] mkl-service               2.4.0            py39h7f8727e_0
[conda] mkl_fft                   1.3.1            py39hd3c417c_0
[conda] mkl_random                1.2.2            py39h51133e4_0
[conda] numpy                     1.23.4           py39h14f4228_0
[conda] numpy-base                1.23.4           py39h31eccc5_0

studyself · November 24, 2022, 6:19am

I have successfully built pytorch from source code on the same machine, but with a different account.
It is strange that I login with the other account, Errors occur.
I try to use the same version of cmake, but it didn’t help.
I try to use the same version of python, neither help.

Just FYI. The successful python collect_env.py:

Collecting environment information...
PyTorch version: 1.14.0a0+gita80e5e7
Is debug build: False
CUDA used to build PyTorch: 11.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.25.0
Libc version: glibc-2.27

Python version: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:18)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-4.15.0-91-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.2.67
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: GeForce GTX 1070
GPU 1: GeForce GTX 1070

Nvidia driver version: 460.91.03
cuDNN version: Probably one of the following:
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7.6.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.7
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==1.14.0a0+gita80e5e7
[conda] magma-cuda112             2.5.2                         1    pytorch
[conda] mkl                       2022.1.0           hc2b9512_224    http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] mkl-include               2022.1.0           h84fe81f_915    http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
[conda] numpy                     1.23.5           py38h7042d01_0    http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
[conda] torch                     1.14.0a0+gita80e5e7           dev_0    <develop>