Failing to build from source aten_op.cc

Hi

Struggling to build from source. Getting the following error with master

[1296/2859] Building CXX object caffe2/CMakeFiles/caffe2.dir/__/aten/src/ATen/native/cpu/SoftMaxKernel.cpp.DEFAULT.cpp.o
[1297/2859] Building CXX object caffe2/CMakeFiles/caffe2.dir/__/aten/src/ATen/native/cpu/UnaryOpsKernel.cpp.DEFAULT.cpp.o
[1298/2859] Building CXX object caffe2/CMakeFiles/caffe2.dir/__/aten/src/ATen/native/cpu/IndexKernel.cpp.DEFAULT.cpp.o
[1299/2859] Building CXX object caffe2/CMakeFiles/caffe2.dir/__/aten/src/ATen/native/cpu/GridSamplerKernel.cpp.DEFAULT.cpp.o
[1300/2859] Building CXX object caffe2/CMakeFiles/caffe2.dir/__/aten/src/ATen/native/cpu/ReduceOpsKernel.cpp.DEFAULT.cpp.o
[1301/2859] Building CXX object caffe2/CMakeFiles/caffe2.dir/__/aten/src/ATen/native/cpu/CopyKernel.cpp.DEFAULT.cpp.o
[1302/2859] Building CXX object caffe2/CMakeFiles/caffe2.dir/__/aten/src/ATen/native/cpu/CrossKernel.cpp.DEFAULT.cpp.o
[1303/2859] Building CXX object caffe2/CMakeFiles/caffe2.dir/contrib/aten/aten_op.cc.o
FAILED: caffe2/CMakeFiles/caffe2.dir/contrib/aten/aten_op.cc.o 
/usr/bin/g++  -DAT_PARALLEL_OPENMP=1 -DCPUINFO_SUPPORTED_PLATFORM=1 -DGFLAGS_IS_A_DLL=0 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DMAGMA_V2 -DNNP_CONVOLUTION_ONLY=0 -DNNP_INFERENCE_ONLY=0 -DONNX_NAMESPACE=onnx_torch -DUSE_C11_ATOMICS=1 -DUSE_CUDA -D_FILE_OFFSET_BITS=64 -D_THP_CORE -Dcaffe2_EXPORTS -I../aten/src -I. -I../ -I../cmake/../third_party/benchmark/include -Icaffe2/contrib/aten -I../third_party/onnx -Ithird_party/onnx -I../third_party/foxi -Ithird_party/foxi -I../caffe2/../torch/csrc/api -I../caffe2/../torch/csrc/api/include -I../caffe2/aten/src/TH -Icaffe2/aten/src/TH -I../caffe2/../torch/../aten/src -Icaffe2/aten/src -Iaten/src -Icaffe2/../aten/src -Icaffe2/../aten/src/ATen -I../caffe2/../torch/csrc -I../caffe2/../torch/../third_party/miniz-2.0.8 -I../aten/src/TH -I../aten/../third_party/catch/single_include -I../aten/src/ATen/.. -Icaffe2/aten/src/ATen -I../caffe2/core/nomnigraph/include -I../third_party/miniz-2.0.8 -I/opt/cuda/include -I../c10/.. -I../third_party/QNNPACK/include -I../third_party/pthreadpool/include -I../third_party/NNPACK/include -I../third_party/cpuinfo/include -I../third_party/fbgemm/include -I../third_party/fbgemm -I../third_party/FP16/include -isystem third_party/gloo -isystem ../cmake/../third_party/gloo -isystem ../cmake/../third_party/googletest/googlemock/include -isystem ../cmake/../third_party/googletest/googletest/include -isystem ../third_party/gemmlowp -isystem ../third_party/neon2sse -isystem ../third_party -isystem /usr/include/opencv4 -isystem ../cmake/../third_party/eigen -isystem /usr/include/python3.7m -isystem /usr/lib/python3.7/site-packages/numpy/core/include -isystem ../cmake/../third_party/cub -isystem /opt/magma/include -isystem include -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow -DHAVE_AVX_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3  -fPIC   -DCAFFE2_USE_GLOO -DCUDA_HAS_FP16=1 -DHAVE_GCC_GET_CPUID -DUSE_AVX -DUSE_AVX2 -DTH_HAVE_THREAD -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wno-write-strings -Wno-unknown-pragmas -Wno-missing-braces -Wno-maybe-uninitialized -fvisibility=hidden -DCAFFE2_BUILD_MAIN_LIB -O2 -pthread -std=gnu++11 -MD -MT caffe2/CMakeFiles/caffe2.dir/contrib/aten/aten_op.cc.o -MF caffe2/CMakeFiles/caffe2.dir/contrib/aten/aten_op.cc.o.d -o caffe2/CMakeFiles/caffe2.dir/contrib/aten/aten_op.cc.o -c ../caffe2/contrib/aten/aten_op.cc
In file included from ../caffe2/contrib/aten/aten_op.cc:1:
../caffe2/contrib/aten/aten_op.h:1:10: fatal error: caffe2/caffe2/contrib/aten/gen_aten_op.h: No such file or directory
 #include "caffe2/caffe2/contrib/aten/gen_aten_op.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[1304/2859] Building CXX object caffe2/CMakeFiles/caffe2.dir/__/aten/src/ATen/native/cpu/DistanceOpsKernel.cpp.DEFAULT.cpp.o
[1305/2859] Building CXX object caffe2/CMakeFiles/caffe2.dir/__/aten/src/ATen/native/cpu/BinaryOpsKernel.cpp.DEFAULT.cpp.o
[1306/2859] Building CXX object caffe2/CMakeFiles/caffe2.dir/__/aten/src/ATen/native/cpu/Activation.cpp.DEFAULT.cpp.o
ninja: build stopped: subcommand failed.
Building wheel torch-1.2.0a0+e07d945
-- Building version 1.2.0a0+e07d945
['cmake',
 '-GNinja',
 '-DBUILDING_WITH_TORCH_LIBS=ON',
 '-DBUILD_BINARY=True',
 '-DBUILD_CAFFE2_OPS=True',
 '-DBUILD_PYTHON=True',
 '-DBUILD_SHARED_LIBS=ON',
 '-DBUILD_TEST=True',
 '-DCAFFE2_STATIC_LINK_CUDA=False',
 '-DCMAKE_BUILD_TYPE=Release',
 '-DCMAKE_CXX_FLAGS= ',
 '-DCMAKE_C_FLAGS= ',
 '-DCMAKE_EXE_LINKER_FLAGS=',
 '-DCMAKE_INSTALL_PREFIX=/home/henry/Downloads/MachineLearning/pytorch/pytorch/torch',
 '-DCMAKE_PREFIX_PATH=/usr/lib/python3.7/site-packages',
 '-DCMAKE_SHARED_LINKER_FLAGS=',
 '-DINSTALL_TEST=True',
 '-DNAMEDTENSOR_ENABLED=False',
 '-DNCCL_EXTERNAL=True',
 '-DNCCL_INCLUDE_DIR=/usr/include',
 '-DNCCL_ROOT_DIR=/usr/',
 '-DNCCL_SYSTEM_LIB=/usr/lib/libnccl.so.2.3.5',
 '-DNUMPY_INCLUDE_DIR=/usr/lib/python3.7/site-packages/numpy/core/include',
 '-DONNX_ML=True',
 '-DONNX_NAMESPACE=onnx_torch',
 '-DPYTHON_EXECUTABLE=/usr/bin/python3',
 '-DPYTHON_INCLUDE_DIR=/usr/include/python3.7m',
 '-DPYTHON_LIBRARY=/usr/lib/libpython3.7m.so.1.0',
 '-DTHD_SO_VERSION=1',
 '-DTORCH_BUILD_VERSION=1.2.0a0+e07d945',
 '-DUSE_ASAN=False',
 '-DUSE_CUDA=True',
 '-DUSE_DISTRIBUTED=True',
 '-DUSE_FBGEMM=True',
 '-DUSE_FFMPEG=False',
 '-DUSE_LEVELDB=False',
 '-DUSE_LMDB=False',
 '-DUSE_MKLDNN=False',
 '-DUSE_NCCL=True',
 '-DUSE_NNPACK=True',
 '-DUSE_NUMPY=True',
 '-DUSE_OPENCV=True',
 '-DUSE_QNNPACK=True',
 '-DUSE_ROCM=False',
 '-DUSE_SYSTEM_EIGEN_INSTALL=OFF',
 '-DUSE_SYSTEM_NCCL=True',
 '-DUSE_TENSORRT=False',
 '/home/henry/Downloads/MachineLearning/pytorch/pytorch']

I’m using the following build command

cd pytorch

python setup.py clean

git pull --all
git submodule sync
git submodule update --init --recursive

CC=gcc \
CXX=g++ \
CUDAHOSTCXX=g++ \
MAX_JOBS=4 \
USE_CUDA=1 \
USE_CUDNN=1 \
USE_OPENCV=1 \
USE_MKLDNN=0 \
BUILD_BINARY=1 \
CUDA_HOME=/opt/cuda \
CUDNN_LIB_DIR=/usr/lib \
CUDNN_INCLUDE_DIR=/usr/include \
MAGMA_HOME=/opt/magma \
TORCH_CUDA_ARCH_LIST="3.5 5.2 6.0 6.1 7.0+PTX" \
TORCH_NVCC_FLAGS="-Xfatbin -compress-all" \
python3 setup.py install --user

Can anyone lend a hand?

Thanks in advance :slight_smile:

Figured it out myself after a little while, it seems USE_NINJA=OFF does the trick. It seems like using Make instead of Ninja causes the build to run normally