Compiling from source with FFMPEG and OPENCV

Hey guys, I’m new to pytorch/caffe2 and am having a really hard time getting it compiled from source with the dependencies I need, which are in this case FFMPEG and OPENCV. I need them compiled together to try to reproduce the environment of a paper that I’m trying to work with.

Well… I’ve tried everything but can’t get it working. I can get it compiled till the end but when I try to run from caffe2.python import core I get the message saying it doesn’t have GPU support. I’m compiling it with the cuda flag and have a titan x at my disposal.

I’m running it on docker with centOS. Using python 2.7, cmake 3.11.4, gcc/g++ 5.4.0, opencv 3.4.4, conda, etc (You can see the rest in the summary below)

That’s the full output I get when compiling it. Did anyone have a similar problem and solved it?

-- ******** Summary ********
--   CMake version         : 3.11.4
--   CMake command         : /usr/local/bin/cmake
--   System                : Linux
--   C++ compiler          : /usr/local/bin/c++
--   C++ compiler version  : 5.4.0
--   CXX flags             :  -Wno-deprecated -fvisibility-inlines-hidden -Wnon-virtual-dtor
--   Build type            : Release
--   Compile definitions   : TH_BLAS_MKL
--   CMAKE_PREFIX_PATH     : /root/minoconda/envs/r21d
--   CMAKE_INSTALL_PREFIX  : /root/miniconda/envs/r21d
--   CMAKE_MODULE_PATH     : /usr/local/pytorch/cmake/Modules;/usr/local/pytorch/cmake/public/../Modules_CUDA_fix
-- 
--   ONNX version          : 1.3.0
--   ONNX NAMESPACE        : onnx_c2
--   ONNX_BUILD_TESTS      : OFF
--   ONNX_BUILD_BENCHMARKS : OFF
--   ONNX_USE_LITE_PROTO   : OFF
--   ONNXIFI_DUMMY_BACKEND : OFF
-- 
--   Protobuf compiler     : 
--   Protobuf includes     : 
--   Protobuf libraries    : 
--   BUILD_ONNX_PYTHON     : OFF
-- Found CUDA with FP16 support, compiling with torch.cuda.HalfTensor
-- Removing -DNDEBUG from compile flags
-- Checking prototype magma_get_sgeqrf_nb for MAGMA_V2 - True
-- Compiling with MAGMA support
-- MAGMA INCLUDE DIRECTORIES: /root/miniconda/envs/r21d/include
-- MAGMA LIBRARIES: /root/miniconda/envs/r21d/lib/libmagma.a
-- MAGMA V2 check: 1
-- Could not find hardware support for NEON on this machine.
-- No OMAP3 processor on this machine.
-- No OMAP4 processor on this machine.
-- Looking for cpuid.h
-- Looking for cpuid.h - found
-- Performing Test HAVE_GCC_GET_CPUID
-- Performing Test HAVE_GCC_GET_CPUID - Success
-- Performing Test NO_GCC_EBX_FPIC_BUG
-- Performing Test NO_GCC_EBX_FPIC_BUG - Success
-- Performing Test C_HAS_AVX_1
-- Performing Test C_HAS_AVX_1 - Failed
-- Performing Test C_HAS_AVX_2
-- Performing Test C_HAS_AVX_2 - Success
-- Performing Test C_HAS_AVX2_1
-- Performing Test C_HAS_AVX2_1 - Failed
-- Performing Test C_HAS_AVX2_2
-- Performing Test C_HAS_AVX2_2 - Success
-- Performing Test CXX_HAS_AVX_1
-- Performing Test CXX_HAS_AVX_1 - Failed
-- Performing Test CXX_HAS_AVX_2
-- Performing Test CXX_HAS_AVX_2 - Success
-- Performing Test CXX_HAS_AVX2_1
-- Performing Test CXX_HAS_AVX2_1 - Failed
-- Performing Test CXX_HAS_AVX2_2
-- Performing Test CXX_HAS_AVX2_2 - Success
-- AVX compiler support found
-- AVX2 compiler support found
-- Performing Test HAS_C11_ATOMICS
-- Performing Test HAS_C11_ATOMICS - Success
-- Atomics: using C11 intrinsics
-- Performing Test BLAS_F2C_DOUBLE_WORKS
-- Performing Test BLAS_F2C_DOUBLE_WORKS - Failed
-- Performing Test BLAS_F2C_FLOAT_WORKS
-- Performing Test BLAS_F2C_FLOAT_WORKS - Success
-- Performing Test BLAS_USE_CBLAS_DOT
-- Performing Test BLAS_USE_CBLAS_DOT - Success
-- Found a library with BLAS API (mkl).
-- Found a library with LAPACK API. (mkl)
-- Found CUDA: /usr/local/cuda (found suitable version "8.0", minimum required is "5.5") 
e[91mdisabling ROCM because NOT USE_ROCM is set
e[0m-- MIOpen not found. Compiling without MIOpen support
-- Found MKLDNN: /root/miniconda/envs/r21d/lib/libmkl_intel_lp64.so;/root/miniconda/envs/r21d/lib/libmkl_gnu_thread.so;/root/miniconda/envs/r21d/lib/libmkl_core.so;-fopenmp;/usr/lib64/libpthread.so;/usr/lib64/libm.so;/usr/lib64/libdl.so  
-- Detecting Intel(R) MKL: trying mklml_intel
-- Detecting Intel(R) MKL: trying mklml_gnu
-- Detecting Intel(R) MKL: trying mklml
-- Detecting Intel(R) MKL: trying mkl_rt
-- Intel(R) MKL: include /root/miniconda/envs/r21d/include
-- Intel(R) MKL: lib /root/miniconda/envs/r21d/lib/libmkl_rt.so
-- Found OpenMP_C: -fopenmp (found version "4.0") 
-- Found OpenMP_CXX: -fopenmp (found version "4.0") 
-- OpenMP lib: -fopenmp
-- Found Doxygen: /usr/bin/doxygen (found version "1.8.5") found components:  doxygen dot 
-- VTune profiling environment is unset
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - found
-- Looking for mmap
-- Looking for mmap - found
-- Looking for shm_open
-- Looking for shm_open - found
-- Looking for shm_unlink
-- Looking for shm_unlink - found
-- Looking for malloc_usable_size
-- Looking for malloc_usable_size - found
-- Performing Test C_HAS_THREAD
-- Performing Test C_HAS_THREAD - Success
-- GCC 5.4.0: Adding gcc and gcc_s libs to link line
-- Check size of long double
-- Check size of long double - done
-- Performing Test COMPILER_SUPPORTS_LONG_DOUBLE
-- Performing Test COMPILER_SUPPORTS_LONG_DOUBLE - Success
-- Performing Test COMPILER_SUPPORTS_FLOAT128
-- Performing Test COMPILER_SUPPORTS_FLOAT128 - Success
-- Performing Test COMPILER_SUPPORTS_SSE2
-- Performing Test COMPILER_SUPPORTS_SSE2 - Success
-- Performing Test COMPILER_SUPPORTS_SSE4
-- Performing Test COMPILER_SUPPORTS_SSE4 - Success
-- Performing Test COMPILER_SUPPORTS_AVX
-- Performing Test COMPILER_SUPPORTS_AVX - Success
-- Performing Test COMPILER_SUPPORTS_FMA4
-- Performing Test COMPILER_SUPPORTS_FMA4 - Success
-- Performing Test COMPILER_SUPPORTS_AVX2
-- Performing Test COMPILER_SUPPORTS_AVX2 - Success
-- Performing Test COMPILER_SUPPORTS_SVE
-- Performing Test COMPILER_SUPPORTS_SVE - Failed
-- Performing Test COMPILER_SUPPORTS_AVX512F
-- Performing Test COMPILER_SUPPORTS_AVX512F - Failed
-- Performing Test COMPILER_SUPPORTS_OPENMP
-- Performing Test COMPILER_SUPPORTS_OPENMP - Success
-- Performing Test COMPILER_SUPPORTS_WEAK_ALIASES
-- Performing Test COMPILER_SUPPORTS_WEAK_ALIASES - Success
-- Performing Test COMPILER_SUPPORTS_BUILTIN_MATH
-- Performing Test COMPILER_SUPPORTS_BUILTIN_MATH - Success
-- Configuring build for SLEEF-v3.2
e[91m   Target system: Linux-4.4.0-139-generic
   Target processor: x86_64
   Host system: Linux-4.4.0-139-generic
   Host processor: x86_64
   Detected C compiler: GNU @ /usr/local/bin/gcc
e[0m-- Using option `-Wall -Wno-unused -Wno-attributes -Wno-unused-result -Wno-psabi -ffp-contract=off -fno-math-errno -fno-trapping-math` to compile libsleef
-- Building shared libs : OFF
-- MPFR : /usr/lib64/libmpfr.so
-- MPFR header file in /usr/include
-- GMP : /usr/lib64/libgmp.so
-- RUNNING_ON_TRAVIS : 0
-- COMPILER_SUPPORTS_OPENMP : 1
-- NCCL operators skipped due to no CUDA support
-- Including IDEEP operators
-- Including image processing operators
-- Including video processing operators
-- MPI operators skipped due to no MPI support
-- Include Observer library
-- Using lib/python2.7/site-packages as python relative installation path
-- Automatically generating missing __init__.py files.
e[91mCMake Warning at CMakeLists.txt:389 (message):
  Generated cmake files are only fully tested if one builds with system glog,
  gflags, and protobuf.  Other settings may generate files that are not well
  tested.


e[0m-- 
-- ******** Summary ********
-- General:
--   CMake version         : 3.11.4
--   CMake command         : /usr/local/bin/cmake
--   System                : Linux
--   C++ compiler          : /usr/local/bin/c++
--   C++ compiler version  : 5.4.0
--   BLAS                  : MKL
--   CXX flags             :  -Wno-deprecated -fvisibility-inlines-hidden -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wno-unused-but-set-variable -Wno-maybe-uninitialized
--   Build type            : Release
--   Compile definitions   : TH_BLAS_MKL;ONNX_NAMESPACE=onnx_c2;MAGMA_V2;USE_C11_ATOMICS=1;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1
--   CMAKE_PREFIX_PATH     : /root/minoconda/envs/r21d
--   CMAKE_INSTALL_PREFIX  : /root/miniconda/envs/r21d
-- 
--   TORCH_VERSION         : 1.0.0
--   CAFFE2_VERSION        : 1.0.0
--   BUILD_ATEN_MOBILE     : OFF
--   BUILD_ATEN_ONLY       : OFF
--   BUILD_BINARY          : OFF
--   BUILD_CUSTOM_PROTOBUF : ON
--     Link local protobuf : ON
--   BUILD_DOCS            : OFF
--   BUILD_PYTHON          : ON
--     Python version      : 2.7.15
--     Python executable   : /root/miniconda/envs/r21d/bin/python
--     Pythonlibs version  : 2.7.15
--     Python library      : /root/miniconda/envs/r21d/lib/python2.7
--     Python includes     : /root/miniconda/envs/r21d/include/python2.7
--     Python site-packages: lib/python2.7/site-packages
--   BUILD_CAFFE2_OPS      : ON
--   BUILD_SHARED_LIBS     : ON
--   BUILD_TEST            : OFF
--   USE_ASAN              : OFF
--   USE_CUDA              : ON
--     CUDA static link    : OFF
--     USE_CUDNN           : ON
--     CUDA version        : 8.0
--     cuDNN version       : 6.0.21
--     CUDA root directory : /usr/local/cuda
--     CUDA library        : /usr/local/cuda/lib64/stubs/libcuda.so
--     cudart library      : /usr/local/cuda/lib64/libcudart_static.a;-pthread;dl;/usr/lib64/librt.so
--     cublas library      : /usr/local/cuda/lib64/libcublas.so;/usr/local/cuda/lib64/libcublas_device.a
--     cufft library       : /usr/local/cuda/lib64/libcufft.so
--     curand library      : /usr/local/cuda/lib64/libcurand.so
--     cuDNN library       : /usr/local/cuda/lib64/libcudnn.so
--     nvrtc               : /usr/local/cuda/lib64/libnvrtc.so
--     CUDA include path   : /usr/local/cuda/include
--     NVCC executable     : /usr/local/cuda/bin/nvcc
--     CUDA host compiler  : /usr/local/bin/gcc
--     USE_TENSORRT        : OFF
--   USE_ROCM              : OFF
--   USE_EIGEN_FOR_BLAS    : 
--   USE_FBGEMM            : OFF
--   USE_FFMPEG            : ON
--   USE_GFLAGS            : ON
--   USE_GLOG              : ON
--   USE_LEVELDB           : ON
--     LevelDB version     : 1.12
--     Snappy version      : 1.1.0
--   USE_LITE_PROTO        : OFF
--   USE_LMDB              : ON
--     LMDB version        : 0.9.22
--   USE_METAL             : OFF
--   USE_MKL               : ON
--   USE_MKLDNN            : ON
--   USE_MOBILE_OPENGL     : OFF
--   USE_NCCL              : OFF
--   USE_NNPACK            : OFF
--   USE_NUMPY             : ON
--   USE_OBSERVERS         : ON
--   USE_OPENCL            : OFF
--   USE_OPENCV            : ON
--     OpenCV version      : 3.4.4
--   USE_OPENMP            : OFF
--   USE_PROF              : OFF
--   USE_QNNPACK           : ON
--   USE_REDIS             : OFF
--   USE_ROCKSDB           : OFF
--   USE_ZMQ               : OFF
--   USE_DISTRIBUTED       : OFF
--   Public Dependencies  : Threads::Threads;caffe2::mkl;glog::glog;caffe2::mkldnn
--   Private Dependencies : qnnpack;cpuinfo;/usr/lib64/liblmdb.so;/usr/lib64/libleveldb.so;/usr/lib64/libsnappy.so;/usr/lib64/libnuma.so;opencv_core;opencv_highgui;opencv_imgproc;opencv_imgcodecs;opencv_videoio;opencv_video;/usr/local/lib/libavcodec.so;/usr/local/lib/libavformat.so;/usr/local/lib/libavutil.so;/usr/local/lib/libswscale.so;fp16;aten_op_header_gen;onnxifi_loader;rt;gcc_s;gcc;dl
-- Configuring done
-- Generating done

The full output is in here: https://gist.github.com/pdrglv/b9e255ec4029f87ebe4e585439fa6b5e

Thank you! :grinning: