Hi,
I am working my way into the source installation of pytorch. I am using ubuntu 18.04, CUDA 10.2 and cuDNN 7. The configuration goes well and I get to 86% when I get this error:
Makefile:159: recipe for target ‘all’ failed
make: *** [all] Error 2
Any idea why this is happening and how to fix it?
Thanks!
Could you check the install log for the reported error and post it here, please?
Thanks for your reply!
I am a newbie here, so I am not too sure what the install logs are. I did check the CMakeError.log file and this is what I see there regarding the error:
Run Build Command(s):/usr/bin/make cmTC_f7d0a/fast && /usr/bin/make -f CMakeFiles/cmTC_f7d0a.dir/build.make CMakeFiles/cmTC_f7d0a.dir/build
make[1]: Entering directory '/home/simulations/pytorch/build/CMakeFiles/CMakeTmp'
Building CXX object CMakeFiles/cmTC_f7d0a.dir/src.cxx.o
/usr/bin/c++ -DHAS_WERROR_CAST_FUNCTION_TYPE -I/home/simulations/anaconda3/envs/pytorch-build/include -I/usr/local/cuda-10.2/include -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -fPIE -Werror=cast-function-type -std=gnu++14 -o CMakeFiles/cmTC_f7d0a.dir/src.cxx.o -c /home/simulations/pytorch/build/CMakeFiles/CMakeTmp/src.cxx
cc1plus: error: -Werror=cast-function-type: no option -Wcast-function-type
CMakeFiles/cmTC_f7d0a.dir/build.make:84: recipe for target 'CMakeFiles/cmTC_f7d0a.dir/src.cxx.o' failed
make[1]: *** [CMakeFiles/cmTC_f7d0a.dir/src.cxx.o] Error 1
make[1]: Leaving directory '/home/simulations/pytorch/build/CMakeFiles/CMakeTmp'
Makefile:140: recipe for target 'cmTC_f7d0a/fast' failed
make: *** [cmTC_f7d0a/fast] Error 2
Is that helpful? Should I check any other log files?
Thanks again! 
The output in the terminal should give you an error message and pointing towards the root cause of the issue. Based on your current output I don’t see what exactly failed.
Hi again,
I tried the installation one more time, just to check if the error repeats and it does. From the terminal, I can see the following:
[ 83%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/CrossKernel.cpp.AVX2.cpp.o
[ 83%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/CopyKernel.cpp.AVX2.cpp.o
[ 84%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/ComplexKernel.cpp.AVX2.cpp.o
[ 84%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/CatKernel.cpp.AVX2.cpp.o
[ 84%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/BlasKernel.cpp.AVX2.cpp.o
[ 84%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/BinaryOpsKernel.cpp.AVX2.cpp.o
[ 84%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/AvgPoolKernel.cpp.AVX2.cpp.o
[ 84%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/AdaptiveMaxPoolKernel.cpp.AVX2.cpp.o
[ 84%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/AdaptiveAvgPoolKernel.cpp.AVX2.cpp.o
[ 84%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/Activation.cpp.AVX2.cpp.o
[ 84%] Linking CXX shared library ../lib/libtorch_cpu.so
[ 84%] Built target torch_cpu
Makefile:159: recipe for target 'all' failed
make: *** [all] Error 2
Thanks for the help!!
The error should be shown before the posted lines containing the error keyword, so you could search for it.
Got it! I looked into the terminal output and found when the error is mentioned. It happens at 69% progress, so not sure why it keeps going until it reaches 84% and displays the error I showed above.
In any case, this is what I found:
[ 69%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/core/adaption.cpp.o
prims_ll.h(198): error: identifier "__funnelshift_r" is undefined
detected during:
instantiation of "uint64_t Primitives<T, RedOp, Fan, Direct, ProtoLL>::DataLoader::loadFinish() [with T=int8_t, RedOp=FuncSum<int8_t>, Fan=FanSymmetric<2>, Direct=1]"
(251): here
instantiation of "void Primitives<T, RedOp, Fan, Direct, ProtoLL>::LLGenericOp<RECV,SEND,SrcBuf,DstBuf>(intptr_t, intptr_t, int, __nv_bool) [with T=int8_t, RedOp=FuncSum<int8_t>, Fan=FanSymmetric<2>, Direct=1, RECV=1, SEND=1, SrcBuf=0, DstBuf=1]"
(384): here
instantiation of "void Primitives<T, RedOp, Fan, Direct, ProtoLL>::recvReduceCopySend(intptr_t, intptr_t, int, __nv_bool) [with T=int8_t, RedOp=FuncSum<int8_t>, Fan=FanSymmetric<2>, Direct=1]"
primitives.h(135): here
instantiation of "void PrimitivesWithoutDirect<RealPrimitives>::directRecvReduceCopySend(intptr_t, intptr_t, intptr_t, int, __nv_bool) [with RealPrimitives=Primitives<int8_t, FuncSum<int8_t>, FanSymmetric<2>, 1, ProtoLL>]"
all_reduce.h(206): here
instantiation of "void <unnamed>::runTreeSplit<T,RedOp,Proto>(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum<int8_t>, Proto=ProtoLL]"
all_reduce.h(372): here
instantiation of "void RunWorkElement<(ncclFunc_t)4, T, RedOp, 0, 0>::run(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum<int8_t>]"
common.h(77): here
instantiation of "void RunWork<Fn, T, RedOp, Algo, Proto>::run(ncclWork *) [with Fn=(ncclFunc_t)4, T=int8_t, RedOp=FuncSum<int8_t>, Algo=0, Proto=0]"
all_reduce.cu(11): here
prims_ll.h(198): error: identifier "__funnelshift_r" is undefined
detected during:
instantiation of "uint64_t Primitives<T, RedOp, Fan, Direct, ProtoLL>::DataLoader::loadFinish() [with T=int8_t, RedOp=FuncSum<int8_t>, Fan=FanAsymmetric<2, 1>, Direct=1]"
(251): here
instantiation of "void Primitives<T, RedOp, Fan, Direct, ProtoLL>::LLGenericOp<RECV,SEND,SrcBuf,DstBuf>(intptr_t, intptr_t, int, __nv_bool) [with T=int8_t, RedOp=FuncSum<int8_t>, Fan=FanAsymmetric<2, 1>, Direct=1, RECV=0, SEND=1, SrcBuf=0, DstBuf=-1]"
(363): here
instantiation of "void Primitives<T, RedOp, Fan, Direct, ProtoLL>::send(intptr_t, int) [with T=int8_t, RedOp=FuncSum<int8_t>, Fan=FanAsymmetric<2, 1>, Direct=1]"
all_reduce.h(224): here
instantiation of "void <unnamed>::runTreeSplit<T,RedOp,Proto>(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum<int8_t>, Proto=ProtoLL]"
all_reduce.h(372): here
instantiation of "void RunWorkElement<(ncclFunc_t)4, T, RedOp, 0, 0>::run(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum<int8_t>]"
common.h(77): here
instantiation of "void RunWork<Fn, T, RedOp, Algo, Proto>::run(ncclWork *) [with Fn=(ncclFunc_t)4, T=int8_t, RedOp=FuncSum<int8_t>, Algo=0, Proto=0]"
all_reduce.cu(11): here
prims_ll.h(198): error: identifier "__funnelshift_r" is undefined
detected during:
instantiation of "uint64_t Primitives<T, RedOp, Fan, Direct, ProtoLL>::DataLoader::loadFinish() [with T=int8_t, RedOp=FuncSum<int8_t>, Fan=FanAsymmetric<1, 2>, Direct=1]"
(251): here
instantiation of "void Primitives<T, RedOp, Fan, Direct, ProtoLL>::LLGenericOp<RECV,SEND,SrcBuf,DstBuf>(intptr_t, intptr_t, int, __nv_bool) [with T=int8_t, RedOp=FuncSum<int8_t>, Fan=FanAsymmetric<1, 2>, Direct=1, RECV=1, SEND=0, SrcBuf=-1, DstBuf=1]"
(369): here
instantiation of "void Primitives<T, RedOp, Fan, Direct, ProtoLL>::recv(intptr_t, int, __nv_bool) [with T=int8_t, RedOp=FuncSum<int8_t>, Fan=FanAsymmetric<1, 2>, Direct=1]"
primitives.h(125): here
instantiation of "void PrimitivesWithoutDirect<RealPrimitives>::directRecv(intptr_t, int) [with RealPrimitives=Primitives<int8_t, FuncSum<int8_t>, FanAsymmetric<1, 2>, 1, ProtoLL>]"
all_reduce.h(243): here
instantiation of "void <unnamed>::runTreeSplit<T,RedOp,Proto>(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum<int8_t>, Proto=ProtoLL]"
all_reduce.h(372): here
instantiation of "void RunWorkElement<(ncclFunc_t)4, T, RedOp, 0, 0>::run(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum<int8_t>]"
common.h(77): here
instantiation of "void RunWork<Fn, T, RedOp, Algo, Proto>::run(ncclWork *) [with Fn=(ncclFunc_t)4, T=int8_t, RedOp=FuncSum<int8_t>, Algo=0, Proto=0]"
all_reduce.cu(11): here
op128.h(51): error: identifier "__funnelshift_r" is undefined
detected during:
instantiation of "void Primitives<T, RedOp, Fan, Direct, ProtoLL128>::loadRegsBegin(uint64_t (&)[WordPerThread], const T *, int) [with T=int8_t, RedOp=FuncSum<int8_t>, Fan=FanSymmetric<2>, Direct=1, WordPerThread=8]"
prims_ll128.h(305): here
instantiation of "void Primitives<T, RedOp, Fan, Direct, ProtoLL128>::GenericOp<RECV,SEND,SrcBuf,DstBuf>(intptr_t, intptr_t, int, __nv_bool) [with T=int8_t, RedOp=FuncSum<int8_t>, Fan=FanSymmetric<2>, Direct=1, RECV=1, SEND=1, SrcBuf=0, DstBuf=1]"
prims_ll128.h(422): here
instantiation of "void Primitives<T, RedOp, Fan, Direct, ProtoLL128>::recvReduceCopySend(intptr_t, intptr_t, int, __nv_bool) [with T=int8_t, RedOp=FuncSum<int8_t>, Fan=FanSymmetric<2>, Direct=1]"
primitives.h(135): here
instantiation of "void PrimitivesWithoutDirect<RealPrimitives>::directRecvReduceCopySend(intptr_t, intptr_t, intptr_t, int, __nv_bool) [with RealPrimitives=Primitives<int8_t, FuncSum<int8_t>, FanSymmetric<2>, 1, ProtoLL128>]"
all_reduce.h(206): here
instantiation of "void <unnamed>::runTreeSplit<T,RedOp,Proto>(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum<int8_t>, Proto=ProtoLL128]"
all_reduce.h(386): here
instantiation of "void RunWorkElement<(ncclFunc_t)4, T, RedOp, 0, 1>::run(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum<int8_t>]"
common.h(77): here
instantiation of "void RunWork<Fn, T, RedOp, Algo, Proto>::run(ncclWork *) [with Fn=(ncclFunc_t)4, T=int8_t, RedOp=FuncSum<int8_t>, Algo=0, Proto=1]"
all_reduce.cu(11): here
prims_ll.h(198): error: identifier "__funnelshift_r" is undefined
detected during:
instantiation of "uint64_t Primitives<T, RedOp, Fan, Direct, ProtoLL>::DataLoader::loadFinish() [with T=int8_t, RedOp=FuncSum<int8_t>, Fan=FanSymmetric<1>, Direct=1]"
(251): here
instantiation of "void Primitives<T, RedOp, Fan, Direct, ProtoLL>::LLGenericOp<RECV,SEND,SrcBuf,DstBuf>(intptr_t, intptr_t, int, __nv_bool) [with T=int8_t, RedOp=FuncSum<int8_t>, Fan=FanSymmetric<1>, Direct=1, RECV=0, SEND=1, SrcBuf=0, DstBuf=-1]"
(363): here
instantiation of "void Primitives<T, RedOp, Fan, Direct, ProtoLL>::send(intptr_t, int) [with T=int8_t, RedOp=FuncSum<int8_t>, Fan=FanSymmetric<1>, Direct=1]"
all_reduce.h(64): here
instantiation of "void <unnamed>::runRing<T,RedOp,Proto>(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum<int8_t>, Proto=ProtoLL]"
all_reduce.h(365): here
instantiation of "void RunWorkElement<(ncclFunc_t)4, T, RedOp, 1, 0>::run(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum<int8_t>]"
common.h(77): here
instantiation of "void RunWork<Fn, T, RedOp, Algo, Proto>::run(ncclWork *) [with Fn=(ncclFunc_t)4, T=int8_t, RedOp=FuncSum<int8_t>, Algo=1, Proto=0]"
all_reduce.cu(11): here
5 errors detected in the compilation of "/tmp/tmpxft_000064a1_00000000-6_all_reduce.cpp1.ii".
/home/simulations/pytorch/build/nccl/obj/collectives/device/Makefile.rules:227: recipe for target '/home/simulations/pytorch/build/nccl/obj/collectives/device/all_reduce_sum_i8.o' failed
make[5]: *** [/home/simulations/pytorch/build/nccl/obj/collectives/device/all_reduce_sum_i8.o] Error 1
Makefile:50: recipe for target '/home/simulations/pytorch/build/nccl/obj/collectives/device/colldevice.a' failed
make[4]: *** [/home/simulations/pytorch/build/nccl/obj/collectives/device/colldevice.a] Error 2
Makefile:25: recipe for target 'src.build' failed
make[3]: *** [src.build] Error 2
CMakeFiles/nccl_external.dir/build.make:129: recipe for target 'nccl_external-prefix/src/nccl_external-stamp/nccl_external-build' failed
make[2]: *** [nccl_external-prefix/src/nccl_external-stamp/nccl_external-build] Error 2
CMakeFiles/Makefile2:1986: recipe for target 'CMakeFiles/nccl_external.dir/all' failed
make[1]: *** [CMakeFiles/nccl_external.dir/all] Error 2
Thanks again!!
The error is raised in:
error: identifier "__funnelshift_r" is undefined
Which device are you using? Funnel shift ops should be available for compute capabilities >= 3.2.
The build uses multiple jobs so when one job fails, the others could continue for a while until they get stopped.
I am using a GTX Geforce 650, which has a computing capability of 3.0. I guess the error makes sense now. Any way to work around it??
I don’t think you would be able to build the current PyTorch version for this device and might need to stick to an older release.
I see! What release would you recommend I use?
Thank you so much for all the help!!
Unfortunately, I’m unsure which release supported sm<3.5 
No problem!! Thanks for all the help!