Undefined reference when building from modified source code

C++
1

Hello Everyone!

I am trying to build PyTorch from source after modifiing this file
import.cpp to let it use a function imported from a C file.

I imported it like this:

extern "C" {
    #include "decryption_utils.h" // a C header, so wrap it in extern "C"
}

The C file is in the same directory with its header.

But when I run:

python setup.py install

I am getting the following error when linking:

Building wheel torch-2.0.0a0+git8bc38ae
-- Building version 2.0.0a0+git8bc38ae
cmake --build . --target install --config Release
[1/164] Linking CXX executable bin/basic
FAILED: bin/basic 
: && /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG -rdynamic -pthread -Wl,-rpath-link,/usr/lib/x86_64-linux-gnu caffe2/CMakeFiles/basic.dir/__/aten/src/ATen/test/basic.cpp.o -o bin/basic  -Wl,-rpath,/home/moh/pytorch/build/lib:  lib/libgtest_main.a  /usr/lib/gcc/x86_64-linux-gnu/9/libgomp.so  /usr/lib/x86_64-linux-gnu/libpthread.so  -Wl,--no-as-needed,"/home/moh/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/home/moh/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  lib/libc10.so  lib/libgtest.a  -pthread && :
/usr/bin/ld: /home/moh/pytorch/build/lib/libtorch_cpu.so: undefined reference to `decrypt_fcontent'
collect2: error: ld returned 1 exit status
[2/164] Linking CXX executable bin/half_test
FAILED: bin/half_test 
: && /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG -rdynamic -pthread -Wl,-rpath-link,/usr/lib/x86_64-linux-gnu caffe2/CMakeFiles/half_test.dir/__/aten/src/ATen/test/half_test.cpp.o -o bin/half_test  -Wl,-rpath,/home/moh/pytorch/build/lib:  lib/libgtest_main.a  /usr/lib/gcc/x86_64-linux-gnu/9/libgomp.so  /usr/lib/x86_64-linux-gnu/libpthread.so  -Wl,--no-as-needed,"/home/moh/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/home/moh/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  lib/libc10.so  lib/libgtest.a  -pthread && :
/usr/bin/ld: /home/moh/pytorch/build/lib/libtorch_cpu.so: undefined reference to `decrypt_fcontent'
collect2: error: ld returned 1 exit status
[3/164] Linking CXX executable bin/broadcast_test
FAILED: bin/broadcast_test 
: && /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG -rdynamic -pthread -Wl,-rpath-link,/usr/lib/x86_64-linux-gnu caffe2/CMakeFiles/broadcast_test.dir/__/aten/src/ATen/test/broadcast_test.cpp.o -o bin/broadcast_test  -Wl,-rpath,/home/moh/pytorch/build/lib:  lib/libgtest_main.a  /usr/lib/gcc/x86_64-linux-gnu/9/libgomp.so  /usr/lib/x86_64-linux-gnu/libpthread.so  -Wl,--no-as-needed,"/home/moh/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/home/moh/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  lib/libc10.so  lib/libgtest.a  -pthread && :
/usr/bin/ld: /home/moh/pytorch/build/lib/libtorch_cpu.so: undefined reference to `decrypt_fcontent'
collect2: error: ld returned 1 exit status
[4/164] Linking CXX executable bin/MaybeOwned_test
FAILED: bin/MaybeOwned_test 
: && /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG -rdynamic -pthread -Wl,-rpath-link,/usr/lib/x86_64-linux-gnu caffe2/CMakeFiles/MaybeOwned_test.dir/__/aten/src/ATen/test/MaybeOwned_test.cpp.o -o bin/MaybeOwned_test  -Wl,-rpath,/home/moh/pytorch/build/lib:  lib/libgtest_main.a  /usr/lib/gcc/x86_64-linux-gnu/9/libgomp.so  /usr/lib/x86_64-linux-gnu/libpthread.so  -Wl,--no-as-needed,"/home/moh/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/home/moh/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  lib/libc10.so  lib/libgtest.a  -pthread && :
/usr/bin/ld: /home/moh/pytorch/build/lib/libtorch_cpu.so: undefined reference to `decrypt_fcontent'
collect2: error: ld returned 1 exit status
[5/164] Linking CXX executab
...

Apparently it fails because of my (imported) function decrypt_fcontent and I am not sure how to solve it, like should I define the header somewhere or so?

Any help will be appreciated a lot!
Thank you
Moh

2

This is the header:

#ifndef DECRYPTION_UTILS_H_
#define DECRYPTION_UTILS_H_

int decrypt_fcontent(char *fpath, char* dcontent);

#endif
3

After modifying pytorch/BUILD.bazel and pytorch/caffe2/serialize/CMakeLists.txt (I am using them there as well) to include the new files, the problem solved.

Also, since I’m using some openssl functions, I needed to link with -lssl -lcrypto, and that by adding them to the LINK_LIBRARIES in pytorch/build/build.ninja.

4

Adding -lssl -lcrypto to pytorch/build/build.ninja is really just a way to make it work, not recomended.

In order to build PyTorch from source with custom modifications, and assuming those modifications are using OpenSSL, then do the following:

  1. First, make sure OpenSSL is installed
  2. Then either:
    • add your implementations directly in your target PyTorch files and just include the OpenSSL library, or
    • if you have your implementations in your own files, then just include those files in your target PyTorch files
  3. If you followed the second approach, then you need to add your files in: build_variables.bzl
  4. For linking, add these to torch/CMakeLists.txt:
target_link_libraries(torch_cpu PUBLIC ssl crypto)
target_link_libraries(torch_cuda PUBLIC ssl crypto)
  1. And build, done.