LAPACK library not found in compilation (Pytorch Mobile, Android, Java)

yingbo · July 15, 2021, 2:45pm

I think someone has the same issue, but the solution is not clear and the download link was removed. Is there an official document to solve the problem?

More details on what I’m facing. In my build.gradle, I tried both org.pytorch:pytorch_android_lite:1.9.0 and org.pytorch:pytorch_android_torchvision:1.9.0. But I got the same error when run forward:

RuntimeError: inverse: LAPACK library not found in compilation
    
        at org.pytorch.NativePeer.forward(Native Method)
        at org.pytorch.Module.forward(Module.java:52)

Shouldn’t this problem be a bug? I don’t understand why the LAPACK is not in the official repo’s jar. Is the only way to solve it compiling PyTorch manually to include a LAPACK library (and the LAPACK library should be downloaded somewhere)?

I created an issue in GitHub too and hope we can get some response: Pytorch Mobile on Android, LAPACK library not found in compilation · Issue #61693 · pytorch/pytorch · GitHub

Thanks~!

kimishpatel · July 15, 2021, 3:03pm

Yes, it seems that in pytorch we dont build with LAPACK for mobile. Hence the jar does not have lapack.

There are two alternatives I see.

If your inverse is on small matrix like 3x3 you can do python implementation of it as part of your model.
Build from source for mobile and add USE_LAPACK=1.

I think 1 can unblock you faster probably.

@IvanKobzarev who is our android side expert can add more context.

yingbo · July 15, 2021, 3:30pm

Thanks @kimishpatel . Unfortunately the matrix is not small and option 1 does not fit.

Is there any documentation on option 2? I can try but I guess we need to prepare a LAPACK library before build the PyTorch?

kimishpatel · July 15, 2021, 4:21pm

I would expect that you do not need to prepare anything. You can build pytorch from source for android but specify USE_LAPACK=1. This can be done in one of the scripts for building android package. I dont honestly know which one though. @IvanKobzarev can suggest more, or I can try to find it later.

yingbo · July 16, 2021, 3:47pm

How to specify USE_LAPACK=1? Should I add it to ENV, e.g. USE_LAPACK=1 ./scripts/build_pytorch_android.sh armeabi-v7a ?

I tried the above, used the two .aar files in lib, and still got the same LAPACK error:

 com.facebook.jni.CppException: inverse: LAPACK library not found in compilation
  
      debug_handle:-1
      
    Exception raised from apply_inverse at ../aten/src/ATen/native/BatchLinearAlgebra.cpp:958 (most recent call first):
    (no backtrace available)
        at org.pytorch.LiteNativePeer.forward(Native Method)
        at org.pytorch.Module.forward(Module.java:52)

If I added changes as in [NOT_FOR_COMMIT][android] Pytorch android build with USE_LAPACK and QML lib by IvanKobzarev · Pull Request #35200 · pytorch/pytorch · GitHub

then I got an error

  [1/1] Linking CXX shared library ../../../../build/intermediates/cmake/release/obj/armeabi-v7a/libpytorch_jni_lite.so                                                                 
  FAILED: ../../../../build/intermediates/cmake/release/obj/armeabi-v7a/libpytorch_jni_lite.so                                                  
  : && /Users/ymiao/Library/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++ --target=armv7-none-linux-androideabi21 --gcc-toolchain=/Users/ymiao/Library
/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64 --sysroot=/Users/ymiao/Library/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64/sysroot -fPIC -g
 -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -march=armv7-a -mthumb -Wformat -Werror=format-securi
ty   -Oz -DNDEBUG  -Wl,--exclude-libs,libgcc.a -Wl,--exclude-libs,libgcc_real.a -Wl,--exclude-libs,libatomic.a -Wl,--build-id -Wl,--fatal-warnings -Wl,--exclude-libs,libunwind.a -Wl,--
no-undefined -Qunused-arguments -shared -Wl,-soname,libpytorch_jni_lite.so -o ../../../../build/intermediates/cmake/release/obj/armeabi-v7a/libpytorch_jni_lite.so CMakeFiles/pytorch_jn
i_lite.dir/src/main/cpp/pytorch_jni_lite.cpp.o CMakeFiles/pytorch_jni_lite.dir/src/main/cpp/pytorch_jni_common.cpp.o  ../../../../build/intermediates/cmake/release/obj/armeabi-v7a/libf
bjni.so -Wl,--gc-sections -Wl,--whole-archive ../../../../src/main/jniLibs/armeabi-v7a/libtorch.a ../../../../src/main/jniLibs/armeabi-v7a/libtorch_cpu.a -Wl,--no-whole-archive ../../.
./../src/main/jniLibs/armeabi-v7a/libc10.a ../../../../src/main/jniLibs/armeabi-v7a/libnnpack.a ../../../../src/main/jniLibs/armeabi-v7a/libXNNPACK.a ../../../../src/main/jniLibs/armea
bi-v7a/libpytorch_qnnpack.a ../../../../src/main/jniLibs/armeabi-v7a/libpthreadpool.a ../../../../src/main/jniLibs/armeabi-v7a/libeigen_blas.a ../../../../src/main/jniLibs/armeabi-v7a/
libcpuinfo.a ../../../../src/main/jniLibs/armeabi-v7a/libclog.a -llibQML libVulkanWrapper.a -landroid -llog -ldl -latomic -lm && :
  /Users/ymiao/Library/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: cannot f
ind -llibQML
  ../../../../src/main/jniLibs/armeabi-v7a/libtorch_cpu.a(BatchLinearAlgebra.cpp.o):BatchLinearAlgebra.cpp:function void at::native::lapackLu<c10::complex<double> >(int, int, c10::comp
lex<double>*, int, int*, int*): error: undefined reference to 'zgetrf_'
  ../../../../src/main/jniLibs/armeabi-v7a/libtorch_cpu.a(BatchLinearAlgebra.cpp.o):BatchLinearAlgebra.cpp:function void at::native::lapackLu<c10::complex<float> >(int, int, c10::compl
ex<float>*, int, int*, int*): error: undefined reference to 'cgetrf_'
.....

Would you please help on how to build from source with the LAPACK support? @kimishpatel @IvanKobzarev

yingbo · July 19, 2021, 2:40am

Forgot mentioning in the above, the source are checked out from tag v1.9.0.

girgink · August 10, 2021, 2:52pm

I’m not sure if it is the case also with Java version, but during compilation PyTorch (1.9.0) BLAS auto detection selects BLIS first, although OpenBLAS exists. BLIS doesn’t have LAPACK support, therefore torch gives “LAPACK library not found in compilation” error. You can check this by looking at the BLAS section of the configuration summary and see if it is indicated as “BLAS: blis”. If it is the case, you can correct it by setting WITH_BLAS=open, or by editing cmake/Modules/FindBLAS.cmake and setting default value to “open” before compilation.

I think OpenBLAS should be checked before BLIS, or should overwrite BLIS if detected.

Paran0 · August 18, 2021, 12:53pm

Is it possible for main developers to publish a build with LAPACK support?

It’s been about week or two since I’m stuck building the pytorch_android with LAPACK …

kimishpatel · August 18, 2021, 2:43pm

@IvanKobzarev can we support this?

@Paran0 my understanding is that you want prebuilt android package with LAPACK that is released from pytorch?

Paran0 · August 18, 2021, 3:03pm

Yeah, we tried with multiple BLAS backend APIs (OpenBLAS, ATLAS, Eigen, … ) All failed in some stage of the building, as I said and you can check from my previous posts we’re trying to build it for about a week or so with no success …

One of my main concerns is when we checked the Dependencies.cmake the LAPACK option is contained in the NOT INTERN_BUILD_MOBILE. Is it possible that the operations depending on LAPACK isn’t supported at all in Mobile device?
There should be some reason in which you guys disabled this option in the first place …

kimishpatel · August 18, 2021, 3:50pm

@Paran0, so I think LAPACK is not built with mobile at the moment. I dont have the historical context as to why. It maybe size related.

Digging deeper into this I see that we use Eigen for BLAS. Eigen does support LAPACK but pytorch build system does not even build for LAPACK as you pointed out. pytorch/Dependencies.cmake at master · pytorch/pytorch · GitHub

In order to change this we likely have to make some changes here: https://github.com/pytorch/pytorch/blob/master/cmake/Modules/FindLAPACK.cmaketo account for eigen, or just build eigen with LAPACK and set USE_LAPACK to 1 here: pytorch/Dependencies.cmake at master · pytorch/pytorch · GitHub

Either way this will take some time to do on our end. @IvanKobzarev can you help with this?

Alternatively, you can also try to follow this. The earlier undefined symbol error you got is because of not building eigen with LAPACK, I think.