[Android] Large model 4GB on Android app is crashing

Mobile
1

Hello all,

I have a working LLM Model of size 4GB, and I’m able to run it successfully on an x86 CPU with Pytorch, and when I try to load the same model from the Android application its crashing.

I have followed the instructions exactly from this website: https://pytorch.org/mobile/android/
Currently, I’m loading the model explicitly from the internal storage, not from the assets, as the model is too large to save as a binary.

Here is my code snippet of the initialization:

private fun initialize(modelPath: String) {
val modelPath = "/storage/emulated/0/Download/model.ptl"
    if (initialized) return
    Logger.appendLog(" Loading model from: $modelPath")
   // module = LiteModuleLoader.load(modelPath)
    module = Module.load(modelPath)
    initialized = true
    Logger.appendLog(" Model loaded successfully.")
}

build.gradle:
implementation(“org.pytorch:pytorch_android_lite:1.13.1”)
implementation(“org.pytorch:pytorch_android_torchvision_lite:1.13.1”)

2

Hey brainchip_india,

I don’t know a good way to run LLMs on LiteInterpreter. We support LLMs and in general way more GenAI functionality with ExecuTorch. The link you are referencing is now updated and points to ExecuTorch as well.

Please give that a try, and if you are still running into issues, let us know. We are also on Discord if you want to hang out with the team there.

3

Could be hitting android’s memory hit. Quantize or split the model or you might wanna test with pytorch_android instead of the lite version

4

Hey @cbilgin and @GerryMan

I have another variant of the LLM model, which is 2GB in size, and I’m able to run it successfully on an Android device. It appears that the crash is due to a memory issue.

I tried cleaning up the cache memory using the lines below, but still no luck. Do you know any legit methods to clean up the RAM during the Model load?

@cbilgin Could you please point me to the Pytorch Discord channel link?

System.gc() PyTorchAndroid.setNumThreads(Runtime.getRuntime().availableProcessors() / 2)

5

Later, I tried to check the adb logs, and there I found that the crash is due to unsupported op[rms norm], see the detailed logs below.

E FATAL EXCEPTION: main Process: com.example.pytorch_new_rag_llm, PID: 29995 com.facebook.jni.CppException: Following ops cannot be found: [aten::rms_norm]. Please check if the operator library is included in the build. If built with selected ops, check if these ops are in the list. If you are a Meta employee, please see fburl.com/missing_ops for a fix. Or post it in https://discuss.pytorch.org/c/mobile/ ()Exception raised from initialize_operators at /home/agunapal/pytorch/torch/csrc/jit/mobile/function.cpp:90

6

Have you tried using ExecuTorch instead?

7

Here you go! PyTorch Edge