On-device ExecuTorch LLM API

[Alex_Zhou1 (Alex Zhou)]

Hi all,

So I want to make a mobile game where the NPCs are basically chatbots. To do so I need to quantize LLMs and deploy them onto mobiles. I have been reading ExecuTorch tutorials but not completely, and since my focus is on computer vision, this is a new field to me really.

To my understanding, I only need LLM APIs to get this idea to work, as opposed to compiling a chatbot app. I was wondering if this is possible? Many thanks in advance.

[larryliu0820 (Mengwei Liu)]

Hi @Alex_Zhou1 thanks for your interest in ExecuTorch!

The high level flow of deploying LLM models to mobile application is:

• Use extension/llm APIs (WIP see more in this RFC: RFC: extension/llm API design · Issue #6558 · pytorch/executorch · GitHub) to quantize, optimize and eventually export to ExecuTorch file format .pte.
• Leverage the existing JNI layer (ExecuTorch Llama Android Demo App — ExecuTorch 0.4 documentation) or iOS examples (ExecuTorch Llama iOS Demo App — ExecuTorch 0.4 documentation) to build a native runner. A C++ Tokenizer implementation is also provided under extension/llm/tokenizer.
• Integrate the runner into your mobile app. The app is responsible to handle I/O (e.g., feeding input into the runner, getting back generated tokens from callback) and artifacts (.pte and tokenizer artifact) management.

For more questions please create an issue in Issues · pytorch/executorch · GitHub