# On-device ExecuTorch LLM API

**URL:** <https://discuss.pytorch.org/t/on-device-executorch-llm-api/212392>\
**Category:** ExecuTorch\
**Created:** [November 1, 2024, 3:35am UTC](https://discuss.pytorch.org/t/on-device-executorch-llm-api/212392 "2024-11-01T03:35:53Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alex\_Zhou1](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alex_zhou1/32/72222_2.png) [@Alex\_Zhou1](https://discuss.pytorch.org/u/Alex_Zhou1)\
**Post date:** [November 1, 2024, 3:35am UTC](https://discuss.pytorch.org/t/on-device-executorch-llm-api/212392/1 "2024-11-01T03:35:53Z")

</div>

Hi all,

So I want to make a mobile game where the NPCs are basically chatbots. To do so I need to quantize LLMs and deploy them onto mobiles. I have been reading ExecuTorch tutorials but not completely, and since my focus is on computer vision, this is a new field to me really.

To my understanding, I only need LLM APIs to get this idea to work, as opposed to compiling a chatbot app. I was wondering if this is possible? Many thanks in advance.

---

<div class="post-metadata">

**Author:** ![larryliu0820](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/larryliu0820/32/66293_2.png) [@larryliu0820](https://discuss.pytorch.org/u/larryliu0820)\
**Post date:** [December 2, 2024, 7:20pm UTC](https://discuss.pytorch.org/t/on-device-executorch-llm-api/212392/2 "2024-12-02T19:20:37Z")

</div>

Hi @Alex_Zhou1 thanks for your interest in ExecuTorch!

The high level flow of deploying LLM models to mobile application is:

- Use `extension/llm` APIs (WIP see more in this RFC: [RFC: extension/llm API design · Issue #6558 · pytorch/executorch · GitHub](https://github.com/pytorch/executorch/issues/6558)) to quantize, optimize and eventually export to ExecuTorch file format `.pte`.
- Leverage the existing JNI layer ([ExecuTorch Llama Android Demo App — ExecuTorch 0.4 documentation](https://pytorch.org/executorch/0.4/llm/llama-demo-android.html)) or iOS examples ([ExecuTorch Llama iOS Demo App — ExecuTorch 0.4 documentation](https://pytorch.org/executorch/0.4/llm/llama-demo-ios.html)) to build a native runner. A C++ Tokenizer implementation is also provided under `extension/llm/tokenizer`.
- Integrate the runner into your mobile app. The app is responsible to handle I/O (e.g., feeding input into the runner, getting back generated tokens from callback) and artifacts (.pte and tokenizer artifact) management.

For more questions please create an issue in [Issues · pytorch/executorch · GitHub](https://github.com/pytorch/executorch/issues)