Other config files

.gitattributes

@@ -33,3 +33,14 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+genie_bundle/genie-t2t-run.exe filter=lfs diff=lfs merge=lfs -text
+genie_bundle/Genie.dll filter=lfs diff=lfs merge=lfs -text
+genie_bundle/libQnnHtpv73Skel.so filter=lfs diff=lfs merge=lfs -text
+genie_bundle/PlatformValidatorShared.dll filter=lfs diff=lfs merge=lfs -text
+genie_bundle/qnn-platform-validator.exe filter=lfs diff=lfs merge=lfs -text
+genie_bundle/QnnGenAiTransformer.dll filter=lfs diff=lfs merge=lfs -text
+genie_bundle/QnnHtp.dll filter=lfs diff=lfs merge=lfs -text
+genie_bundle/QnnHtpNetRunExtensions.dll filter=lfs diff=lfs merge=lfs -text
+genie_bundle/QnnHtpPrepare.dll filter=lfs diff=lfs merge=lfs -text
+genie_bundle/QnnHtpv73Stub.dll filter=lfs diff=lfs merge=lfs -text
+genie_bundle/QnnSystem.dll filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,430 @@
+# LLM On-Device Deployment
+
+In this tutorial we will show an end to end workflow deploying large language
+models (LLMs) to Snapdragon® platforms such as Snapdragon® 8 Elite,
+Snapdragon® 8 Gen 3 (e.g., Samsung Galaxy S24 family) and Snapdragon® X Elite
+(e.g. Snapdragon® based Microsoft Surface Pro). We will use
+[Qualcomm AI Hub](https://aihub.qualcomm.com/) to compile the models to QAIRT
+context binaries and run them with Genie from the [QAIRT
+SDK](https://qpm.qualcomm.com/#/main/tools/details/Qualcomm_AI_Runtime_SDK).
+
+We will use Llama3 8B as a running example. Other LLMs from [AI Hub
+Models](https://github.com/quic/ai-hub-models/tree/main/qai_hub_models/models)
+will work with the same flow.
+
+## Overview
+
+We will walk you through the follow steps:
+
+1. Get access to [Llama 3 weights from Hugging Face](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct).
+2. Use Qualcomm [AI Hub
+Models](https://github.com/quic/ai-hub-models/tree/main/qai_hub_models/models) to export Llama 3 using AI Hub.
+3. Prepare assets required by Qualcomm Genie, the inference runtime for LLMs.
+4. Run the LLM on device with an example prompt on Android / Windows PC with Snapdragon®.
+
+Note that because this is a large model, it may take 4-6 hours to generate required assets.
+
+If you have any questions, please feel free to post on [AI Hub Slack channel](https://aihub.qualcomm.com/community/slack)
+
+## Device Requirements
+
+| Model name | Minimum Compile QAIRT SDK version | Supported devices |
+| --- | --- | --- |
+| Llama-v2-7B-Chat | 2.27.0 | Snapdragon® 8 Elite<br>Snapdragon® 8 Gen 3<br>Snapdragon® X Elite<br>Snapdragon® X Plus |
+| Llama-v3-8B-Instruct | 2.27.0 | Snapdragon® 8 Elite<br>Snapdragon® X Elite<br>Snapdragon® X Plus |
+| Llama-v3.1-8B-Instruct | 2.27.7 | Snapdragon® 8 Elite |
+| Llama-v3.1-8B-Instruct | 2.28.0 | Snapdragon® X Elite<br>Snapdragon® X Plus |
+| Llama-v3.2-3B-Instruct | 2.27.7 | Snapdragon® 8 Elite<br>Snapdragon® 8 Gen 3 (Context length 2048) |
+| Llama-v3.2-3B-Instruct | 2.28.0 | Snapdragon® X Elite<br>Snapdragon® X Plus |
+| Llama-SEA-LION-v3.5-8B-R | 2.28.0 | Snapdragon® 8 Elite<br>Snapdragon® X Elite<br>Snapdragon® X Plus |
+| Llama3-TAIDE-LX-8B-Chat-Alpha1 | 2.27.0 | Snapdragon® 8 Elite<br>Snapdragon® X Elite<br>Snapdragon® X Plus |
+| Baichuan2-7B | 2.27.7 |  Snapdragon® 8 Elite |
+| Qwen2-7B-Instruct | 2.27.7 |  Snapdragon® 8 Elite |
+| Mistral-7B-Instruct-v0.3 | 2.27.7 |  Snapdragon® 8 Elite |
+| Phi-3.5-Mini-Instruct | 2.29.0 | Snapdragon® 8 Elite<br>Snapdragon® X Elite<br>Snapdragon® 8 Gen 3 |
+| IBM-Granite-v3.1-8B-Instruct | 2.30.0 | Snapdragon® 8 Elite<br>Snapdragon® X Elite |
+
+Device requirements:
+
+- Android 15
+- At least Genie SDK from QAIRT (or QNN) SDK 2.29.0 (earlier versions have issues with long prompts).
+- Hexagon architecture v73 or above (please see [Devices](https://app.aihub.qualcomm.com/devices/) list).
+- 16GB memory or more for 7B+ or 4096 context length models.
+- 12GB memory or more for 3B+ models (and you may need to adjust down context length).
+
+> [!IMPORTANT]
+> Please make sure device requirements are met before proceeding.
+
+## Required Software
+
+The following packages are required:
+
+1. [QAIRT SDK](https://qpm.qualcomm.com/#/main/tools/details/Qualcomm_AI_Runtime_SDK) (see [QNN SDK](https://qpm.qualcomm.com/#/main/tools/details/qualcomm_ai_engine_direct) for versions prior to 2.32)
+2. [qai-hub-models](https://pypi.org/project/qai-hub-models/) and any extras for your desired model.
+3. [qai-hub](https://pypi.org/project/qai-hub/)
+
+### QAIRT Installation
+
+Typically we recommend using the same QAIRT SDK version that AI Hub uses to compile
+the assets. You can find this version by clicking the job links posted printed
+by the export command.
+
+Go to [QAIRT
+SDK](https://qpm.qualcomm.com/#/main/tools/details/Qualcomm_AI_Runtime_SDK) (or [QNN SDK](https://qpm.qualcomm.com/#/main/tools/details/qualcomm_ai_engine_direct) for older versions) and
+follow the installation instructions. Note that the first time after log in you
+would be redirected to QPM home page. Click on the link again to get to the
+QAIRT download page.
+
+If you are on a Mac laptop, we recommend using
+[Docker](https://www.docker.com/) to install qpm-cli to extract the `.qik` file.
+
+If successful, you should see a message with the install path. This will depend on
+the platform and can look like this:
+
+```text
+/opt/qcom/aistack/qairt/<version>
+C:\Qualcomm\AIStack\QAIRT\<version>
+```
+
+Set your `QNN_SDK_ROOT` environment variable to point to this directory. On
+Linux or Mac you would run:
+
+```bash
+export QNN_SDK_ROOT=/opt/qcom/aistack/qairt/<version>
+```
+
+On Windows, you can search the taskbar for "Edit the system environment
+variables".
+
+### Python Packages
+
+Following standard best practices, we recommend creating a virtual environment specifically for
+exporting AI Hub models. The following steps can be performed on Windows,
+Linux, or Mac. On Windows, you can either install x86-64 Python (since package
+support is limited on native ARM64 Python) or use Windows Subsystem for Linux
+(WSL).
+
+#### Create Virtual Environment
+
+Create a [virtualenv](https://virtualenv.pypa.io/en/latest/) for `qai-hub-models` with Python 3.10.
+You can also use [conda](https://conda.io/projects/conda/en/latest/user-guide/install/index.html).
+
+For clarity, we recommend creating a virtual env:
+
+```bash
+python3.10 -m venv llm_on_genie_venv
+```
+
+#### Install `qai-hub-models`
+
+In a shell session, install `qai-hub-models` in the virtual environment:
+
+```bash
+source llm_on_genie_venv/bin/activate
+pip install -U "qai-hub-models[llama-v3-8b-instruct]"
+```
+
+Replace `llama-v3-8b-instruct` with the desired llama model from [AI Hub
+Model](https://github.com/quic/ai-hub-models/tree/main/qai_hub_models/models).
+Note to replace `_` with `-` (e.g. `llama_v3_8b_instruct` -> `llama-v3-8b-instruct`)
+
+Make sure Git is installed in your environment. This command should work:
+
+```bash
+git --version
+```
+
+Ensure at least 80GB of memory (RAM + swap). On Ubuntu (including through WSL) you can check it by
+
+```bash
+free -h
+```
+
+Increase swap size if needed.
+
+We use
+[qai-hub-models](https://github.com/quic/ai-hub-models/tree/main/qai_hub_models/)
+to adapt Huggingface Llama models for on-device inference.
+
+## Acquire Genie Compatible QNN binaries from AI Hub
+
+### [Llama Only] Setup Hugging Face token
+
+Setting up Hugging Face token is required only for the Llama model family.
+Request model access on Hugging Face for Llama models. For instance, you can [apply here](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) to access Llama 3.2 3B model.
+
+Set up Hugging Face token locally by following the instructions [here](https://huggingface.co/docs/huggingface_hub/en/guides/cli).
+
+### Download or Generate Genie Compatible QNN Binaries
+
+Some of the models can be downloaded directly from [AI
+Hub](https://aihub.qualcomm.com). For Llama, it has to be exported through [AI Hub
+Models](https://github.com/quic/ai-hub-models/tree/main/qai_hub_models/models).
+
+To generate the Llama assets, we will run a single command that performs the
+following steps:
+
+1. Download model weights from Hugging Face. You will need to sign the Llama
+license if you haven't already done so.
+
+2. Upload models to AI Hub for compilation.
+
+3. Download compiled context binaries. Note that there are multiple binaries as
+   we have split up the model.
+
+Make a directory to put in all deployable assets. For this example we use
+
+```bash
+mkdir -p genie_bundle
+```
+
+#### [Optional] Upgrade PyTorch
+
+The export command below may take 4-6 hours. It takes an additional 1-2 hours
+on PyTorch versions earlier than 2.4.0. We recommend upgrading PyTorch first:
+
+```bash
+pip install torch==2.4.0
+```
+
+This version is not yet supported in general by AI Hub Models but will work
+for the below export command.
+
+Note that the export also requires a lot of memory (RAM + swap) on the host
+device (for Llama 3, we recommend 80 GB). If we detect that you have less
+memory than recommended, the export command will print a warning with
+instructions of how to increase your swap space.
+
+#### For Android on Snapdragon® 8 Elite
+
+```bash
+python -m qai_hub_models.models.llama_v3_8b_instruct.export --device "Snapdragon 8 Elite QRD" --skip-inferencing --skip-profiling --output-dir genie_bundle
+```
+
+For Snapdragon 8 Gen 3, please use `--device "Snapdragon 8 Gen 3 QRD"`.
+
+#### For Windows on Snapdragon® X Elite
+
+```bash
+python -m qai_hub_models.models.llama_v3_8b_instruct.export --device "Snapdragon X Elite CRD" --skip-inferencing --skip-profiling --output-dir genie_bundle
+```
+
+Note: For older devices, you may need to adjust the context length using
+`--context-length <context-length>`.
+
+The `genie_bundle` would now contain both the intermediate models (`token`,
+`prompt`) and the final context binaries (`*.bin`). Remove the intermediate
+models to have a smaller deployable artifact:
+
+```bash
+# Remove intermediate assets
+rm -rf genie_bundle/{prompt,token}
+```
+
+## Prepare Genie Configs
+
+### Tokenizer
+
+To download the tokenizer, go to the source model's Hugging Face page and go to "Files
+and versions." You can find a Hugging Face link through the model card on
+[AI Hub](https://aihub.qualcomm.com/). This will take you to the Qualcomm Hugging Face page,
+which in turn will have a link to the source Hugging Face page. The file will be named `tokenizer.json`
+and should be downloaded to the `genie_bundle` directory. The tokenizers are only hosted on the source Hugging Face page.
+
+| Model name | Tokenizer | Notes |
+| --- | --- | --- |
+| Llama-v2-7B-Chat | [tokenizer.json](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/blob/main/tokenizer.json) | |
+| Llama-v3-8B-Instruct | [tokenizer.json](https://huggingface.co/meta-llama/Meta-Llama-3-8B/blob/main/tokenizer.json) | |
+| Llama-v3.1-8B-Instruct | [tokenizer.json](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/blob/main/tokenizer.json) | |
+| Llama-SEA-LION-v3.5-8B-R | [tokenizer.json](https://huggingface.co/aisingapore/Llama-SEA-LION-v3.5-8B-R/blob/main/tokenizer.json) | |
+| Llama-v3.2-3B-Instruct | [tokenizer.json](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct/blob/main/tokenizer.json) | |
+| Llama3-TAIDE-LX-8B-Chat-Alpha1 | [tokenizer.json](https://huggingface.co/taide/Llama3-TAIDE-LX-8B-Chat-Alpha1/blob/main/tokenizer.json) | |
+| Baichuan2-7B | [tokenizer.json](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/baichuan2_7b_quantized/v2/tokenizer.json) | |
+| Qwen2-7B-Instruct | [tokenizer.json](https://huggingface.co/Qwen/Qwen2-7B-Instruct/blob/main/tokenizer.json) | |
+| Phi-3.5-Mini-Instruct | [tokenizer.json](https://huggingface.co/microsoft/Phi-3.5-mini-instruct/blob/main/tokenizer.json) | To see appropriate spaces in the output, remove lines 193-196 (Strip rule) in the tokenizer file. |
+| Mistral-7B-Instruct-v0.3 | [tokenizer.json](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/blob/main/tokenizer.json) | |
+| IBM-Granite-v3.1-8B-Instruct | [tokenizer.json](https://huggingface.co/ibm-granite/granite-3.1-8b-base/blob/main/tokenizer.json) | |
+
+### [Optional] Use the Windows PowerShell LLM Runner
+
+**Do not use this script to create your Genie bundle if you are building Windows ChatApp. Continue with the rest of the tutorial instead.**
+
+The easiest path to running an LLM on a Windows on Snapdragon® device is to use the [PowerShell implementation](powershell/)
+of the rest of this tutorial. It will automatically generate the appropriate configuration files and execute `genie-t2t-run.exe`
+on a prompt of your choosing.
+
+### Genie Config
+
+Check out the [AI Hub Apps repository](https://github.com/quic/ai-hub-apps)
+using Git:
+
+```bash
+git clone https://github.com/quic/ai-hub-apps.git
+```
+
+Now run (replacing `llama_v3_8b_instruct` with the desired model id):
+
+```bash
+cp ai-hub-apps/tutorials/llm_on_genie/configs/genie/llama_v3_8b_instruct.json genie_bundle/genie_config.json
+```
+
+For Windows laptops, please set `use-mmap` to `false`.
+
+If you customized context length by adding `--context-length` to the export
+command, please open `genie_config.json` and modify the `"size"` option (under
+`"dialog"` -> `"context"`) to be consistent.
+
+In `genie_bundle/genie_config.json`, also ensure that the list of bin files in
+`ctx-bins` matches with the bin files under `genie_bundle`. Genie will look for
+QNN binaries specified here.
+
+### HTP Backend Config
+
+Copy the HTP config template:
+
+```bash
+cp ai-hub-apps/tutorials/llm_on_genie/configs/htp/htp_backend_ext_config.json.template genie_bundle/htp_backend_ext_config.json
+```
+
+Edit `soc_model` and `dsp_arch` in `genie_bundle/htp_backend_ext_config.json`
+depending on your target device (should be consistent with the `--device` you
+specified in the export command):
+
+| Generation               | `soc_model` | `dsp_arch` |
+|--------------------------|--------|----------|
+| Snapdragon® Gen 2        | 43     | v73      |
+| Snapdragon® Gen 3        | 57     | v75      |
+| Snapdragon® 8 Elite      | 69     | v79      |
+| Snapdragon® X Elite      | 60     | v73      |
+| Snapdragon® X Plus       | 60     | v73      |
+
+## Collect & Finalize Genie Bundle
+
+When finished with the above steps, your bundle should look like this:
+```
+genie_bundle/
+   genie_config.json
+   htp_backend_ext_config.json
+   tokenizer.json
+   <model_id>_part_1_of_N.bin
+   ...
+   <model_id>_part_N_of_N.bin
+```
+
+where <model_id> is the name of the model. This is the name of the json you copied from `configs/genie/<model_name>.json`.
+
+## Run LLM on Device
+
+You have three options to run the LLM on device:
+
+ 1. Use the `genie-t2t-run` CLI command.
+ 2. Use the [CLI Windows ChatApp](https://github.com/quic/ai-hub-apps/tree/main/apps/windows/cpp/ChatApp) (Windows only).
+ 3. Use the [Android ChatApp](https://github.com/quic/ai-hub-apps/tree/main/apps/android/ChatApp).
+
+### Prompt Formats
+
+All the LLMs have different formats. To get sensible output from the LLMs, it is important to use the correct prompt format for the model. These can also be found on the Hugging Face repository for each of the model. Adding samples for a few models here.
+
+| Model name | Sample Prompt |
+| --- | --- |
+| Llama-v2-7B-Chat | &lt;s&gt;[INST] &lt;&lt;SYS&gt;&gt;You are a helpful AI Assistant.&lt;&lt;/SYS&gt;&gt;[/INST]&lt;/s>&lt;s&gt;[INST]What is France's capital?[/INST] |
+| Llama-v3-8B-Instruct <br> Llama-v3.1-8B-Instruct <br> Llama-v3.2-3B-Instruct | <&#124;begin_of_text&#124;><&#124;start_header_id&#124;>user<&#124;end_header_id&#124;>\n\nWhat is France's capital?<&#124;eot_id&#124;><&#124;start_header_id&#124;>assistant<&#124;end_header_id&#124;> |
+| Llama3-TAIDE-LX-8B-Chat-Alpha1 | <&#124;begin_of_text&#124;><&#124;start_header_id&#124;>system<&#124;end_header_id&#124;>\n\n你是一個來自台灣的AI助理，你的名字是 TAIDE，樂於以台灣人的立場幫助使用者，會用繁體中文回答問題<&#124;eot_id&#124;>\n<&#124;start_header_id&#124;>user<&#124;end_header_id&#124;>\n\n介紹台灣特色<&#124;eot_id&#124;>\n<&#124;start_header_id&#124;>assistant<&#124;end_header_id&#124;> |
+| Llama-SEA-LION-v3.5-8B-R (non-thinking mode) | <&#124;begin_of_text&#124;><&#124;start_header_id&#124;>system<&#124;end_header_id&#124;>\n\ndetailed thinking off<&#124;eot_id&#124;><&#124;start_header_id&#124;>user<&#124;end_header_id&#124;>\n\nThủ đô của Việt Nam là thành phố nào?<&#124;eot_id&#124;><&#124;start_header_id&#124;>assistant<&#124;end_header_id&#124;>\n\n&lt;think&gt;\n\n&lt;/think&gt;>\n\n |
+| Llama-SEA-LION-v3.5-8B-R (thinking mode) | <&#124;begin_of_text&#124;><&#124;start_header_id&#124;>system<&#124;end_header_id&#124;>\n\ndetailed thinking on<&#124;eot_id&#124;><&#124;start_header_id&#124;>user<&#124;end_header_id&#124;>\n\nThủ đô của Việt Nam là thành phố nào?<&#124;eot_id&#124;><&#124;start_header_id&#124;>assistant<&#124;end_header_id&#124;>\n\n&lt;think&gt;\nHere is my thinking:\n |
+| Qwen2-7B-Instruct | <&#124;im_start&#124;>system\nYou are a helpful AI Assistant<&#124;im_end&#124;><&#124;im_start&#124;>What is France's capital?\n<&#124;im_end&#124;>\n<&#124;im_start&#124;>assistant\n |
+| Phi-3.5-Mini-Instruct | <&#124;system&#124;>\nYou are a helpful assistant. Be helpful but brief.<&#124;end&#124;>\n<&#124;user&#124;>What is France's capital?\n<&#124;end&#124;>\n<&#124;assistant&#124;>\n |
+| Mistral-7B-Instruct-v0.3 | &lt;s&gt;[INST] You are a helpful assistant\n\nTranslate 'Good morning, how are you?' into French.[/INST] |
+| IBM-Granite-v3.1-8B-Instruct | <&#124;start_of_role&#124;>system<&#124;end_of_role&#124;>You are a helpful AI assistant.<&#124;end_of_text&#124;>\n <&#124;start_of_role&#124;>user<&#124;end_of_role&#124;>What is France's capital?<&#124;end_of_text&#124;>\n <&#124;start_of_role&#124;>assistant<&#124;end_of_role&#124;>\n |
+
+### 1. Run Genie On-Device via `genie-t2t-run`
+
+#### Genie on Windows with Snapdragon® X
+
+Copy Genie's shared libraries and executable to our bundle.
+(Note you can skip this step if you used the powershell script to prepare your bundle.)
+
+```bash
+cp $QNN_SDK_ROOT/lib/hexagon-v73/unsigned/* genie_bundle
+cp $QNN_SDK_ROOT/lib/aarch64-windows-msvc/* genie_bundle
+cp $QNN_SDK_ROOT/bin/aarch64-windows-msvc/genie-t2t-run.exe genie_bundle
+```
+
+In Powershell, navigate to the bundle directory and run
+
+```bash
+./genie-t2t-run.exe -c genie_config.json -p "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nWhat is France's capital?<|eot_id|><|start_header_id|>assistant<|end_header_id|>"
+```
+
+Note that this prompt format is specific to Llama 3.
+
+#### Genie on Android
+
+Copy Genie's shared libraries and executable to our bundle.
+
+```bash
+# For 8 Gen 2
+cp $QNN_SDK_ROOT/lib/hexagon-v73/unsigned/* genie_bundle
+# For 8 Gen 3
+cp $QNN_SDK_ROOT/lib/hexagon-v75/unsigned/* genie_bundle
+# For 8 Elite
+cp $QNN_SDK_ROOT/lib/hexagon-v79/unsigned/* genie_bundle
+# For all devices
+cp $QNN_SDK_ROOT/lib/aarch64-android/* genie_bundle
+cp $QNN_SDK_ROOT/bin/aarch64-android/genie-t2t-run genie_bundle
+```
+
+Copy `genie_bundle` from the host machine to the target device using ADB and
+open up an interactive shell on the target device:
+
+```bash
+adb push genie_bundle /data/local/tmp
+adb shell
+```
+
+On device, navigate to the bundle directory:
+
+```bash
+cd /data/local/tmp/genie_bundle
+```
+
+Set `LD_LIBRARY_PATH` and `ADSP_LIBRARY_PATH` to the current directory:
+
+```bash
+export LD_LIBRARY_PATH=$PWD
+export ADSP_LIBRARY_PATH=$PWD
+```
+
+Then run:
+
+```bash
+./genie-t2t-run -c genie_config.json -p "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nWhat is France's capital?<|eot_id|><|start_header_id|>assistant<|end_header_id|>"
+```
+
+#### Sample Output
+
+```text
+Using libGenie.so version 1.1.0
+
+[WARN]  "Unable to initialize logging in backend extensions."
+[INFO]  "Using create From Binary List Async"
+[INFO]  "Allocated total size = 323453440 across 10 buffers"
+[PROMPT]: <|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nWhat is France's capital?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
+
+[BEGIN]: \n\nFrance's capital is Paris.[END]
+
+[KPIS]:
+Init Time: 6549034 us
+Prompt Processing Time: 196067 us, Prompt Processing Rate : 86.707710 toks/sec
+Token Generation Time: 740568 us, Token Generation Rate: 12.152884 toks/sec
+```
+
+### 2. Sample C++ Chat App Powered by Genie SDK
+
+We provide a sample C++ app to show how to build an application using the Genie SDK.
+See [CLI Windows ChatApp](https://github.com/quic/ai-hub-apps/tree/main/apps/windows/cpp/ChatApp) for more details.
+
+### 3. Sample Android Chat App Powered by Genie SDK
+
+We provide a sample Android (Java and C++ app) to show how to build an application using the Genie SDK for mobile.
+See [Android ChatApp](https://github.com/quic/ai-hub-apps/tree/main/apps/android/ChatApp) for more details.

configs/genie/baichuan2_7b.json

@@ -0,0 +1,59 @@
+{
+    "dialog": {
+        "version": 1,
+        "type": "basic",
+        "context": {
+            "version": 1,
+            "size": 4096,
+            "n-vocab": 125696,
+            "bos-token": 1,
+            "eos-token": 2
+        },
+        "sampler": {
+            "version": 1,
+            "seed": 42,
+            "temp": 0.8,
+            "top-k": 40,
+            "top-p": 0.95
+        },
+        "tokenizer": {
+            "version": 1,
+            "path": "tokenizer.json"
+        },
+        "engine": {
+            "version": 1,
+            "n-threads": 3,
+            "backend": {
+                "version": 1,
+                "type": "QnnHtp",
+                "QnnHtp": {
+                    "version": 1,
+                    "use-mmap": true,
+                    "spill-fill-bufsize": 0,
+                    "mmap-budget": 0,
+                    "poll": false,
+                    "cpu-mask": "0xe0",
+                    "kv-dim": 128,
+                    "rope-theta": 10000,
+                    "pos-id-dim" : 64,
+                    "allow-async-init": false
+                },
+                "extensions": "htp_backend_ext_config.json"
+            },
+            "model": {
+                "version": 1,
+                "type": "binary",
+                "binary": {
+                    "version": 1,
+                    "ctx-bins": [
+                        "weight_sharing_model_1_of_5.serialized.bin",
+                        "weight_sharing_model_2_of_5.serialized.bin",
+                        "weight_sharing_model_3_of_5.serialized.bin",
+                        "weight_sharing_model_4_of_5.serialized.bin",
+                        "weight_sharing_model_5_of_5.serialized.bin"
+                    ]
+                }
+            }
+        }
+    }
+}

configs/genie/ibm_granite_v3_1_8b_instruct.json

@@ -0,0 +1,63 @@
+{
+    "dialog": {
+        "version": 1,
+        "type": "basic",
+        "context": {
+            "version": 1,
+            "size": 4096,
+            "n-vocab": 49155,
+            "bos-token": 0,
+            "eos-token": 0,
+            "pad-token": 0
+        },
+        "sampler": {
+            "version": 1,
+            "seed": 42,
+            "temp": 0.8,
+            "top-k": 40,
+            "top-p": 0.95
+        },
+        "tokenizer": {
+            "version": 1,
+            "path": "tokenizer.json"
+        },
+        "engine": {
+            "version": 1,
+            "n-threads": 3,
+            "backend": {
+                "version": 1,
+                "type": "QnnHtp",
+                "QnnHtp": {
+                    "version": 1,
+                    "use-mmap": true,
+                    "spill-fill-bufsize": 0,
+                    "mmap-budget": 0,
+                    "poll": true,
+                    "cpu-mask": "0xe0",
+                    "kv-dim": 128,
+                    "allow-async-init": false
+                },
+                "extensions": "htp_backend_ext_config.json"
+            },
+            "model": {
+                "version": 1,
+                "type": "binary",
+                "binary": {
+                    "version": 1,
+                    "ctx-bins": [
+                        "weight_sharing_model_1_of_5.serialized.bin",
+                        "weight_sharing_model_2_of_5.serialized.bin",
+                        "weight_sharing_model_3_of_5.serialized.bin",
+                        "weight_sharing_model_4_of_5.serialized.bin",
+                        "weight_sharing_model_5_of_5.serialized.bin"
+                    ]
+                },
+                "positional-encoding": {
+                    "type": "rope",
+                    "rope-dim": 64,
+                    "rope-theta": 10000000
+                }
+            }
+        }
+    }
+}

configs/genie/llama_v2_7b_chat.json

@@ -0,0 +1,57 @@
+{
+    "dialog": {
+        "version": 1,
+        "type": "basic",
+        "context": {
+            "version": 1,
+            "size": 1024,
+            "n-vocab": 32000,
+            "bos-token": 1,
+            "eos-token": 2
+        },
+        "sampler": {
+            "version": 1,
+            "seed": 42,
+            "temp": 0.8,
+            "top-k": 40,
+            "top-p": 0.95
+        },
+        "tokenizer": {
+            "version": 1,
+            "path": "tokenizer.json"
+        },
+        "engine": {
+            "version": 1,
+            "n-threads": 4,
+            "backend": {
+                "version": 1,
+                "type": "QnnHtp",
+                "QnnHtp": {
+                    "version": 1,
+                    "spill-fill-bufsize": 0,
+                    "use-mmap": true,
+                    "mmap-budget": 0,
+                    "poll": true,
+                    "pos-id-dim": 64,
+                    "cpu-mask": "0xe0",
+                    "kv-dim": 128,
+                    "allow-async-init": false
+                },
+                "extensions": "htp_backend_ext_config.json"
+            },
+            "model": {
+                "version": 1,
+                "type": "binary",
+                "binary": {
+                    "version": 1,
+                    "ctx-bins": [
+                        "Llama2_Part1.bin",
+                        "Llama2_Part2.bin",
+                        "Llama2_Part3.bin",
+                        "Llama2_Part4.bin"
+                    ]
+                }
+            }
+        }
+    }
+}

configs/genie/llama_v3_1_8b_instruct.json

@@ -0,0 +1,69 @@
+{
+    "dialog": {
+        "version": 1,
+        "type": "basic",
+        "context": {
+            "version": 1,
+            "size": 4096,
+            "n-vocab": 128256,
+            "bos-token": -1,
+            "eos-token": [128001, 128009, 128008]
+        },
+        "sampler": {
+            "version": 1,
+            "seed": 42,
+            "temp": 0.8,
+            "top-k": 40,
+            "top-p": 0.95
+        },
+        "tokenizer": {
+            "version": 1,
+            "path": "tokenizer.json"
+        },
+        "engine": {
+            "version": 1,
+            "n-threads": 3,
+            "backend": {
+                "version": 1,
+                "type": "QnnHtp",
+                "QnnHtp": {
+                    "version": 1,
+                    "use-mmap": true,
+                    "spill-fill-bufsize": 0,
+                    "mmap-budget": 0,
+                    "poll": true,
+                    "cpu-mask": "0xe0",
+                    "kv-dim": 128,
+                    "allow-async-init": false
+                },
+                "extensions": "htp_backend_ext_config.json"
+            },
+            "model": {
+                "version": 1,
+                "type": "binary",
+                "binary": {
+                    "version": 1,
+                    "ctx-bins": [
+                        "llama_v3_1_8b_instruct_part_1_of_5.bin",
+                        "llama_v3_1_8b_instruct_part_2_of_5.bin",
+                        "llama_v3_1_8b_instruct_part_3_of_5.bin",
+                        "llama_v3_1_8b_instruct_part_4_of_5.bin",
+                        "llama_v3_1_8b_instruct_part_5_of_5.bin"
+                    ]
+                },
+                "positional-encoding": {
+                    "type": "rope",
+                    "rope-dim": 64,
+                    "rope-theta": 500000,
+                    "rope-scaling": {
+                        "rope-type": "llama3",
+                        "factor": 8.0,
+                        "low-freq-factor": 1.0,
+                        "high-freq-factor": 4.0,
+                        "original-max-position-embeddings": 8192
+                    }
+                }
+            }
+        }
+    }
+}

configs/genie/llama_v3_1_sea_lion_3_5_r_8b_chat.json

@@ -0,0 +1,69 @@
+{
+    "dialog": {
+        "version": 1,
+        "type": "basic",
+        "context": {
+            "version": 1,
+            "size": 4096,
+            "n-vocab": 128256,
+            "bos-token": -1,
+            "eos-token": [128001, 128009, 128008]
+        },
+        "sampler": {
+            "version": 1,
+            "seed": 42,
+            "temp": 0.85,
+            "top-k": 40,
+            "top-p": 0.6
+        },
+        "tokenizer": {
+            "version": 1,
+            "path": "tokenizer.json"
+        },
+        "engine": {
+            "version": 1,
+            "n-threads": 3,
+            "backend": {
+                "version": 1,
+                "type": "QnnHtp",
+                "QnnHtp": {
+                    "version": 1,
+                    "use-mmap": true,
+                    "spill-fill-bufsize": 0,
+                    "mmap-budget": 0,
+                    "poll": true,
+                    "cpu-mask": "0xe0",
+                    "kv-dim": 128,
+                    "allow-async-init": false
+                },
+                "extensions": "htp_backend_ext_config.json"
+            },
+            "model": {
+                "version": 1,
+                "type": "binary",
+                "binary": {
+                    "version": 1,
+                    "ctx-bins": [
+                        "llama_v3_1_sea_lion_3_5_r_8b_chat_part_1_of_5.bin",
+                        "llama_v3_1_sea_lion_3_5_r_8b_chat_part_2_of_5.bin",
+                        "llama_v3_1_sea_lion_3_5_r_8b_chat_part_3_of_5.bin",
+                        "llama_v3_1_sea_lion_3_5_r_8b_chat_part_4_of_5.bin",
+                        "llama_v3_1_sea_lion_3_5_r_8b_chat_part_5_of_5.bin"
+                    ]
+                },
+                "positional-encoding": {
+                    "type": "rope",
+                    "rope-dim": 64,
+                    "rope-theta": 500000,
+                    "rope-scaling": {
+                        "rope-type": "llama3",
+                        "factor": 8.0,
+                        "low-freq-factor": 1.0,
+                        "high-freq-factor": 4.0,
+                        "original-max-position-embeddings": 8192
+                    }
+                }
+            }
+        }
+    }
+}

configs/genie/llama_v3_2_3b_instruct.json

@@ -0,0 +1,67 @@
+{
+    "dialog": {
+        "version": 1,
+        "type": "basic",
+        "context": {
+            "version": 1,
+            "size": 4096,
+            "n-vocab": 128256,
+            "bos-token": -1,
+            "eos-token": [128001, 128009, 128008]
+        },
+        "sampler": {
+            "version": 1,
+            "seed": 42,
+            "temp": 0.8,
+            "top-k": 40,
+            "top-p": 0.95
+        },
+        "tokenizer": {
+            "version": 1,
+            "path": "tokenizer.json"
+        },
+        "engine": {
+            "version": 1,
+            "n-threads": 3,
+            "backend": {
+                "version": 1,
+                "type": "QnnHtp",
+                "QnnHtp": {
+                    "version": 1,
+                    "use-mmap": true,
+                    "spill-fill-bufsize": 0,
+                    "mmap-budget": 0,
+                    "poll": true,
+                    "cpu-mask": "0xe0",
+                    "kv-dim": 128,
+                    "allow-async-init": false
+                },
+                "extensions": "htp_backend_ext_config.json"
+            },
+            "model": {
+                "version": 1,
+                "type": "binary",
+                "binary": {
+                    "version": 1,
+                    "ctx-bins": [
+                        "llama_v3_2_3b_instruct_part_1_of_3.bin",
+                        "llama_v3_2_3b_instruct_part_2_of_3.bin",
+                        "llama_v3_2_3b_instruct_part_3_of_3.bin"
+                    ]
+                },
+                "positional-encoding": {
+                    "type": "rope",
+                    "rope-dim": 64,
+                    "rope-theta": 500000,
+                    "rope-scaling": {
+                        "rope-type": "llama3",
+                        "factor": 8.0,
+                        "low-freq-factor": 1.0,
+                        "high-freq-factor": 4.0,
+                        "original-max-position-embeddings": 8192
+                    }
+                }
+            }
+        }
+    }
+}

configs/genie/llama_v3_8b_instruct.json

@@ -0,0 +1,60 @@
+{
+    "dialog": {
+        "version": 1,
+        "type": "basic",
+        "context": {
+            "version": 1,
+            "size": 4096,
+            "n-vocab": 128256,
+            "bos-token": 128000,
+            "eos-token": 128001,
+            "eot-token": 128009
+        },
+        "sampler": {
+            "version": 1,
+            "seed": 42,
+            "temp": 0.8,
+            "top-k": 40,
+            "top-p": 0.95
+        },
+        "tokenizer": {
+            "version": 1,
+            "path": "tokenizer.json"
+        },
+        "engine": {
+            "version": 1,
+            "n-threads": 3,
+            "backend": {
+                "version": 1,
+                "type": "QnnHtp",
+                "QnnHtp": {
+                    "version": 1,
+                    "use-mmap": true,
+                    "spill-fill-bufsize": 0,
+                    "mmap-budget": 0,
+                    "poll": true,
+                    "pos-id-dim": 64,
+                    "cpu-mask": "0xe0",
+                    "kv-dim": 128,
+                    "rope-theta": 500000,
+                    "allow-async-init": false
+                },
+                "extensions": "htp_backend_ext_config.json"
+            },
+            "model": {
+                "version": 1,
+                "type": "binary",
+                "binary": {
+                    "version": 1,
+                    "ctx-bins": [
+                        "llama_v3_8b_instruct_part_1_of_5.bin",
+                        "llama_v3_8b_instruct_part_2_of_5.bin",
+                        "llama_v3_8b_instruct_part_3_of_5.bin",
+                        "llama_v3_8b_instruct_part_4_of_5.bin",
+                        "llama_v3_8b_instruct_part_5_of_5.bin"
+                    ]
+                }
+            }
+        }
+    }
+}

configs/genie/llama_v3_taide_8b_chat.json

@@ -0,0 +1,60 @@
+{
+    "dialog": {
+        "version": 1,
+        "type": "basic",
+        "context": {
+            "version": 1,
+            "size": 4096,
+            "n-vocab": 128256,
+            "bos-token": 128000,
+            "eos-token": 128001,
+            "eot-token": 128009
+        },
+        "sampler": {
+            "version": 1,
+            "seed": 42,
+            "temp": 0.8,
+            "top-k": 40,
+            "top-p": 0.95
+        },
+        "tokenizer": {
+            "version": 1,
+            "path": "tokenizer.json"
+        },
+        "engine": {
+            "version": 1,
+            "n-threads": 3,
+            "backend": {
+                "version": 1,
+                "type": "QnnHtp",
+                "QnnHtp": {
+                    "version": 1,
+                    "use-mmap": true,
+                    "spill-fill-bufsize": 0,
+                    "mmap-budget": 0,
+                    "poll": true,
+                    "pos-id-dim": 64,
+                    "cpu-mask": "0xe0",
+                    "kv-dim": 128,
+                    "rope-theta": 500000,
+                    "allow-async-init": false
+                },
+                "extensions": "htp_backend_ext_config.json"
+            },
+            "model": {
+                "version": 1,
+                "type": "binary",
+                "binary": {
+                    "version": 1,
+                    "ctx-bins": [
+                        "llama_v3_taide_8b_chat_part_1_of_5.bin",
+                        "llama_v3_taide_8b_chat_part_2_of_5.bin",
+                        "llama_v3_taide_8b_chat_part_3_of_5.bin",
+                        "llama_v3_taide_8b_chat_part_4_of_5.bin",
+                        "llama_v3_taide_8b_chat_part_5_of_5.bin"
+                    ]
+                }
+            }
+        }
+    }
+}

configs/genie/mistral_7b_instruct_v0_3.json

@@ -0,0 +1,59 @@
+{
+    "dialog": {
+        "version": 1,
+        "type": "basic",
+        "context": {
+            "version": 1,
+            "size": 4096,
+            "n-vocab": 128256,
+            "bos-token": 1,
+            "eos-token": 2,
+            "eot-token": 128009
+        },
+        "sampler": {
+            "version": 1,
+            "seed": 42,
+            "temp": 0.8,
+            "top-k": 40,
+            "top-p": 0.95
+        },
+        "tokenizer": {
+            "version": 1,
+            "path": "tokenizer.json"
+        },
+        "engine": {
+            "version": 1,
+            "n-threads": 3,
+            "backend": {
+                "version": 1,
+                "type": "QnnHtp",
+                "QnnHtp": {
+                    "version": 1,
+                    "spill-fill-bufsize": 0,
+                    "use-mmap": true,
+                    "mmap-budget": 0,
+                    "poll": true,
+                    "pos-id-dim": 64,
+                    "cpu-mask": "0xe0",
+                    "kv-dim": 128,
+                    "rope-theta": 10000,
+                    "allow-async-init": false
+                },
+                "extensions": "htp_backend_ext_config.json"
+            },
+            "model": {
+                "version": 1,
+                "type": "binary",
+                "binary": {
+                    "version": 1,
+                    "ctx-bins": [
+                        "weight_sharing_model_1_of_4.serialized.bin",
+                        "weight_sharing_model_2_of_4.serialized.bin",
+                        "weight_sharing_model_3_of_4.serialized.bin",
+                        "weight_sharing_model_4_of_4.serialized.bin"
+                    ]
+                }
+            }
+        }
+    }
+}

configs/genie/phi_3_5_mini_instruct.json

@@ -0,0 +1,170 @@
+{
+    "dialog": {
+      "version": 1,
+      "type": "basic",
+      "context": {
+        "version": 1,
+        "n-vocab": 32064,
+        "size": 4096,
+        "bos-token": 1,
+        "eos-token": [
+          32007,
+          32001,
+          32000
+        ]
+      },
+      "sampler": {
+        "version": 1,
+        "seed": 42,
+        "temp": 0.8,
+        "top-k": 40,
+        "top-p": 0.95
+      },
+      "tokenizer": {
+        "version": 1,
+        "path": "tokenizer.json"
+      },
+      "engine": {
+        "version": 1,
+        "n-threads": 3,
+        "backend": {
+          "version": 1,
+          "type": "QnnHtp",
+          "QnnHtp": {
+            "version": 1,
+            "use-mmap": true,
+            "spill-fill-bufsize": 0,
+            "mmap-budget": 0,
+            "poll": true,
+            "cpu-mask": "0xe0",
+            "kv-dim": 96,
+            "allow-async-init": false
+          },
+          "extensions": "htp_backend_ext_config.json"
+        },
+        "model": {
+          "version": 1,
+          "type": "binary",
+          "binary": {
+            "version": 1,
+            "ctx-bins": [
+              "weight_sharing_model_1_of_4.serialized.bin",
+              "weight_sharing_model_2_of_4.serialized.bin",
+              "weight_sharing_model_3_of_4.serialized.bin",
+              "weight_sharing_model_4_of_4.serialized.bin"
+            ]
+          },
+          "positional-encoding": {
+            "type": "rope",
+            "rope-dim": 48,
+            "rope-theta": 10000,
+            "rope-scaling": {
+              "rope-type": "longrope",
+              "factor": 32,
+              "original-max-position-embeddings": 4096,
+              "long-factor": [
+                1.0800000429153442,
+                1.1100000143051147,
+                1.1399999856948853,
+                1.340000033378601,
+                1.5899999141693115,
+                1.600000023841858,
+                1.6200000047683716,
+                2.620000123977661,
+                3.2300000190734863,
+                3.2300000190734863,
+                4.789999961853027,
+                7.400000095367432,
+                7.700000286102295,
+                9.09000015258789,
+                12.199999809265137,
+                17.670000076293945,
+                24.46000099182129,
+                28.57000160217285,
+                30.420001983642578,
+                30.840002059936523,
+                32.590003967285156,
+                32.93000411987305,
+                42.320003509521484,
+                44.96000289916992,
+                50.340003967285156,
+                50.45000457763672,
+                57.55000305175781,
+                57.93000411987305,
+                58.21000289916992,
+                60.1400032043457,
+                62.61000442504883,
+                62.62000274658203,
+                62.71000289916992,
+                63.1400032043457,
+                63.1400032043457,
+                63.77000427246094,
+                63.93000411987305,
+                63.96000289916992,
+                63.970001220703125,
+                64.02999877929688,
+                64.06999969482422,
+                64.08000183105469,
+                64.12000274658203,
+                64.41000366210938,
+                64.4800033569336,
+                64.51000213623047,
+                64.52999877929688,
+                64.83999633789062
+              ],
+              "short-factor": [
+                1,
+                1.0199999809265137,
+                1.0299999713897705,
+                1.0299999713897705,
+                1.0499999523162842,
+                1.0499999523162842,
+                1.0499999523162842,
+                1.0499999523162842,
+                1.0499999523162842,
+                1.0699999332427979,
+                1.0999999046325684,
+                1.1099998950958252,
+                1.1599998474121094,
+                1.1599998474121094,
+                1.1699998378753662,
+                1.2899998426437378,
+                1.339999794960022,
+                1.679999828338623,
+                1.7899998426437378,
+                1.8199998140335083,
+                1.8499997854232788,
+                1.8799997568130493,
+                1.9099997282028198,
+                1.9399996995925903,
+                1.9899996519088745,
+                2.0199997425079346,
+                2.0199997425079346,
+                2.0199997425079346,
+                2.0199997425079346,
+                2.0199997425079346,
+                2.0199997425079346,
+                2.0299997329711914,
+                2.0299997329711914,
+                2.0299997329711914,
+                2.0299997329711914,
+                2.0299997329711914,
+                2.0299997329711914,
+                2.0299997329711914,
+                2.0299997329711914,
+                2.0299997329711914,
+                2.0799996852874756,
+                2.0899996757507324,
+                2.189999580383301,
+                2.2199995517730713,
+                2.5899994373321533,
+                2.729999542236328,
+                2.749999523162842,
+                2.8399994373321533
+              ]
+            }
+          }
+        }
+      }
+    }
+  }

configs/genie/qwen2_7b_instruct.json

@@ -0,0 +1,58 @@
+{
+    "dialog": {
+        "version": 1,
+        "type": "basic",
+        "context": {
+            "version": 1,
+            "size": 4096,
+            "n-vocab": 152064,
+            "bos-token": -1,
+            "eos-token": 151645
+        },
+        "sampler": {
+            "version": 1,
+            "seed": 42,
+            "temp": 0.8,
+            "top-k": 40,
+            "top-p": 0.95
+        },
+        "tokenizer": {
+            "version": 1,
+            "path": "tokenizer.json"
+        },
+        "engine": {
+            "version": 1,
+            "n-threads": 3,
+            "backend": {
+                "version": 1,
+                "type": "QnnHtp",
+                "QnnHtp": {
+                    "version": 1,
+                    "use-mmap": true,
+                    "spill-fill-bufsize": 0,
+                    "mmap-budget": 0,
+                    "poll": false,
+                    "pos-id-dim": 64,
+                    "cpu-mask": "0xe0",
+                    "kv-dim": 128,
+                    "rope-theta": 1000000,
+                    "allow-async-init": false
+                },
+                "extensions": "htp_backend_ext_config.json"
+            },
+            "model": {
+                "version": 1,
+                "type": "binary",
+                "binary": {
+                    "version": 1,
+                    "ctx-bins": [
+                        "weight_sharing_model_1_of_4.serialized.bin",
+                        "weight_sharing_model_2_of_4.serialized.bin",
+                        "weight_sharing_model_3_of_4.serialized.bin",
+                        "weight_sharing_model_4_of_4.serialized.bin"
+                    ]
+                }
+            }
+        }
+    }
+}

configs/htp/htp_backend_ext_config.json.template

@@ -0,0 +1,21 @@
+{
+    "devices": [
+        {
+            "soc_model": <TODO>,
+            "dsp_arch": "<TODO>",
+            "cores": [
+                {
+                    "core_id": 0,
+                    "perf_profile": "burst",
+                    "rpc_control_latency": 100
+                }
+            ]
+        }
+    ],
+    "memory": {
+        "mem_type": "shared_buffer"
+    },
+    "context": {
+        "weight_sharing_enabled": true
+    }
+}

genie_bundle/Genie.dll

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:415408862c6d4d00dcb6311fbe09295616488e34fef7c56021d78a19124ab018
+size 6075904

genie_bundle/PlatformValidatorShared.dll

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:148b5d54dd5c66886bd54fc2fc61492414d04ff8dd31ce8a2395596219cdc852
+size 10963968

genie_bundle/QnnGenAiTransformer.dll

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2fe7907763b53c1a6f576844a3cc8a2e984adc5ca74fca8aa68e9023b0c5b64
+size 516096

genie_bundle/QnnHtp.dll

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45371a736f3fa4a131fb51031959d6b771577ec2ecc2ea2ef25c690ac94f77ab
+size 2168832

genie_bundle/QnnHtpNetRunExtensions.dll

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:85aa19646293a0b6319345064b034ce36f6a89935b6ab98fad5221af3640c6e8
+size 835584

genie_bundle/QnnHtpPrepare.dll

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6202a3516c606545ab28fc4db0e554f46839aef9c178065cc9f637e82024d5bf
+size 62586880

genie_bundle/QnnHtpv73Stub.dll

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:14bba834d18b6491b0da343f2ef37d6497f5f6f43991d84ea4954916ae09419d
+size 328704

genie_bundle/QnnSystem.dll

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1bc04d39a24d81932b545293e7de1dccc632d3b52c9841dc5e8cb4276142bbeb
+size 2276352

genie_bundle/genie-t2t-run.exe

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1afe8a72063d6780b100e9a75f161c681b309c9e2af8640a28f1e4c344269453
+size 228864

genie_bundle/genie_config.json

@@ -0,0 +1,67 @@
+{
+    "dialog": {
+        "version": 1,
+        "type": "basic",
+        "context": {
+            "version": 1,
+            "size": 4096,
+            "n-vocab": 128256,
+            "bos-token": -1,
+            "eos-token": [128001, 128009, 128008]
+        },
+        "sampler": {
+            "version": 1,
+            "seed": 42,
+            "temp": 0.8,
+            "top-k": 40,
+            "top-p": 0.95
+        },
+        "tokenizer": {
+            "version": 1,
+            "path": "C:\\ai-hub-apps\\tutorials\\llm_on_genie\\genie_bundle\\tokenizer.json"
+        },
+        "engine": {
+            "version": 1,
+            "n-threads": 3,
+            "backend": {
+                "version": 1,
+                "type": "QnnHtp",
+                "QnnHtp": {
+                    "version": 1,
+                    "use-mmap": false,
+                    "spill-fill-bufsize": 0,
+                    "mmap-budget": 0,
+                    "poll": true,
+                    "cpu-mask": "0xe0",
+                    "kv-dim": 128,
+                    "allow-async-init": false
+                },
+                "extensions": "C:\\ai-hub-apps\\tutorials\\llm_on_genie\\genie_bundle\\htp_backend_ext_config.json"
+            },
+            "model": {
+                "version": 1,
+                "type": "binary",
+                "binary": {
+                    "version": 1,
+                    "ctx-bins": [
+                        "C:\\ai-hub-apps\\tutorials\\llm_on_genie\\genie_bundle\\llama_v3_2_3b_instruct_part_1_of_3.bin",
+                        "C:\\ai-hub-apps\\tutorials\\llm_on_genie\\genie_bundle\\llama_v3_2_3b_instruct_part_2_of_3.bin",
+                        "C:\\ai-hub-apps\\tutorials\\llm_on_genie\\genie_bundle\\llama_v3_2_3b_instruct_part_3_of_3.bin"
+                    ]
+                },
+                "positional-encoding": {
+                    "type": "rope",
+                    "rope-dim": 64,
+                    "rope-theta": 500000,
+                    "rope-scaling": {
+                        "rope-type": "llama3",
+                        "factor": 8.0,
+                        "low-freq-factor": 1.0,
+                        "high-freq-factor": 4.0,
+                        "original-max-position-embeddings": 8192
+                    }
+                }
+            }
+        }
+    }
+}

genie_bundle/htp_backend_ext_config.json

@@ -0,0 +1,21 @@
+{
+    "devices": [
+        {
+            "soc_model": 60,
+            "dsp_arch": "v73",
+            "cores": [
+                {
+                    "core_id": 0,
+                    "perf_profile": "burst",
+                    "rpc_control_latency": 100
+                }
+            ]
+        }
+    ],
+    "memory": {
+        "mem_type": "shared_buffer"
+    },
+    "context": {
+        "weight_sharing_enabled": true
+    }
+}

genie_bundle/libQnnHtpv73Skel.so

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a96ea19f8bb55d576ab3721787d2502bc744064cca5f7f4aa267b39b802d45b5
+size 9311684

genie_bundle/qnn-platform-validator.exe

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:75b259422c65c7e970280a732059f451637ea1048bafdab00bcf45d37fd90047
+size 637952

platformValidator/output/Result.csv

@@ -0,0 +1,3 @@
+Platform Validator Results:
+Backend,Hardware Backend,Library Prerequisites,Library Version,Backend Core Version,Unit Test,Overall Result
+DSP,Supported,Found,Not Queried,Not Queried,Passed,Passed

powershell/LlmUtils.ps1

@@ -0,0 +1,188 @@
+# ------------------------------------------------------------------------
+# Copyright (c) 2025 Qualcomm Innovation Center, Inc. All rights reserved.
+# SPDX-License-Identifier: BSD-3-Clause
+# ------------------------------------------------------------------------
+
+function Copy-Genie {
+<#
+.SYNOPSIS
+    Copy enough of QNN to be able to run genie-t2t-run.exe.
+#>
+
+    param(
+        [Parameter(Mandatory = $true)]
+        [string]$QnnSdkRoot,
+
+        [Parameter(Mandatory = $true)]
+        [string]$Destination,
+
+        [Parameter(Mandatory = $true)]
+        [string]$DspArch,
+
+        [boolean]$Clean = $true
+    )
+
+    if ($Clean) {
+        Remove-Item -Path $Destination\*.* -Include *.cat, *.dll, *.exe, *.lib, *.so
+    }
+
+    $Arch = "aarch64-windows-msvc"
+
+    Copy-Item -Destination $Destination `
+        $QnnSdkRoot\bin\$Arch\genie-t2t-run.exe, `
+        $QnnSdkRoot\bin\$Arch\qnn-platform-validator.exe, `
+        $QnnSdkRoot\lib\$Arch\Genie.dll, `
+        $QnnSdkRoot\lib\$Arch\PlatformValidatorShared.dll, `
+        $QnnSdkRoot\lib\$Arch\QnnGenAiTransformer.dll, `
+        $QnnSdkRoot\lib\$Arch\QnnGenAiTransformerModel.dll, `
+        $QnnSdkRoot\lib\$Arch\QnnHtp.dll, `
+        $QnnSdkRoot\lib\$Arch\QnnHtpNetRunExtensions.dll, `
+        $QnnSdkRoot\lib\$Arch\QnnHtpPrepare.dll, `
+        $QnnSdkRoot\lib\$Arch\QnnHtp${DspArch}Stub.dll, `
+        $QnnSdkRoot\lib\$Arch\QnnHtp${DspArch}CalculatorStub.dll, `
+        $QnnSdkRoot\lib\$Arch\QnnSystem.dll, `
+        $QnnSdkRoot\lib\hexagon-$DspArch\unsigned\libCalculator_skel.so, `
+        $QnnSdkRoot\lib\hexagon-$DspArch\unsigned\libQnnHtp${DspArch}.cat, `
+        $QnnSdkRoot\lib\hexagon-$DspArch\unsigned\libQnnHtp${DspArch}Skel.so `
+}
+
+function Get-ModelFileNames {
+<#
+.SYNOPSIS
+    Get a list of a model's context binaries.
+#>
+    param (
+        [Parameter(Mandatory = $true)]
+        [string]$GenieConfigPath
+    )
+
+    $genieConfig = (Get-Content -Path $GenieConfigPath) | ConvertFrom-Json
+    return $genieConfig.dialog.engine.model.binary."ctx-bins"
+}
+
+function Test-ModelFiles {
+<#
+.SYNOPSIS
+    Test if all files required to run a model are available.
+#>
+    param (
+        [string]$GenieConfigTemplatePath,
+        [string]$BundleRoot,
+        [string]$TokenizerPath
+    )
+
+    if (-Not (Test-Path $GenieConfigTemplatePath)) {
+        Write-Error "Genie config template $GenieConfigTemplatePath does not exist."
+        return $false
+    }
+
+    $modelFileNames = Get-ModelFileNames -GenieConfigPath $GenieConfigTemplatePath
+    foreach ($modelFileName in $modelFileNames) {
+        $modelFilePath = Join-Path $BundleRoot $modelFileName
+        if (-Not (Test-Path $modelFilePath)) {
+            Write-Error "Model file $modelFilePath does not exist."
+            return $false
+        }
+    }
+
+    if (-Not (Test-Path $TokenizerPath)) {
+        Write-Error "Tokenizer not found in $TokenizerPath. See LLM on Genie tutorial for more info."
+        return $false
+    }
+
+    return $true
+}
+
+function Test-SdkDirectories {
+<#
+.SYNOPSIS
+    Check if the QNN and QAIHM directories are valid.
+#>
+
+    param(
+        [Parameter(Mandatory = $true)]
+        [string]$QnnSdkRoot,
+
+        [Parameter(Mandatory = $true)]
+        [string]$QaihmRoot
+    )
+
+    if (-Not $QnnSdkRoot) {
+        Write-Error "QNN SDK root has not been set. Set Env[QNN_SDK_ROOT] or pass -QnnSdkRoot argument."
+        return $false
+    }
+
+    if (-Not (Test-Path -Path $QnnSdkRoot -PathType Container)) {
+        Write-Error "QNN SDK root '$QnnSdkRoot' is not a directory."
+        return $false
+    }
+
+    if (-Not $QaihmRoot) {
+        Write-Error "ai-hub-models repo root has not been set. Set Env[QAIHM_ROOT] or pass -QaihmRoot argument."
+        return $false
+    }
+
+    if (-Not (Test-Path -Path $QaihmRoot -PathType Container)) {
+        Write-Error "ai-hub-models repo root '$QaihmRoot' is not a directory."
+        return $false
+    }
+
+    return $true
+}
+
+function ThrowIfLastFailed() {
+    param (
+        [string]$ErrorMessage = "Last command failed."
+    )
+    if (-Not $?) {
+        throw $ErrorMessage
+    }
+}
+
+function Update-ModelPaths {
+<#
+.SYNOPSIS
+    Prefix bare file references in the given genie.config text with our bundle root.
+#>
+    param (
+        [Parameter(Mandatory = $true)]
+        [string]$GenieConfigTemplatePath,
+
+        [Parameter(Mandatory = $true)]
+        [string]$BundleRoot,
+
+        [Parameter(Mandatory = $true)]
+        [string]$TokenizerPath,
+
+        [Parameter(ValueFromPipeline = $true, Mandatory = $true)]
+        [string]$GenieConfigLine
+    )
+
+    BEGIN {
+        $BundleRootEscaped = $BundleRoot.Replace("\", "\\")
+        $TokenizerPathEscaped = $TokenizerPath.Replace("\", "\\")
+        $ModelFileNames = Get-ModelFileNames -GenieConfigPath $GenieConfigTemplatePath
+    }
+
+    PROCESS {
+        foreach ($line in $GenieConfigLine) {
+            $line = $line.Replace("`"path`": `"tokenizer.json`"", "`"path`": `"$TokenizerPathEscaped`"")
+            $line = $line.Replace("`"extensions`": `"htp_backend_ext_config.json`"", "`"extensions`": `"$BundleRootEscaped\\htp_backend_ext_config.json`"")
+
+            foreach ($modelFile in $ModelFileNames) {
+                $line = $line.Replace("`"$modelFile`"", "`"$BundleRootEscaped\\$modelFile`"")
+            }
+
+            $line
+        }
+    }
+}
+
+function Write-Status {
+    param(
+        [Parameter(Mandatory = $true)]
+        [string]$Message
+    )
+
+    Write-Host -ForegroundColor Green $Message
+}

powershell/README.md

@@ -0,0 +1,41 @@
+# Genie PowerShell script for Windows
+
+This script simplifies the steps in the [LLM On-Device Deployment](..)
+tutorial. It is assumed that you have followed the tutorial and prepared a
+Genie bundle with the context binaries and tokenizer.
+
+From the parent directory (`llm_on_genie`), please run the command in a
+PowerShell terminal:
+
+```powershell
+.\powershell\RunLlm1.ps1
+```
+
+For more information, run:
+
+```powershell
+Get-Help .\powershell\RunLlm1.ps1
+```
+
+If it says that you do not have permission to run PowerShell scripts, you may
+need to enable this first with this command:
+
+```powershell
+Set-ExecutionPolicy -Scope CurrentUser Unrestricted -Force
+```
+
+## UTF-8 support
+
+If you want to use Unicode characters in your prompts, we recommend globally
+enabling UTF-8 on Windows:
+
+* Open Control Panel -> Region -> Administrative -> Change system locale...
+* Tick the "Beta: Use Unicode UTF-8 for worldwide language support"
+* Restart Windows
+
+If you want to change this only for a single PowerShell session, execute:
+
+```powershell
+chcp 65001
+[Console]::OutputEncoding = [System.Text.Encoding]::UTF8]
+```

powershell/RunLlm.ps1

@@ -0,0 +1,135 @@
+# ------------------------------------------------------------------------
+# Copyright (c) 2025 Qualcomm Innovation Center, Inc. All rights reserved.
+# SPDX-License-Identifier: BSD-3-Clause
+# ------------------------------------------------------------------------
+
+<#
+.SYNOPSIS
+    Run large language models exported by Qualcomm AI Hub.
+
+.DESCRIPTION
+    This script handles all of the tasks required to run supported LLMs on
+    a Windows on Snapdragon device. Use the ai-hub-models Python package to
+    export the model and provide the associated tokenizer.json. Next, this
+    will produce all necessary configuration files, set up QNN/Genie, and
+    run genie-t2t-run.exe.
+
+.EXAMPLE
+    # 1. Export the model
+    python -m qai_hub_models.models.llama_v3_2_3b_instruct.export --device "Snapdragon X Elite CRD" --skip-inferencing --skip-profiling --output-dir genie_bundle
+
+    # 2. Download tokenizer to genie_bundle\tokenizer.json (see ..\README.md for download links).
+
+    # 3. Run the model on this device.
+    & .\RunLlm.ps1 -ModelName llama_v3_2_3b_instruct -BundleRoot genie_bundle -RawPrompt "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nHow many dogs are there in space?<|eot_id|><|start_header_id|>assistant<|end_header_id|>"
+
+.LINK
+    https://github.com/quic/ai-hub-apps/tree/main/tutorials/llm_on_genie/
+
+.LINK
+    https://github.com/quic/ai-hub-models/
+#>
+param (
+    [Parameter(Mandatory = $true,
+               HelpMessage = "The name of the AI Hub model to run,`n  e.g., llama_v3_2_3b_instruct.")]
+    [string]$ModelName,
+
+    [Parameter(Mandatory = $true,
+               HelpMessage = "Path to the directory containing the model's context binaries.`n  e.g., genie_bundle")]
+    [string]$BundleRoot,
+
+    [Parameter(HelpMessage = "Path to the model's tokenizer.json. Default: `$BundleRoot\tokenizer.json")]
+    [string]$TokenizerPath,
+
+    [Parameter(Mandatory = $true,
+               HelpMessage = "A raw prompt string, including system prompt header. Please note that this is model-specific; details in ..\README.md.`n  e.g., <|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nWhat is France's capital?<|eot_id|><|start_header_id|>assistant<|end_header_id|>")]
+    [string]$RawPrompt,
+
+    [Parameter(HelpMessage = "Path to the QNN SDK.")]
+    [string]$QnnSdkRoot = $Env:QNN_SDK_ROOT,
+
+    [Parameter(HelpMessage = "Override path to the ai-hub-models repository.")]
+    [string]$QaihmRoot,
+
+    [Parameter(HelpMessage = "This device's soc_model. 60 = Snapdragon X Elite / X Plus.")]
+    [int]$SocModel = 60,
+
+    [Parameter(HelpMessage = "This device's dsp_arch. v73 = Snapdragon X Elite / X Plus and many others.")]
+    [string]$DspArch = "v73"
+)
+
+# Make utility functions available
+$ScriptRoot = (Resolve-Path -Path "$(Split-Path -Parent $MyInvocation.MyCommand.Definition)").Path
+. $ScriptRoot\LlmUtils.ps1
+
+if (-Not $TokenizerPath) {
+    $TokenizerPath = Join-Path $BundleRoot "tokenizer.json"
+}
+$TokenizerPath = (Resolve-Path -Path $TokenizerPath)
+
+if (-Not $QaihmRoot) {
+    $QaihmRoot = (Resolve-Path -Path $ScriptRoot\..\..\..)
+}
+
+# Turn $BundleRoot into an absolute path
+$BundleRoot = (Resolve-Path -Path $BundleRoot)
+
+# Location of the sample Genie configs in the ai-hub-models repo
+$GenieConfigsDir = "$QaihmRoot\tutorials\llm_on_genie\configs"
+$HtpConfigTemplatePath = "$GenieConfigsDir\htp\htp_backend_ext_config.json.template"
+$GenieConfigTemplatePath = "$GenieConfigsDir\genie\$ModelName.json"
+
+# Ensure the SDKs and other files are available
+if (-Not (Test-SdkDirectories -QnnSdkRoot $QnnSdkRoot -QaihmRoot $QaihmRoot)) {
+    exit 1
+}
+if (-Not (Test-ModelFiles -GenieConfigTemplatePath $GenieConfigTemplatePath -BundleRoot $BundleRoot -TokenizerPath $TokenizerPath)) {
+    exit 1
+}
+
+# Where the config files need to go
+$HtpConfigPath = "$BundleRoot/htp_backend_ext_config.json"
+$GenieConfigPath = "$BundleRoot/genie_config.json"
+
+# Substitute two placeholder strings in the HTP config.
+# Using -Encoding ascii in the Out-File invocation is required to avoid
+# cryptic JSON parsing errors.
+Write-Status "Creating $HtpConfigPath from template in $HtpConfigTemplatePath."
+(Get-Content -Path $HtpConfigTemplatePath) `
+    -replace "`"soc_model`": <TODO>", "`"soc_model`": $SocModel" `
+    -replace "`"dsp_arch`": `"<TODO>`"", "`"dsp_arch`": `"$DspArch`"" |
+    Out-File -Encoding ascii "$HtpConfigPath"
+ThrowIfLastFailed
+
+# Disable mmap on this Windows device and update some paths so we don't have to
+# change directories to run Genie.
+# As above, ascii encoding is required.
+Write-Status "Creating $GenieConfigPath from template in $GenieConfigTemplatePath."
+(Get-Content -Path "$GenieConfigTemplatePath") `
+    -replace "`"use-mmap`": true", "`"use-mmap`": false" ` |
+    Update-ModelPaths -GenieConfigTemplatePath $GenieConfigTemplatePath -BundleRoot $BundleRoot -TokenizerPath $TokenizerPath |
+    Out-File -Encoding ascii "$GenieConfigPath"
+ThrowIfLastFailed
+
+Write-Status "Setting ADSP_LIBRARY_PATH = `"$BundleRoot`"."
+$OldAdspLibraryPath = $Env:ADSP_LIBRARY_PATH
+$Env:ADSP_LIBRARY_PATH = $BundleRoot
+
+# I haven't been able to get Genie to run from $QnnSdkRoot so copy it into $BundleRoot and run from there.
+# Note that setting ADSP_LIBRARY_PATH and PATH _is_ sufficient for QNN (including the validator),
+# but _not_ for Genie on Windows.
+Write-Status "Copying Genie and $DspArch binaries to $BundleRoot."
+Copy-Genie -QnnSdkRoot $QnnSdkRoot -Destination $BundleRoot -DspArch $DspArch
+
+# Check if QNN thinks our host is set up properly.
+# Note that qnn-platform-validator always returns 0 so we can't easily bail on failure.
+# Its output is a malformed CSV so even that is hard to use :-/
+& $BundleRoot\qnn-platform-validator.exe --backend dsp --testBackend
+
+# Run Genie
+$GenieCmd = "$BundleRoot\genie-t2t-run.exe"
+$GenieArgs = @("-c", "$BundleRoot\genie_config.json", "-p", $RawPrompt)
+Write-Status "Running $GenieCmd $GenieArgs"
+& $GenieCmd $GenieArgs
+
+$Env:ADSP_LIBRARY_PATH = $OldAdspLibraryPath