--- license: mit language: - zh - en base_model: - deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B pipeline_tag: text-generation library_name: transformers tags: - Context - DeepSeek-R1-Distill-Qwen-1.5B --- # DeepSeek-R1-Distill-Qwen-1.5B This version of DeepSeek-R1-Distill-Qwen-1.5B has been converted to run on the Axera NPU using **w8a16** and **w4a16** quantization. This model has been optimized with the following LoRA: Compatible with Pulsar2 version: 4.2 ## Feature - Support for longer contexts, in this sample it's 2k - Support context dialogue - System prompt kvcache is supported ## Convert tools links: For those who are interested in model conversion, you can try to export axmodel through the original repo : https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B and https://huggingface.co/jakiAJK/DeepSeek-R1-Distill-Qwen-1.5B_GPTQ-int4 [Pulsar2 Link, How to Convert LLM from Huggingface to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/appendix/build_llm.html) [AXera NPU AXEngine LLM Runtime](https://github.com/AXERA-TECH/ax-llm/tree/ax-context) [AXera NPU AXCL LLM Runtime](https://github.com/AXERA-TECH/ax-llm/tree/axcl-context) ### Convert script The follow show how to convert DeepSeek-R1-Distill-Qwen-1.5B ``` pulsar2 llm_build --input_path deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \ --output_path deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B-ax650 \ --hidden_state_type bf16 --kv_cache_len 2047 --prefill_len 128 \ --last_kv_cache_len 128 \ --last_kv_cache_len 256 \ --last_kv_cache_len 384 \ --last_kv_cache_len 512 \ --last_kv_cache_len 640 \ --last_kv_cache_len 768 \ --last_kv_cache_len 896 \ --last_kv_cache_len 1024 \ --last_kv_cache_len 1152 \ --last_kv_cache_len 1280 \ --last_kv_cache_len 1408 \ --last_cache_len 1536 \ --chip AX650 -c 1 --parallel 8 ``` ## Support Platform - AX650 - AX650N DEMO Board - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html) - [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html) - AX630C - *TBD* |Chips|w8a16|w4a16| DDR | Flash | |--|--|--|--|--| |AX650| 12 tokens/sec| 17 tokens/sec | 2.3GB | 2.3GB | ## How to use Download all files from this repository to the device ``` root@ax650:/mnt/qtang/llm-test/deepseek-r1-1.5b-ctx# tree -L 1 . |-- README.md |-- config.json |-- deepseek-r1-1.5b-ax650 |-- deepseek-r1-1.5b-int4-ax650 |-- deepseek-r1_tokenizer |-- deepseek-r1_tokenizer.py |-- figures |-- main_ax650 |-- main_axcl_aarch64 |-- main_axcl_x86 |-- post_config.json |-- run_deepseek-r1_1.5B_ax650.sh |-- run_deepseek-r1_1.5B_int4_ax650.sh |-- run_deepseek-r1_1.5b_axcl_aarch64.sh |-- run_deepseek-r1_1.5b_axcl_x86.sh |-- run_deepseek-r1_1.5b_int4_axcl_aarch64.sh `-- run_deepseek-r1_1.5b_int4_axcl_x86.sh 4 directories, 16 files ``` #### Start the Tokenizer service ``` root@ax650:/mnt/qtang/llm-test/deepseek-r1-1.5b-ctx# python3 deepseek-r1_tokenizer_uid.py Server running at http://127.0.0.1:12345 ``` #### System prompt cache - The System prompt can be preset through the configuration file from `--system_prompt` - The System prompt can be cached in the form of kv cache to a specified folder for quick loading at the next run time from `--kvcache_path` - This folder needs to be created manually before running, for example `mkdir kvcache` ``` root@ax650:/mnt/qtang/llm-test/deepseek-r1-1.5b-ctx# cat run_deepseek-r1_1.5b_ax650.sh ./main_ax650 \ --template_filename_axmodel "deepseek-r1-1.5b-ax650/qwen2_p128_l%d_together.axmodel" \ --axmodel_num 28 \ --url_tokenizer_model "http://127.0.0.1:12345" \ --filename_post_axmodel "deepseek-r1-1.5b-ax650/qwen2_post.axmodel" \ --filename_tokens_embed "deepseek-r1-1.5b-ax650/model.embed_tokens.weight.bfloat16.bin" \ --tokens_embed_num 151936 \ --tokens_embed_size 1536 \ --use_mmap_load_embed 1 \ --live_print 1 ``` #### Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board Open another terminal and run `run_deepseek-r1_1.5b_ax650.sh` ``` root@ax650:/mnt/qtang/llm-test/deepseek-r1-1.5b-ctx# ./run_deepseek-r1_1.5b_ax650.sh [I][ Init][ 110]: LLM init start [I][ Init][ 34]: connect http://127.0.0.1:12345 ok [I][ Init][ 57]: uid: 96cbe293-ff24-4011-b3a1-b5ab95234990 bos_id: 151646, eos_id: 151643 3% | ██ | 1 / 31 [2.07s<64.23s, 0.48 count/s] tokenizer init ok[I][ Init][ 26]: LLaMaEmbedSelector use mmap 100% | ████████████████████████████████ | 31 / 31 [8.76s<8.76s, 3.54 count/s] init post axmodel ok,remain_cmm(8114 MB)[I][ Init][ 188]: max_token_len : 2047 [I][ Init][ 193]: kv_cache_size : 256, kv_cache_num: 2047 [I][ Init][ 201]: prefill_token_num : 128 [I][ Init][ 205]: grp: 1, prefill_max_token_num : 1 [I][ Init][ 205]: grp: 2, prefill_max_token_num : 128 [I][ Init][ 205]: grp: 3, prefill_max_token_num : 256 [I][ Init][ 205]: grp: 4, prefill_max_token_num : 384 [I][ Init][ 205]: grp: 5, prefill_max_token_num : 512 [I][ Init][ 205]: grp: 6, prefill_max_token_num : 640 [I][ Init][ 205]: grp: 7, prefill_max_token_num : 768 [I][ Init][ 205]: grp: 8, prefill_max_token_num : 896 [I][ Init][ 205]: grp: 9, prefill_max_token_num : 1024 [I][ Init][ 209]: prefill_max_token_num : 1024 [I][ load_config][ 282]: load config: { "enable_repetition_penalty": false, "enable_temperature": true, "enable_top_k_sampling": true, "enable_top_p_sampling": false, "penalty_window": 20, "repetition_penalty": 1.2, "temperature": 0.9, "top_k": 10, "top_p": 0.8 } [I][ Init][ 218]: LLM init ok Type "q" to exit, Ctrl+c to stop current running [I][ GenerateKVCachePrefill][ 275]: input token num : 11, prefill_split_num : 1 prefill_grpid : 2 [I][ GenerateKVCachePrefill][ 315]: input_num_token:11 [I][ main][ 228]: precompute_len: 11 [I][ main][ 229]: system_prompt: prompt >> 你是谁 [I][ SetKVCache][ 529]: prefill_grpid:2 kv_cache_num:128 precompute_len:11 input_num_token:6 [I][ SetKVCache][ 532]: current prefill_max_token_num:896 [I][ Run][ 658]: input token num : 6, prefill_split_num : 1 [I][ Run][ 684]: input_num_token:6 [I][ Run][ 807]: ttft: 256.87 ms 您好！我是由中国的深度求索（DeepSeek）公司开发的智能助手DeepSeek-R1。如您有任何任何问题，我会尽我所能为您提供帮助。 您好！我是由中国的深度求索（DeepSeek）公司开发的智能助手DeepSeek-R1。如您有任何任何问题，我会尽我所能为您提供帮助。 [N][ Run][ 921]: hit eos,avg 17.68 token/s [I][ GetKVCache][ 498]: precompute_len:91, remaining:933 ```