---
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
- vLLM
- AWQ
base_model:
- deepseek-ai/DeepSeek-V3.2
base_model_relation: quantized
---
# DeepSeek-V3.2-AWQ
Base model: [deepseek-ai/DeepSeek-V3.2](https://www.modelscope.cn/models/deepseek-ai/DeepSeek-V3.2)
```
Note:
1. Tested on Hopper device, we don't know if
ada / ampere devices could run this repo yet.
2. Waiting for official chat_template.jinja;
The file in this repo is borrowed from v3.1
with thinking mode turned off by default.
To enable thinking mode, include:
extra_body = {"chat_template_kwargs": {"thinking": True}}
in the post requests.
```
### 【Dependencies / Installation】
As of **2025-12-02**, make sure your system has cuda12.8 installed.
Then, create a fresh Python environment (e.g. python3.12 venv) and run:
```bash
# install vllm
pip install vllm==0.11.2
# install deep_gemm
git clone https://github.com/deepseek-ai/DeepGEMM.git
cd DeepGEMM/third-party
git clone https://github.com/NVIDIA/cutlass.git
git clone https://github.com/fmtlib/fmt.git
cd ../
git checkout v2.1.1.post3
pip install . --no-build-isolation
```
or
```
uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly
uv pip install git+https://github.com/deepseek-ai/DeepGEMM.git@v2.1.1.post3 --no-build-isolation # Other versions may also work. We recommend using the latest released version from https://github.com/deepseek-ai/DeepGEMM/releases
```
see [Official vLLM Deepseek-V3.2 Guide](https://docs.vllm.ai/projects/recipes/en/latest/DeepSeek/DeepSeek-V3_2-Exp.html)
### 【vLLM Startup Command】
Note: It could take a little while to load, if `--enable-expert-parallel` is enabled;
```
export VLLM_USE_DEEP_GEMM=0 # ATM, this line is a "must" for Hopper devices
export TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1
export VLLM_USE_FLASHINFER_MOE_FP16=1
export VLLM_USE_FLASHINFER_SAMPLER=0
export OMP_NUM_THREADS=4
CONTEXT_LENGTH=32768
vllm serve \
__YOUR_PATH__/QuantTrio/DeepSeek-V3.2-AWQ \
--served-model-name MY_MODEL_NAME \
--enable-auto-tool-choice \
--tool-call-parser deepseek_v31 \
--reasoning-parser deepseek_v3 \
--swap-space 16 \
--max-num-seqs 32 \
--max-model-len $CONTEXT_LENGTH \
--gpu-memory-utilization 0.9 \
--tensor-parallel-size 8 \
--enable-expert-parallel \ # optional
--speculative-config '{"model": "__YOUR_PATH__/QuantTrio/DeepSeek-V3.2-AWQ", "num_speculative_tokens": 1}' \ # optional, 50%+- throughput increase is observed
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
```
### 【Logs】
```
2025-12-02
1. Initial commit
```
### 【Model Files】
| File Size | Last Updated |
|-----------|--------------|
| `338 GiB` | `2025-12-02` |
### 【Model Download】
```python
from huggingface_hub import snapshot_download
snapshot_download('QuantTrio/DeepSeek-V3.2-AWQ', cache_dir="your_local_path")
```
### 【Overview】
# DeepSeek-V3.2: Efficient Reasoning & Agentic AI
Technical Report👁️
## Introduction
We introduce **DeepSeek-V3.2**, a model that harmonizes high computational efficiency with superior reasoning and agent performance. Our approach is built upon three key technical breakthroughs:
1. **DeepSeek Sparse Attention (DSA):** We introduce DSA, an efficient attention mechanism that substantially reduces computational complexity while preserving model performance, specifically optimized for long-context scenarios.
2. **Scalable Reinforcement Learning Framework:** By implementing a robust RL protocol and scaling post-training compute, *DeepSeek-V3.2* performs comparably to GPT-5. Notably, our high-compute variant, **DeepSeek-V3.2-Speciale**, **surpasses GPT-5** and exhibits reasoning proficiency on par with Gemini-3.0-Pro.
- *Achievement:* 🥇 **Gold-medal performance** in the 2025 International Mathematical Olympiad (IMO) and International Olympiad in Informatics (IOI).
3. **Large-Scale Agentic Task Synthesis Pipeline:** To integrate **reasoning into tool-use** scenarios, we developed a novel synthesis pipeline that systematically generates training data at scale. This facilitates scalable agentic post-training, improving compliance and generalization in complex interactive environments.
We have also released the final submissions for IOI 2025, ICPC World Finals, IMO 2025 and CMO 2025, which were selected based on our designed pipeline. These materials are provided for the community to conduct secondary verification. The files can be accessed at `assets/olympiad_cases`.
## Chat Template
DeepSeek-V3.2 introduces significant updates to its chat template compared to prior versions. The primary changes involve a revised format for tool calling and the introduction of a "thinking with tools" capability.
To assist the community in understanding and adapting to this new template, we have provided a dedicated `encoding` folder, which contains Python scripts and test cases demonstrating how to encode messages in OpenAI-compatible format into input strings for the model and how to parse the model's text output.
A brief example is illustrated below:
```python
import transformers
# encoding/encoding_dsv32.py
from encoding_dsv32 import encode_messages, parse_message_from_completion_text
tokenizer = transformers.AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V3.2")
messages = [
{"role": "user", "content": "hello"},
{"role": "assistant", "content": "Hello! I am DeepSeek.", "reasoning_content": "thinking..."},
{"role": "user", "content": "1+1=?"}
]
encode_config = dict(thinking_mode="thinking", drop_thinking=True, add_default_bos_token=True)
# messages -> string
prompt = encode_messages(messages, **encode_config)
# Output: "<｜begin▁of▁sentence｜><｜User｜>hello<｜Assistant｜>Hello! I am DeepSeek.<｜end▁of▁sentence｜><｜User｜>1+1=?<｜Assistant｜>"
# string -> tokens
tokens = tokenizer.encode(prompt)
# Output: [0, 128803, 33310, 128804, 128799, 19923, 3, 342, 1030, 22651, 4374, 1465, 16, 1, 128803, 19, 13, 19, 127252, 128804, 128798]
```
Important Notes:
1. This release does not include a Jinja-format chat template. Please refer to the Python code mentioned above.
2. The output parsing function included in the code is designed to handle well-formatted strings only. It does not attempt to correct or recover from malformed output that the model might occasionally generate. It is not suitable for production use without robust error handling.
3. A new role named `developer` has been introduced in the chat template. This role is dedicated exclusively to search agent scenarios and is designated for no other tasks. The official API does not accept messages assigned to `developer`.
## How to Run Locally
The model structure of DeepSeek-V3.2 and DeepSeek-V3.2-Speciale are the same as DeepSeek-V3.2-Exp. Please visit [DeepSeek-V3.2-Exp](https://github.com/deepseek-ai/DeepSeek-V3.2-Exp) repo for more information about running this model locally.
Usage Recommendations:
1. For local deployment, we recommend setting the sampling parameters to `temperature = 1.0, top_p = 0.95`.
2. Please note that the DeepSeek-V3.2-Speciale variant is designed exclusively for deep reasoning tasks and does not support the tool-calling functionality.
## License
This repository and the model weights are licensed under the [MIT License](LICENSE).
## Citation
```
@misc{deepseekai2025deepseekv32,
title={DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models},
author={DeepSeek-AI},
year={2025},
}
```
## Contact
If you have any questions, please raise an issue or contact us at [service@deepseek.com](service@deepseek.com).
