Mike0021/Qwen3.8-27B-GGUF

Qwen3.8-27B GGUF

Source-faithful and architecture-aware GGUF conversions of Qwen/Qwen3.8-27B, including its native vision projector and one-layer MTP draft model.

This release was independently built from source revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 with llama.cpp revision 1692f9e50bb20fd96b963af38a282daf78feea64. All 18 source safetensors shards were verified against their Hub LFS SHA-256 digests before conversion.

Files

Main model

File	Size	Purpose
Qwen3.8-27B-Q4_K_M.gguf	18.97 GB	Recommended practical model. Conservative 5.64 BPW mixed quantization.
Qwen3.8-27B-Q8_0.gguf	28.60 GB	Near-lossless high-quality quantization.
Qwen3.8-27B-BF16.gguf	53.81 GB	Maximum-fidelity BF16 conversion.

The recommended Q4 file is deliberately not a stock Q4_K_M quant. Its FFN and embedding matrices use Q4_K, the final output and attention-output projections use Q6_K, and all other attention plus Gated DeltaNet/SSM matrices use Q8_0. This protects the model's architecture-sensitive paths while keeping the file under 19 GB.

Vision projector

Download one projector to use images or video. It is not needed for text-only inference.

File	Size	Purpose
mmproj-Qwen3.8-27B-Q8_0.gguf	0.63 GB	Recommended practical projector.
mmproj-Qwen3.8-27B-BF16.gguf	0.93 GB	Maximum-fidelity projector.

MTP speculative-decoding sidecar

These files are optional. They enable Qwen3.8's native one-layer multi-token-prediction draft model; they do not change the target model's final sampling distribution.

File	Size	Purpose
mtp-Qwen3.8-27B-Q4_0.gguf	1.68 GB	Smallest and usually best practical draft sidecar.
mtp-Qwen3.8-27B-Q8_0.gguf	3.16 GB	Higher-fidelity draft sidecar.
mtp-Qwen3.8-27B-BF16.gguf	5.95 GB	Maximum-fidelity draft sidecar.

Download

Recommended text + vision pair:

hf download Mike0021/Qwen3.8-27B-GGUF \
  Qwen3.8-27B-Q4_K_M.gguf \
  mmproj-Qwen3.8-27B-Q8_0.gguf \
  --local-dir ./Qwen3.8-27B-GGUF

Add mtp-Qwen3.8-27B-Q4_0.gguf to that command if you want speculative decoding.

llama.cpp usage

This new architecture requires llama.cpp revision 1692f9e or a tested newer revision. Other GGUF runtimes and older GUIs may not support Qwen3.8 yet.

Text chat with thinking disabled:

llama-cli \
  -m Qwen3.8-27B-Q4_K_M.gguf \
  -ngl all -c 32768 --jinja --conversation \
  --chat-template-kwargs '{"enable_thinking":false}'

Image understanding:

llama-cli \
  -m Qwen3.8-27B-Q4_K_M.gguf \
  --mmproj mmproj-Qwen3.8-27B-Q8_0.gguf \
  --image image.jpg \
  --image-min-tokens 1024 \
  -p "Describe this image precisely." \
  -ngl all -c 32768 -n 512 --jinja

MTP speculative decoding:

llama-cli \
  -m Qwen3.8-27B-Q4_K_M.gguf \
  -md mtp-Qwen3.8-27B-Q4_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -ngl all -ngld all -c 32768 --jinja --conversation \
  --chat-template-kwargs '{"enable_thinking":false}'

The model's native context window is 262,144 tokens. Start with a smaller runtime context such as 32K unless you need the full window, because KV and recurrent-state memory grow with the configured context and concurrency.

Thinking and sampling

The exact source Jinja template is embedded in every text and MTP GGUF. Thinking defaults to xhigh. Select another supported effort explicitly:

--chat-template-kwargs \
  '{"enable_thinking":true,"reasoning_effort":"low","preserve_thinking":true}'

Supported effort values are low, medium, and xhigh. Use {"enable_thinking":false} to disable thinking. At the pinned llama.cpp revision, pass the effort through chat_template_kwargs; top-level OpenAI API reasoning_effort forwarding is still being completed in llama.cpp PR #26941.

Qwen's source model card recommends these starting points:

Mode	Temperature	Top-p	Top-k	Presence penalty
Thinking	1.0	0.95	20	0.0
Non-thinking	0.7	0.8	20	1.5

Disabling thinking does not automatically change the sampler settings.

Validation and provenance

The release passed the following checks before upload:

• GGUF v3 metadata: qwen35, 64 target layers, 262,144 context, 248,320-token vocabulary.
• Exact source chat-template comparison.
• Expected tensor counts: 851 text, 334 projector, and 18 MTP tensors.
• Expected type distributions for every BF16, Q8, and Q4 artifact.
• Full SHA-256 checksums (see SHA256SUMS).
• CUDA tensor checks and deterministic text-generation smoke tests.
• Native image/projector and MTP speculative-decoding smoke tests.

The exact conversion commands, tool revisions, tensor policy, and validation details are in CONVERSION.md. Raw conversion and quantization logs are included under logs/.

Why no importance-matrix Q4?

llama.cpp supports importance-matrix quantization, but the architecture was new at conversion time and no representative Qwen3.8-specific calibration plus held-out evaluation was available. An unvalidated calibration corpus could bias the remaining Q4 FFNs. This release therefore follows ggml-org's conservative, architecture-aware reference recipe, which already keeps the attention and DeltaNet/SSM paths at Q8. Any future imatrix build should be published as a separate variant with its calibration provenance and held-out KLD/task results.

License

The original model and these converted weights are distributed under the Apache License 2.0. See the Qwen/Qwen3.8-27B model card for the upstream model's documentation and intended-use guidance.