Qwen3.8-27B-Pi-W4A8

W4A8 (int4 weights, dynamic int8 activations, group 128) compressed-tensors quantization of bytkim/Qwen3.8-27B-pi for fast text serving on Intel Arc GPUs. Follows the base model license (Apache-2.0).

Recipe

Setting	Value
Tool	Intel auto-round 0.15.1, auto_round:llm_compressor (CT pack-quantized) export
Weights	int4 symmetric, group 128
Activations	dynamic int8 (act_bits 8, per-token)
Calibration	nvidia/OpenCodeInstruct (128 samples x 2048 tokens)
torch.compile	explicitly disabled (Triton inductor failure on XPU)
Kept in BF16	lm_head, visual tower (model.visual.* ignored)
Quant time	~6.5 h on one Intel Arc Pro B70 (GPU1)

Speed (measured, not estimated)

Single-stream decode, 128 streamed tokens requested (76 generated, natural stop; temperature 0, prompt: "Explain why the sky is blue in one short paragraph"), OpenAI-compatible /v1/chat/completions streaming, tokens counted after time-to-first-token:

Checkpoint	Decode	TTFT	Backend
This repo (Pi W4A8-dynamic)	33.0 t/s (3/3 runs)	0.08 s	dynamic act-quant path
This repo + DFlash draft (--spec-method dflash, block 8)	~55 t/s (54.6–56.4, 3 runs)	0.09 s warm	speculative decode, acceptance 2.71/8
LeoHyperlink/Sage-1.5-W4A16 (same box and protocol, W4A16 reference)	104.3 t/s	0.06 s	XPUwNa16LinearKernel (fused)

Dynamic activation quantization costs ~3x vs the fused W4A16 path on this XPU stack. Coherent output on all runs, byte-identical between plain and speculative decode (temperature 0).

Speculative decoding (DFlash)

The 1.9B DFlash draft (block-diffusion drafter for Qwen3.8) was served alongside the quant on a single B70 (--spec-model incoai/Qwen3.8-27B-DFlash2 --spec-tokens 8, --per-request-spec-decode-metrics summary): ~55 t/s, a 1.7x speedup over plain decode, same prompt and protocol. Acceptance: mean length 2.71 per 8-token block, draft rate 21.4% (48/224 tokens over 28 steps) — modest because the draft was trained for base Qwen3.8 while the target is the pi-quant; a matched draft would do better. First-request TTFT was 1.33s cold (draft warmup), then 0.09s warm.

Fit note: target + draft weights (20.3 GB) + graphs (4.9 GB) leave almost no room for KV cache on one 32 GB card — the spec serve needed --max-model-len 1024 --max-num-seqs 16 --gpu-memory-utilization 0.95 (KV need 1.68 GB vs 1.64 GB available at len 1024 / seqs 64; halving seqs closed the gap). Batch-1 decode speed is unaffected by these reservations. Tensor-parallel (TP=2) was attempted to relieve this but is blocked on this box: oneCCL cannot initialize (mem_to_ipc_handle: invalid_fd, all IPC-exchange modes fail).

Benchmarks

Humanity's Last Exam (HLE): 12.14%

Full text-only set (2,158/2,158 samples), paper-faithful config via inspect_evals/hle (CAIS team's original.yaml-equivalent: hle/json_judge scorer, single attempt, no multi-modal):

Metric	Value
Original accuracy	12.14% (262/2158; 3 samples unscored)
Stderr	0.70%
Run wall time	4 d 9 h
Tokens	57.6 M total — 54.3 M reasoning (≈25 k reasoning tok/sample, thinking xhigh)

Setup:

• Solver = this quant (thinking default/xhigh, temperature 0, max_tokens 32768, streaming forced), served via a vLLM TP=2 OpenAI-compatible endpoint.
• Grader = the same quant answering a JSON-judge prompt (max_grader_attempts 1).

Caveats:

• Self-graded: no external judge (e.g. o3-mini) was used — the quant judges its own answers. Plan-level judge disagreement adds unknown bias, plausibly downwards.
• Text-only, no tools: the base card's HLE 55.1 is with tools; this run is single-shot text-only. The two numbers must not be read against each other as a quantization recovery claim.
• Raw per-sample transcripts and scores: benchmarks/hle-full.eval (inspect_ai .eval archive).

SciCode: 4.6% test (3/65), 33.3% val (5/15)

Upstream inspect_ai harness from the SciCode repo (github.com/scicode-bench/SciCode), no background prompts, temperature 0, streaming, test_data.h5 numeric grading:

Split	Main-problem solve	Substep correctness	Tokens
test (65 problems)	4.6% (3/65)	30.2% (88/291)	9.0 M
validation (15 problems)	33.3% (5/15)	48.0% (24/50)	0.84 M

Caveats:

• Run background-free (zero-shot, no doc-comment context in the prompt) — a strictly harder setting than the paper's default with-background mode; the val:test gap (33% → 4.6%) reflects that the test split is novel research problems rather than routine physics code.
• Several substeps (≈12) ended in "fail to extract python code" — the model's output did not yield a parseable script under the harness's extraction rules, scoring 0 regardless of the underlying reasoning quality.
• Raw logs: benchmarks/scicode-test.eval, benchmarks/scicode-val.eval.

Tested hardware / software

• GPU: Intel Arc Pro B70 32 GB (Battlemage Xe2), single GPU
• Serving stack: vLLM-XPU 0.32.0 fork, torch 2.x +XPU, oneCCL
• Serve flags: --trust-remote-code --max-model-len 8192 --max-num-seqs 128 --gpu-memory-utilization 0.9 --language-model-only --kv-cache-dtype fp8 --enable-prefix-caching plus --default-chat-template-kwargs '{"enable_thinking": false}'. Note: this is a Mamba-hybrid model — only 159 Mamba cache blocks fit at 0.9 utilization, so max_num_seqs must stay at or below 159 (default 256 fails engine init). ZE_AFFINITY_MASK=0.0 crashes engine init on this build; GPU pinned via ONEAPI_DEVICE_SELECTOR=level_zero:N.

Usage

pip install "vllm>=0.32"   # XPU fork (vllm-xpu) on Intel GPUs
vllm serve LeoHyperlink/Qwen3.8-27B-Pi-W4A8 --trust-remote-code \
  --max-model-len 8192 --max-num-seqs 128 --gpu-memory-utilization 0.9 --language-model-only

Files

• model-0000[1-4]-of-00004.safetensors (~18.5 GB) + model.safetensors.index.json — quantized weights (MTP heads mtp.* included in the main checkpoint)
• model_extra_tensors.safetensors (849 MB) — raw passthrough tensors, kept for faithfulness. vLLM 0.32.0 cannot load them alongside the CT main checkpoint; text-only serving does not need them.
• quantization_config.json — compressed-tensors pack-quantized spec
• config.json, tokenizer files, preprocessor/processor configs, chat_template.jinja, generation_config.json
• benchmarks/hle-full.eval, benchmarks/scicode-test.eval, benchmarks/scicode-val.eval — raw inspect_ai eval logs (full per-sample transcripts + scores)
• .eval_results/hle.yaml — machine-readable HLE score card (Hub eval-results schema; SciCode's dataset is README-only as it is not a registered benchmark)

Limitations

• HLE 12.14% is text-only and self-graded (the quant judges itself, no o3-mini judge); the base card's HLE 55.1 is a with-tools agentic setting — see Benchmarks caveats, they are not comparable.
• SciCode 4.6% is a no-background zero-shot run; with-background and tool-using settings score materially higher.
• Not yet run: SWE-bench Pro and Terminal-Bench 2.1 (harness setup pending).
• MTP speculative decoding is NOT enabled by this checkpoint (MTP weights ship in the main checkpoint but are unused; --spec-method qwen3_5_mtp exists in the serving build and is untested). Speculative decoding WAS validated via the DFlash draft path instead (see above).
• Tensor-parallel serving is blocked on this box (oneCCL IPC init failure); all serves above are single-GPU.
• The visual tower ships unquantized and was not exercised (--language-model-only serving).

Citation

No new training was done for this quant — if you use it, cite the base model (bytkim/Qwen3.8-27B-pi) and link back to this repo for the quantization recipe and serving numbers.