cleyesode/Qwen3.8-Flash-Next-RAZOR-96B-A6B-NVFP4

Model Card

Source Model: Nickyang/Qwen3.8-Flash-Next-RAZOR-96B-A6B-E384of512

THIS REPO:

Excellent 25% expert-prune by Nickyang utilizing new RAZOR methodology. cleyesode further reduced memory requirements via NVFP4 quantization for use on Blackwell sm120 gpus, FP8 PLE, BF16 MTP.

For even smaller memory footprints, check out:

cleyesode/Qwen3.8-Flash-Next-RAZOR-65B-A6B-NVFP4 50% expert-prune quantized to NVFP4, FP8 PLE.

cleyesode/Qwen3.8-Flash-Next-RAZOR-65B-A6B-NVFP4-PLE-MTP-FP8 50% expert-prune quantized to NVFP4, FP8 PLE & MTP.

Ready to serve on latest vLLM branch.

Quantization via NVIDIA/Model-Optimizer custom qwen4exp workflow:

Component	Precision
Routed experts	NVFP4 W4A4 (group 16, FP8 block scales, FP32 global scales)
Shared expert, router, attention, QSA indexer, GatedDeltaNet, hyper-connections, embeddings, lm_head, vision, MTP	BF16
PLE n-gram table	FP8 E4M3, one scale per table
KV cache	Not quantized
Please see post-quantization verification_report for details.

PLE isolated, model sharded for reduced memory pressure.

Calibrated on curated samples from Nickyang/RazorCal:

Domain	Count	Percentage	Description / Sub-sources
Coding	57	~24.1%	Algorithmic Python/C++, competitive programming, and GitHub SWE issue resolution.
Math	41	~17.3%	High-level math derivations, AIME-style problems, calculus, and geometry.
Chinese-STEM	33	~13.9%	Multilingual math, coding, and science MCQs written in Chinese.
Instruction Following	32	~13.5%	Strict formatting constraints (e.g., word count, excluded words, required letters).
World Knowledge	30	~12.7%	Trivia, history, reading comprehension, and translation tasks.
Tool Calling	25	~10.5%	Multi-turn agentic workflows using external APIs (weather, stocks, ecommerce).
Science (STEM)	19	~8.0%	Chemistry, physics, biology, and thermodynamics (both MCQs and long-form).

This model demonstrates very impressive retention of pre-prune / pre-quantize abilities - benchmarks in-progress.

MTP is present but untested here, updates will be posted as they become available. See other model listings above for MTP metrics.

Serving with vLLM

Example: 8 x 5060Ti 16GB | tensor parallel 8 with expert parallelism | single sequence at full 262k context

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export VLLM_USE_V2_MODEL_RUNNER=1
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False

vllm serve cleyesode/Qwen3.8-Flash-Next-RAZOR-96B-A6B-NVFP4 \
    --served-model-name Qwen3.8-Flash-Next-RAZOR-96B-A6B-NVFP4 \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --gpu-memory-utilization .90 \
    --max-model-len 262144 \
    --max-num-seqs 1 \
    --max-num-batched-tokens 2048 \
    --kv-cache-dtype fp8 \
    --kv-offloading-size 8 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml \
    --no-enable-flashinfer-autotune \
    --engram-config '{\"cpu_offload\":\"true\"}' \
    --trust-remote-code

---

Source Repo Model Card:

Qwen3.8-Flash-Next-RAZOR-96B-A6B-E384of512

Qwen3.8-Flash-Next with 25% of its routed experts removed using RAZOR. Each main MoE layer retains 384 of 512 experts. The separate MTP expert bank is also reduced to 384 experts, using the unobserved-layer policy described below.

No gradient updates, recovery training or new quantization were applied. The surviving expert weights and router rows are slices of the base checkpoint; non-target tensor bytes are unchanged.

Property	Base checkpoint	This checkpoint
Routed experts per MoE layer	512	384
Routed experts removed	0%	25%
Main language model, excluding n-gram tables, MTP and vision	125,743,653,760 (~125.744B)	95,528,936,320 (~95.529B)
N-gram embedding tables, stored separately	51,200,245,760 (~51.200B)	51,200,245,760 (~51.200B), unchanged
Distinct stored MTP parameters	2,607,150,848 (~2.607B)	1,977,677,568 (~1.978B)
Vision encoder	448,931,056 (~0.449B)	448,931,056 (~0.449B), unchanged
Complete stored floating-point parameters	179,999,981,424 (~180.000B)	149,155,790,704 (~149.156B)
Activated parameters, using the base model's reported convention	~6B	~6B
Routed experts per token (top-k)	10	10, unchanged
Shared experts per main MoE layer	1	1, unchanged
Main MoE decoder layers / stored MTP MoE layers	48 / 1	48 / 1
Weight dtype	BF16	BF16

Naming scope: 96B rounds the main language-model count, including token embeddings and the LM head but excluding n-gram lookup tables, MTP and the vision encoder. The n-gram weights are still included in the download: they have not been deleted. HF's automatic parameter badge counts the full stored checkpoint and may therefore show approximately 149B, rather than the 96B used in the name. The 35 integer buffer elements are not learned parameters.

The official card separates its headline 125B language model with 6B activated from 51B n-gram and 4B MTP. The table above follows that separation but counts distinct stored tensors exactly. In particular, the official revision contains 2.607B distinct MTP tensor elements; the approximate 4B headline is not substituted for that storage count. A6B retains the official activated-parameter convention and is not a new throughput or FLOP measurement.

The other released budget is Qwen3.8-Flash-Next-RAZOR-65B-A6B-E256of512.

Selection and calibration

• Main layers: rank experts by the RMS of the local output change when a selected expert is removed and the router promotes its highest-ranked unselected expert. Keep the highest-scoring 384 experts per layer.
• Aggregation: RMS conditional on the calibration tokens routed to each expert, not a corpus sum and not an arithmetic mean.
• Calibration evidence: 2,151,120 valid tokens packed into 66 rows of up to 32,768 tokens, covering all 48 main MoE layers. Existing additive statistics were reused to recompute the Refill-RMS selection; this was not a fresh collection on the currently released RazorCal.
• Corpus: the historical MoECal_v2.5 experiment corpus. Do not interpret this release as having been calibrated on the current public RazorCal. calibration_info.json records the statistics-pack digest and calibration geometry. No calibration examples are included here.
• MTP: the ordinary text calibration does not execute mtp.layers.0. Its experts were selected by the L2 norm of their original router rows (mtp_policy=router_norm), an explicit heuristic rather than measured RAZOR saliency. Its keep-set is named mtp_0.

This release is a newly exported Refill-RMS checkpoint, not the older CDMp2/LOO checkpoint with a changed label. The 25% main-layer selection differs from that older experiment by 309 retained expert choices across the 48 layers.

The base revision is de4b8e4d43b917e7706784d8bb445c9af86a3540. kept_expert_indices.json gives original expert indices in output order. pruning_info.json records method=razor, aggregation=rms, the full keep-set and the MTP policy. These artifacts allow the exact slicing operation to be inspected independently of the calibration data.

Native architecture and loading

The official architecture is qwen4_exp / Qwen4ExpForConditionalGeneration, not qwen3_next. Keep the original configuration: substituting an older Qwen architecture is not supported. The conditional model preserves the vision branch, hyper-connections, PLE/n-gram embeddings, linear attention and QSA.

The validation environment used Transformers 5.16.1 and BF16/FP32 native kernels. Transformers 5.8.1 does not contain this model implementation. Use AutoModelForImageTextToText for the conditional checkpoint rather than allowing an automatic causal-LM loader to drop the vision branch. The original tokenizer, chat template, generation defaults and image/video preprocessing configurations are included.

Approximately 298.31 GB are required for tensor payloads alone in BF16, before activations, caches, conversion buffers and runtime overhead. In particular, the large n-gram embedding affects device placement. This release does not prescribe a verified full-size serving topology. Optional FLA/causal-convolution kernels may require CUDA even when the model code has a PyTorch fallback; use a compatible native environment.

The RAZOR codebase describes the pruning method. Verification of this release used a Qwen3.8-capable development checkout; a published package or earlier Git revision without qwen4_exp support cannot verify it merely by accepting its file paths.

Verification and limitations

Completed checks for this checkpoint:

• All 1,658 stored tensors were checked against the source and keep-set: all surviving expert rows and router rows in matching order, all non-target tensor bytes, the configuration, and the complete keep manifest.
• License bytes and tokenizer/template assets were checked; rewritten JSON auxiliary files were compared by their complete parsed contents, not formatting.
• A real-pretrained-weight comparison of the first native decoder layer on two short text batches matched an independently sliced source reference with zero difference.
• Separate tiny-native-model regressions covered FP32/BF16 routing and refill, PLE/ngram, hyper-connections, QSA, streaming windows, query chunking and export/reload logits. Those tests are not full-size benchmarks.

verification_report.json records storage and auxiliary checks. release_info.json records the public naming, parameter scope and additional verification boundaries.

Not established: full-size end-to-end log-probability equivalence, benchmark retention, multimodal task quality, long-context generation quality or MTP speculative-decoding behavior. The prefix check is not a whole-model forward check. Expert pruning is lossy; successful slicing verification does not guarantee accuracy, robustness, termination behavior or performance without regression. Evaluate on the intended workload before deployment.

License and attribution

The weights are a derivative of Qwen3.8-Flash-Next and remain subject to Qwen Community License 1.0, included verbatim as LICENSE. This is not Apache-2.0. The license includes conditions for large commercial services and separate-license requirements for certain commercial Model-as-a-Service and AI Work Assistant uses; consult the complete upstream terms rather than assuming unrestricted commercial hosting.

RAZOR contributes expert selection and slicing, not newly trained weights. Cite the RAZOR paper for the pruning method and the official Qwen3.8-Flash-Next model card for the base model.