RedHatAI/gemma-4-31B-it-speculator.dflash2

Model Overview

Model Architecture: Gemma4ForConditionalGeneration
- Input: Text
- Output: Text
Model Optimizations:
- Speculative Decoding Algorithm: DFlash2
- Draft Model: 5-layer qwen3 backbone (DFlash2DraftModel)
Release Date: 2026-10-14
Version: 1.0
Model Developers: RedHatAI

A DFlash2 speculative-decoding draft model for google/gemma-4-31B-it, trained with Speculators.

Model Specifications


Base Model	google/gemma-4-31B-it — use the `/chat/completions` endpoint with the verifier's chat template
Chat Template	Inherited from `google/gemma-4-31B-it`
Format	Safetensors
License	Apache-2.0 (the Gemma-4 verifier is subject to its own license and terms)
Validation Hardware	Not tracked

Training Details

Trained with the Speculators library (ref 442ce1bf4e2f70dd0b45f36e63bd5849fbb35912, vLLM base image v0.29.0) on regenerated google/gemma-4-31B-it responses over mlabonne/open-perfectblend prompts.

Setting	Value
Speculator type	`dflash2`
Verifier	`google/gemma-4-31B-it`
Draft transformer layers	5
Target layer ids	`1, 17, 29, 47, 59`
Sliding window	2,048
Block size	8
Draft vocab size	full verifier vocabulary (not pruned)
Max anchors	1,024
Total sequence length	8,192
Data workers / prefetch factor	2 / 1
Hidden-states backend	Mooncake (32 GiB global segment, 8 GiB local buffer, 8 writer threads)
Loss function	mixed: `0.1 * CE + 0.9 * TV`
Optimizer	Muon (lr 6e-4, weight decay 0.01, muon_lr 0.001)
Scheduler	cosine
Epochs	2
Gradient checkpointing	off
FSDP sharding	on
Trainer hardware	1 node, 4x NVIDIA A100-SXM4-80GB
Hidden-state extractor hardware	4x A100-SXM4-80GB, TP=2, DP=2, max_model_len 32,000, gpu-memory-utilization 0.92

Full cluster training recipe (YAML)

metadata:
  name: gemma4-31b-dflash2-bs8

images:
  builds:
    shared:
      vllm_base: docker.io/vllm/vllm-openai:v0.29.0
      speculators_ref: "442ce1bf4e2f70dd0b45f36e63bd5849fbb35912"

extractor:
  replicas: 1
  tensor_parallel_size: 2
  data_parallel_size: 2
  trust_remote_code: true
  max_model_len: 32000
  resources:
    gpu_count: 4
    node_selector:
      nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB
  vllm_args: ["--gpu-memory-utilization", "0.92"]

data_preparation:
  defaults:
    sources:
      - hf:nm-research/gemma-4-31b-it-responses-open-perfectblend

trainers:
  defaults:
    nodes: 1
    resources:
      gpu_count: 4
      node_selector:
        nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB
    config:
      speculator_type: dflash2
      verifier:
        verifier_name_or_path: google/gemma-4-31B-it
        trust_remote_code: true
      draft:
        draft_vocab_size: null
        num_layers: 5
        target_layer_ids: [1, 17, 29, 47, 59]
        sliding_window: 2048
      data:
        hidden_states_backend: mooncake
        total_seq_len: 8192
        num_workers: 2
        prefetch_factor: 1
        max_anchors: 1024
      generation:
        on_missing: generate
        on_generate: delete
        request_timeout: 180
        max_retries: 20
        max_consecutive_generation_failures: 20
      loss:
        loss_fn: '{"ce":0.1,"tv":0.9}'
      optimizer:
        optimizer: muon
        lr: 6e-4
        weight_decay: 0.01
        muon_lr: 0.001
      scheduler:
        scheduler_type: cosine
      trainer:
        epochs: 2
        checkpoint_freq: 0.1
        log_freq: 100
        fsdp_shard: true
        gradient_checkpointing: false
      dflash:
        block_size: 8

Deployment

vllm serve google/gemma-4-31B-it \
  --tensor-parallel-size 4 \
  --speculative-config '{
    "model": "RedHatAI/gemma-4-31B-it-speculator.dflash2",
    "num_speculative_tokens": 7,
    "draft_tensor_parallel_size": 4,
    "method": "dflash"
  }'

Use the verifier's chat template via the /chat/completions endpoint.

Evaluation

Evaluated with Speculators 0.8.0 on vLLM 0.29.0 against the RedHatAI/speculator_benchmarks dataset (evaluate.py ... throughput, 200 requests, max concurrency 16, temperature 0, enable_thinking: true).

Per-position token acceptance rates across datasets:

Dataset	Pos 0	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Avg. Length
HumanEval	93.33%	86.27%	79.71%	73.61%	67.95%	62.67%	57.78%	6.21
math_reasoning	96.23%	92.25%	88.38%	84.43%	79.99%	75.53%	71.46%	6.88
qa	78.86%	61.26%	47.40%	37.58%	30.73%	25.17%	20.59%	4.02
question	82.31%	65.10%	52.27%	42.67%	35.90%	30.65%	26.26%	4.35
rag	88.08%	76.18%	64.92%	55.80%	48.09%	41.28%	35.15%	5.10
summarization	82.13%	65.81%	51.62%	40.81%	31.57%	24.49%	19.10%	4.16
tool_call	84.07%	69.21%	56.50%	46.84%	38.87%	32.89%	27.51%	4.56
translation	86.28%	74.86%	63.85%	55.11%	47.51%	40.88%	34.96%	5.03
writing	82.26%	65.08%	52.14%	42.53%	35.80%	30.60%	26.22%	4.35

Performance benchmarking

Throughput vs. interactivity, comparing the google/gemma-4-31B-it verifier with no speculator against the verifier served with this DFlash2 draft model (7 speculative tokens), at matched request concurrency (N=1,2,4,8,16,32,64,128):

HumanEval	math_reasoning

At the highest-throughput point measured (N=128 concurrent requests):

Subset	Baseline (tok/s)	DFlash2 (tok/s)	Speedup
HumanEval	2,890	4,098	1.42x
math_reasoning	3,502	4,753	1.36x

Configuration and command

Configuration

temperature: 0
max output tokens: 1024
concurrency (N): 1, 2, 4, 8, 16, 32, 64, 128
repetitions: 1 run per point
vLLM version: 0.29.0
GuideLLM version: 0.7.4
Verifier served with: --tensor-parallel-size 4 --gpu-memory-utilization 0.90 --max-model-len 8192 --max-num-seqs 256 --max-num-batched-tokens 16384 --no-enable-prefix-caching

Command

throughput_interactivity.py collect \
  --target http://localhost:8120 \
  --model google/gemma-4-31B-it \
  --tokenizer google/gemma-4-31B-it \
  --dataset RedHatAI/speculator_benchmarks \
  --subset HumanEval \
  --max-tokens 1024 \
  --repeats 1 \
  --extra-body temperature=0 \
  --streams 1,2,4,8,16,32,64,128

Optimization

This model implements DFlash2 speculative decoding: a 5-layer draft model proposes up to 7 tokens per step, which the google/gemma-4-31B-it verifier accepts or rejects in parallel. The average accepted length in the table above (4.0–6.9 tokens, depending on subset) quantifies the expected decode-throughput speedup — confirmed directly by the measured 1.5x+ throughput gains above.