memra-bench — benchmark artifacts for memra

Reproduction artifacts for the performance claims in github.com/avifenesh/memra — a from-scratch LLM inference engine (Rust + CUDA, OpenAI-compatible serving) tuned for RTX PRO 6000 Blackwell (sm_120a) and RTX 5090, with a compile-gated H100 (sm_90a) lane. If a claim depends on a specific artifact, the artifact is public here.

Refreshed 2026-08-21 against memra main 73d79a3cc8 (latest release v0.99.0). Two renames since this card's first revision: the engine is memra (formerly bw24), and the env prefix is MEMRA_* (formerly BW24_*) — every BW24_* incantation in older revisions of this card is dead. Per-cell configs, flag-verified against the engine's audited catalog (docs/FLAGS.md), live in CONFIGS.md in this repo.

Drafts (drafts/<model>/) — the standard regime, one file per model

Every memra spec board row runs one trimmed draft file built by the documented regime (docs/DRAFT-REGIME.md): FR-Spec ranks derived from the model's own generations (never transferred between models — foreign ranks measured −12 acceptance pts on an identical tokenizer), MTP block extracted byte-verbatim from the published model GGUF, head requantized NVFP4 after trimming (measured zero acceptance cost), block Q4_K_M (measured faster AND higher acceptance than Q8_0). Serve with MEMRA_MTP_DRAFT=<draft.gguf> — no other flags (winners are defaults). Server spelling: MEMRA_MODELS="alias=/trunk.gguf+/draft.gguf". Proof of attach is a log line, not the absence of an error: [mtp-draft] loading external MTP draft: <path> or [worker] <name>: regime draft attached (<path>).

Directory	Source model (exact bytes)	Tracked board row (e2e tok/s short / medium / long-agentic)
drafts/qwen35-9b-nvfp4/	Qwen3.5-9B NVFP4 MTP GGUF	K=3: 281.0 / 211.7 / 187.1
drafts/qwen36-27b-nvfp4/	nvidia/Qwen3.6-27B-NVFP4 → Q4_K_M GGUF	K=3: 116.4 / 101.2 / 86.0
drafts/qwen36-35b-a3b-iq4xs/	unsloth Qwen3.6-35B-A3B UD-IQ4_XS	K=2: 302.4 / 253.0 / 270.7
drafts/qwen36-27b-unsloth-nvfp4/	unsloth/Qwen3.6-27B-NVFP4 → GGUF	no board row — plain parity with the nvidia artifact; spec long-agentic win / medium loss (jsonl 2026-07-17)

Rows are the tracked RTX 5090 Laptop spec cells at memra main 73d79a3cc8 (medians of N=5 same-session interleaved reps, measured 2026-08-02, no flags; full conditions and raw per-run logs referenced from docs/PERFORMANCE.md). The long-agentic column is SAMPLED (temp 0.7): rejection-sampling spec decode is distribution-exact; short/medium are greedy.

Each directory carries the draft (draft-owntrim-nvfp4head-q4blk.gguf) and the rank file it was built from (owngen-ranks-32768.txt, one token id per line, rank order — derived 2026-07-17/18, 218-prompt mixed corpus, ~110k own-generated tokens per model).

Use ours for these exact models. For any other model, requant, or finetune, build your own in two commands (a finetune's distribution moved, so its draft must too):

./target/release/frspec-owngen model.gguf ranks.gguf 32768        # ranks from the model's OWN generations
tools/make-trimmed-draft.sh model.gguf ranks.gguf.txt draft.gguf  # extract + trim + quantize

Validate before trusting: frspec-owngen model.gguf out.gguf --validate A/Bs baseline-vs-trimmed spec e2e and prints a GOOD/WASH/BAD verdict.

drafts/kf4/ — archived experiment, no released config consumes these

These directories were built for an experimental NVFP4 KV-cache format arm (kv-fp4 lane, 2026-07-20). That arm was never merged: at memra main 73d79a3cc8 the KV-cache format knobs accept MEMRA_KV_K = q8_0 | fp8 and MEMRA_KV_V = q5_1 | q4_0 | fp8 only (defaults q8_0/q5_1; the arm survives on the engine repo's archive/kv-fp4 ref). The files are kept as evidence of regime law 1 — a new numeric config re-derives each model's draft from its own generations under that config. That lane's 2026-07-20 verdicts, for the record: 9B wash, 27B kf4-draft win (+2.5% e2e), 35B old-draft stays.

Gemma ranks (drafts/gemma4-*/)

Gemma drafters are separate assistant GGUFs, not NextN/MTP heads — they attach with MEMRA_DRAFT=<assistant.gguf> (the seam trap: MEMRA_MTP_DRAFT is the qwen-family attach and refuses this format; on the server the gemma spec route arms via MEMRA_GEMMA4_SPEC — unset with a drafter = K=5). The FR-Spec trim applies at load: MEMRA_GEMMA_DRAFT_RANKS=<ranks.txt>; the serve-time adaptive trim (MEMRA_GEMMA_TRIM_ADAPT, on by default with ranks — 512 spare head slots, 0 = static trim) learns coverage escapes live from prompt + verify tokens and persists them to <ranks>.learned. Own-gen rank files per model (regime law 1):

Directory	Model	Note
drafts/gemma4-26b-a4b-qat/	Gemma-4 26B-A4B QAT Q4_0	trim adopted (own-gen e2e-neutral vs corpus ranks, correct provenance)
drafts/gemma4-31b-qat/	Gemma-4 31B QAT Q4_0	the adaptive trim flipped the 31B chat cell from −17% to +2.5% and made trim ≥ untrimmed on every measured cell (2026-07-19)
drafts/gemma4-e4b-qat/	Gemma-4 E4B QAT Q4_0	ranks published; serving stays untrimmed (measured e2e wash — small head)

Legacy trims (top level, Qwen3.6-27B)

The pre-regime artifacts the earlier boards used — superseded by drafts/ but kept because published claims referenced them:

File	Ranking	Status
mtp-Qwen3.6-27B-Q4_K_M-frspec32768.gguf	generic corpus-frequency	superseded by drafts/qwen36-27b-nvfp4/
mtp-Qwen3.6-27B-Q4_K_M-frspec-code75-32768.gguf	code-skewed corpus	superseded
mtp-Qwen3.6-27B-Q4_K_M-frspec-balanced32768.gguf	balanced corpus	superseded

Prompts + configs

prompts/ holds the exact board prompts — p1-code-short.txt, p2-code-medium.txt, p3-agentic-long.txt plus the well-formed p3-agentic-long-v2.txt the sampled protocol uses — and the long-context depth documents p4-16k.txt / p5-32k.txt / p6-64k.txt. CONFIGS.md carries the per-cell engine configs. Protocol: interleaved same-session reps (N≥2–3 medians, both orders), power state pinned per window — research/benchmarks.md.

Related

• Engine: github.com/avifenesh/memra — every published number's raw JSONL under research/, wins and losses both; boards in docs/PERFORMANCE.md.
• Flagship (Qwen3.8-27B) ranks + drafts: Avifenesh/Qwen3.8-27B-NVFP4-MTP-GGUF — three ranks flavours; the safetensors path trims at load via MEMRA_FRSPEC_TRIM=<ranks.txt>, no separate draft file.
• Avifenesh/Hy3-REAP-Layer103p5-bw24 — the ~100 GB Hy3 expert overlay memra serves on a 24 GB card via VRAM→RAM→NVMe spill (the repo keeps the engine's former name).
• No card? A hosted instance runs at inference.tiyuvta.ai.