one week of Muse Glimmer in production, guided decoding at high volume (vLLM, single RTX 5090)
We run a financial research pipeline that generates a few thousands structured documents per day, all schema-constrained (guided decoding with Pydantic schemas, so xgrammar under the hood). We put Glimmer in production the day of the release and it has been running our full workload since then, on one RTX 5090 with vLLM. As far as we can tell nobody pushed this model through this exact code path at volume yet, because every bug we hit was still virgin. So here is what breaks, how we fixed it, and some numbers that surprised us. Maybe it saves a week to the next team.
The setup
vLLM with a small set of local patches (details below), fp8 KV cache, vision enabled, structured output on almost every request. Reasoning dial on low. Nothing exotic in the launch flags except this one:
--kv-cache-memory 5905580032
We size the KV pool explicitly instead of letting the utilization heuristic decide, because with the vision tower loaded the automatic split was leaving GPU memory on the table.
What breaks under guided decoding
Four distinct defects, all reported upstream:
- xgrammar #831, schemas with
additionalPropertiescan walk the grammar into an ordering trap where the model gets cornered into emitting keys in an order the grammar then refuses. Shows up as impossible-to-satisfy states on perfectly valid schemas. - xgrammar #832, control tokens are not declared correctly to the grammar matcher, so tokens the runtime considers special can leak into (or be refused from) constrained output.
- vllm #52146, the EOS handling under grammar: the engine can accept an end-of-sequence in the middle of an open JSON string. We first saw it on the vision path, a response dying mid-string while the grammar said everything was fine. The root cause is the stop token set, not the grammar itself: the server was not using the full stop set from
generation_config.json, so a token the model legitimately uses as a stop was treated as printable (and the reverse case exists too). - sglang #34631, on the sglang side this time: guided decoding resumes one token too early for channel-framed reasoning formats. Glimmer writes its own channel headers (
<|start|>assistant to=user<|message|>) before the answer body, and the grammar gets applied on the header tokens themselves. In our tests 46.7% of the JSON responses collapsed to the shortest schema-valid placeholder because of it. We reported it while we were still comparing both engines.
On top of the reports we carry four small local patches while waiting for upstream: the grammar start detection for this tokenizer family, correct grammar arming on short responses (there is a window at the start of generation where the grammar is not armed yet, and a model that answers very fast can slip unconstrained tokens through it), min_p actually applied when speculative decoding is on (it was silently ignored, and it must apply after temperature), and the stop-token set built from generation_config.json. Happy to share the diffs if someone wants them, they are a few dozen lines each.
One more trap that costed us an evening: disable_any_whitespace=true does not disable whitespace, it imposes a canonical whitespace form. The name is misleading. With it off, we measured around 8% of constrained responses degenerating into whitespace runaway. Turn it on.
Numbers that surprised us
The KV pool is the headline. The hybrid attention (3 sliding-window layers for 1 global) gives an absurdly light cache. With our 5.5 GiB KV budget, fp8, vision loaded, we get a pool of 623,837 tokens, that is about 4.7 concurrent full 131k contexts on a single consumer card. Our real workload is mostly 8k to 32k prompts, so in practice the concurrency is bounded by compute, not by memory. We came from architectures where KV was always the wall, here it just is not.
Speculative decoding was a negative result for us. We benched a NVFP4 (W4A16) drafter. Acceptance rates were healthy (about 54% first position, 21% second), so the modeling side works. But the quantized drafter is compute-slow on this card and the end-to-end throughput went down: at batch 32 we measured 895 tok/s with the drafter versus 1,637 tok/s without. If your batch sizes are small the math may be different, but at high batch, on this GPU, spec decode with a NVFP4 drafter is a loss. We reverted.
Low reasoning effort wins for structured extraction. For our extraction and composition tasks, the low setting gave us equal or better outputs than high, several times faster. If your use case is "fill this schema from this document", do not pay for long thinking traces, this model does not need them for that. The terse traces other people noticed are real.
A small operational one at the end: if you healthcheck the server, probe with a real generation of at least a few hundreds tokens. Some of the failure modes above only show after 384+ tokens, a 10 token ping stays green while production burns.
Where this model fits
We are not claiming it is the best model in absolute, it is not (and the hallucination numbers published by Artificial Analysis are consistent with what you would expect if you ask it open questions from parametric knowledge, do not do that). But for tool-use style structured work where all the informations come from the prompt, on a single consumer GPU, with high concurrency needs, we did not find better. The combination of the light KV, the fast decode and the terse reasoning is exactly the profile for high-volume constrained generation.
If you are deploying it for a similar workload and hit something not listed here, we are interested to hear it.
Follow-up on the vision side, since the original report was mostly about guided decoding on text. We spent a day chasing why our chart-reading eval read better under SGLang than under vLLM (97.9 vs 91.7 exact+directional on 100 hand-checked slide pages), and the answer is worth sharing because we were wrong twice before getting it right.
First wrong lead: the resize filter. vLLM's processor resizes with LANCZOS (faithful to what the checkpoint declares), SGLang substitutes BICUBIC. The pixel delta is real and measurable, 85 percent of it concentrated on glyph edges, textbook ringing. Looked like a smoking gun. It is not: we exposed the filter as a knob and ran the A/B both ways on two engine versions -- BICUBIC came out WORSE than LANCZOS on vLLM in both cases. A perfectly characterized pixel delta with a plausible physical mechanism, and zero causal role. The test that could invert the direction was the only thing that settled it.
Second wrong lead: an engine regression. Also refuted by bisect.
The actual answer had two parts. One, run-to-run variance on this kind of eval is about 6 points exact at identical config (multi-figure pages stochastically hitting the token cap), so a chunk of the gap was never real. Two, and this is the part that matters for anyone doing multimodal extraction under guided decoding: our eval schema allowed additional properties, and compact-whitespace grammar mode interacts very badly with permissive schemas -- the model starts dumping its answer into an empty string key that the schema technically allows. Details and the bisect table are in an issue we filed on xgrammar (mlc-ai/xgrammar #851). With a production-shaped schema (additionalProperties false) the whole gap evaporates: 91.7 exact / 97.9 exact+directional on vLLM, 100/100 calls parsed, faster per page than every permissive run. Same engine, same flags, same weights.
So the summary for vision on this model under vLLM: leave the preprocessing alone, it is already the faithful one. Tighten your schemas. And if you compare two engines, make sure your harness schema is shaped like what you would actually serve, because ours was shaped to measure key drift and ended up manufacturing the very gap we were investigating.