pipecat-ai/phonellm-mini-alpha-2-nvfp4

PhoneLLM Alpha 2: a blazingly fast, open-weights text model designed to be integrated into voice agent pipelines.

This new version is available in two variants: Mini for speed, and Max for accuracy.

Check out our blog post to find out more about the improvements we made, and how we trained the models:

https://www.daily.co/blog/phonellm-alpha-2/

---

Pipecat PhoneLLM Mini Alpha 2 (NVFP4)

The NVFP4 quantization of PhoneLLM Mini Alpha 2, a full-parameter supervised fine-tune of NVIDIA Nemotron 3 Nano 30B-A3B for phone voice agents with tool calling.

• Quantization: NVIDIA Model Optimizer (ModelOpt) post-training quantization (MAMBA_MOE_NVFP4_AGGRESSIVE_CFG): every linear layer is NVFP4 except lm_head, the six attention layers and the projections of six Mamba layers, which stay bf16. FP8 KV-cache scales are included. Calibrated on 1,000 assistant turns from 288 PhoneLLM training conversations. ~19 GB. NVFP4 runs natively on Blackwell GPUs (B200, B300, RTX PRO 6000); Hopper falls back to slower emulation.
• Thinking is disabled by default: the bundled chat_template.jinja defaults enable_thinking to false (the model was trained without reasoning). Serve without a reasoning parser.
• Recommended settings: temperature=0.

Serving with vLLM

Smoke-tested recipe (vLLM 0.30, one B200); throughput has not been benchmarked yet.

vllm serve pipecat-ai/phonellm-mini-alpha-2-nvfp4 \
  --served-model-name phonellm-mini-alpha-2-nvfp4 \
  --trust-remote-code \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --kv-cache-dtype bfloat16 \
  --enable-prefix-caching \
  --async-scheduling \
  --mamba-ssm-cache-dtype float32 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --moe-backend flashinfer_cutlass

• Keep --kv-cache-dtype bfloat16: the checkpoint carries FP8 KV-cache scales, so without the flag vLLM silently uses an FP8 KV cache. FP8 KV holds roughly 40 % more tokens, but on PhoneLLM Alpha 1 NVFP4 it cost 0.5 PhoneBench points (bf16 KV: within 0.04 of the bf16 model). Switch to --kv-cache-dtype fp8 only if you need the capacity.
• Keep --moe-backend flashinfer_cutlass: vLLM 0.30's default NVFP4 MoE kernel on B200 (FLASHINFER_TRTLLM, which pads this model's hidden size from 2688 to 2816) produces degenerate text with CUDA graphs enabled — NVIDIA's own Nemotron 3 Nano NVFP4 checkpoint does the same.
• Do not add --enforce-eager: CUDA graphs are 5-7x faster decode on this model.
• No --reasoning-parser: the bundled template keeps thinking off, so replies and tool calls are parsed directly. (With a reasoning parser, a request that omits enable_thinking would have its tool calls filed under reasoning.)
• Keep 128-256 requests in flight per GPU for batch workloads; raise --max-model-len only if needed.

License

PhoneLLM Mini Alpha 2 is released under the BSD 2-Clause License.

PhoneLLM is a derivative work of NVIDIA Nemotron 3 Nano 30B-A3B, which is licensed under the NVIDIA Nemotron Open Model License. Under Section 3 (Redistribution) of that license, if you redistribute this model or your own derivatives of it, you must (a) include a copy of the NVIDIA Nemotron Open Model License, and (b) retain the NVIDIA copyright and attribution notices. Our BSD 2-Clause terms apply to our modifications and to the model as a whole, as Section 3 permits; the NVIDIA license continues to apply to the underlying Nemotron work. "Nemotron" and "NVIDIA" are trademarks of NVIDIA Corporation, used here only to describe the origin of the base model.