swiss-ai

/

Apertus-8B-Instruct-2509

Like 492

Follow

Swiss AI Initiative 1.42k

Hardware: ASUS TUF Gaming RTX 3090 (24 GB VRAM)
vLLM: v0.19.1 (v0.19.0 was the first release with native Apertus model support; v0.19.1 adds LoRA)
Deployment: vLLM served via the official vllm/vllm-openai container image, fronted by a FastAPI application server. Coqui TTS also runs on the same GPU for voice synthesis.

Quantization

--quantization bitsandbytes
--dtype auto

4-bit bitsandbytes (QLoRA-compatible) quantization brings the model from ~16 GB down to approximately 6 GB of VRAM, essential on a 24 GB card that has to share with other processes. --dtype auto lets vLLM pick the right compute dtype to complement the quantization without manual tuning.

Gotcha: --quantization bitsandbytes requires using --load-format safetensors together with --safetensors-load-strategy eager. Without the eager load strategy, bitsandbytes can fail to quantize correctly on load.

KV Cache

--kv-cache-dtype fp8

fp8 KV cache halves the memory footprint of the attention cache compared to fp16/bf16. This matters more than it might seem: KV cache scales non-linearly with context length and sequence count. With a 20K context window and another process (TTS) competing for VRAM, fp16 KV cache consistently caused OOM. fp8 resolved it.

GPU Memory Utilization

--gpu-memory-utilization 0.73

This is the fraction of GPU VRAM that vLLM is allowed to claim for its memory pool (model weights + KV cache). The default is 0.90, which would starve Coqui TTS. I landed on 0.73 empirically: low enough that TTS synthesis doesn't OOM, high enough to keep a useful KV cache. If you're running vLLM alone on this card, you can push this to 0.90.

Context Window

--max-model-len 20480

The model supports up to 65K tokens, but allocating the full KV cache for 65K on a shared 24 GB card isn't feasible. 20,480 covers real-world use (conversation history, tool definitions, system prompt, response) without exhausting VRAM.

Concurrency

--max-num-seqs 4

Maximum parallel in-flight requests. 4 is a practical ceiling given the VRAM budget — a higher value increases KV cache demand and risks OOM under load. For a home/personal server with occasional concurrent users, 4 is comfortable.

Here is a theoretical breakdown of GPU VRAM needed to support the full 65k token limit of Apertus at FP8 and 0.90 GPU memory utilization.

Prefix Caching

--enable-prefix-caching
--no-enable-chunked-prefill

Prefix caching reuses KV cache entries for common prompt prefixes (system prompt, tool definitions, repeated context). In practice I am seeing ~35% cache hit rate on conversational workloads, which meaningfully reduces prompt processing time on repeat calls.

Chunked prefill is disabled because it conflicts with prefix caching in vLLM v0.19.x when bitsandbytes quantization is active.

CUDAGraphs

By default, vLLM uses CUDA graph capture to optimize GPU kernel execution. An older workaround flag — --enforce-eager — disables this and falls back to eager execution.

Gotcha (and important lesson): I had originally added --enforce-eager to work around segfaults during container startup. It turned out this was a red herring as the real cause was using a generic base container image rather than the official vllm/vllm-openai image. The official image includes the correct CUDA/PyTorch runtime that vLLM expects. Switching to it resolved all segfault issues, including a separate LoRA-related segfault. Removing --enforce-eager then came for free, and the result was significant: tool call response times dropped from ~4 seconds to ~1.25 seconds — roughly a 3–4× improvement.

If you're seeing mysterious segfaults or crashes: check your base image before reaching for --enforce-eager.

LoRA Support

--enable-lora
--max-lora-rank 64
--lora-modules '{"name":"home-control","path":"...","base_model_name":"swiss-ai/Apertus-8B-Instruct-2509"}'

vLLM v0.19.1 supports loading LoRA adapters at startup and routing requests to them by name. A few things that tripped me up:

• base_model_name is required in the JSON. Omitting it causes vLLM to set it to null and reject requests.
• Use /v1/chat/completions, not /v1/completions. LoRA routing by name returns 404 on the completions endpoint.
• --lora-modules JSON quoting in supervisord: supervisord parses the command= directive via Python's shlex.split(). Wrap the JSON in single quotes to preserve it as a single token.

Quality note: My first LoRA was trained exclusively on home-control tool-calling examples. The result was a model that tried to call tools for every query — including conversational ones where it should just reply in prose. I have disabled it for now. The lesson: if your training data only contains one response type, the adapter will bias toward that type regardless of context. Training data needs to cover the full range of expected behaviors, including "do nothing, just answer."

Passthrough Chat Template

Apertus uses a custom chat template that formats the prompt with role tokens (<|im_start|>, <|im_end|>, etc.). If your application layer is already formatting the prompt before sending it to vLLM, you need to prevent vLLM from applying the template a second time:

"chat_template": "{% for message in messages %}{{ message.content }}{% endfor %}"

This requires --trust-request-chat-template on the vLLM server. Without it, the server rejects custom templates with a 400 error.

Model Quality: A Candid Note

With all of the above in place, the model performs well at its intended tasks — instruction following, tool calling, conversational interaction. Response times are fast.

For general knowledge queries, particularly anything requiring specific historical facts or dates, the 8B parameter size shows some limitations. Asked about the Ferrari F1 team, the model confidently stated it was "founded in 126", apparently conflating century with year, or perhaps just confabulating. Lowering temperature to 0.3 makes it consistently wrong rather than variably wrong; it doesn't fix the underlying knowledge gap.

I feel that this is just a known tradeoff at the 8B scale. For a home assistant that needs factual trivia accuracy, you'd want retrieval augmentation. For home automation, personal context, and instruction following, 8B at this speed is excellent.

Full vLLM Command (summary)

python3 -m vllm.entrypoints.openai.api_server \
  --model swiss-ai/Apertus-8B-Instruct-2509 \
  --quantization bitsandbytes \
  --dtype auto \
  --load-format safetensors \
  --safetensors-load-strategy eager \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.73 \
  --max-model-len 20480 \
  --max-num-seqs 4 \
  --enable-prefix-caching \
  --no-enable-chunked-prefill \
  --enable-lora \
  --max-lora-rank 64 \
  --trust-request-chat-template

Have any questions? Leave me a comment! I'll do my best to answer.