bumblebuttpow/MiniCPM5-2B-heretic-abliterated-MLX-8bit

MiniCPM5-2B-heretic-abliterated MLX 8-bit

MLX 8-bit quantization (8.5 bits/weight, group size 64) of insraq/MiniCPM5-2B-heretic-abliterated, a Heretic v1.4.0 abliteration (decensoring) of openbmb/MiniCPM5-2B. Runs on Apple Silicon via mlx-lm. ~2.5 GB on disk, ~2.8 GB peak memory, ~87 tok/s on an M4 Pro.

Converted from the fp16 safetensors (not the GGUF) so no double-quantization loss.

Important: this repo includes a rope config fix

The source repo's config.json was exported with transformers v5, which nests the rope settings as rope_parameters: {rope_theta: ...}. mlx-lm's llama loader only reads the flat rope_theta key and silently falls back to the default (10000) when it is missing. The result: the model runs with a rope theta 500x too small, which is coherent at short context and collapses into word salad somewhere past ~12k tokens, regardless of quant, sampler, or thinking mode.

This repo's config.json carries the flat "rope_theta": 5000000 key, so it works correctly out of the box. With the fix, long context behaves as the base model card claims: verified correct last-line retrieval from 70k-token documents on a 2B model.

If you convert transformers-v5-era re-exports to MLX yourself, check for nested rope_parameters and add the flat key before serving.

Usage

from mlx_lm import load, generate

model, tokenizer = load("bumblebuttpow/MiniCPM5-2B-heretic-abliterated-MLX-8bit")

Or serve it:

mlx_lm.server --model bumblebuttpow/MiniCPM5-2B-heretic-abliterated-MLX-8bit

Sampling recommendations (measured)

This merge has a sensitive think channel. Recommendations:

• temperature 0.6, top_k 20, repetition_penalty 1.05 — stable. Higher temperature (including the base model's recommended 1.0) causes the think channel to verbatim-loop on trivial prompts.
• Thinking mode: it deliberates at length. Give it max_tokens of 800+ or the answer stays trapped in the reasoning block.
• Instant-answer mode: pass chat_template_kwargs: {"enable_thinking": false} to pre-close the think block (supported by the chat template).

Verification

• 8-bit quant from fp16 safetensors via mlx_lm 0.31.3.
• Short-context QA, 26k and 70k-token retrieval prompts: coherent, correct retrieval, no degeneration (after the rope fix above).
• Verified both solo and pipeline-parallel across two Macs with identical outputs.

Attribution

• Abliteration: insraq (Heretic v1.4.0, KL 0.0391, refusals 5/100)
• Base model: openbmb/MiniCPM5-2B (Apache 2.0)
• Quant + rope fix: bumblebuttpow