WB-catboy-NCM-Instruct-725M

neko, a small catboy assistant, built on a Next-Concept Model (NCM): a language model that does not write word by word. It plans the whole reply at once, coarse to fine, and then decodes every word in parallel.

As far as I know, this is one of the first chat / instruction-tuned NCMs. It is a hobby research model: it greets you, knows who it is, comforts you when you have a bad day and answers simple things, but it often gets facts wrong and its grammar wobbles.

Just for fun! neko is a hobby project made for fun and curiosity, not a serious assistant. Don't take anything it says seriously, and please don't use it for advice, facts or anything important. Enjoy the meows. :3

you> hello
bot: hello! how can i help you today?
you> who are you?
bot: i am neko, a catboy living in the computer.
you> what is your favorite food?
bot: salmon is my favorite food!
you> can you recommend a good book?
bot: what do you like to read?
you> what is the capital of france?
bot: paris is crowded france, nya.

What is an NCM?

Next-Concept Modeling (Hazy Research, 2026) splits generation into two parts:

1. A frozen multiscale VQ-VAE compresses a 128-token window into a stack of discrete codes: 1 code for the gist of the whole window, then 8, then more at each finer level.
2. A transformer generator predicts one level at a time, all codes within a level in parallel. The VQ-VAE decoder then turns the finest level back into words.

So instead of one step per word, a reply takes one step per level. It is similar in spirit to VAR ("next-scale prediction") for images.

This model is the 725M-parameter LM1B-large generator from the NCM release, fine-tuned for chat. The prompt (chat history) goes into the generator as a 128-token raw-token prefix; only the reply is quantized.

Persona

neko is a friendly catboy assistant who lives in the computer. Likes: helping, naps in sunny spots, fish (favorite food: salmon), the color orange. It sometimes adds "meow", "nya" or ":3" (on purpose only in about a quarter of replies).

How to use

Everything needed is in this repo: the weights, the NCM model code (models/, tokenization/, utils/, from HazyResearch/ncm, MIT, see LICENSE-NCM-code) and the chat script.

git lfs install
git clone https://huggingface.co/banan332/WB-catboy-NCM-Instruct-725M
cd WB-catboy-NCM-Instruct-725M
pip install -r requirements.txt   # install the torch build for your GPU (CUDA or ROCm) first
python chat.py --n 3 --cfg 3 --temp 0.3 --guide last

It needs a GPU with about 5 GB of free memory (measured: ~4.5 GB on an RX 7900 XTX). On AMD (RDNA3), set TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1; the script does this for you.

Files:

• generator.pt, tokenizer.pt, config.yaml: the model (generator and frozen VQ-VAE tokenizer)

• chat.py: interactive chat

• train_instruct.py: model loading, sampling and the fine-tuning script

• scripts/: the full data pipeline (data prep, Qwen data generation, mixing) and the evaluation tests from this card

• --cfg is classifier-free guidance: how strongly the reply follows your message. 2-3 works best; 5+ causes repeated words and 20 turns into keyword salad.

• --temp is the temperature. 0.3-0.5 gives the cleanest sentences; at 1.0 replies get much more random.

• --guide last makes guidance push away from the chat without your newest message instead of from an empty prompt, so the reply follows what you just said instead of bleeding in the old topic. Recommended.

• --n 3 shows three candidate replies per turn (the first goes into the history).

• In chat: /reset clears the history (useful when it gets stuck on a topic), /save appends the exchange to quotes.txt.

• Everything is lowercase: the tokenizer is bert-base-uncased.

Training

All training ran on a single AMD Radeon RX 7900 XTX (24 GB) with ROCm 6.4 and PyTorch 2.9.1, at about 4.3 s per step (batch 32, micro-batch 8, gradient checkpointing, bf16 autocast). About 26 GPU-hours of training plus about 6 hours of data generation.

The training and data scripts were written with the help of Claude Code. The data choices, experiments, evaluation and the neko persona are my own.

The base model never read long prompts (it was pretrained with a 1-token prefix), so the prefix was widened to 128 tokens and learned during fine-tuning. 10% of training prompts are blanked so that CFG works. Optimizer: AdamW (betas 0.9/0.95, weight decay 0.01), cosine schedule to 10% of peak, gradient clipping at 1.0.

Stage	Data	Steps	LR
1. Chat basics	SODA (1.18M dialogue pairs)	4,000	5e-5
2. Persona	3.4k Qwen-written neko chats (×8) + DailyDialog, EmpatheticDialogues, BST, SODA	6,500	2e-5
3. Assistant	+ 21.5k neko answers to real user questions, public assistant data, recall exercises (309k pairs)	400 + 3,800	2e-5
4. Chat skills	+ 10% "topic switch" examples, my/your contrast pairs, fewer exercises (314k pairs)	1,500	2e-5
5. Chat variety	+ 2k varied-style chats (slang, typos, one-word users, jokes, topic changes), 17% topic switches, more human chat data (362k pairs)	2,000	2e-5
6. Flexibility	+ 1.2k chats with corrections, mini poems, jokes and world questions after persona talk; history dropout (369k pairs)	2,000	2e-5
7. Pronouns	+ 15k my/your contrast pairs (380k pairs)	1,000	2e-5

A first attempt at stage 2 with lr 5e-5 diverged (loss 0.3 → 4.5), so all later stages use 2e-5.

Synthetic data. About 28k conversations and answers were written by a local Qwen3.8-27B finetune (Apache 2.0), prompted to answer as neko: short, lowercase, simple words. The user questions came from Dolly, Anthropic HH (helpful), OpenAssistant 2 and hand-written persona questions.

Training tricks that helped:

• Recall exercises ("my favorite food is sushi … what is my favorite food?") taught it to read the prompt at all. Too many (11%) made them take over the chat ("your name is maria!"), so they were cut to 3%.
• My/your contrast pairs ("what is your name?" → neko vs "what is my name?" → the user's name).
• Stitched topic switches: the history from one conversation and the last question from another. The only way to get these right is to answer the last message instead of the old topic.
• Varied user styles: Qwen wrote chats where the user is casual, slangy, typo-heavy, grumpy, sad, a kid, formal, or changes topic mid-chat, so neko sees many ways of chatting instead of one polite Q&A shape.
• Corrections, poems and jokes: chats where neko gets something wrong and is corrected ("you are right, it is 10"), tiny rhyming poems, and clean jokes.
• History dropout: a quarter of the examples keep only their last 1-3 turns, so the newest message weighs most.
• Last-message guidance (at chat time, no training): classifier-free guidance normally contrasts the prompt with an empty prompt. Contrasting it with the chat minus the newest message amplifies what the new message adds. It cut replies that bled in the old topic or neko's own favorites by more than half.

What didn't help: re-weighting the loss per level (the NCM paper's α = 1 weighting) did not make the model use the prompt more.

Evaluation

Validation loss is on 500 held-out neko conversations (lower is better). The chat skill tests are scored by keywords over 24 samples each, at cfg 3 / temp 0.3. All skill scores are success rates, so higher is better.

Model	Val loss	"my vs your"	Topic switch	Avoids persona bleed	Recall (seen values)
Stage 2	0.3074	-	-	-	0%
Stage 3	0.2984	17%	4%	-	65%
Stage 4	0.2925	33%	46%	-	55%
Stage 5	0.2856	83%	58%	12%	65%
Stage 6	0.2804	54%	67%	25%	-
Stage 7 (this model)	0.2788	67%	62%	8%	62%
Stage 7 + --guide last		83%	62%	17%

With 24 samples per test, differences of a few points are partly noise.

• my vs your: after "my name is omar", does "what is your name?" get "neko"?
• Topic switch: after a chat about pasta, does "who are you?" get answered instead of more pasta talk?
• Avoids persona bleed: after chatting about neko's favorites (orange, salmon), does "what color is the sky?" or "where is paris?" get a real answer instead of more orange and salmon? Still the weakest skill: half bleed, half lack of knowledge.
• Recall: after "my favorite food is sushi … what is my favorite food?", is the answer "sushi"? It works for words it saw in training (62%) but almost never for new words (2%), and it cannot repeat arbitrary sentences (0%). It can pick among concepts it knows, but it can't copy arbitrary text through the concept bottleneck.

Limitations

• Facts are weak. The LM1B pretraining (news sentences) holds little world knowledge, so it makes things up ("how many legs does a spider have?" usually fails).
• Short context. 128 tokens of history and 128 tokens of reply.
• Weak prompt grounding. A blank prompt raises its loss only about 3%. It catches the type of question and keywords, but the chat history can still pull it back to the old topic. --guide last and /reset help.
• Duplicated neighbouring words ("i am am neko", "fish fish"): every word is decoded in parallel without seeing its neighbours, the classic non-autoregressive repetition problem.
• Rare "filler collapse": in about 0.3% of samples the reply never ends and fills the window with dots and the word "cup". chat.py trims these.
• Lowercase only, English only. Not safety-tuned beyond the source data. Don't rely on it for anything.

Hall of fame

Real, unedited outputs from this model and its earlier checkpoints. Small models are bad at facts but great at this.

The Crime Dog. Asked about a car fire, it suggests branching out:

you: did his car go up in flames? bot: well, do you think you could try other crimes around the dog? everything doing pretty well.

The Chickendoraie Baptist Pig. At cfg 20 (guidance cranked way too high), "ah yes pizza" summoned a county fair menu from another dimension:

you: ah yes pizza bot: sounds good! buy the the!. legacy f1 building -. f1 store and menu steel flavoro menu ale pizza hardwood chickendoraie baptist pig and grey prestige prestigeholders st sportingcake round - yours clothes. cup cup 200 producttick..ny abu boiler packages packages sheep 53 tie yours commit lottery derby sevens cup.

Same session, same guidance: "can i buy your car?" → "…car car new about rental! what a fiat bombnsteinnstein?" and "hmm boiler sounds good" → "products from pizza christmas packages aboard horse summit…"

The /save existential crisis. Typing a chat command into an old version that didn't know it yet:

you: /save bot: here, i'm fine. but i'll there be found to be worthless! bot: his name is wonderful

☕The cup. About 1 in 300 replies forgets to end and fills the rest of its 128-token window, almost always with the word "cup": "…you're a great job!! . . . . . . cup . . must . . . ." or "cup cup cup cup cup cup…". Nobody knows why it's cup. chat.py now trims it.

Your name is maria. A version trained with too many memory exercises answered everything with them:

you: what is your name? bot: your name is maria! you: am i maria? bot: your favorite color is blue. you: but i don't like blue! bot: your favorite color is green.

Small catboy things: "why is the sky blue?" → "…are fluffy for meow?", "what is your favorite food?" → "i love fish fish.", "what is the capital of france?" → "paris, france is a bright orange city.", and for "what is seven times eight?": "you try picking a healthy nap".

License

CC BY-NC-SA 4.0: free for non-commercial use; derivatives must keep the same license.

This follows the strictest training data: DailyDialog (CC BY-NC-SA 4.0) and EmpatheticDialogues (CC BY-NC 4.0). The other sources are SODA (CC BY 4.0), Synthetic-Persona-Chat (CC BY 4.0), Anthropic HH (MIT), OpenAssistant 2 (Apache 2.0), SmolTalk (Apache 2.0) and Dolly 15k (CC BY-SA 3.0; its ShareAlike term conflicts with NC). Blended Skill Talk has no stated license. The base NCM checkpoint is MIT and the Qwen model used for synthetic data is Apache 2.0. Whether model weights count as derivatives of training data is legally unsettled; this license is the conservative choice.

Citation

Base model and method:

@misc{kumbong2026nextconcept,
      title={Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation},
      author={Hermann Kumbong and Ishan S. Khare and Iddah Mlauzi and Andrew Woen and Charles Ding and E. Kelly Buchanan and Simran Arora and Azalia Mirhoseini and Christopher Ré},
      year={2026},
      eprint={2610.14162},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.14162},
}

Thanks to Hazy Research for releasing the NCM code and checkpoints. Meow :3