Release v22: requests behind an opening no, bare topics, what's new in X, lowercase names

.gitattributes

@@ -36,3 +36,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
 grpo-best-q8_0.gguf filter=lfs diff=lfs merge=lfs -text
 sft-v19-q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+sft-v22-q8_0.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -89,19 +89,20 @@ These are real outputs from the released weights, greedy:
 | in | out |
 |---|---|
 | `Spanish: no busques la tienda de ropa, dime la dosis de paracetamol` | `dime la dosis de paracetamol` |
-| `Italian: scusa il disturbo, mi diresti che tempo fa domani a Bologna?` | `dimmi che tempo fa domani a Bologna` |
 | `English: book me a table for friday and also cancel my dentist` | `book me a table for friday`<br>`cancel my dentist` |
+| `Italian: no aspetta, riassumimi cosa ci siamo detti` | `riassumi cosa ci siamo detti` |
 | `Italian: guarda un po', questo sacchetto della spesa è tutto rotto.` | `NO_REQUEST` |
+| `Italian: ciao come stai` | `NO_REQUEST` |
 
-A negation is dropped, an apology is dropped, one message asking for two
-things becomes two lines, and something that is not a request at all
-answers with the sentinel.
+A negation is dropped, one message asking for two things becomes two lines,
+a request behind an opening "no" is kept, and something that is not a
+request at all answers with the sentinel.
 
 ## What is in this repository
 
 | file | size | what it is |
 |---|---|---|
-| `sft-v19-q8_0.gguf` | 300 MB | what the engine serves, through llama-server |
+| `sft-v22-q8_0.gguf` | 292 MB | what the engine serves, through llama-server |
 | `model.safetensors` | 536 MB | the same weights fused, BF16, 236 tensors |
 | `model.safetensors.index.json` | | the index over that single file |
 | `tokenizer.json`, `tokenizer_config.json` | 33 MB | the tokenizer as the fuse wrote it |
@@ -109,127 +110,82 @@ answers with the sentinel.
 | `chat_template.jinja` | | present because the fuse writes it — **not the surface this model reads**, see above |
 
 The GGUF's sha256 is
-`33b6cc73c7d306c9c81aa96c5a8ddb9dbfd46a5ac5d0388a2de45b5229928c56`.
+`58b750b4bba4ed9cf1ea30316f1c5ad131316efaedd56cfd9440b488b2ae88f4`.
 The engine pins it and refuses anything else, which is what makes a routing
 decision reproducible.
 
-Both files carry the same weights: the fused final adapter of training run
-v19, adapter sha256 `1082985953ed6c33…`, quantised to Q8_0. Q8_0 and not
-Q4: measured on this seat, at temperature 0 a Q8_0 is stable to the byte
-across runs and a Q4 is not.
+Both files carry the same weights: checkpoint 21,000 of training run v22,
+chosen on the validation split, quantised to Q8_0. Q8_0 and not Q4:
+measured on this seat, at temperature 0 a Q8_0 is stable to the byte across
+runs and a Q4 is not.
 
 ## What it is measured to do
 
-Measured against the two things worth measuring against: the model it was
-fine-tuned from, and doing no repair at all. Two benches over different
-corpora, because a model that wins one of them is a model chosen by which
-table got printed.
+Every number below compares these weights with the previous release of this
+model (training run v19), both served by the same llama-server build with
+the same greedy call, on the same day. Pairs are counted where the two
+disagree, which is where the information is, with an exact two-sided sign
+test.
 
-**400 routing tasks over 343 graphs**, of which 78 ask for more than one
-capability and 160 name a day. These exact released weights against the
-untrained base, every column the bench reports:
+**481 real questions people asked the engine**, in Italian, held out from
+every training and selection step and routed through the full path the
+engine takes (the router, the repair when the route is unsure, admission
+only when every line is confident):
 
-| | this model | [`gemma-3-270m`](https://huggingface.co/google/gemma-3-270m), untrained |
+| | this model | previous release |
 |---|---|---|
-| **exact**, every capability the question asks for, and no other | **32** | 24 |
-| **safe**, a proper subset reached, nothing wrong run | 6 | 1 |
-| **abstained**, nothing executed at all | 298 | 197 |
-| **wrong execution**, something ran that should not have | **1** | 0 |
-| **answered in the wrong language** | **6** | **123** |
-| **lost the day** the question named, of 160 | 57 | 55 |
-| **echoed the question back** instead of restating it | 64 | 106 |
-| of those echoes, ones that still routed exactly | 9 | 20 |
-| **compound questions answered whole**, of 78 | 0 | 0 |
-
-Read the columns together rather than one at a time.
-
-The **language** column is what the fine-tune buys and it is the largest
-effect in any table here: 123 becomes 6.
-
-The **abstention** column is the honest cost. This model refuses far more
-often than the base does, 298 against 197. It abstains rather than guess,
-and abstaining is why its wrong executions stay at one. A rewriter that
-guesses would score better on exact and worse on everything that matters.
-
-The one wrong execution is worth reading whole. *Vorrei sapere quanto devo
-per la visita di martedì scorso. Non ho ricevuto la fattura.* — gold
-`pagamento_fattura` — came back as *dimmi quanto devo per la visita di
-martedì scorso*: the date kept, the second sentence gone, and with it the
-one word that decided. `visita` then carried the line confidently to
-`prenota_visita` (top 0.392, margin 0.148, both over the served bars),
-where the original message had put `pagamento_fattura` first but without
-the margin to execute (0.086) — which is why it was in repair at all. The
-failure is a dropped clause, not an invented action: the model wrote
-nothing the person did not say. Recovered by re-running the bench, which
-is greedy and seeded, so the same 400 tasks reproduce every column of
-this table to the digit.
-
-The **echo** column says how often the model gave the question back
-unchanged. It halves, 106 to 64, which is the fine-tune learning that
-restating is the job.
-
-The **date** column barely moves, 55 to 57, and the **compound** column does
-not move at all: neither the base nor this model answers a single one of
-the 78 questions asking for two capabilities on this bench. Those are the
-two things the training did not buy.
-
-What bounds the compound number: in the engine, every line of the
-restatement is routed on its own, and nothing decides where to split
-except the router's own scores on each line. A message restated into one
-right line and one missing one still reaches the capability it named —
-that is what the **safe** row counts. *Answered whole* is the
-all-or-nothing reading of that; the seat's failure mode on a compound is
-a partial answer, not a wrong one.
-
-**39 held-out questions on a live workspace**, written by three blind
-judges against the capabilities in it, in two languages:
-
-| | exact |
-|---|---|
-| **this model** | **16** |
-| routing the question as written, no repair | 12 |
-| `google/gemma-3-270m`, untrained | 9 |
+| **exact**, every capability asked and no other | **159** | 138 |
+| **safe**, a proper subset reached, nothing wrong run | 10 | 10 |
+| **wrong execution** | 12 | 12 |
+| questions where only this model is right / only the previous one is | **23 / 2** | p = 0.00002 |
+
+The gain is where it was aimed: requests behind an opening "no", "ok" or
+"I didn't get it", bare statistical topics, "what's new in X", and names
+written in lowercase, which the previous release answered with
+`NO_REQUEST`. Of the 118 questions it still fails on a capability it can
+reach, 66 are decided by the router, not the rewrite: the right capability
+comes first but below the bars. 38 route elsewhere and 13 are still
+answered with the sentinel.
+
+**900 routing tasks over graphs from another corpus**, five languages, two
+in five naming a day:
+
+| | this model | previous release |
+|---|---|---|
+| exact | 56 | 59 |
+| compound questions answered whole | 4 | 5 |
+| safe | 17 | 11 |
+| abstained | 696 | 713 |
+| wrong execution | **4** | 4 |
+| answered in the wrong language | 2 | 1 |
+| **lost the day** the question named | **125** | 112 |
+| echoed the question back | 123 | 150 |
+
+Disagreements split 26 to 39 against this model, p = 0.14: no difference
+on this bench is significant, and the day column is the one to read.
+
+**100 probes written before training**, 20 per family, none equal to a
+training row or to a real question above: 52 of 80 requests routed to the
+right capability with every name kept (previous release 43), and 20 of 20
+non-requests answered `NO_REQUEST` (previous 20).
+
+**The held-out test split** of the training data, 963 rows, whole
+completion equal to the target: 263 (previous 260).
 
-The base model scoring **below** "no repair at all" is the whole argument
-for fine-tuning something for this seat: an untrained 270M put here makes
-routing worse than leaving the question alone. Of the 10 compound questions
-in that set — one message asking for two things — this model answers 2
-whole.
+## What it does not fix, which a card naming only the gains would hide
 
-**Its own training gate**, on 1000 held-out examples: 261 exact (26.1%),
-927 of 1000 answers well formed, and the language-balance gate passed with
-a 7-point gap between English and Italian against a 10-point bar.
+**Dates.** On the 900-task bench it loses the day the question named more
+often than the previous release, 125 against 112, and the restatement reads
+as a clean request with the day simply gone. If the day matters to your
+routing, measure that before you serve this.
 
-## What it does not fix, which a card naming only the gains would hide
+**It can drop the verb of a request.** `Italian: scusa il disturbo, mi
+diresti che tempo fa domani a Bologna?` comes back as `dimmi se domani a
+Bologna è previsto`: "che tempo fa" is gone.
 
-**The sentinel does not catch every non-request.** `guarda un po', questo
-sacchetto della spesa è tutto rotto` answers `NO_REQUEST`, but `ciao come
-stai` comes back verbatim rather than refused. If nothing downstream checks
-that the restatement is a request, a greeting will be routed.
-
-**It sometimes rewrites a word rather than restating a sentence.** Measured:
-`Bitte reserviere mir einen Platz im Klettergarten für Freitag` comes back
-as `reserviere mir einen Platz im Kletterpark für Freitag` — `Klettergarten`
-became `Kletterpark`. The restatement is only ever compared against
-capability descriptions and never executed, which is why this is tolerable
-here; in any design where the rewrite is acted on, it is not.
-
-**Dates.** In the 400-task run above, 160 of the tasks name a day. This
-model loses the day in **57 of them** — more than a third. It is the
-weakest column it has, and it is weak in a way that does not announce
-itself: the restatement reads as a clean request, with the "yesterday" or
-the "before Easter" simply gone. If the day matters to your routing,
-measure that before you serve this.
-
-**It is not distinguishable from its own neighbours.** Ten checkpoints of
-this run were compared on the same 400 tasks pair by pair rather than by
-totals — counting only the tasks where two checkpoints disagree, which are
-the only ones carrying information. Not one difference reaches
-significance: the widest gap in the table is 7 tasks to 1, p = 0.07. With
-around thirty exact answers out of four hundred the natural fluctuation is
-about five, which is most of the ranking. Treat the order inside a run as
-noise: a checkpoint one or two points ahead of another inside the same run
-has not been shown to be better than it.
+**It still refuses some requests.** `Italian: che novità ci sono in Django
+6?` answers `NO_REQUEST`. Of the 80 request probes, 10 still come back as
+the sentinel, 15 route to another capability and 3 lose a name.
 
 ### What bounds all of this
 
@@ -250,8 +206,8 @@ restatement's to lose.
 
 **A restatement that does not win by enough margin routes nothing.** The
 router applies a margin and a floor, and below them the system abstains and
-asks for a rephrasing. That is why the wrong-execution column is 1 in 400
-while the abstention column is 298: the failure mode is refusing, not acting
+asks for a rephrasing. That is why the wrong-execution column is 4 in 900
+while the abstention column is 696: the failure mode is refusing, not acting
 wrongly.
 
 **The descriptions are the other half.** What a question is matched against
@@ -276,7 +232,7 @@ handed an answer with no account of how it was reached.
 Served with llama-server, reached by the engine as one of four model seats:
 
 ```
-llama-server -m sft-v19-q8_0.gguf --host 127.0.0.1 --port 9250 -ngl 99 -c 2048 --jinja
+llama-server -m sft-v22-q8_0.gguf --host 127.0.0.1 --port 9250 -ngl 99 -c 2048 --jinja
 ```
 
 It is loaded only when routing is genuinely unsure, never as the default
@@ -284,7 +240,7 @@ path, and released after an idle window.
 
 ### On CPU alone
 
-Measured on an Apple M-series machine with `-ngl 0 -t 4`: no GPU offload at
+Measured on this architecture at this size, on an Apple M-series machine with `-ngl 0 -t 4`: no GPU offload at
 all, four threads, and nothing in the server log naming Metal or a GPU.
 
 | | latency | tokens/s |
@@ -296,7 +252,7 @@ all, four threads, and nothing in the server log naming Metal or a GPU.
 **762 MB resident** while serving.
 
 ```
-llama-server -m sft-v19-q8_0.gguf --host 127.0.0.1 --port 9250 -ngl 0 -t 4 -c 2048
+llama-server -m sft-v22-q8_0.gguf --host 127.0.0.1 --port 9250 -ngl 0 -t 4 -c 2048
 ```
 
 What this does and does not establish: it runs usefully with no accelerator,
@@ -311,7 +267,21 @@ under heat while these did not.
 **[`Gramscii-IT/semantic-repair-routing`](https://huggingface.co/datasets/Gramscii-IT/semantic-repair-routing)**
 — published, and everything below can be recounted from it.
 
-**84,819 pairs**: 82,819 train, 1,000 validation, 1,000 test. Each pair is a
+The base is that dataset's **84,819 pairs** (82,819 train, 1,000 validation,
+1,000 test), cleaned by deterministic rules before training: template
+leaks, changed values, English targets for other languages, repeated lines
+and generic "category" lines removed, which leaves 80,297 train, 966
+validation and 963 test rows. To them this release adds **2,515 rows** in
+Italian and English, generated by Qwen3.6-35B-A3B and kept only when every
+line's top capability on the engine's own roster was the one the row was
+written for, every number and name of the message survived, and no row
+shared most of its trigrams with a probe, a real question or another row.
+They cover a request about the conversation itself, a bare statistical
+topic, "what's new in X", names in lowercase or with a typo, long noisy
+messages carrying two or three requests, and acknowledgements that ask for
+nothing. Each is repeated twice in training, about 6% of it.
+
+The base dataset, as published: Each pair is a
 message somebody could plausibly write and the requests inside it restated
 plainly, one per line.
 
@@ -389,9 +359,10 @@ and where they stopped being.
 
 ## Training
 
-Fine-tuned from [`google/gemma-3-270m`](https://huggingface.co/google/gemma-3-270m) with `mlx_lm lora`, full fine-tune.
-The released weights are the run's final adapter, chosen by a gate on 1,000
-held-out examples rather than by the best number on a validation curve.
+Fine-tuned from [`google/gemma-3-270m`](https://huggingface.co/google/gemma-3-270m) with `mlx_lm lora`, full fine-tune, all
+18 blocks. The released weights are checkpoint 21,000, chosen among the
+run's checkpoints by exact answers on the validation split (301 of 1,077),
+never on the test split or the benches above.
 
 ## Where all of this happened
 
@@ -409,16 +380,15 @@ One desktop machine, and nothing left it.
 | the fine-tune | |
 |---|---|
 | method | `mlx_lm lora`, full fine-tune |
-| iterations | 20,704 |
-| tokens seen | 2,383,728 |
-| peak memory | 7.3 GB |
-| speed | 5.6 iterations/s, 610 tokens/s |
-| wall clock | 11:15 to 12:42, about 90 minutes |
+| iterations | 21,276 |
+| tokens seen | 2,327,566 |
+| peak memory | 5.4 GB |
+| speed | about 6 iterations/s |
+| wall clock | 08:27 to 09:30, about an hour |
 
 Generators and judge ran on that same machine as local llama.cpp servers on
 loopback. No message here was written by a hosted API, and none was sent to
-one to be judged. The unified memory is why: a 35B generator and a
-fine-tune only share a desk when CPU and GPU read the same 64 GB.
+one to be judged.
 
 A model that decides where a question goes does not need a cluster or a
 month. It needs a narrow job, data built for it, and a bench honest enough

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f3a2d08287ab45528b19d6fa512f7bbcd2e6718d7264d0fce4d955c1a140af40
+oid sha256:eb85980247d466cb2b5a6e1b32d40a35f6b0551b968c344a19449656cfda3005
 size 536222768

sft-v19-q8_0.gguf → sft-v22-q8_0.gguf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:33b6cc73c7d306c9c81aa96c5a8ddb9dbfd46a5ac5d0388a2de45b5229928c56
-size 299748064
+oid sha256:58b750b4bba4ed9cf1ea30316f1c5ad131316efaedd56cfd9440b488b2ae88f4
+size 291545056