bench table + honest verdict: KL gains do not convert on wikitext; ~-.02 task acc both arms, mechanism differences survive as the result
Browse files
README.md
CHANGED
|
@@ -59,10 +59,25 @@ gauge that takes an absolute value.
|
|
| 59 |
|
| 60 |
## Standardized bench
|
| 61 |
|
| 62 |
-
0-shot
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 66 |
|
| 67 |
## Artifacts
|
| 68 |
|
|
|
|
| 59 |
|
| 60 |
## Standardized bench
|
| 61 |
|
| 62 |
+
0-shot via lm-eval-harness (acc_norm where defined, else acc):
|
| 63 |
+
|
| 64 |
+
| config | arc_easy | hellaswag | piqa | winogrande | mean |
|
| 65 |
+
|---|---|---|---|---|---|
|
| 66 |
+
| stock 0.5B | 0.5888 | 0.5251 | 0.7051 | 0.5564 | **0.5939** |
|
| 67 |
+
| + aleph | 0.5960 | 0.4669 | 0.6779 | 0.5675 | **0.5771** |
|
| 68 |
+
| + mlp | 0.5968 | 0.4743 | 0.6746 | 0.5462 | **0.5730** |
|
| 69 |
+
| teacher 1.5B | 0.7584 | 0.6822 | 0.7601 | 0.6290 | **0.7074** |
|
| 70 |
+
|
| 71 |
+
**The honest verdict**: the wikitext-KL distillation did not convert to
|
| 72 |
+
task capability — both arms trade ~.02 of 0-shot accuracy for their
|
| 73 |
+
~25% KL improvement, *identically*, closing none of the 0.5B→1.5B gap
|
| 74 |
+
(.11). The finding indicts the distillation **corpus/objective**, not
|
| 75 |
+
either mechanism: matching the teacher's distribution on encyclopedic
|
| 76 |
+
text is not the same thing as acquiring its task behavior. The
|
| 77 |
+
mechanistic differences between the arms (placement election, signed
|
| 78 |
+
regime, both fully detachable) survive as the real result of this
|
| 79 |
+
round; an instruct-domain distillation corpus is the named next
|
| 80 |
+
revision.
|
| 81 |
|
| 82 |
## Artifacts
|
| 83 |
|