AbstractPhil commited on
Commit
48f2b42
·
verified ·
1 Parent(s): f7a1e49

bench table + honest verdict: KL gains do not convert on wikitext; ~-.02 task acc both arms, mechanism differences survive as the result

Browse files
Files changed (1) hide show
  1. README.md +19 -4
README.md CHANGED
@@ -59,10 +59,25 @@ gauge that takes an absolute value.
59
 
60
  ## Standardized bench
61
 
62
- 0-shot `arc_easy / hellaswag / piqa / winogrande` (lm-eval-harness),
63
- four rows: stock 0.5B, +aleph, +mlp, and the 1.5B teacher as the
64
- ceiling. **Running now — the table lands in
65
- `adopt0/adopt0_bench.json` and here when complete.**
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
66
 
67
  ## Artifacts
68
 
 
59
 
60
  ## Standardized bench
61
 
62
+ 0-shot via lm-eval-harness (acc_norm where defined, else acc):
63
+
64
+ | config | arc_easy | hellaswag | piqa | winogrande | mean |
65
+ |---|---|---|---|---|---|
66
+ | stock 0.5B | 0.5888 | 0.5251 | 0.7051 | 0.5564 | **0.5939** |
67
+ | + aleph | 0.5960 | 0.4669 | 0.6779 | 0.5675 | **0.5771** |
68
+ | + mlp | 0.5968 | 0.4743 | 0.6746 | 0.5462 | **0.5730** |
69
+ | teacher 1.5B | 0.7584 | 0.6822 | 0.7601 | 0.6290 | **0.7074** |
70
+
71
+ **The honest verdict**: the wikitext-KL distillation did not convert to
72
+ task capability — both arms trade ~.02 of 0-shot accuracy for their
73
+ ~25% KL improvement, *identically*, closing none of the 0.5B→1.5B gap
74
+ (.11). The finding indicts the distillation **corpus/objective**, not
75
+ either mechanism: matching the teacher's distribution on encyclopedic
76
+ text is not the same thing as acquiring its task behavior. The
77
+ mechanistic differences between the arms (placement election, signed
78
+ regime, both fully detachable) survive as the real result of this
79
+ round; an instruct-domain distillation corpus is the named next
80
+ revision.
81
 
82
  ## Artifacts
83