Eclipse-Senpai commited on
Commit
c973df6
·
verified ·
1 Parent(s): 71bff9e

Upload folder using huggingface_hub

Browse files
Files changed (3) hide show
  1. README.md +58 -108
  2. charts/comparison.svg +294 -266
  3. charts/effort.svg +236 -194
README.md CHANGED
@@ -12,79 +12,75 @@ tags:
12
 
13
  # min-spark
14
 
15
- min-spark is a 5.76M-parameter decoder-only language model trained on 10.01B
16
- tokens. It is the first release in the min-spark family of sub-10M models.
17
 
18
- The model runs a small weight-shared core repeatedly instead of stacking many
19
- distinct layers. Each pass through the core is one *loop*. The number of loops
20
- is a dial you control at inference time: fewer loops is faster, more loops is
21
- more capable. We expose three presets, called **effort levels**.
22
 
23
- | Effort | Compute | Use when |
24
- |--------|---------|----------|
25
- | min-spark-low | quickest | high-throughput, short replies |
26
- | min-spark-medium | balanced | general use (default) |
27
- | min-spark-high | highest quality | you want the best answer per token |
28
 
29
- ## How to use
30
 
31
- The model loads with standard Transformers. The repository is currently
32
- private, so authenticate first with `hf auth login` (or set `HF_TOKEN`).
 
 
 
 
 
 
 
 
 
33
 
34
  ```python
35
  from transformers import AutoModelForCausalLM, AutoTokenizer
36
 
37
  model = AutoModelForCausalLM.from_pretrained(
38
- "MinimaLabs/min-spark", trust_remote_code=True
 
39
  ).to("cuda")
40
  tokenizer = AutoTokenizer.from_pretrained(
41
- "MinimaLabs/min-spark", trust_remote_code=True
 
42
  )
43
 
44
  prompt = "The meaning of life is"
45
  inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
46
- out = model.generate(**inputs, effort="high", max_new_tokens=64)
47
- print(tokenizer.decode(out[0], skip_special_tokens=True))
 
 
 
 
48
  ```
49
 
50
- With `pipeline`:
51
 
52
  ```python
53
  from transformers import pipeline
54
 
55
  pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
56
- print(pipe(prompt, effort="high", max_new_tokens=64)[0]["generated_text"])
 
57
  ```
58
 
59
- Two constraints to know up front.
60
-
61
- - **Single sequence.** Generation runs one sequence at a time. Batched
62
- generation and left padding are not supported; right-padded batches work
63
- for scoring, which is what lm-eval uses.
64
- - **No KV cache yet.** It arrives in min-spark 1.1. For a 5.76M model this is
65
- rarely a bottleneck, but long generations re-encode the prefix each step.
66
-
67
- A transformers-free path is bundled as `generate.py`:
68
 
69
  ```bash
70
  python generate.py -p "The meaning of life is" -e high
71
  ```
72
 
73
- ## Benchmarks
74
 
75
- Scores are zero-shot, evaluated with lm-eval 0.4.12, batched and
76
- masked/train-consistent. BLiMP is accuracy; ARC-Easy, ARC-Challenge,
77
- HellaSwag, and PIQA are length-normalized accuracy; WikiText-2 is byte
78
- perplexity, lower is better.
79
 
80
- Across the three effort levels, min-spark sits in the mid-60s on BLiMP,
81
- low-to-mid 30s on ARC-Easy, and low-20s on ARC-Challenge and HellaSwag. On
82
- PIQA it lands in the low-50s. The best WikiText-2 byte perplexity is 2.77.
83
- Higher effort helps on BLiMP, where medium and high reach about 69; on the
84
- other tasks the three levels stay close, which means the common-sense tasks
85
- are less sensitive to loop count than grammar is.
86
 
87
- ### By effort level
 
 
 
 
88
 
89
  ![min-spark accuracy by effort](charts/effort.svg)
90
 
@@ -94,99 +90,53 @@ are less sensitive to loop count than grammar is.
94
  | min-spark-medium | 69.19% | 37.08% | 22.78% | 27.92% | 54.30% | 2.7747 |
95
  | min-spark-high | 69.18% | 37.08% | 22.87% | 27.91% | 54.35% | 2.7747 |
96
 
97
- ### Against other small models
98
 
99
- The chart below takes the best score min-spark reaches on each benchmark
100
- across effort levels, and compares it to published results from comparable
101
- sub-10M models: [GPT-S2-5M](https://huggingface.co/AxiomicLabs/GPT-S2-5M),
102
- [SLM-10M](https://huggingface.co/LiodonAI/SLM-10M), and
103
- [michel-nano-v2](https://huggingface.co/finnianx/michel-nano-v2). Scores are
104
- from those models' published cards, so some cells are blank where a model does
105
- not report that benchmark.
106
 
107
- ![min-spark best score vs small-model peers](charts/comparison.svg)
108
 
109
- min-spark matches or beats the comparison set on ARC-Easy (37.1 vs the next
110
- best 35.9) and ARC-Challenge (23.2 vs 23.5 for SLM-10M, above the rest). On
111
- HellaSwag and PIQA it sits in the same band as the peers. BLiMP is reported
112
- only by michel-nano-v2 (72.5), where min-spark lands at 69.2.
113
 
114
- ### Against KeyLM
115
 
116
- The prior release in this line, [KeyLM-75M](https://huggingface.co/MinimaLabs/KeyLM-75M),
117
- is a 75.25M-parameter model trained on 18B tokens, thirteen times the size of
118
- min-spark. min-spark reaches roughly comparable common-sense numbers at a
119
- fraction of the parameter budget. KeyLM reports ARC at 29.9 average and
120
- HellaSwag at 29.7; min-spark reaches 37.1 on ARC-Easy, 23.2 on ARC-Challenge,
121
- and 27.9 on HellaSwag. On PIQA, KeyLM scores 60.0 and min-spark reaches 54.4.
122
 
123
- The gap on PIQA and the ARC average is the honest picture: the looped
124
- architecture compresses a lot of the parameter-count advantage, but a 13x
125
- parameter difference still shows up on the harder relational tasks. What the
126
- comparison does show is that a 5.76M looped model is in the same general band
127
- as a 75M standard model on a large share of these tasks, at less than a tenth
128
- of the inference cost. KeyLM's reported numbers are zero-shot accuracy from
129
- its own card.
130
 
131
  ## Architecture
132
 
133
- min-spark is a tied-embedding, looped decoder. A byte-level BPE vocabulary of
134
- 4,096 feeds an embedding, a single prelude block, then a body of three
135
- distinct grouped-query-attention blocks that are run K times, then a coda
136
- block and a tied unembedding. Each loop pass adds a per-loop LoRA and a
137
- transient loop embedding; residuals use Deep-Delta projections. There are 11
138
- block applications per token.
139
-
140
- The model runs 11 block applications per token, which is why a 5.76M
141
- parameter count behaves more like a somewhat larger standard model.
142
 
143
- The interactive architecture graph is rendered by
144
- [hfviewer](https://hfviewer.com/), a graph viewer that traces the model and
145
- lets you zoom from the top-level shape down to individual blocks. The embed
146
- below is a summary card; open it to reach the full graph with block-level
147
- detail. The first render can take a few minutes while hfviewer traces the
148
- model.
149
 
150
  [![min-spark architecture](https://hfviewer.com/api/card.svg?source=MinimaLabs%2Fmin-spark&granularity=auto&animated=false)](https://hfviewer.com/MinimaLabs/min-spark)
151
 
152
- The three effort levels map to the internal loop count K: min-spark-low is
153
- K=2, min-spark-medium is K=3, min-spark-high is K=4.
154
 
155
  ## Training
156
 
157
- min-spark was trained on 10.01B tokens drawn from a filtered fineweb-edu and
158
- finemath-4plus mix, with a warmup-stable-decay schedule and a 20% cooldown.
159
- The released checkpoint is at step 182,875 of that run.
160
 
161
- `model.safetensors` sha256:
162
- `a121c6a316e54e490918d93d3d265460d45addbdad49b8448c514e5f61232c41`
163
 
164
- `tokenizer.json` sha256:
165
- `d671f013ebdb06a55e48d047f12621c3311310fdcc344683d434d294a909ac51`
166
 
167
- ## Reproducing the benchmarks
168
-
169
- The bundled `run_lmeval.py` reproduces the published numbers exactly. Run it
170
- with the effort you want to check:
171
 
172
  ```bash
173
- python run_lmeval.py --effort medium --tasks blimp,arc_easy,arc_challenge,hellaswag,piqa,wikitext
 
 
174
  ```
175
 
176
- Add `--limit N` to bound runtime. The stock `--model hf` path in lm-eval also
177
- works; the measured difference between it and the published methodology is
178
- recorded in `deltas.json`.
179
 
180
  ## Limitations
181
 
182
- - A 5.76M model is small. Factual recall, multi-step reasoning, and
183
- long-range coherence are weak.
184
- - The base model is not instruction-tuned, aligned, or filtered for safety.
185
- It completes text as-is, including harmful or offensive continuations.
186
- - Context length is 512 tokens.
187
- - Generation is single-sequence, and there is no KV cache yet.
188
- - Scores were collected at a 512-token context; longer contexts are not
189
- validated.
190
 
191
  ## License
192
 
 
12
 
13
  # min-spark
14
 
15
+ **min-spark is a 5.76M-parameter language model with native effort levels.** It introduces controllable depth to the sub-10M model class. The same checkpoint can produce a quick completion or spend more computation on the same prompt, selected with one inference argument.
 
16
 
17
+ The model was trained on 10.01B tokens. Its looped decoder reuses a compact transformer core across multiple passes, giving a small model the computation of a deeper network while keeping the parameter count fixed.
 
 
 
18
 
19
+ ## Introducing native effort levels
 
 
 
 
20
 
21
+ Effort levels have traditionally been associated with large reasoning models, where they adjust a thinking budget or modify the surrounding prompt. min-spark brings the idea directly into the language model. Each effort level changes the model's internal computation by selecting a different number of passes through its shared core.
22
 
23
+ | Effort | Character | Recommended use |
24
+ |---|---|---|
25
+ | `low` | Fastest | High-throughput completion |
26
+ | `medium` | Balanced | General generation |
27
+ | `high` | Most compute | Highest available quality |
28
+
29
+ Effort is a generation-time choice. All three levels use the same weights and tokenizer.
30
+
31
+ ## Usage
32
+
33
+ min-spark is compatible with Transformers and requires remote code loading.
34
 
35
  ```python
36
  from transformers import AutoModelForCausalLM, AutoTokenizer
37
 
38
  model = AutoModelForCausalLM.from_pretrained(
39
+ "MinimaLabs/min-spark",
40
+ trust_remote_code=True,
41
  ).to("cuda")
42
  tokenizer = AutoTokenizer.from_pretrained(
43
+ "MinimaLabs/min-spark",
44
+ trust_remote_code=True,
45
  )
46
 
47
  prompt = "The meaning of life is"
48
  inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
49
+ outputs = model.generate(
50
+ **inputs,
51
+ effort="high",
52
+ max_new_tokens=64,
53
+ )
54
+ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
55
  ```
56
 
57
+ The same model works with the Transformers pipeline API:
58
 
59
  ```python
60
  from transformers import pipeline
61
 
62
  pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
63
+ result = pipe(prompt, effort="high", max_new_tokens=64)
64
+ print(result[0]["generated_text"])
65
  ```
66
 
67
+ A lightweight Transformers-free generation script is included in the repository:
 
 
 
 
 
 
 
 
68
 
69
  ```bash
70
  python generate.py -p "The meaning of life is" -e high
71
  ```
72
 
73
+ Generation currently runs one sequence at a time. Right-padded batches are supported for evaluation. KV caching is planned for a future release, so long generations recompute the prompt at each step. The context window is 512 tokens.
74
 
75
+ ## Evaluation
 
 
 
76
 
77
+ Scores below come from zero-shot evaluation with lm-eval 0.4.12. BLiMP uses accuracy. ARC-Easy, ARC-Challenge, HellaSwag, and PIQA use length-normalized accuracy. WikiText-2 is reported as byte-level perplexity, where lower is better.
 
 
 
 
 
78
 
79
+ min-spark reaches 69.19% on BLiMP at medium effort. ARC-Easy reaches 37.08%. ARC-Challenge reaches 23.21%. HellaSwag reaches 27.92%. PIQA reaches 54.35%. Its best WikiText-2 byte perplexity is 2.7747.
80
+
81
+ ### Effort levels
82
+
83
+ The table shows how the model responds to additional internal computation. Grammar improves most clearly from low to medium effort. The common-sense tasks remain close across the three settings.
84
 
85
  ![min-spark accuracy by effort](charts/effort.svg)
86
 
 
90
  | min-spark-medium | 69.19% | 37.08% | 22.78% | 27.92% | 54.30% | 2.7747 |
91
  | min-spark-high | 69.18% | 37.08% | 22.87% | 27.91% | 54.35% | 2.7747 |
92
 
93
+ ### Comparison with small models
94
 
95
+ The comparison places min-spark alongside published results for GPT-S2-5M, SLM-10M, and michel-nano-v2. It covers the benchmarks reported across this group.
 
 
 
 
 
 
96
 
97
+ ![min-spark compared with small-model peers](charts/comparison.svg)
98
 
99
+ min-spark reaches 37.08% on ARC-Easy, the highest score in this comparison. Its 23.21% ARC-Challenge result is close to SLM-10M at 23.46%. HellaSwag reaches 27.92%, and PIQA reaches 54.35%. michel-nano-v2 records the highest BLiMP result in the group at 72.52%, while min-spark reaches 69.19%.
 
 
 
100
 
101
+ ### Comparison with KeyLM
102
 
103
+ KeyLM is a 75.25M-parameter language model trained on 18B tokens. min-spark uses less than one thirteenth of its parameter count while reaching a similar range on several small-model evaluations.
 
 
 
 
 
104
 
105
+ KeyLM reports 29.9% on its ARC average, 29.7% on HellaSwag, and 60.0% on PIQA. min-spark reaches 37.08% on ARC-Easy, 23.21% on ARC-Challenge, 27.92% on HellaSwag, and 54.35% on PIQA. The comparison highlights the value of repeated computation in a compact model. min-spark closes much of the size gap on these tasks while retaining a 5.76M parameter footprint.
 
 
 
 
 
 
106
 
107
  ## Architecture
108
 
109
+ min-spark uses a tied-embedding, looped decoder with 5,758,572 parameters. Its 4,096-token byte-level BPE vocabulary feeds an embedding with model width 288. A prelude block prepares the hidden state, followed by a body of three distinct grouped-query attention blocks. The body is repeated according to the selected effort level before a coda block produces the final representation. The input embedding and output projection share weights.
 
 
 
 
 
 
 
 
110
 
111
+ Each pass includes loop-specific low-rank adaptation and a transient loop embedding. Deep-Delta residual projections provide the recurrent update. Across the full computation, the model applies 11 transformer blocks per token. The internal loop count is K=2 for low effort, K=3 for medium effort, and K=4 for high effort.
 
 
 
 
 
112
 
113
  [![min-spark architecture](https://hfviewer.com/api/card.svg?source=MinimaLabs%2Fmin-spark&granularity=auto&animated=false)](https://hfviewer.com/MinimaLabs/min-spark)
114
 
115
+ Open the image to explore the full architecture graph in hfviewer, including the block-level structure and parameter flow.
 
116
 
117
  ## Training
118
 
119
+ min-spark was pretrained as a decoder-only base model on 10.01B tokens. The corpus combines filtered FineWeb-Edu with Finemath-4plus. Training used a warmup, stable learning-rate phase, and decay schedule with a 20% cooldown. The released checkpoint is the final checkpoint selected from this training run.
 
 
120
 
121
+ The model is released for research and experimentation with compact language models, weight-shared decoders, and controllable inference compute.
 
122
 
123
+ ## Reproducing the evaluation
 
124
 
125
+ The evaluation runner is included in the repository. It supports each effort level and the benchmark suite used for this card.
 
 
 
126
 
127
  ```bash
128
+ python run_lmeval.py \
129
+ --effort medium \
130
+ --tasks blimp,arc_easy,arc_challenge,hellaswag,piqa,wikitext
131
  ```
132
 
133
+ Use `--limit N` to run a smaller evaluation during development.
 
 
134
 
135
  ## Limitations
136
 
137
+ min-spark is a small base language model. It is not instruction-tuned and does not provide conversational alignment or safety filtering. Factual recall, multi-step reasoning, and long-form coherence are limited by its scale and training objective.
138
+
139
+ The model supports a 512-token context window. Generation accepts one sequence at a time and currently has no KV cache. Results were collected with a 512-token context and should not be assumed to transfer to longer inputs.
 
 
 
 
 
140
 
141
  ## License
142
 
charts/comparison.svg CHANGED
charts/effort.svg CHANGED