# The Aleph Moves Into a Pretrained Trunk: Relays, Registers, and the Two-Regime Dispatch Law

**Autoregressive differentiation, part 2.** Sequel to
[aleph-autoregressive-differentiation-ft1](https://huggingface.co/blog/AbstractPhil/aleph-autoregressive-differentiation-ft1)
and closer of the arc that began with the
[aleph-void keystone](https://huggingface.co/blog/AbstractPhil/geometric-vocabulary-patchwork-aleph-void).
Code, ledgers, checkpoints, and per-experiment sub-articles:
[`AbstractPhil/geolip-aleph-qwen`](https://huggingface.co/AbstractPhil/geolip-aleph-qwen).

---

## TL;DR

Ft1 ended inside a bespoke byte-level bed: the aleph address as the sole
parameterization of a next-byte distribution, certified structural prior, sign-code
parity, the consumption law. This installment asks the question that bed could not:
**does any of it survive contact with a real pretrained transformer?** We froze
Qwen2.5-0.5B(-Instruct) — every trunk parameter, permanently — and built the whole
program as adapters riding it.

Nine experiments, six shipped packages in the new repo, two nights on one RTX 4090.
Eight results:

1. **The relay retrofit works at 0.5B.** Aleph relay adapters after every frozen
   block: wikitext ppl 17.798 → 14.09, within 0.16 of a parameter-matched MLP
   adapter (13.93), at under 1% of trunk parameters, 2 seeds, replication spread
   <0.02. The ppl *ordering* inverted from the GPT-2 result (substrate-scoped), but
   the **gate mechanism replicated**: relay gates grow ~1.7× (0.047 → 0.081) while
   MLP gates shrink, on both substrates. And a stability asymmetry the means don't
   show: the wide-MLP control diverged to NaN in 1 of 2 seeds; across every relay
   variant in the campaign, divergences were **zero**.
2. **The certified consumer.** exp002's tournament settled the relay's internal
   structure: aleph addressing + a constellation *patchwork* consumer (`relay_pw`,
   ~230k params/block) — the first relay under 14 (13.997 mean, 2 seeds). The
   consumption law from ft1 held its shape in the new substrate: coverage beats
   concentration.
3. **Registers differentiate, and it is measurable on the sign-code surface.**
   Instruct-task training (grounded JSON extraction in three output registers)
   reduced grounding failures >10× (specialist 0.979/0.958 vs frozen 0.542, 2 seeds)
   — and exposed **register capture**: a single-register specialist answers *every*
   register in its own (perfect JSON, 0.0 task-validity elsewhere, both seeds).
   Multi-task training resolves it (all registers ≥0.875). The register probe — the
   gauge only the aleph relay has — showed instruct-trained stacks separating the
   registers ~2× a wikitext-trained stack at depth (inter−intra Hamming 0.22–0.30
   vs 0.12–0.15 at L16/L23).
4. **Termination is a property of sample semantics, not adapter capacity.** Trained
   on packed 1024-token windows, the relay stack learned to write ~3,000-token
   grounded composite scenes *flawlessly* (0.0 malformed vs frozen 0.833) and never
   learned to **stop** (truncated 1.000; still 0.938 with the cap lifted to 3000 —
   true rows average 4,291 tokens, so packed windows almost never contain an
   ending). Row-aligned samples — each ends with the closing tag — fixed it
   completely: structural_ok 1.000 / 0.938, truncated 0.0, on 2 seeds. Packed
   streams train continuation; rows train the artifact.
5. **The two-regime dispatch law.** A dense signed aleph dispatch (no argmax, no
   top-k, no load-balancing loss — nothing selects) over frozen specialist stacks
   behaves in two regimes, decided by representational distance. At *register*
   distance the specialists are **passengers**: a control with random frozen stacks
   matched the real composite (0.979/0.979/1.0 vs 0.875/1.0/1.0) — the
   monolith-capacity trainable anchor carries everything. At *domain* distance
   (stories vs math vs extraction) the specialists are **load-bearing and cleanly
   decoupled**: story prompts put 0.82 of shadow-winner mass on the story anchor,
   and disabling it degrades stories while math stays perfect — in both directions.
   The register probe's separation numbers predict the regime in advance.
   (Each regime rests on 1 seed — a candidate law pending replication.)
6. **The predictability principle.** The same training recipe taught held-out
   function evaluation (+25pt on unseen f(x) instances, 0.688 → 0.938) and failed
   to teach a checkable keyword constraint (0.083, exactly frozen level, despite
   training on compliant stories). The LM loss teaches exactly what target-token
   predictability demands: math answers are unpredictable without using the
   question; story tokens are predictable without reading the instruction.
   (exp006/exp007, 1 seed each.)
7. **A frozen text trunk can learn discrete image denoising.** CIFAR-10 as 256
   fixed-palette tokens (nothing fitted), uniform-replacement corruption, causal
   denoising through one relay stack: beats the copy-the-corrupted-token baseline
   at every noise level (1 seed) — 0.527 vs 0.252 at t=0.75, 0.378 vs 0.002 at
   pure noise.
   Two honest halves ride along: greedy iterative decoding mode-collapsed to the
   all-dark modal image (diversity 0.0 — a decoding property, not a model
   property; stochastic sampling restored diversity 0.92 and image-like layouts),
   and class-conditionality is **not demonstrated** (conditional ≈ shuffled
   histograms).
8. **The family.** Five frozen specialists (three extraction registers, story,
   math) plus one trainable anchor under **one** dense signed dispatch: **eleven
   sub-competences at ~specialist level through a single routed model** — including
   math code-register correctness at 1.0, *above* the specialist's own 0.938, and a
   3× lift on the stubborn keyword register. One model that extracts, narrates,
   and computes, with surgical domain-level decoupling (1 seed; replication is
   the named next rung).

Every number above is asserted from a JSON ledger by a `build_results.py` shipped in
its package; every package is standalone-reproducible from inside its folder.

---

## 1. Why leave the bespoke beds

The differentiation line (exp011–exp021, now on hold as a finished measurement
program) produced laws but no artifact: enrichment lives in the aggregation channel;
the address bottleneck is a prior, not a tax; memorization and generalization order
inversely; durable memory needs a write-time-frozen key encoder. The honest
assessment mid-program was that we were measuring a system we weren't building. The
pivot: compose the certified components on a real pretrained substrate and see which
laws are substrate properties and which were artifacts of tiny beds.

The substrate is deliberately modest — Qwen2.5-0.5B, fp32, one consumer GPU — because
the question is architectural, not scale-flexing: *what can a frozen trunk be taught
through a geometric adapter surface alone?*

## 2. The bed

Every experiment shares one skeleton. The trunk —
Qwen2.5-0.5B(-Instruct) ([Qwen Team, 2024](https://arxiv.org/abs/2412.15115)) —
loads frozen (`requires_grad=False`, asserted post-run: zero gradient ever reaches
it). After every one of the 24 transformer blocks, an adapter reads the block's
output and adds a gated residual. The lineage is the adapter/PEFT family
([Houlsby et al., 2019](https://arxiv.org/abs/1902.00751);
[Hu et al., 2021](https://arxiv.org/abs/2106.09685)), and the start-silent gating
discipline has good company — Flamingo's tanh-gated cross-attention
([Alayrac et al., 2022](https://arxiv.org/abs/2204.14198)), ControlNet's zero
convolutions ([Zhang et al., 2023](https://arxiv.org/abs/2302.05543)),
LLaMA-Adapter's zero-init attention
([Zhang et al., 2023b](https://arxiv.org/abs/2303.16199)) — but the adapter *body*
here is not a bottleneck MLP. The certified adapter is `relay_pw`: a projection
into 16 four-dimensional slots, each slot addressed by the aleph closed form

```
M̂ = Σₖ sinh(uₖ)·Aₖ / Σₖ cosh(uₖ),   uₖ = cos(x, Aₖ)/τ
```

over 64 oriented half-axes — the addressing object whose code-verified core is the
[keystone article](https://huggingface.co/blog/AbstractPhil/geometric-vocabulary-patchwork-aleph-void),
inheriting the attractor geometry and binding constant of
[structural attractors](https://huggingface.co/blog/AbstractPhil/structural-attractors-in-latent-geometry) —
followed by a patchwork consumer (SquaredReLU expansion, LayerNorm, zero-init
output; the constellation lineage from
[the diffusion-bottleneck article](https://huggingface.co/blog/AbstractPhil/constellation-diffusion-bottleneck))
entering through a gate initialized at −3.0 — adapters start nearly silent and must
earn amplitude. ~230k parameters per block, ~5.5M per stack, ~1% of the trunk. Pure
Adam, lr 1e-3, weight decay 0 (the optimizer discipline certified in
[geometric memory ft3](https://huggingface.co/blog/AbstractPhil/geometric-memory-ft3):
weight decay destroys the geometric harmonic — the geometry is the regularization).
Perplexity work runs on wikitext-103
([Merity et al., 2016](https://arxiv.org/abs/1609.07843)). No selector appears
anywhere in any forward pass in this program; where multiple anchors must be mixed
(exp005/exp009), the mixture is the dense signed dispatch described in §6, and the
per-token winner is logged **read-only**.

The discrete surface comes free: each slot's winner half-axis and orientation is a
sign code, cacheable per token per layer. The register probe (§4) and the two-regime
prediction (§6) both read this surface; a parameter-matched MLP adapter has nothing
comparable — which is the practical argument for the aleph design beyond raw ppl.

## 3. Retrofit and refinement (exp001, exp002)

exp001 asked the bare question: do relays retrofit? Five runs, two seeds: frozen
17.798 → relay 14.09; the parameter-matched MLP control 13.93. The GPT-2 ordering
(relays beat MLPs) inverted here — worth stating plainly: **which adapter wins on ppl
is substrate-scoped.** What replicated across substrates is the mechanism: relay
gates grow ~1.7× while MLP gates shrink, and relay training never diverged where the
wide MLP control lost 1 of 2 seeds to NaN.

exp002 ran the architecture tournament (12 runs): multi-tau strobing, depth, width,
ordering, consumers. Verdicts: `relay_pw` certified at 13.997 (first sub-14 relay,
2 seeds); width and strobe saturated; **coverage beats concentration** (spreading
the budget across all 24 blocks beat concentrating it on the deepest 12,
−0.48 ppl) — the consumption law's
signature, surviving translation from bespoke byte-beds to a pretrained trunk;
ordering budget-stable at 6k steps.

## 4. Registers and the discrete gauge (exp003)

The instrument: 22k COCO captions
([Lin et al., 2014](https://arxiv.org/abs/1405.0312)) rendered through three
structurally distinct output registers (grounded literal extraction / bracketed
generics / positional placeholders) in native messages+tools format, caption-hash
holdout, validators re-implemented from the dataset's own rules
([json-coco-format](https://huggingface.co/datasets/AbstractPhil/json-coco-format)).

Hallucination — ungrounded content in generation, the standing failure mode
surveyed in [Ji et al., 2023](https://arxiv.org/abs/2202.03629) — is measured here
as a per-leaf grounding rate, and the reduction is quantified: the task_1
specialist grounds 0.979/0.958 of
held-out extractions vs frozen 0.542 — grounding failures cut from 46% to 2–4%.
Register capture, discovered: that same specialist answers *every* register in its
own — perfect JSON shape, 0.0 task-validity — both seeds, with cross-register ppl
damage. Multi-task training resolves capture (all registers ≥0.875).

The probe is the part that only exists because the adapter is an aleph: sign-codes
collected at four depths for the same captions under four registers; separation =
inter-register minus intra-register Hamming. Instruct-trained stacks: 0.22–0.30 at
L16/L23. Wikitext-trained stack, same architecture: 0.12–0.15. Instruct training
*reorganizes* the discrete surface into differentiated registers — and this gauge
is what later predicts dispatch behavior (§6).

Instrument honesty, ledgered: the first validity pass used a non-greedy regex that
truncated nested tool-call JSON — a *perfect* generation scored 0. Caught via a
ppl-1.05-but-validity-0 contradiction; v2 validators re-judged every checkpoint
post-hoc; both generations of rows ship in the ledger.

## 5. Termination is sample semantics (exp004)

Scaling the register from ~200-token outputs to multi-kilotoken composite scenes
produced the cleanest process finding of the program. The targets are fused
multi-entity JSON from a deterministic-first vision pipeline — Grounding DINO
([Liu et al., 2023](https://arxiv.org/abs/2303.05499)), SAM
([Kirillov et al., 2023](https://arxiv.org/abs/2304.02643)), Depth Anything V2
([Yang et al., 2024](https://arxiv.org/abs/2406.09414)), SigLIP
([Zhai et al., 2023](https://arxiv.org/abs/2303.15343)) fused into one grounded
scene per image
([qwen-deepfashion-fused](https://huggingface.co/datasets/AbstractPhil/qwen-deepfashion-fused),
[qwen-synth-characters-fused](https://huggingface.co/datasets/AbstractPhil/qwen-synth-characters-fused));
polygons excluded.

Act 1 (packed 1024-token windows): the relay learned the composite register
flawlessly — json_error 0.0 vs frozen 0.833, held-out ppl 4.063 → 1.427 — and
truncated **every single generation** at the 1400 cap. Lifting the cap to 3000
recovered almost nothing (0.938 still unclosed), and the one scene that closed was
structurally near-perfect. The v2 cache then measured what the v1 estimate had
missed: true rendered rows average **4,291 tokens** (max 9,032). A packed 1024
window almost never contains an ending; stopping was never in the gradient.
(Sequence packing is standard practice for throughput —
[Krell et al., 2021](https://arxiv.org/abs/2107.02027) — but its *semantics* for
what a small adapter can learn about document boundaries is exactly what this gate
measured.)

Act 2 (row-aligned SFT: every sample is one complete conversation ending with the
closing tag, ≤4096 tokens, over-cap rows skipped and counted): structural_ok
**1.000** (16/16) on seed 0, 0.938 on seed 1 (one malformed JSON; truncation 0.0 on
both). The identical architecture that scored truncated 1.000 under packed windows
scores ok 1.000 under rows. For long structured emission through frozen-trunk
adapters, **window semantics are part of the task definition.**

The caveats ship asserted: entity over-emission (1.667/1.311× the reference — the
model invents plausible extras) and weak caption-grounding. Construction is solved;
*faithfulness* is the next line.

An ops law fell out too, the expensive way: at 4096-token rows with an adapter after
every block, fp32 activations plus the seq×vocab logits exceed 24GB — and the
Windows driver's sysmem-fallback **silently** spills to shared memory instead of
OOMing (42.8GB observed; steps at PCIe speed; the tell is ~100W draw at "100%
util"). Gradient checkpointing + chunked cross-entropy + a hard memory-fraction cap:
66GB spill → 8.8GB peak at 1.03s/step. The shipped beds print peak memory at step 50.

## 6. The dispatch, its confound, and the two-regime law (exp005, exp009)

Mixture-of-experts, in its standard form, is a *selector*: a gate picks top-k
experts and a load-balancing loss fights the collapse that selection invites
([Shazeer et al., 2017](https://arxiv.org/abs/1701.06538);
[Fedus et al., 2021](https://arxiv.org/abs/2101.03961)). This program's standing
failure class forbids exactly that — comparative selectors near the aleph collapse
its paths — so the question is asked doctrine-legally, closest in spirit to fully
dense soft mixtures ([Puigcerver et al., 2023](https://arxiv.org/abs/2308.00951))
but with signed weights and a geometric key: anchors are whole frozen relay stacks;
per block, mixing weights are `wₖ = sinh(uₖ)/Σⱼcosh(uⱼ)` with `u = cos(Px, C)/τ`,
key projection **frozen at init** (the write-time-frozen key law from the
[predecessor line's exp021](https://huggingface.co/AbstractPhil/geolip-aleph-differentiation),
satisfied by construction), codebook trainable, weights dense and signed. No
selection event exists, and no load-balancing loss — nothing pushes usage toward
uniformity except the task mixture itself; the argmax|u| "shadow winner" is logged
read-only.

exp005 (three frozen register specialists + one trainable anchor, 1 seed): the routed
composite serves all registers (0.875/1.0/1.0), usage stays dense (perplexity
3.74–3.91 of 4, all paths alive — no collapse), the shadow winner leans to the
domain anchor. The headline died in review before it shipped: the trainable anchor
is itself monolith-capacity (5,524,008 params ≈ the multi-task stack's 5,517,864)
trained on all registers interleaved — the exact recipe that already resolved
capture — and the composite's numbers equal that monolith's seed-0 numbers exactly.
So the package shipped with its own control: specialists replaced by **random**
frozen stacks. The control *matched* (0.979/0.979/1.0). At register distance, the
specialists were passengers; removal of one even ticked its neighbors up.

exp009 (1 seed) re-ran the architecture where the anchors are far apart: three extraction
registers + a story anchor (four request registers) + a math anchor (four
ask-registers), five frozen stacks + one trainable, one dispatch. Eleven
sub-competences through one model: extraction 0.854/1.0/1.0; stories 1.0 on
continuation/instruct/JSON; math perfect on all four registers including code at
1.0 — above its own specialist. Usage concentrated *without a selector*: story
prompts 0.820 on the story anchor, math 0.508 on math, extraction registers
blended. And the decouple tests are surgical in both directions: story anchor off →
stories fall (instruct 1.0 → 0.667) while math stays perfect everywhere; math
anchor off → math degrades (code 1.0 → 0.75) while stories hold untouched.

Together: **whether a frozen anchor carries load or rides along is a function of
representational separation — and the sign-code register probe measures that
separation before you ever build the dispatch.** Registers (sep 0.2–0.3) blend;
domains (sep 0.35–0.5) specialize. The MoE design variable is not the gate; it is
the distance between what the anchors know. Both regimes stand on 1 seed each — a
candidate law, not yet replicated.

## 7. The predictability principle (exp006, exp007)

Two anchors (1 seed each) trained by the identical recipe, judged by checkable constraints:

- **Story anchor** (TinyStories,
  [Eldan & Li, 2023](https://arxiv.org/abs/2305.07759); four request methods):
  continuation, instruct,
  and JSON registers all reach 1.0 validity (frozen: 0.667/0.833/0.833). The
  keyword register — "write a story using these three words," fully checkable —
  stays at **0.083, exactly frozen level**, despite every training story
  containing its keywords.
- **Math anchor** (synthetic, four ask-methods, ground truth computable): format
  locks to 1.0 everywhere (frozen NL format: 0.312), correctness never degrades,
  and held-out f(x) evaluation improves 0.688 → 0.938 (audited: 5 of 16 judged
  code problems have zero recall channel — unseen coefficient combinations).

Same trunk, same adapter, same optimizer, same steps. The difference is in the
targets: an arithmetic answer is unpredictable without using the question, so the
loss routes computation through the adapter; a story's tokens are predictable
without reading the instruction, so the constraint is ignorable and stays
unlearned. **Teacher forcing teaches what target-token predictability demands —
nothing more.** This is teacher forcing's known blind spot
([Williams & Zipser, 1989](https://doi.org/10.1162/neco.1989.1.2.270); the
train/inference mismatch that motivated scheduled sampling,
[Bengio et al., 2015](https://arxiv.org/abs/1506.03099)) meeting shortcut learning
([Geirhos et al., 2020](https://arxiv.org/abs/2004.07780)) in its cleanest form:
the shortcut is *ignoring the instruction entirely*, and a matched-everything pair
of tasks makes it measurable. (The family's mixture pressure later tripled keyword
compliance to 0.25 — real but far from solved; a keyword-first target format is
the recorded follow-on.)

## 8. Images as tokens through a text trunk (exp008)

The most speculative directive — *a CIFAR-10 diffusion model in tokens through the
gated adapter, if possible* — gets a split verdict (1 seed), all halves ledgered.
The reference points: images as autoregressive token streams is iGPT
([Chen et al., 2020](https://proceedings.mlr.press/v119/chen20s.html)); discrete
diffusion over token corruption is D3PM
([Austin et al., 2021](https://arxiv.org/abs/2107.03006)) and its masked-parallel
cousin MaskGIT ([Chang et al., 2022](https://arxiv.org/abs/2202.04200)); and
frozen language models acquiring non-text competence through small trainable
surfaces is the Frozen/universal-computation line
([Tsimpoukelli et al., 2021](https://arxiv.org/abs/2106.13884);
[Lu et al., 2021](https://arxiv.org/abs/2103.05247)). This bed sits at their
intersection with one deliberate deviation: **no fitted tokenizer** — where iGPT
used k-means color clusters and the VQ family learns its codebook
([van den Oord et al., 2017](https://arxiv.org/abs/1711.00937)), this program's
failure class forbids fitted quantizers near the aleph, so the palette is fixed
arithmetic: CIFAR-10 ([Krizhevsky, 2009](https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf))
at 16×16, 8 uniform levels per channel, palette id = 512 fixed vocabulary slots.
Corruption replaces tokens uniformly at rate t; the training sample is `[class,
level, corrupted×256, SEP, clean×256]` with loss on the clean half; inference
walks the noise schedule down.

The denoiser is real: teacher-forced reconstruction beats the identity baseline
(copy the corrupted token: (1−t)+t/512) at every level — 0.840 vs 0.751 (t=0.25),
0.689 vs 0.501, 0.527 vs 0.252, and 0.378 vs 0.002 at t=1.0, where the corrupted
view carries nothing and reconstruction comes entirely from learned
position/smoothness/class priors. The frozen trunk's own zero-shot 0.259 at t=1.0
is a finding in itself — an emergent copy-local-context strategy.

The two honest halves: greedy argmax in the iterative refinement loop
mode-collapsed to a single all-dark image (diversity 0.0; conditional == shuffled
to four decimals) — deterministic decoding in an iterative denoiser collapses to
the global mode, the visual sibling of greedy text degeneration
([Holtzman et al., 2019](https://arxiv.org/abs/1904.09751)), and a *decoding*
property: multinomial sampling from the same checkpoint restores diversity 0.92 —
horizon lines, sky/ground splits, coherent palettes in the shipped grids. And
class-conditionality is **not demonstrated**: conditional histogram distance ≈
shuffled. A frozen text trunk + 5.5M adapter learned image *statistics*, not yet
image *classes*.

## 9. What ships, what's next

Six packages (`exp004_composite` through `exp009_family`), each with scrubbed
standalone code, a self-asserting results builder run green before upload, the full
JSON ledger including every failed act and instrument bug, checkpoints, and a
sub-article. Two nights, one 4090, ~60 logged GPU runs.

The recorded next rungs: the family with the image-token anchor joined;
sub-capacity trainable anchors (make specialists load-bearing at register
distance); keyword-first targets for constraint binding; faithfulness training
against entity over-emission; and the geofractal tower port — the family's anchors
are homogeneous relay stacks, which is exactly what the tower machinery wants.

The aleph started this arc as a codebook differentiation device in toy
reconstruction beds. It ends this installment as the addressing surface of a
composable adapter family on a frozen pretrained transformer — with its discrete
sign-code gauge doing real predictive work that a parameter-matched MLP cannot
offer: telling you, before you build the mixture, whether your experts will carry
or coast.

---

## This program's corpus

The aleph line is self-published and cumulative; the installments this article
stands on directly:

- [aleph-autoregressive-differentiation-ft1](https://huggingface.co/blog/AbstractPhil/aleph-autoregressive-differentiation-ft1)
  — the predecessor: bottleneck prior (7/7), the consumption law and the
  comparative-selector pathology this program's dispatch design obeys, sign-code
  parity, shell transit, projective objective-independence.
- [geometric-vocabulary-patchwork-aleph-void](https://huggingface.co/blog/AbstractPhil/geometric-vocabulary-patchwork-aleph-void)
  — the keystone: the aleph closed form, code-verified.
- [structural-attractors-in-latent-geometry](https://huggingface.co/blog/AbstractPhil/structural-attractors-in-latent-geometry)
  — the binding constant and the attractor geometry the addressing objects
  inherit.
- [constellation-diffusion-bottleneck](https://huggingface.co/blog/AbstractPhil/constellation-diffusion-bottleneck)
  — the 16s constellation system the patchwork consumer descends from.
- [geometric-tri-band-ft2](https://huggingface.co/blog/AbstractPhil/geometric-tri-band-ft2)
  — the projective codebook law, carried here unchanged.
- [geometric-memory-ft3](https://huggingface.co/blog/AbstractPhil/geometric-memory-ft3)
  — the optimizer discipline (pure Adam, wd 0): weight decay destroys the
  geometric harmonic; the geometry is the regularization.
- [geolip-svae-transformer-ft1](https://huggingface.co/blog/AbstractPhil/geolip-svae-transformer-ft1)
  — the two-year differentiation survivor: reconstruction-cultivated sign codes,
  the origin of "read the signs, not the probabilities."

Repositories: [geolip-aleph-qwen](https://huggingface.co/AbstractPhil/geolip-aleph-qwen)
(this line: code, ledgers, checkpoints, per-experiment sub-articles) and
[geolip-aleph-differentiation](https://huggingface.co/AbstractPhil/geolip-aleph-differentiation)
(the measurement program, exp011–exp021). Datasets:
[json-coco-format](https://huggingface.co/datasets/AbstractPhil/json-coco-format),
[qwen-deepfashion-fused](https://huggingface.co/datasets/AbstractPhil/qwen-deepfashion-fused),
[qwen-synth-characters-fused](https://huggingface.co/datasets/AbstractPhil/qwen-synth-characters-fused).

## References

- Alayrac, J.-B., et al. (2022). *Flamingo: a Visual Language Model for Few-Shot
  Learning.* [arXiv:2204.14198](https://arxiv.org/abs/2204.14198)
- Austin, J., Johnson, D. D., Ho, J., Tarlow, D., & van den Berg, R. (2021).
  *Structured Denoising Diffusion Models in Discrete State-Spaces.*
  [arXiv:2107.03006](https://arxiv.org/abs/2107.03006)
- Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015). *Scheduled Sampling
  for Sequence Prediction with Recurrent Neural Networks.*
  [arXiv:1506.03099](https://arxiv.org/abs/1506.03099)
- Chang, H., Zhang, H., Jiang, L., Liu, C., & Freeman, W. T. (2022). *MaskGIT:
  Masked Generative Image Transformer.*
  [arXiv:2202.04200](https://arxiv.org/abs/2202.04200)
- Chen, M., et al. (2020). *Generative Pretraining from Pixels.* ICML 2020.
  [PMLR 119](https://proceedings.mlr.press/v119/chen20s.html)
- Eldan, R., & Li, Y. (2023). *TinyStories: How Small Can Language Models Be and
  Still Speak Coherent English?*
  [arXiv:2305.07759](https://arxiv.org/abs/2305.07759)
- Fedus, W., Zoph, B., & Shazeer, N. (2021). *Switch Transformers: Scaling to
  Trillion Parameter Models with Simple and Efficient Sparsity.*
  [arXiv:2101.03961](https://arxiv.org/abs/2101.03961)
- Geirhos, R., et al. (2020). *Shortcut Learning in Deep Neural Networks.*
  [arXiv:2004.07780](https://arxiv.org/abs/2004.07780)
- Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019). *The Curious
  Case of Neural Text Degeneration.*
  [arXiv:1904.09751](https://arxiv.org/abs/1904.09751)
- Houlsby, N., et al. (2019). *Parameter-Efficient Transfer Learning for NLP.*
  [arXiv:1902.00751](https://arxiv.org/abs/1902.00751)
- Hu, E. J., et al. (2021). *LoRA: Low-Rank Adaptation of Large Language Models.*
  [arXiv:2106.09685](https://arxiv.org/abs/2106.09685)
- Ji, Z., et al. (2023). *Survey of Hallucination in Natural Language Generation.*
  [arXiv:2202.03629](https://arxiv.org/abs/2202.03629)
- Kirillov, A., et al. (2023). *Segment Anything.*
  [arXiv:2304.02643](https://arxiv.org/abs/2304.02643)
- Krell, M. M., Kosec, M., Perez, S. P., & Fitzgibbon, A. (2021). *Efficient
  Sequence Packing without Cross-contamination: Accelerating Large Language
  Models without Impacting Performance.*
  [arXiv:2107.02027](https://arxiv.org/abs/2107.02027)
- Krizhevsky, A. (2009). *Learning Multiple Layers of Features from Tiny Images.*
  [Technical report](https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf)
- Lin, T.-Y., et al. (2014). *Microsoft COCO: Common Objects in Context.*
  [arXiv:1405.0312](https://arxiv.org/abs/1405.0312)
- Liu, S., et al. (2023). *Grounding DINO: Marrying DINO with Grounded
  Pre-Training for Open-Set Object Detection.*
  [arXiv:2303.05499](https://arxiv.org/abs/2303.05499)
- Lu, K., Grover, A., Abbeel, P., & Mordatch, I. (2021). *Pretrained Transformers
  as Universal Computation Engines.*
  [arXiv:2103.05247](https://arxiv.org/abs/2103.05247)
- Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016). *Pointer Sentinel
  Mixture Models.* [arXiv:1609.07843](https://arxiv.org/abs/1609.07843)
- Puigcerver, J., Riquelme, C., Mustafa, B., & Houlsby, N. (2023). *From Sparse
  to Soft Mixtures of Experts.*
  [arXiv:2308.00951](https://arxiv.org/abs/2308.00951)
- Qwen Team (2024). *Qwen2.5 Technical Report.*
  [arXiv:2412.15115](https://arxiv.org/abs/2412.15115)
- Shazeer, N., et al. (2017). *Outrageously Large Neural Networks: The
  Sparsely-Gated Mixture-of-Experts Layer.*
  [arXiv:1701.06538](https://arxiv.org/abs/1701.06538)
- Tsimpoukelli, M., et al. (2021). *Multimodal Few-Shot Learning with Frozen
  Language Models.* [arXiv:2106.13884](https://arxiv.org/abs/2106.13884)
- van den Oord, A., Vinyals, O., & Kavukcuoglu, K. (2017). *Neural Discrete
  Representation Learning.* [arXiv:1711.00937](https://arxiv.org/abs/1711.00937)
- Williams, R. J., & Zipser, D. (1989). *A Learning Algorithm for Continually
  Running Fully Recurrent Neural Networks.* Neural Computation, 1(2), 270–280.
  [doi:10.1162/neco.1989.1.2.270](https://doi.org/10.1162/neco.1989.1.2.270)
- Yang, L., et al. (2024). *Depth Anything V2.*
  [arXiv:2406.09414](https://arxiv.org/abs/2406.09414)
- Zhai, X., Mustafa, B., Kolesnikov, A., & Beyer, L. (2023). *Sigmoid Loss for
  Language Image Pre-Training.*
  [arXiv:2303.15343](https://arxiv.org/abs/2303.15343)
- Zhang, L., Rao, A., & Agrawala, M. (2023). *Adding Conditional Control to
  Text-to-Image Diffusion Models.*
  [arXiv:2302.05543](https://arxiv.org/abs/2302.05543)
- Zhang, R., et al. (2023b). *LLaMA-Adapter: Efficient Fine-tuning of Language
  Models with Zero-init Attention.*
  [arXiv:2303.16199](https://arxiv.org/abs/2303.16199)

*Code and ledgers: [geolip-aleph-qwen](https://huggingface.co/AbstractPhil/geolip-aleph-qwen).
Predecessor line and laws: [geolip-aleph-differentiation](https://huggingface.co/AbstractPhil/geolip-aleph-differentiation).*