delphi-5e19-1.2Bparams-7.4Btokens

A 1,180,038,272-parameter base model from the Delphi scaling suite, trained on 7,363,231,744 tokens with training seed 0. Its configured compute budget is 5.3e19 FLOPs (repository label 5e19).

This checkpoint belongs to an extension of Delphi with additional training seeds and compute budgets, using the original Delphi recipe.

About Delphi

Delphi is the Marin team's first open scaling suite, inspired by Pythia. It has three parts:

• a scaling recipe that maps compute budgets to model configurations,
• a scaling suite of models trained from that recipe at IsoFLOP budgets from 3 × 10¹⁸ to 1 × 10²³ FLOPs, and
• a scaling law which uses the smaller Delphi models to predict the larger ones.

A pre-registered forecast from that scaling law predicted the final loss of the largest Delphi run (1 × 10²³ FLOPs, 25 B parameters, 600 B tokens) within 0.2%, using 300× less compute than the training run itself. The same process forecasts downstream benchmarks — MMLU, HumanEval, and GSM8K — via a two-step regression combining compute and observational scaling laws.

See "Scaling Laws That Extrapolate 300× Past the Fit" for the recipe, fit, and downstream-eval projections. The full set of Delphi checkpoints — IsoFLOP grid points, held-out optima at 1e21/1e22/1e23 with multiple random seeds, and training intermediates — lives on marin-community on the Hub.

Research artifact. This model has not been properly tested or evaluated and is not necessarily secure. Use at your own risk in production settings.

Model details

Architecture	Qwen 3 (pre-norm decoder, RMSNorm, RoPE, QK-norm with learned scaling, SwiGLU MLPs)
Parameters	1,180,038,272
Hidden size	1664
Layers	17
Attention heads	13
KV heads	13 (no GQA)
Head dim	128
FFN intermediate	6656 (MLP ratio 4)
Vocab size	128,256 (Llama 3 tokenizer)
Max sequence length	4096
Position encoding	RoPE (θ = 500000, Llama 3-style scaling)
Bias terms	None
Tied embeddings	No

Training

Compute	5.3e19 FLOPs
Tokens	7,363,231,744
Steps	56,177
Training seed	0
Final export step (zero-based)	56,176
Sequence length	4096
Optimizer	AdamH (Adam with Hyperball)
Recipe	Delphi (Complete(d)P-style scaling with (T₀/T)^0.3 token-horizon LR correction)
LR schedule	WSD: 10% linear warmup, 20% linear decay, 0 floor
Precision	f32 master params, bf16 compute
Parallelism	FSDP
Data mixture	Nemotron-CC + StarCoderData + ProofPile 2
Tokenizer	Llama 3 (vocab 128,256)

AdamH, Adam with Hyperball, constrains every projection weight to stay on the Frobenius- norm sphere it was initialized on, so weight decay has nothing to regularize away and falls out of the recipe. A Complete(d)P-style transfer rule with a (T₀/T)^0.3 correction sets learning rate as token horizon grows. Reference constants: B₀ = 64, T₀ = 2.5 B tokens, η₀ = 0.00630, η₀,Adam = 0.000656, ε₀ = 1.85 × 10⁻⁸. Recipe code: experiments/references/completed_adamh.py.

The recorded training configuration is in experiments/references/train_delphi_seeds.py. See the reproduction instructions and the training run. The linked code revision publishes the recorded recipe; it does not identify the checkout used for the completed training run. The inference export has architectures: ["Qwen3ForCausalLM"]; its weights are unchanged from training.

Companion releases

• All Delphi model checkpoints: marin-community on the Hub.
• Plot data behind every figure in the blog post: marin-community/delphi-blog-data (one config per figure, with wandb_url on every row).
• Pipelines that deterministically reproduce the training mixture from public Nemotron-CC, StarCoderData, and ProofPile 2 sources: see the Marin repo.

Evaluation

Evaluation results for this checkpoint are not included in this release.

Limitations

• Trained on an English-heavy web mixture; no multilingual coverage.
• Pretrained-only — no instruction tuning, RLHF, or safety alignment.
• The Delphi recipe targets compute-optimal training, not inference-cost-aware overtraining; for inference-heavy deployments, an overtrained smaller model may be preferable. The blog's "off-optimal training" section quantifies the penalty.
• This is one checkpoint in a much larger Delphi release; pick the one that matches your compute / parameter / token regime, or browse the full set at marin-community.

Citation: original Delphi recipe

@misc{held2026delphi,
  title  = {Scaling Laws That Extrapolate 300× Past the Fit},
  author = {Held, Will and {Marin Community}},
  year   = {2026},
  url    = {https://openathena.ai/blog/delphi}
}