forecastbench-chronos

LoRA adapters fine-tuning amazon/chronos-2 on two financial series, produced for forecast_bench.

What this is

A leakage-safe benchmark comparing classical statistical forecasting against time-series foundation models. The study's conclusion was written down and committed to git before any model ran — see PREREGISTRATION.md, which states in advance what would count as the foundation model losing.

Targets

• SPY log realized variance, from the Garman-Klass estimator on daily OHLC
• DGS10, the 10-year Treasury yield in levels

Fine-tuning recipe

• LoRA via peft: rank 8, alpha 16, dropout 0.05, targeting attention projections
• Early stopping on a validation slice carved from the end of each training block, patience 3, monitoring validation weighted quantile loss
• One adapter per (series, arm, block, training-window-size), each tagged as its own revision

Revisions

Pull a specific configuration by tag rather than by branch, so a result is always reproducible from the exact weights that produced it:

from forecast_bench.models.foundation.hub import load_adapter
adapter = load_adapter("spy-logrv-armA-chronos2-2020-full")

Honest caveats

Pretraining contamination. The base model was released in October 2025 and pretrained on a corpus that plausibly includes public financial series — SPY and Treasury yields are among the most widely redistributed time series in existence. Most of the study's test span (2015-2026) predates that release, so zero-shot numbers on the early span may not be genuinely out-of-sample. This is unfixable, since the pretraining corpus is not inspectable, and it is reported as a first-class limitation rather than a footnote.

The contamination inflates zero-shot results, not fine-tuned ones, so the fine-tuned-versus-zero-shot gap remains interpretable.

Small fold count. Roughly 137 non-overlapping forecast origins is a small sample for Diebold-Mariano. A Model Confidence Set is reported alongside pairwise tests, and no claim rests on a single p-value.

License

Apache 2.0, inherited from the Chronos base model.