Debk/saarthi-default-prediction

SAARTHI — MSME Default Prediction

Credit default-prediction models trained entirely on public datasets for IDBI Innovate 2026 (Track 04, Default Prediction Model). No bank data was used at any point.

The headline design choice: SAARTHI ships pre-trained and then fine-tunes on a lender's own book, rather than cold-starting on every upload.

• Training data: 2,875,926 labelled loans across 8 public credit datasets (plus 12M synthetic card transactions for self-supervised pre-training of the sequence encoder).
• Live application: https://koushikdeb.duckdns.org · Code: https://github.com/DevDaring/IDBI_SAARTHI · Demo video: https://youtu.be/WGtfB9VSff4

Fine-tune on your own loan book

A bank maps its columns once and runs one command. Everything runs locally; no data leaves the bank.

pip install -r finetune/requirements.txt
python finetune/finetune.py --data my_book.csv --mapping finetune/mapping_example.json --out my_model.joblib
python finetune/predict.py  --model my_model.joblib --data new_loans.csv --out scored.csv

finetune.py downloads global_canonical.joblib, maps your columns onto the 15 canonical fields, ranks them within your book (the same transform used in pre-training), adds the global model's PD as a prior, and trains a LightGBM model on your fields plus that prior with a strict 60/15/25 fit / calibrate / test split. It prints test-fold AUC, PR-AUC, Brier and ECE for the global model alone and for the fine-tuned model. predict.py scores new loans (no outcome column needed). See finetune/FINETUNE.md.

Example run on a 2,500-loan synthetic MSME book (625-loan test fold): global model alone AUC 0.526 → fine-tuned 0.746. This gap is expected: the global model does not transfer reliably to a new lender (see the leave-one-dataset-out result below), which is exactly why the bank's own model is trained locally.

Methodology — why these numbers are trustworthy

Every metric below comes from a strict three-way split:

Fold	Share	Used for
fit	60%	training the boosters
calibrate	15%	fitting the isotonic calibrator only
test	25%	never seen by either — all reported metrics

This matters: a common shortcut fits the probability calibrator and then measures calibration error on that same fold, which drives ECE artificially toward zero. Here the calibrator never sees the test fold, so the reported ECE is a real out-of-sample calibration estimate.

Per-dataset specialist models

Full native feature set per corpus. Ensemble of LightGBM + XGBoost + CatBoost, isotonic-calibrated.

Dataset	n	Default rate	Test AUC	PR-AUC	ECE	Published reference
sba	897,167	0.1756	0.9800	0.9126	0.0015	~0.95 (literature)
lending_club	1,369,566	0.2124	0.7263	0.4104	0.0024	~0.70-0.73 (leakage-free)
home_credit	307,511	0.0807	0.7632	0.2449	0.0011	0.805 (Kaggle winner)
gmsc	150,000	0.0668	0.8522	0.3629	0.0041	~0.87 (Kaggle winner)
taiwan	30,000	0.2212	0.7698	0.5243	0.0074	~0.78 (literature)
german	1,000	0.3000	0.7668	0.5505	0.0680	~0.79 (literature)
berka	682	0.1111	0.8558	0.5725	0.0295	n/a
amex	120,000	0.2598	0.9580	0.8858	0.0050	~0.96 AUC (winners; the 0.80 headline is a different metric)
hc2024	—	—	failed	—	—	~0.86 (Kaggle winner)

Note on sba: its 0.980 is driven mainly by term and approval_year. Both are known at approval, but approval year also captures the economic cycle (e.g. 2006–08 vintages), and only terminal outcomes (paid in full / charged off) are kept. It is a genuine test-fold result, not evidence that small-business default is this predictable in general. Splits are random (stratified), not out-of-time.

• sba — US Small Business Administration loans - closest public analogue to MSME lending
• lending_club — 2007-2018 consumer loans, charge-off label
• home_credit — Home Credit 2018 application + bureau aggregates
• gmsc — Give Me Some Credit, 90+ DPD within 2 years
• taiwan — Taiwan credit-card default, 6-month repayment panel
• german — UCI German Credit, 1000 rows
• berka — PKDD'99 Czech bank, pre-origination transaction aggregates
• amex — American Express monthly statement panel
• hc2024 — Home Credit 2024 model-stability competition

Pooled global model (ships with the app)

Trained across corpora in a shared 15-field canonical credit vocabulary so it can score any loan book that maps onto it.

• Test AUC: 0.8605
• PR-AUC: 0.5795
• ECE: 0.0018 · Brier: 0.0868
• Test rows: 270,421
• Corpora pooled: sba, lending_club, home_credit, gmsc, taiwan, german, berka

Leave-one-dataset-out transfer — a negative result

Train on every corpus except one, then score the held-out corpus cold. This is the honest proxy for "will it transfer to a book it has never seen?" — the question that actually matters for deployment.

It does not transfer. Most hold-outs land at or below chance, which means the pooled model is not merely uninformative on an unseen corpus, it is anti-predictive: the feature→outcome relationships invert across lending domains.

Held-out corpus	n	Raw pooling	Rank-normalised
berka	682	0.4612	0.5155
german	1,000	0.4646	0.4022
gmsc	150,000	0.7786	0.7711
home_credit	300,000	0.4991	0.5785
lending_club	300,000	0.5228	0.5399
sba	300,000	0.4103	0.2574
taiwan	30,000	0.7038	0.6989
mean		0.5486	0.5377

Normalisation did NOT rescue it

The obvious hypothesis is scale mismatch — these corpora are denominated in USD, DM, NT$ and CZK, so a "loan amount" of 50,000 means different things in each. Converting every numeric feature to its within-corpus percentile rank tests that hypothesis directly, and it fails: mean transfer AUC moved 0.5486 → 0.5377, i.e. no better. SBA in particular degrades from 0.4103 to 0.2574.

The mechanism is therefore relational inversion, not scale. SBA is small-business lending, where a longer term and a larger SBA-guaranteed principal typically indicate a better-vetted, collateral-backed loan — the opposite of the consumer-credit corpora that dominate the pooled training set. Rank-normalising preserves that inverted ordering perfectly, which is why it cannot help.

Practical consequence: a single pooled "foundation" model for credit risk is not supported by this evidence. Domain-matched training plus fine-tuning on the lender's own book is the defensible architecture, which is what SAARTHI does.

Ablation — does transaction-as-language earn its place?

A CoLES-style contrastive encoder (GRU + InfoNCE over disjoint sub-sequence views) was pre-trained on unlabelled transaction streams, then its 256-d embedding was tested against the tabular features.

Corpus	Tabular	Sequence only	Tabular + sequence	Lift
berka (pre-loan transactions only, corrected)	0.8575	0.7727	0.8471	-0.0104
amex	0.9590	0.9156	0.9585	-0.0006

Correction (2026-10-05): an earlier version of this card reported a Berka lift of +0.027 AUC. That ablation used embeddings built from each account's full transaction history, including post-loan transactions. Re-run on pre-loan transactions only, the lift is −0.0104, and across six seeds it is −0.005 ± 0.050 (171 test loans), i.e. no measurable gain.

What holds: on Amex (941 hand-engineered aggregates over the same statement data) the embedding adds nothing — the aggregates already capture what the encoder learns. The embedding is a substitute for feature engineering, not an addition to it.

Note also that on Amex the 256-d embedding alone reaches 0.9156 AUC with no hand-engineered features at all, versus 0.9590 for 941 engineered columns.

Linear probe on frozen embeddings

Logistic regression on the frozen encoder output — measures how much default signal the unsupervised embedding captured on its own.

• amex: AUC 0.8717 (n=60,000)
• berka, pre-loan transactions only: AUC 0.6799 (n=682). (The 0.7783 in probe_results.json used full-history sequences and is not leak-free.)

Known limitations

• Berka aggregates only pre-origination transactions. Using the full history yields AUC 1.0000 because 71% of an account's transactions occur after the loan date and encode the repayment behaviour that defines the label. The leaked figure is not reported here.
• Licences: Amex, Home Credit and Give Me Some Credit are Kaggle-competition datasets used under their competition terms for research; Home Credit and GMSC are part of the pooled global model's training data. No dataset rows are redistributed here — only trained model weights and metrics.
• The pooled model uses a small shared vocabulary, yet within each corpus it stays close to the full-feature specialists (e.g. GMSC 0.863 vs 0.852, Taiwan 0.769 vs 0.770, SBA 0.959 vs 0.980); a predictor that knows only which corpus a row came from reaches 0.634. It still does not transfer to an unseen corpus (above).
• No out-of-time validation, no hyper-parameter tuning and a single seed: no confidence intervals. Small corpora (German 250 test rows, Berka 171) are noisy.
• No Indian MSME data was available; GST-behavioural fields are simulated against the schema IDBI published, not learned from real filings.

Files

File	What it is
global_canonical.joblib	pooled global model + isotonic calibrator
specialist_<corpus>.joblib	per-corpus specialist ensembles
coles.pt	transaction-sequence encoder checkpoint (bi-GRU, 0.89M parameters)
finetune/finetune.py	fine-tune on your own loan book (60/15/25 split, isotonic calibration)
finetune/predict.py	score new loans with a fine-tuned model
finetune/mapping_example.json	example column mapping (bank-style names → canonical fields)
finetune/FINETUNE.md	step-by-step guide
metrics.json	every number above, machine-readable
ablation_sequence.json	sequence-embedding ablation

Usage

Score a frame that already holds the 15 canonical columns (b['features']). Numeric fields must be within-book percentile ranks (as in training); can_sector uses the categories the model was trained with. finetune/finetune.py does all of this for you.

import joblib, numpy as np, pandas as pd
from huggingface_hub import hf_hub_download

b = joblib.load(hf_hub_download("Debk/saarthi-default-prediction", "global_canonical.joblib"))
X = df.reindex(columns=b["features"])                              # canonical vocabulary
num = [c for c in b["features"] if c not in b["cat_cols"]]
X[num] = X[num].apply(pd.to_numeric, errors="coerce").rank(pct=True)  # within-book percentile ranks
levels = b["members"]["lgbm"].booster_.pandas_categorical[0]
X["can_sector"] = pd.Categorical(X["can_sector"].where(X["can_sector"].isin(levels)), categories=levels)
raw = []
for m in b["members"].values():
    Xm = X.assign(can_sector=X["can_sector"].astype(str)) if type(m).__name__.startswith("CatBoost") else X
    raw.append(m.predict_proba(Xm)[:, 1])
pd_calibrated = b["calibrator"].predict(np.mean(raw, axis=0))     # probability of default

Citation

@misc{saarthi2026,
  title  = {SAARTHI: explainable MSME default early-warning},
  author = {Koushik Deb},
  year   = {2026},
  note   = {IDBI Innovate 2026, Track 04. https://huggingface.co/Debk/saarthi-default-prediction}
}

---

Generated 2026-08-31 from metrics files; updated 2026-10-05 (fine-tuning package, corrected Berka sequence ablation, licence wording).