Spaces:

---

multimodalart

/

jev-decision-index

Like 863

Running

Update (2026-09-25): the 9B row now points to JevK5-9B v0.3.3, trained for 1.5 epochs on the same data (details in the comment below). The 9B numbers under "What we measured ourselves" are v0.3's. The optional JevK5-Lite row is also in a comment below.

Hi! Thanks again for the lift run in #13. We released JevK5 v0.3, and a 9B trained on the same data. Could you run both?

What to run

• Same runtime and readout as the 0.2.2 run: pip install "jevk5 @ git+https://github.com/allebee/jevk5@v0.3.0", then JevK5("alibiserikbay/JevK5").probabilities(state, question), or JevK5("alibiserikbay/JevK5-9B"). Questions with more than 16 options take the same knockout path as in #13, so the lifted adapter should work unchanged. The one runtime change in 0.3.0: each repo's jevk5_config.json now also carries its own second temperature for more than 16 options (knockout_temperature), so please use 0.3.0, not 0.2.2. Without the key (v0.2) nothing changes.
• The v0.3 4B replaces v0.2 at the same repo. v0.2 stays at the Hub tag v0.2, so by your superseded policy the 4B row would move to v0.3. The 9B is a new row.
• Cost: the 9B takes about 2-3x the 4B's time per pass. On JevBench's short items on an H100, p50 is 31 ms against 13.6 ms (the 9B measured while another job shared the GPU).

Declared overlap (both models, same training data)

• Test or validation items in training: none that we know of. Every public and teacher-written training row was checked against the test and validation text of 35 index benchmarks and JevBench's public items. A row was dropped for an exact match (after normalizing) or for any shared 8-word sequence. On 2026-09-25 we re-checked the final training set for exact question matches against ARC, MMLU, OpenBookQA, CommonsenseQA, HellaSwag, WinoGrande and GSM8K test and validation, and found none. That includes the 15 ARC test rows your notes find duplicated in ARC train. GPQA and HLE are gated and were not checked; no source is built from them.
• No MMLU, MMLU-Pro or BBH at all, in any split. The 980 MMLU-Pro test ids that v0.1/v0.2 trained on are not in v0.3. MMLU's auxiliary_train was dropped too (it is mostly RACE, which is for non-commercial research only).
• Other splits of the same dataset in training (rows in training; train splits, deduplicated against your test and validation items):
  • 26/27 ARC-Easy/Challenge: train, 1,186
  • 51 OpenBookQA: train, 791
  • 52 CommonsenseQA: train, 1,186
  • 30 GSM8K: train, 2,373
  • 28 WinoGrande: winogrande_xl train, 2,373
  • 29 HellaSwag: train, 2,373
  • 4 BANKING77: train, 791 (each shown with at most 16 intents)
  • 5 CLINC150+OOS: plus train, 1,186 (at most 16 intents each)
  • 10 SGD: train (30 of 127 files), 1,186
  • 37 Amazon ESCI: train (3 of 11 shards), 1,186
  • 62 When2Call: train_pref, 1,978
  • 40 iSarcasmEval: train.En, 1,581
  • 59 RAGTruth: train, 1,186
  • 61 HoVer: train claims with their cited Wikipedia introductions, 1,582
  • 64 New Yorker matching: fold 0 train, 791
• Teacher-written questions:
  • 53 of the 3,270 Qwen3.6-27B questions (the same ones v0.2 trained on) share at least one 8-word sequence with 11 ContractNLI (41) or 10 SGD (12) test/dev text. They were reported and kept.
  • GPT-6 Luna questions with any hit were dropped: 40 ContractNLI, 6 SGD, 3 ARC, 3 ToolRet and 2 RAGTruth.
• Not in training: ANLI and NLI4CT. Their train splits were used only in an unreleased development variant.
• Selection used held-out teacher questions, a hand-written set, and held-out rows of the train splits above. No index test item was used. Our many-options checks (BANKING77, CLINC150, MASSIVE) use train-split items only, and none of those items is in the training data.

What we measured ourselves (for context, not index numbers)

• Train splits, every intent offered, each model with its own second temperature:
  • The v0.3 4B is worse than v0.2 on BANKING77 (0.652 against 0.690, ECE 0.044) and better on CLINC150 (0.700 against 0.666, ECE 0.056).
  • JevK5-9B: 0.734 on BANKING77 and 0.780 on CLINC150 (ECE 0.044 and 0.061).
• Index proxy (chance-corrected skill on held-out train-split rows of 16 index benchmarks): v0.2 0.620, v0.3 4B 0.731, 9B 0.762.

Model cards: JevK5, JevK5-9B. Thanks for running the index!