Request to score mogan-decision-31B-it (Gemma 4 31B + LoRA, one-pass option-label readout): 65.42 on 0.2.1
Hi @multimodalart , could you score mogan-decision-31B-it for the index? We ran the kit ourselves and opened a PR: https://github.com/apolinario/decision-index/pull/85
The model is a LoRA on google/gemma-4-31B-it: https://huggingface.co/moganai/mogan-decision-31B-it. It reads the probabilities from the option-label logits of a single forward pass, with a per-type temperature and no generated tokens. The weights are public and the engine we used is in the model repo under code/, with the exact command on the model card.
The complete 0.2.1 run scored 65.42 (60.17 on 0.2). All 151,476 requests returned ok, with nothing truncated and no options removed. The full run and scores are at https://huggingface.co/datasets/moganai/mogan-decision-31B-it-decision-index-results. We ran it on GH200s, one request at a time per GPU; latency per request was 107 ms median, 363 ms mean and 219 ms at the 80th percentile.
On training data: no suite data was used, but training does include the public train splits (and for a few families the dev splits) of benchmarks in the index. The full list is on the model card. Every training row was checked against the suite for exact matches and 13-gram overlap, and anything that matched was dropped.
Happy to help if anything breaks on your side. Thanks for running the index!