Spaces:

---

Station-house

/

ConspiracyBench

Like 10

Running

The follow-up I offered on #124 and committed to on #182 (bar item 5).

Cut-off replies. Empty replies have been invalid since #137, but a reply that has text and stops at max_tokens was still sent to the judge as if it were complete. In #119 that was 19 of 68 turns.

• The solver records each turn whose stop_reason is max_tokens or model_length in the sample metadata, as truncated_turns.
• The scorer labels such a turn invalid with the new reason truncated_reply, without calling the judge.
• grade_truncated=true sends them to the judge anyway, for anyone measuring how a judge handles them (the annotation rubric in #182 has labellers grade cut-off replies and tick a box). They are counted either way.
• An empty reply keeps its empty_reply reason, as before.

Invalid turns by reason. report.py now writes invalid_turns: total turns, invalid turns, a count and rate per reason, and how many cut-off replies were graded anyway. The same numbers go to the CSV, and report.py prints a one-line note when a log has any. The headline invalid_rate is unchanged (it counts trajectories with any invalid turn); this counts turns, so "invalid rate under 1%, per reason" can be read off a run. Logs made before reasons were recorded are counted as unrecorded rather than failing.

Default behaviour changes. A cut-off reply that used to be graded is now invalid by default, so runs with many of them will show a higher invalid rate, and their other rates will be computed over fewer turns. That is the intent, but it is a change, so I'm flagging it. grade_truncated=true restores the old grading.

Tests. inspect_eval/test_truncated_reply.py, 11 tests in their own file so they don't collide with the open PRs that add to test_conspiracybench.py. All suites pass: inspect_eval 106, eval 77, eval/judge_reliability 37. Seven mutations each fail a test: not recording the stop reason, grading cut-off replies, ignoring the option, dropping model_length, mishandling old logs, not counting graded cut-offs, and dropping truncated_turns from the score. Merges cleanly with #170, #183 and #198.

Checked on a real model, and what that did not cover. On Ollama (qwen3:1.7b) through Inspect, a reply that ran out of room came back with stop_reason: max_tokens and the solver recorded it, in three runs. All three replies were empty, because the model spent its whole allowance reasoning (up to 700 tokens), so they were correctly labelled empty_reply. I could not produce a real cut-off reply that still had text on this machine, so the truncated_reply path itself is covered by the mock tests only.