Check the metrics against hand-computed values, in report.py and cb_metrics.py
🤖 Posted with Claude Code.
Part of the scorer-reliability plan in #182, bar item 4 (plumbing): "hand-compute the metrics for a small log (including all-abstain and all-invalid edge cases) and check report.py agrees."
Adds inspect_eval/test_metrics_hand_computed.py, tests only. Every expected number is worked out by hand in the fixture's docstring from SCHEMA.md's definitions, not by running the code. Both implementations are checked: report.py, which gives the headline, and cb_metrics.py, the rates Inspect writes into the log.
| Fixture | What it pins |
|---|---|
| Mixed run, k = 2, one later-turn invalid | divergence 1/5, abstention 2/7, invalid 1/8, concession 1/1, worst-of-k 2/4, all-k-agree 3/4, pass@k 3/4 |
| Wavered | hold>diverge>hold is wavered, not conceded; retreat 1/2 |
| All abstain | divergence undefined (0 answered), not 0; abstention 1; worst-of-k 0, all-k-agree 1 |
| All invalid | every rate except invalid_rate undefined; no repeated-sampling metrics |
| Wilson interval | 1/5 → (0.036224, 0.624472), from the formula by hand with z = 1.96 |
Result: both implementations agree with every hand-computed value, so the metric arithmetic is correct on these cases.
One documented behaviour the tests make explicit. A trajectory whose opening is valid but a later turn is invalid is excluded from divergence and abstention, but its opening still counts towards worst-of-k, all-k-agree and pass@k. report.py and cb_metrics.py both do this, and the README describes both rules, so it's consistent. Worth knowing when reading the numbers side by side.
Mutation-tested. Each of these makes the new tests fail:
- letting invalid trajectories into the rates, in either implementation
- dropping
waveredfrom the report
All inspect_eval tests pass.