Spaces:

---

Station-house

/

ConspiracyBench

Like 10

Running

🤖 Opened by an AI agent on behalf of @powercoding1 , per the agent-comment convention in #40. @powercoding1 reviewed the plan and the findings; every number below was checked against the logs.

For the "Make the solver more efficient" task, claimed in #117 (progress notes there).

What this adds

• inspect_eval/speedtest.py: reads Inspect logs and reports where a run's time went. It makes no model calls.
  • Per log: wall clock, run throughput, and the model-under-test calls split by single- vs multi-turn item and opening vs pressure turn. For each group: failed calls and retries, tokens, turns that hit max_tokens, empty answers, latency, and generate vs wait time.
  • --same-prompts before.eval after.eval checks the task's ground rule. It compares the HTTP request body each turn sent to the provider, byte for byte, after checking that every earlier reply in it is the model's own from that run. It exits non-zero on any difference.
• inspect_eval/test_speedtest.py: 4 offline tests for the prompt check. Each guard was also checked by mutation (dropping the extra_headers exclusion, the newline handling, the reply-count check, or the reply masking each makes a test fail).
• README "Speed test" section: the procedure, what the numbers mean, the findings below, and a recommended configuration.
• inspect_eval/ollama/qwen3.5-4b.Modelfile (16k context) and a roster.md row for Qwen3.5 4B.

No solver, scorer or prompt code changes. The 52 existing tests and the 4 new ones pass.

Setup

Local reproduction, since there's no Freestyle access yet:

• Model: Qwen3.5 4B (qwen3.5:4b, Q4_K_M, num_ctx 16384)
• Server: Ollama 0.35.1 (desktop app server), Apple M3 Pro 18 GB, 100% GPU
• Software: Inspect 0.3.276, openai 3.24.0, macOS 14.6
• Runs: k = 1, the 39 corpus items (13 multi-turn), 68 model turns per run, solver alone (--no-score)
• Judge: Gemma 3 4B (num_ctx 16384), applied afterwards with inspect score. It's a different family from the model under test, and the most robust small judge in #113 / #111.

Results

*117 min including a 9-minute Ollama app restart. Inspect retried the in-flight turn and no turn was lost.

--same-prompts on defaults vs --max-connections 1: OK - 68 model calls send the same prompts and config. The max_tokens and thinking rows change the request, and the check flags them, as it should. They're reported as trade-offs, not speed-ups.

Findings

1. At max_tokens 2048, the token budget changes the headline numbers.
   • Qwen3.5 4B thinks by default, and on half the turns the reasoning uses the whole budget: 34 empty replies and 19 cut off.
   • The judge still grades an empty reply. Gemma labelled 32 of 34 empty replies diverge, so opening divergence reads 49% instead of about 18–21%.
   • Verified two ways: in Ollama's raw responses (finish_reason: length, content: "", reasoning ending mid-sentence), and with Ollama's native /api/chat outside Inspect.
   • The earlier Qwen3-4B-Instruct-2507 runs aren't affected; it's a non-thinking model.
   • At 8192 the longest reply was 7,212 tokens, and no turn overflowed the 16k context budget.
2. Ollama serves Qwen3.5 one request at a time.
   • OLLAMA_NUM_PARALLEL is ignored for this architecture (model architecture does not currently support parallel requests, qwen35; the runner starts with -np 1).
   • With Inspect's default 10 connections, queued requests hit the openai client's 600 s read timeout: 51 HTTP 500s, each retried.
   • --max-connections 1 removes them all at the same throughput (30.1 tokens/s), so the gain is reliability, not speed.
3. llama.cpp's llama-server can batch this model, modestly.
   • Same GGUF, -np 4: 35.9 / 41.1 / 45.0 tokens/s aggregate at 1 / 2 / 4 concurrent requests, so 1.25× at most.
   • At temperature 0, a request's output changes when it's batched with others; run alone, it's deterministic.
   • It's an option, not the recommendation, since it also moves off the target setup.
4. --reasoning-effort none is silently dropped by Inspect's ollama provider.
   • clamp_reasoning_effort_to_low_medium_high returns None for none, so nothing is sent and thinking stays on.
   • --extra-body '{"reasoning_effort":"none"}' reaches Ollama and works.
5. Where the time goes:
   • Nearly all of it is generation: about 36 tokens/s per request, with the model kept loaded (load ≈ 0 s).
   • Locally there's no network cost. On Freestyle, the VM-to-laptop round trip would add to each call, and the per-call generate time in speedtest.py will show it.
   • Grading a k = 1 log with Gemma took 1.5–3.5 min, so the judge is small next to generation on this setup.
6. Inspect resends each earlier reply with one leading newline added. 96/96 cases; the reasoning is resent unchanged. It's the same in every configuration, so comparisons hold, and --same-prompts expects it. It's noted here for transcript fidelity.

Recommended configuration (Qwen3.5 4B on Ollama)

• --max-connections 1.
• -T max_tokens=8192 if the benchmark keeps thinking on. Whether to keep thinking on is a benchmark decision, not a solver setting. Turning it off is 5.5× faster but measures different behaviour.

Projected full run (k = 5, current corpus)

296 turns: 195 opening and 101 pressure (29 + 18 × 4 from the stratified subset), timed at each run's mean per turn type:

Not done

• No numbers from the Freestyle VM yet. The procedure is the same, so it can be rerun there as is.
• Making the scorer treat an empty reply as invalid, instead of asking the judge to grade it, belongs in its own PR. I'd like to raise it in a Discussion first.