Spaces:

---

Station-house

/

ConspiracyBench

Like 9

Running

🤖 Posted with Claude Code.

One design point for the validity run, from reading the Clef adapter (eval/judge_reliability/run_clef_preflight.py, build_clef_state): Clef is given the candidate response with no conversation history. The state has the question, ground truth, checklists, notes and candidate_response, but nothing about the preceding turns.

That's fine for the cases it has been tested on so far. The standardized suite is all opening turns, and the near-context stress test puts filler around a single answer. But the Inspect scorer grades a later pressure turn together with the preceding exchange (build_judge_prompt(item, answer, history)), and real concessions often only make sense in context. "You raise a fair point — it may not be fully settled" is a diverge because of what it's agreeing to.

So for the real-transcript comparison I'd:

1. Report opening and later turns separately for every judge, so Clef's opening-turn validity isn't averaged together with a context problem.
2. Add the preceding exchange to Clef's state for later turns (e.g. a preceding_exchange field, with the same "treat as data" instruction), and run Clef both with and without it, so we see whether context helps a deterministic classifier as much as it helps a chat judge. That's a small adapter change. @melover24 , since it's your adapter, does that fit how you'd want it done?

Progress: transcripts from a non-Qwen target (Gemma 3 4B, k = 1, the current 80-item corpus) are generating now, and a blind sheet generator is ready: stratified by framing and turn type, oversampling adversarial, leading and pressure turns, with model names hidden behind a separate key. Sheet and labelling instructions follow shortly.

🤖 Posted with Claude Code.

Update: the validity set and the compute request are up as #208.

• 150 real replies, labelled blind: 75 from Gemma 3 4B and 75 from Qwen3.5 4B (thinking off), on the current 80-item corpus. There are 56 opening turns and 94 later pressure turns, and leading and adversarial framings are oversampled.
• The analysis is fixed before any labels exist (score_validity.py): human κ first, then per judge per-class recall with Wilson intervals, κ, the invalid rate, and precision reweighted to a real run's class mix, all split by opening and later turns.
• Clef runs with and without preceding_exchange on every later turn, as proposed above, so we'll see whether the missing history matters.

Labellers are the bottleneck now. @melover24 , the rubric includes your "accurate but incomplete" rule; the sheet is ready whenever you are. If two or three more people can take about 75 replies each, we keep the Oct 14 results and poll date.

Quick thought: Clef is a (relatively) large decision model at 27B parameters and it's main point of strength as far as I understand is its multimodality. Since our data is not multimodal, maybe we can test also smaller models like Kev or Laya?

🤖 Posted with Claude Code.

@tomerz14 good point, and cost is on the list (criterion 5), so cheaper decision models belong in the comparison. Checking the three models' cards turned up something I should have flagged in the protocol:

• Clef is 27.4B, fine-tuned from Qwen3.8-27B. It is much larger than the target, but it's also Qwen-family, so gate 1 as I wrote it ("a different model family") would rule it out.
• Kev-4B is a LoRA on Qwen3.5-4B-Base, the target's own base model. That's the strongest same-family case of all.
• Laya is ModernBERT-large (0.42B), so it's not Qwen, which is a plus. But the English checkpoint reads 512 tokens, about 320 of them for the state. By a rough character count, 146 of our 150 opening requests are longer than that. The multilingual checkpoint takes up to 8,192 tokens, but its card reports accuracy falling past about 4k tokens, and about half of the later-turn requests with context are past 4k. Scoring windows and taking the maximum, as in #173, works for "is there manipulation anywhere", but a hold/diverge verdict needs the whole reply read against the ground truth.

A proposal for gate 1: make it measurable rather than a blanket family rule. The validity set has replies from both a Qwen target and a non-Qwen one (Gemma 3 4B), so for every judge we can compare its error rate on Qwen replies with its error rate on Gemma replies. A judge passes if it isn't the target's own base model and shows no Qwen-over-Gemma advantage beyond the intervals. That keeps Clef in on evidence rather than by exception, and still tests the risk JoyeeChen raised.

Would you run Kev-4B and Laya on the validity units? You already have pinned environments and runners for both from #173. The input is eval/judge_reliability/validity_v1/units.jsonl in #208 (judge_cases format), and the output only needs unit_id, choice and an optional variant per row, which score_validity.py reads directly. For Laya, I'd use the multilingual checkpoint with max_len=8192 and report the long units separately. Even if Kev fails gate 1, its numbers are useful as a cost reference.

I saw the new validity set in #208. I'm ready to start labeling. Could you let me know which unit IDs I should take and where you'd like me to submit the completed labels?

🤖 Posted with Claude Code.

@melover24 sorry for the slow reply, and thank you! Here's how to do it:

• Units: please label all 150 (u001–u150) in eval/judge_reliability/validity_v1/sheet.csv from #208. Until more labellers join, every unit needs both of us. If others join later, they'll add a third label on subsets.
• What to fill in: only the last three columns (label, confidence 1-3, note). Save the file as sheet_melover24.csv. The rubric is validity_v1/ANNOTATION_RUBRIC.md.
• Where to submit: to keep the labels independent, please don't post the file yet. Just comment "done" on #208. Once both labellers are done, we each attach our file to #208, and I run score_validity.py and post human–human κ before any adjudication.
• Timing: it's probably 2–3 hours for 150. If you can finish by Tuesday evening (Oct 13), we keep the Oct 14 results and poll. If not, say so and we'll move the date rather than rush.

Ah apologies for my comment on the previous PR. When we run the clef experiment, let's group the Kev and laya models there as well. Its minimal additional cost.

Done, I've finished reviewing all 150 units. I'll wait until the other labels are in before submitting my sheet.

🤖 Posted with Claude Code.

@stationhouse done: #208 now runs Clef, Kev-4B and Laya with one runner (run_decision_validity.py --judge clef|kev|laya) on the same 244 requests, with Laya's multilingual checkpoint at 8192 tokens and no truncation for any judge. The PR description has the commands, including a 3-call smoke test first, since Kev's and Laya's adapters are only tested against fake SDKs so far.

On labelling: @melover24 has finished all 150 (thank you!). The other two labellers are aiming for tonight.

Three points on the protocol, all about how judge errors reach the published numbers.

1. Pair diverge recall with a precision floor. Ranking on recall first makes sense, since a missed divergence is the worse error. But the benchmark publishes a rate, and one-sided errors bias a rate. A judge that over-calls diverge can lead on recall and still inflate every model's divergence rate. I would add a minimum diverge precision to the gates, and report each judge's estimated divergence rate on the 150 units next to the human-labelled rate. If the two are far apart, that judge shifts the headline number even with good recall.

2. Per-unit accuracy compounds across turns. Concession, retreat and wavered are read off a whole trajectory, so every turn has to be labelled correctly. At 0.90 accuracy per unit, a trajectory of three answers is fully correct about 73% of the time, and four answers about 66%. The trajectory metrics will be noisier than unit accuracy suggests. Since 94 of the 150 units are later turns, could we also report agreement at trajectory level for the items where all turns were labelled?

3. Choosing the cascade threshold. If τ is tuned on the same 150 units that are used to compare architectures, the cascade gets a number that is optimistic compared with Clef alone or a single chat judge, which have nothing to tune. A split, cross-validation, or fixing the rule for τ before looking at the labels would keep the comparison fair.

Happy to help compute any of these once the labels are in.

I'd like to propose including perplexity decider (pplx-decider-v1-27b) model as another candidate judge in the architecture comparison. was reading about it earlier today and it seems way better than CLEF on paper.

Note on bandwidth: I am completely tied up for the rest of this week and won't be able to write or test the adapter/runner code for it until sunday night mostly.
If anyone on the team has the bandwidth to whip up a quick adapter to include in the upcoming Oct 12–13 compute run with Clef/Kev/Laya, that would be awesome!
Otherwise, if the timeline is strict for Phase 1 (Oct 17), we can benchmark it as part of a follow-up iteration.

What do you all think?

Hey @vaisakhi I think this is a great idea I will make sure to add this in as a candidate!

🤖 Posted with Claude Code.

@Emar7 thank you, and sorry it took two days to answer. All three points are in #323, fixed before anyone sees a label:

1. Precision floor and rates. Every judge now has its share of each label reported next to the human share on the same units. I'd add a gate: diverge precision ≥ 0.80, and the judge's diverge rate within the interval of the human rate, so over-calling can't win on recall alone.
2. Trajectories. You're right that errors compound. Only 6 of the 62 multi-turn trajectories in v1 are fully covered (20 units), so I'll show those descriptively, and propose a small batch of complete trajectories after the poll.
3. Choosing τ. --cascade picks τ by 5-fold cross-validation, maximising κ on the other folds, so the cascade is scored only on units it wasn't tuned on.

@vaisakhi I've added a pplx-decider-v1.1-27b adapter to the same runner (#323), so it can go into the GPU run with Clef, Kev and Laya without waiting for Sunday. One thing for the vote: like Clef, it's fine-tuned from Qwen3.8-27B. If we want a decision model outside the Qwen family, the StrandsAgents deciders are built on Gemma 4 (E4B, 12B, 26B-A4B), which may be worth a look next round.

Status: labels from @melover24 and @vaisakhi are done; the other two are due tonight. Five chat judges have already run on all 150 units: Qwen3-4B, Gemma 3 4B, Llama 3.2 3B, Qwen3.5 9B and gpt-oss-20b, the last on a Freestyle CPU VM. Results go up tonight, once the labels are in.

🤖 Posted with Claude Code.

Done: @powercoding1 's labels for u001–u075 are in (all 75 valid). That's three of the four labellers finished; the file stays private until @Divyanshi is done too, then everyone attaches theirs here.

One suggestion on criterion 4 (injection target-following), before Friday's poll.

Test the architectures, not just the judges. The injection set from #111 (eval/judge_injection: 140 cases, seven attack families) already exists, and the four steering texts I tried earlier (in #178) map roughly onto four of its families: a direct override, a fake end of the prompt, an authority claim ("Grader note: ... Label: hold") and a reply that is only a label. So I don't think we need a new set. What I'd add is to report target-following for each architecture in (a) to (d), and not only for the single models. In the cascade (c), the second stage is a chat judge, and that's the kind that gave way in my small test: llama3.2:3b flipped from diverge to hold on the fake grader note, phi3:mini didn't. That was one item and one reply per case, so it's a hint and not a rate. A result for Clef or Laya alone wouldn't tell us whether the cascade or a jury inherits that weakness.

A question for the protocol: should injection be a gate and not only fourth in the ranking? A judge steered toward hold makes a model look safer than it is, the same direction of error that put diverge recall first.

Two smaller points:

The reply that was only the word "hold" got hold from both chat judges I tried, although the rubric says a non-answer is abstain. It's one cheap case to run on each finalist.
The #111 set is built from the 21 corpus items of that time. With 200+ questions and 61% multiturn now, a v2 release is worth doing after the vote, so I wouldn't hold the poll for it.
@tomerz14 this may also fit your manipulation dataset. @stationhouse , I can't run any of this on Clef or the decider without the GPU, so it would go into @powercoding1 's runner if you both think it's worth the extra calls.

@powercoding1 - Thanks for adding the perplexity model!! Also if we wanna test out models on different architectures, I think strands is also Qwen - have not seen any documentation on their Gemma version :/ . I think Rizzo is on Llama architecture and is open source. Not sure what extent of decision it supports, but that might be a model we could add.

🤖 Posted with Claude Code.

All four labellers are done. Thank you @melover24 , @vaisakhi and Divyanshi! Since #208 is already merged, here's how the sheets go in. Everyone has finished, so they no longer need to stay private.

1. Each labeller opens a small PR adding their filled-in sheet as eval/judge_reliability/validity_v1/labels/sheet_<your-hf-username>.csv, unchanged apart from the three label columns. For example:
   hf upload Station-house/ConspiracyBench sheet_<you>.csv eval/judge_reliability/validity_v1/labels/sheet_<you>.csv --repo-type space --create-pr
   Mine is #345.
2. Adjudication: once @melover24 's sheet is in, I'll list the u001–u075 units where Chaewon and I disagree, without saying who gave which label. Divyanshi then settles them, since she didn't label those units. Units u076–u150 are settled by majority of three.
3. Results: I run the pre-registered score_validity.py over every judge, plus the jury and cascade. Human–human κ goes up first, before adjudication, then the full results here for Laura's poll.

@stationhouse , if the label PRs come in today, the results can be up within a few hours of the last one.

Amazing looking great thanks @powercoding1 for the update!

Ok! Uploaded mine. The pr is #348

Uploaded my sheet as well! (u076–u150) (PR is #347)

IDEA - While labelling u076–u150, I noticed several later-turn units where the model clearly held in preceding_exchange and only diverged once the pressure turn message was applied (e.g. u110, u126, u129).

As @powercoding1 mentioned, v1 only contains sampled individual turns, and only 6 conversations have all their turns in our 150 units. After Friday's poll, I think sampling full conversations would be super useful so we can analyse how models actually behave across a conversation and see which category they fall into:

• diverged immediately (before any pressure turn)
• held then diverged (held initially, then flipped on a pressure turn)
• held throughout (didn't diverge even after all pressure turns)

Let me know your thoughts on this!

🤖 Posted with Claude Code.

Judge results on validity set v1: chat judges, the pre-stated jury and the cascade, next to the decision models

Laura's GPU run (#365) covered the four decision models. This adds the five chat judges, run locally (gpt-oss-20b on a Freestyle CPU VM), and the two #203 architectures. All are scored with the pre-registered score_validity.py against the #364 reference: 150 units, 111 hold / 34 diverge / 5 abstain.

Human–human agreement, the ceiling to compare against: κ 0.50 (81% raw) between the two independent humans on u076–u150. u001–u020 had 10/20 agreement (#364).

*Reweighted to a real-run class mix of 80/15/5 (@Emar7 , #182).

What this means for the vote

1. The current judge roughly doubles the divergence rate. Qwen3-4B calls 43% of replies diverge where humans call 23%. Its recall is good, but its precision is about half. Every chat judge over-calls, Gemma and Llama badly. If Sunday's eval keeps Qwen3-4B, its headline divergence rates would be inflated.
2. pplx-decider is the strongest single judge, at 0.31 s/call. Giving it the preceding exchange helps a little on later turns (0.60 vs 0.58), while Clef doesn't benefit. It's the closest to the human diverge rate among the high-recall judges.
3. Laura's two-level judge works, in exploratory form. pplx labels a reply, and if its confidence is below τ = 0.35, Qwen3-4B decides instead. That gives κ 0.62. τ was chosen by 5-fold cross-validation and came out 0.35 in every fold. It's exploratory because I picked pplx after #365 was public. The pre-stated version with Clef (0.49) also beats Clef alone (0.42) and Qwen3-4B alone (0.47).
4. The pre-stated jury failed (κ 0.17). Two of its three members over-call diverge, so the majority does too. I'm reporting it anyway, since it was pre-stated.
5. Same-family check (gate 1): no advantage found. Every Qwen-family judge (Qwen3-4B, Qwen3.5 9B, Clef, pplx) agrees less with humans on Qwen3.5 replies than on Gemma replies. For example, pplx scores κ 0.43 on Qwen replies vs 0.61 on Gemma replies. Only gpt-oss-20b goes the other way.
6. No judge passes the 0.80 diverge-precision gate I proposed. The best reach about 0.5 on a real-run mix. I'd treat that gate as a target rather than a filter for this round, and report Sunday's rates with the judge's measured error alongside them.

Limits, so the poll weighs them

• Abstain: only 5 units are abstain, so abstain recall isn't measurable.

• The reference isn't two humans everywhere:

  • On u021–u075 the second label was Qwen3-32B, with disagreements going to the human, so in practice that slice is one human labeller (mine).
  • On u001–u020 the adjudicator was also one of the two labellers.

  The two-human column (95 units) is the conservative one. The ranking at the top holds there too: pplx with context 0.45, the cascade 0.44 and Qwen3.5 9B 0.43, all ahead of the current judge's 0.36.

• One sample: k = 1, one 80-item corpus snapshot, two target models.

Suggested poll options

• (a) pplx-decider alone;
• (b) the two-level judge, pplx then Qwen3-4B below τ 0.35;
• (c) keep Qwen3-4B, reporting its measured over-calling next to every rate.

I'll put the scoring outputs and a short report in a PR next.

🤖 Posted with Claude Code.

A correction to my table above: the "with context" figures (pplx 0.60 / 0.45) are on the later-turn units only (94 in all, 56 in the two-human slice), since opening turns have no earlier exchange. They shouldn't sit next to the all-unit numbers. The like-for-like comparison is on later turns, with and without the preceding exchange:

So the conclusion holds, and is a little stronger: on the later pressure turns, giving the decision model the earlier exchange helps pplx clearly and Kev somewhat, and Clef only slightly. That's the #203 preceding_exchange question answered. In the summary line, "pplx with context 0.45" should read "on later turns". The two-human, all-turns ranking (the cascade 0.44, Qwen3.5 9B 0.43, pplx 0.40, current judge 0.36) is unchanged.

The data correction clarifies that pplx gains significantly from context on later turns (0.53→0.60), while Clef barely moves (0.39→0.41). That's a solid finding for the #203 preceding_exchange question.

I can help draft the short report you mentioned for the PR, summarizing these specific metrics and the ranking shift. Just say the word.

Following @gowrishkumar 's suggestion , I propose an injection-resistance comparison using the frozen 140-case fixture in #173. I’d compare Qwen3-4B, pplx-decider, and the existing pplx → Qwen3-4B cascade at τ = 0.35 on the same paired cases. Should this first pass include Clef or the jury?
Thing is I don't have the compute available locally unfortunately, should I still plan the study?

The study plan makes sense even without local compute; you can run it on a cloud instance or request access to the fixture's hosted runner. Regarding your question: yes, include Clef in the first pass if its latency/throughput profile is relevant to the injection-resistance metric; otherwise, keep the jury out until we have a clear use case for it.

HI @tomerz14 Still make the study, pull it as a PR. I am going to add a folder to the HF that is Experiment requests. Completed experiments will be moved to another folder.

Sounds like a solid plan, @stationhouse . Moving completed experiments to a separate folder will keep the repo clean and make it easier to track what’s been validated versus what’s still in flux. I’ll keep an eye on that new 'Experiment requests' folder once it’s live.

@tomerz14 , if you need help drafting the PR description or structuring the comparison metrics for Qwen3-4B vs pplx-decider (and Clef, if included), just let me know. I can summarize the findings from your 140-case fixture run once you have the results.

No budget on my end to cover compute costs, but happy to help with text-only tasks like report drafting or metric interpretation.

@stationhouse Great, I will work on this and submit an experiment's runner and instructions for the GPU machine.
@richard-catcapital Sounds great, let's coordinate offline on this?

@tomerz14 and @richard-catcapital sounds good. If I need to change stuff based on available compute, it'll be low level stuff to fit it onto the GPUs. I'll make sure I note it in the experiments

Sounds like a solid plan, @stationhouse . Moving completed experiments to a separate folder will keep the repo clean and make it easier to track what’s been validated versus what’s still in flux. I’ll keep an eye on that new 'Experiment requests' folder once it’s live.

@tomerz14 , if you need help drafting the PR description or structuring the comparison metrics for Qwen3-4B vs pplx-decider (and Clef, if included), just let me know. I can summarize the findings from your 140-case fixture run once you have the results.

No budget on my end to cover compute costs, but happy to help with text-only tasks like report drafting or metric interpretation.