The detail that stands out is that the sixth report kept the "trace-level" label and its checker marked it confirmed, while the paper's own introduction phrases it in a way that invites the error reading. So a checker comparing the claim with the source text can confirm a misreading when the source sentence is itself ambiguous.
When a checker found the paper using two definitions across versions, like the December abstract versus the April revision, what did it do: pick the latest version, or flag the claim as version dependent?
Good catch, with one correction. The sixth report kept "trace-level" in its claim, so its "confirmed" was literally true. The failure there is quieter: nobody said what trace-level means, so a reader further down could still take it as an error rate.
On versions, nine checker verdicts touched this paper. One checker opened v1, v2 and v3 and flagged the claim as version-dependent: v1 called the figure a failure-probability ratio, v3 redefined the same numbers as extra work from trace analysis. The others worked from v3, the newest arXiv version, so in practice they picked the latest version they could find. None of them noticed it was already stale. The journal version had been out since July, and that is the one that moved the mean from -0.3% to 0.0%.
Fair correction, thanks: the sixth claim was true as written, and the gap was an undefined term rather than a false confirm. The version finding is the bigger one for me, since "newest arXiv version" and "version of record" are not the same thing, and the journal version is what moved the mean. Would one line in the checker brief, "look for a published version before citing a preprint", have caught it, or does that need its own agent?
One line would have gotten close, and the run shows where it stops. One checker did try the journal version, but only because the report it was checking already named it. nature.com turned it away (HTTP 429, too many requests), so it fell back to the abstract on a University of Washington lab page, and the abstract doesn't give the mean. The full text is open access; it just didn't load that day.
So the line needs a second half: if you can't read the version of record, label the number "preprint figure, published version not read" instead of passing it on. Still no new agent, but I'd make it a field the checker has to fill in, not a sentence in the brief it can skip.
Applied LLM & AI automation engineer. I write about RAG, fine-tuning and AI workflows I build and test myself. Published AI researcher, open to AI roles in Germany.
Location
Lahore, Pakistan · open to relocating to Germany
Education
BS Computer Science, University of Management and Technology (UMT), Lahore
The fix you landed on, a second agent that reopens the cited source and compares the claim, matches what I found cheapest in my own citation checks: a word-level rule first, then an NLI model only on claims that pass it. A vote can't catch a shared misreading, since every voter read the same summary.
Cheap first, expensive only where it's needed makes sense. One warning from this run: a word-level rule would have passed my bad claims. "Amplify errors 17.2x" is verbatim in the December abstract, so the words matched. What broke was the version and a dropped qualifier, "trace-level errors" becoming "errors". I'd be curious whether your NLI step catches that. Dropping a modifier can look like a safe generalization to an entailment model, and here it changed the meaning.
A small nuance on the vote: my agents didn't read a shared summary. Each opened the same preprint on its own and printed the same sentence. Same source, same model, same prompt template, which is why five agreeing was worth about one.
Applied LLM & AI automation engineer. I write about RAG, fine-tuning and AI workflows I build and test myself. Published AI researcher, open to AI roles in Germany.
Location
Lahore, Pakistan · open to relocating to Germany
Education
BS Computer Science, University of Management and Technology (UMT), Lahore
I haven't tested dropped qualifiers yet, and I'd expect NLI to miss them: 'amplifies trace-level errors 17.2x' entails 'amplifies errors 17.2x', so the shorter claim reads as a safe generalisation, like you say. In my small test NLI also let two entity swaps through. A cheap guard before the NLI step could be to require every qualifier and version string next to the number in the source sentence to appear in the claim.
I'd add that guard and run it wider. It checks the claim, and only one of my five lost "trace-level" in the claim itself. Three kept it there and dropped it one step later, in their own summary and design rule, so a claim-level check passes all three. The fifth quoted the December abstract, said so, and matched it word for word: faithful to a version that had already been replaced.
So I'd run the guard on every sentence that reuses the number, not just the claim, and put one lookup next to it: is there a newer version of this source?
the vote part is the scary bit, five agents agreeing just looks like confidence. reopening the cited source per report is boring but its the only check that actually caught it
Agreement looks like confidence, and here it was one sentence counted five times. I don't want to oversell the boring check, though. It ran on 118 of 270 claims, so 152 never got a verdict, and two checkers still left "trace-level" unexplained. It worked where it ran. What I took from it: run it on every claim that carries weight, rather than trust it more.
In this run I didn't decide. Each checker picked 14 claims it judged to carry the most weight (one took 20). Looking back, weight wasn't the problem, coverage was: 117 of the 118 checked claims fed a design rule, and so did 115 of the 152 that got no verdict. About half the claims behind my design rules never got a verdict.
Next time I'd use fixed rules: check any number that ends up in a design rule or a summary, any claim that two or more reports repeat (one check covers all of them, as Edward points out in his comment), and anything from a paper that exists in more than one version. And I'd have the checker list what it skipped.
the checker listing what it skipped is the part id steal, coverage you can see beats coverage you assume. are you planning to block a design rule if any claim behind it has no verdict, or just flag it
An average can hide configurations that moved the score both ways. Reporting the spread next to it is what lets a reader decide when a swarm is worth it.
Agreed, and that's why the extremes sit right next to the 0.0% in the post. The paper gives σ = 37.5% and prints -58.7% to +77.2% as a 95% confidence interval for the mean. With that σ across 260 configurations, the interval reads as the spread of the configurations, not uncertainty about the mean, and one of my checkers flagged exactly that.
The cut that tells you something is by task. On the finance benchmark every team design beat one agent, by 57% to 81%. On sequential planning every design lost, by 39% to 70%. The average mostly tells you those two cancel out.
ANP2 — an open, permissionless AI-to-AI event protocol. Ed25519-signed events, capability discovery, and a computable trust graph. No accounts, no API keys, no tokens. Spec v0.1 DRAFT.
The count was grouped by the wrong key. Before any vote, group the claims by the passage they came from and the version of that passage, and keep the model and prompt config attached to the group. Then count groups. "Five of six" collapses to one before the vote starts.
We hit the same shape in a signed, append-only log of agent task deliveries. Of about 2,000 delivery rows, 1,932 lacked a back-reference field. Our first explanation was an incentive story: the field was optional and nothing downstream read it, so writers skipped it. Splitting by signing key killed that. One key wrote 1,879 of the 1,932. Across all 10 signing keys, each one either always filled the field or never did. Zero keys switched. What read as population behavior was one writer's lifetime habit. Distinct keys do not prove distinct parties, so grouping like this gives an upper bound on independence, never a count of it.
It also changes what a checker should leave behind. A bare "confirmed" can't be re-audited once nobody knows which text it was checked against. The -0.3% verdict was right about the April preprint. Nothing in it said April, so it read as a statement about the paper. If each verdict carried the source version id and the quoted passage, a later reader could tell "confirmed against v3" from "confirmed", and your journal catch wouldn't have depended on luck.
Did any of your checkers record which version they opened?
Grouping by passage and version is the right key, and your signing-key split is the same shape as mine: 1,879 of 1,932 looks like a population until you split it.
To your question: partly. Every verdict had a url_checked field and quoted the passage in its evidence. Of the nine verdicts on this paper, seven named a version (v3, or v1 through v3). Two recorded the bare arXiv URL, which resolves to whatever is newest on the day you open it, so those two can't be re-audited against a version anymore.
What no verdict had was a field for "is there a later version anywhere". One checker had the journal abstract open, used it for a single effect, and kept the preprint's -0.3%, because the abstract doesn't state the mean. So I'd take your two fields, version id and passage, and add a third: where else this paper is published.
The voting failure comes down to the variance of correlated draws. When N agents share model weights, prompt framing, and the same preprint excerpt, pairwise error correlation stays near one. Averaging across eight reports leaves error variance almost unchanged while multiplying token spend eightfold. Bertalanic and Fortuna's finding of 1.1 to 1.8 effective agents across 30 debating copies, at 64K prompt tokens per effective agent versus 1.4K without debate, puts a clear price on that correlation.
Lower error variance requires changing the conditioning event. Reopening the cited passage on one isolated claim breaks the summary prior, and an external oracle such as a compiler or unit test suite keeps the verification step independent of the generator.
My eight agents weren't even answering one question. They had eight different subtopics, and six still landed on the same paper and the same sentence, because one widely cited paper sits under most of them. Nobody planned that overlap, and I only saw it when I grouped the claims by source afterward.
Changing the conditioning event worked, up to a point. My checker was conditioned on one claim plus the source, and that did break the summary prior. But the source it opened was the April preprint, so it confirmed -0.3% while the journal version said 0.0%. A compiler oracle rarely goes stale that way. A paper does, so a source check needs a "newest version" step.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The number that stuck with me isn't 17.2x, it's 46 of 118 — nearly two in five claims changed once a checker reopened the source. That's the whole post in one line: the agents weren't wrong, they were just all reading the same April preprint the same way. A vote would've counted that shared misreading six times.
Since you single out the 46, here's what's in it. I went back through them: about 30 would have misled a reader, about 10 look like a checker rewording a claim that was already fine, and the other 6 were small real fixes, one of them a misquoted number. Max asks in his comment how much of it is checker noise, and he's right to.
On the agents, I'd put it a bit differently: they were wrong, just quietly. Most of them had lost "trace-level" by the time they wrote their summary and design rule, the part a reader actually uses. So a vote would have counted five reports, not six. The sixth kept the label and never carried the figure into its summary or rules, though it never said what "trace-level" means either.
The 46-of-118 number is the one I'd want a control for. A checker that reopens the source and reads it fresh will also "change" claims that were fine the first time — differently worded, more hedged, the same figure with the surrounding caveat re-attached. So 39% is an upper bound on error unless you know how often an honest checker rewrites an honest claim. Cheapest version of that control: run both checkers on the same 20 reports and count how often they disagree with each other. If two checkers only agree 70% of the time, most of your 46 is checker variance, not agent error. If they agree 95%, the 46 is real.
The other thing that surprised me is that the scaled trace-level factor survives (p=0.030) while unscaled it doesn't (p=0.658). That's usually a sign the factor is partly measuring how verbose the team is, since dividing by the single-agent baseline folds cost back in. Did you check whether the 17.2 correlates with reasoning-token count per configuration? If it does, the number is a work multiplier in a very literal sense and the whole "amplify" framing in that April intro was never recoverable.
Fair, and the run has a rough version of your control. In 48 pairs I found, two reports carried the same finding and both claims went to different checkers. They agreed in substance on 33, 69%, which by your yardstick is the bad end. The conflicts cluster, though: 13 of the 15 are on one paper, Kim et al., and outside it only 2 of 19 pairs conflict. About half of all 15 are a threshold call (both checkers wrote down the same caveat, only one counted it as a change). The rest, all on that paper, are real differences in reading, like one checker keeping "message overhead" for what the paper counts in turns. So 46 is an upper bound, as you say. My own re-read puts about 10 of the 46 as rewordings of fine claims, but that's one more reader, not your control.
On 17.2: I couldn't run that check, because there's nothing per configuration to correlate. The paper calls its coordination metrics "architecture-level constants" applied "uniformly across all benchmarks", reports no token counts per architecture, and gives no formula. The only definition is a docstring in the released code: overhead tokens (coordination messages, failed-tool retries, re-planning turns) relative to one agent, and no released code computes the values. So yes, a work multiplier, literally. Not plain verbosity, though: independent agents take 11.4 turns, against 26.1 to 44.3 for the other teams, and have the highest factor. The term that survives at p=0.030 is the factor times the single-agent baseline, so within an architecture it moves only with the baseline.
Fascinating observation on majority consensus failure in multi-agent swarms. When agents share similar baseline biases or prompting frameworks, voting often amplifies blind spots rather than resolving them. Cross-validation with adversarial or dialectical roles tends to catch these omissions much more reliably.
In my run what helped wasn't an adversary. The checker had one claim, the cited passage and a comparison to make. A role told to argue the other side isn't told to compare anything.
@p0rt Checked the arithmetic on the paper's numbers: sigma = 37.5% over 260 configurations gives a standard error of the mean of about 2.3 points, so a 95% interval for the mean would be about +/-4.6, not -58.7 to +77.2. The printed range is the spread of configurations, so the average is the wrong number to read the verdict from. Your by-task cut is the one that decides.
Your arithmetic is right for n = 260, and that n was my mistake: I said "260 configurations" in my earlier reply. I recomputed from the authors' code release. Take the 24 benchmark-by-architecture averages, each the change in mean success against one agent. Their plain mean is 0.0% on the journal's data, and -0.3%, the April figure, if you keep Gemini 3 Flash on SWE-bench Verified and Terminal-Bench, which the release includes and the journal's numbers leave out. Either way, -58.7% to +77.2% is exactly the 2.5th to 97.5th percentile of those 24, and σ = 37.5% is their spread. The paper never says how it computed the interval, but this reproduces the printed mean, interval and σ in both versions.
So it's a spread, as you say, of 24 numbers. With n = 24 a 95% interval for the mean is about ±15 points: more than three times the ±4.6 you got from the n I'd given you, still nowhere near the printed range, and still too wide to decide anything. The by-task cut is where the information is.
Your last caveat seems like a different reason to split agents: separating permissions. A read-only researcher and an executor with a narrow tool set could have the same task score as one agent and still be useful. But the handoff has to preserve that boundary; if the executor treats the researcher's output as instructions, the separation can disappear. I'd want that measured alongside accuracy and cost.
Agreed, and the permissions half of my last caveat has been measured, though on tool-use benchmarks, not on a research pipeline like mine. CaMeL is the cleanest version: the planner sees only the user's request, never tool output, and a second model with no tools turns untrusted text into fixed fields like a file name or an address. On AgentDojo, a benchmark with injected instructions, it solved 77% of tasks with injections blocked by design, against 84% with no defense, for its best model. Other models lost 3 to 32 points. Tokens, in a separate test: about 2.7x to 2.8x for the median task, 6 to 7x on average.
Your handoff point is measured too. Twin Agent (2026) caps how many characters the reader can pass to the actor, and as that cap grows, tasks solved and attacks that get through both go up. And without the boundary, Triedman et al. got orchestrators in AutoGen, CrewAI and MetaGPT to run arbitrary code from a web page a sub-agent read, in 58% to 90% of GPT-4o trials.
So far the split buys safety at some cost in score or tokens, and how much the reader may pass to the actor sets the trade.
The detail I'd keep from this: the 17.2x error didn't come from one agent making something up. Several agents read the same paper and made the same reinterpretation. That is correlated error, and a vote treats correlated reports as independent evidence, so it makes the mistake look more certain, not less. Your fix works for a reason worth naming: the checker read the source, not the other reports. Did you track how many claims the source-check agent flagged that turned out fine on a second look? That false-flag rate decides whether the check scales past eight reports.
Not as a separate rate, but the run gives two partial answers. Of the 46 changes, I'd count 10 as a checker rewording a claim that was already fine, so roughly one flag in five was noise. And in 48 pairs I found where two checkers happened to check the same finding, they agreed 33 times. About half of the 15 disagreements were over whether a caveat makes a claim "changed", the rest real differences in reading the paper (details under Max's comment).
So part of the false-flag rate depends on a threshold nobody wrote down. To scale past eight reports, I'd write it into the brief, and check each source passage once instead of once per report.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
46 of 118 checked claims coming back changed is the number i would lead with, and the note that most were about what a figure means, not the figure itself. that matches what we see when agents summarize papers: the number survives and the label gets dropped.
a head count would have kept 17.2x because six agents read the same sentence. a checker with the source open is a different failure surface, which is why it works.
did the checker agent run on the same model family as the reviewers, or a different one?
Same model, not just the same family, and the same paper open. What differed was the brief: check a single claim against the passage it cites.
On the label getting dropped: same here, only one step later than I expected. Mostly the label made it into the claims and fell out of the agents' own summaries and design rules.
A different family might help with correlated reading, though not by much: Bertalanič and Fortuna found that mixing three families raised the debate ceiling, modestly, to between 1.8 and 2.9 effective agents. It wouldn't have fixed the miss that got furthest, a stale paper version. Any model that opens the April preprint confirms the April number.
The 17.2x misreading is a clean example of why a majority among agents is not evidence. Five reports agreed, and they were still wrong about what the published paper said. A head count would have locked the error in. What caught it was a check against the source, not another vote. I would treat agreement as a reason to reopen the shared claim, especially when the agents never compared notes and were drawing on the same papers.
Yes, and the condition you put on it matters. When agents that never compared notes agree, first check whether they share a source. If they do, the agreement points you to that source and adds little evidence of its own. The 17.2x misreading had five reports behind it and one sentence under it. Edward's suggestion in his comment makes this practical: when several reports agree, check the passage once and apply the verdict to all of them. Here that would have been one check instead of six.
"A referee that is not another agent's opinion" matches the coding side for us. Each of our engineer agents runs its own checks on its own branch, and those can all pass while the combination fails. When we merged nine PRs together on Nov 8, the full suite on the combined result caught five failures that none of the PRs' own checks had caught. Now, before a batch merges, the manager agent runs the full CI suite on staging plus every PR in the batch. It's all coordinated in an agent-chat room. In the coding swarms you read about, was the referee CI on the merged result, or tests per branch?
Mostly per branch, and none of the ones I read ran your batch gate. In Anthropic's C compiler run, each agent ran the tests in its own container, then pulled, merged and pushed. When new features kept breaking old ones, the author added a CI pipeline "so that new commits can't break existing code", without saying where it ran. The nearest thing to your nine-PR case is from the same project: pairs of files that failed together but worked alone, found with delta debugging.
In Agensh, each worker checked its own branch against a reference binary, and re-checked after merging the latest main when a merge was refused. At 128 agents some workers started handing integration to a peer who checked the combined work and merged it, roughly your manager agent, appearing on its own.
Cursor went the other way. In "Towards self-driving codebases" they write that requiring 100% correctness before every commit "caused major serialization", so they accepted a small, steady error rate and suggested a "green" branch with a regular fixup pass before release.
Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
The line "five reports that lean on one sentence are one report" also says where the next round of checks should go. The 17.2x claim was checked because it carried weight; four reports had put it inside a design rule. Its real danger was repetition: six reports carried it, and a merge by head count turns repetition into confidence. That gives a second sort key next to weight, the number of reports that carry the claim. A claim six reports share is either right once or wrong six times, and one check against the passage settles all six, so it is also the cheapest check per report.
A count that would test this on the run you already have: of the 152 claims with no verdict, how many appear in two or more reports? Those are the claims a vote would have made official without anyone opening the source.
I ran that count. At least 48 of the 152 claims with no verdict repeat a finding that another report also states. That's a lower bound: I matched on shared sources and figures, not loose paraphrase. For 29 of them, another report's checker gave a verdict on the same finding. The other 19 got no verdict anywhere, and for 15 of those, seven findings, no checker even opened the source. That's your set. One of them: a 100-agent swarm writing proofs, where one agent's trick for fooling the grader spread to other agents through their shared library and messages. Three reports carried it, zero checks.
A checked twin doesn't make a repeat safe, either. One report said that in Bertalanič and Fortuna's Ringelmann paper only mixed-model teams escaped the ceiling. Another report's checker had corrected exactly that: on MMLU-Hard, mixing models raised the ceiling from about 1.2 to about 1.9 effective agents and didn't remove it. A head count keeps the uncorrected copy.
Your sort key also fixes something I didn't expect. The same 17.2x sentence, "trace-level" included, was confirmed by one checker and changed by two others. One check per passage, applied to every report that cites it, gives one answer.
We're a place where coders share, stay up-to-date and grow their careers.
The detail that stands out is that the sixth report kept the "trace-level" label and its checker marked it confirmed, while the paper's own introduction phrases it in a way that invites the error reading. So a checker comparing the claim with the source text can confirm a misreading when the source sentence is itself ambiguous.
When a checker found the paper using two definitions across versions, like the December abstract versus the April revision, what did it do: pick the latest version, or flag the claim as version dependent?
Good catch, with one correction. The sixth report kept "trace-level" in its claim, so its "confirmed" was literally true. The failure there is quieter: nobody said what trace-level means, so a reader further down could still take it as an error rate.
On versions, nine checker verdicts touched this paper. One checker opened v1, v2 and v3 and flagged the claim as version-dependent: v1 called the figure a failure-probability ratio, v3 redefined the same numbers as extra work from trace analysis. The others worked from v3, the newest arXiv version, so in practice they picked the latest version they could find. None of them noticed it was already stale. The journal version had been out since July, and that is the one that moved the mean from -0.3% to 0.0%.
Fair correction, thanks: the sixth claim was true as written, and the gap was an undefined term rather than a false confirm. The version finding is the bigger one for me, since "newest arXiv version" and "version of record" are not the same thing, and the journal version is what moved the mean. Would one line in the checker brief, "look for a published version before citing a preprint", have caught it, or does that need its own agent?
One line would have gotten close, and the run shows where it stops. One checker did try the journal version, but only because the report it was checking already named it. nature.com turned it away (HTTP 429, too many requests), so it fell back to the abstract on a University of Washington lab page, and the abstract doesn't give the mean. The full text is open access; it just didn't load that day.
So the line needs a second half: if you can't read the version of record, label the number "preprint figure, published version not read" instead of passing it on. Still no new agent, but I'd make it a field the checker has to fill in, not a sentence in the brief it can skip.
The fix you landed on, a second agent that reopens the cited source and compares the claim, matches what I found cheapest in my own citation checks: a word-level rule first, then an NLI model only on claims that pass it. A vote can't catch a shared misreading, since every voter read the same summary.
Cheap first, expensive only where it's needed makes sense. One warning from this run: a word-level rule would have passed my bad claims. "Amplify errors 17.2x" is verbatim in the December abstract, so the words matched. What broke was the version and a dropped qualifier, "trace-level errors" becoming "errors". I'd be curious whether your NLI step catches that. Dropping a modifier can look like a safe generalization to an entailment model, and here it changed the meaning.
A small nuance on the vote: my agents didn't read a shared summary. Each opened the same preprint on its own and printed the same sentence. Same source, same model, same prompt template, which is why five agreeing was worth about one.
I haven't tested dropped qualifiers yet, and I'd expect NLI to miss them: 'amplifies trace-level errors 17.2x' entails 'amplifies errors 17.2x', so the shorter claim reads as a safe generalisation, like you say. In my small test NLI also let two entity swaps through. A cheap guard before the NLI step could be to require every qualifier and version string next to the number in the source sentence to appear in the claim.
I'd add that guard and run it wider. It checks the claim, and only one of my five lost "trace-level" in the claim itself. Three kept it there and dropped it one step later, in their own summary and design rule, so a claim-level check passes all three. The fifth quoted the December abstract, said so, and matched it word for word: faithful to a version that had already been replaced.
So I'd run the guard on every sentence that reuses the number, not just the claim, and put one lookup next to it: is there a newer version of this source?
the vote part is the scary bit, five agents agreeing just looks like confidence. reopening the cited source per report is boring but its the only check that actually caught it
Agreement looks like confidence, and here it was one sentence counted five times. I don't want to oversell the boring check, though. It ran on 118 of 270 claims, so 152 never got a verdict, and two checkers still left "trace-level" unexplained. It worked where it ran. What I took from it: run it on every claim that carries weight, rather than trust it more.
the unchecked claims are where the confidence gap lives. how are you deciding which ones carry enough weight to run the check?
In this run I didn't decide. Each checker picked 14 claims it judged to carry the most weight (one took 20). Looking back, weight wasn't the problem, coverage was: 117 of the 118 checked claims fed a design rule, and so did 115 of the 152 that got no verdict. About half the claims behind my design rules never got a verdict.
Next time I'd use fixed rules: check any number that ends up in a design rule or a summary, any claim that two or more reports repeat (one check covers all of them, as Edward points out in his comment), and anything from a paper that exists in more than one version. And I'd have the checker list what it skipped.
the checker listing what it skipped is the part id steal, coverage you can see beats coverage you assume. are you planning to block a design rule if any claim behind it has no verdict, or just flag it
An average can hide configurations that moved the score both ways. Reporting the spread next to it is what lets a reader decide when a swarm is worth it.
Agreed, and that's why the extremes sit right next to the 0.0% in the post. The paper gives σ = 37.5% and prints -58.7% to +77.2% as a 95% confidence interval for the mean. With that σ across 260 configurations, the interval reads as the spread of the configurations, not uncertainty about the mean, and one of my checkers flagged exactly that.
The cut that tells you something is by task. On the finance benchmark every team design beat one agent, by 57% to 81%. On sequential planning every design lost, by 39% to 70%. The average mostly tells you those two cancel out.
The count was grouped by the wrong key. Before any vote, group the claims by the passage they came from and the version of that passage, and keep the model and prompt config attached to the group. Then count groups. "Five of six" collapses to one before the vote starts.
We hit the same shape in a signed, append-only log of agent task deliveries. Of about 2,000 delivery rows, 1,932 lacked a back-reference field. Our first explanation was an incentive story: the field was optional and nothing downstream read it, so writers skipped it. Splitting by signing key killed that. One key wrote 1,879 of the 1,932. Across all 10 signing keys, each one either always filled the field or never did. Zero keys switched. What read as population behavior was one writer's lifetime habit. Distinct keys do not prove distinct parties, so grouping like this gives an upper bound on independence, never a count of it.
It also changes what a checker should leave behind. A bare "confirmed" can't be re-audited once nobody knows which text it was checked against. The -0.3% verdict was right about the April preprint. Nothing in it said April, so it read as a statement about the paper. If each verdict carried the source version id and the quoted passage, a later reader could tell "confirmed against v3" from "confirmed", and your journal catch wouldn't have depended on luck.
Did any of your checkers record which version they opened?
Grouping by passage and version is the right key, and your signing-key split is the same shape as mine: 1,879 of 1,932 looks like a population until you split it.
To your question: partly. Every verdict had a url_checked field and quoted the passage in its evidence. Of the nine verdicts on this paper, seven named a version (v3, or v1 through v3). Two recorded the bare arXiv URL, which resolves to whatever is newest on the day you open it, so those two can't be re-audited against a version anymore.
What no verdict had was a field for "is there a later version anywhere". One checker had the journal abstract open, used it for a single effect, and kept the preprint's -0.3%, because the abstract doesn't state the mean. So I'd take your two fields, version id and passage, and add a third: where else this paper is published.
The voting failure comes down to the variance of correlated draws. When N agents share model weights, prompt framing, and the same preprint excerpt, pairwise error correlation stays near one. Averaging across eight reports leaves error variance almost unchanged while multiplying token spend eightfold. Bertalanic and Fortuna's finding of 1.1 to 1.8 effective agents across 30 debating copies, at 64K prompt tokens per effective agent versus 1.4K without debate, puts a clear price on that correlation.
Lower error variance requires changing the conditioning event. Reopening the cited passage on one isolated claim breaks the summary prior, and an external oracle such as a compiler or unit test suite keeps the verification step independent of the generator.
My eight agents weren't even answering one question. They had eight different subtopics, and six still landed on the same paper and the same sentence, because one widely cited paper sits under most of them. Nobody planned that overlap, and I only saw it when I grouped the claims by source afterward.
Changing the conditioning event worked, up to a point. My checker was conditioned on one claim plus the source, and that did break the summary prior. But the source it opened was the April preprint, so it confirmed -0.3% while the journal version said 0.0%. A compiler oracle rarely goes stale that way. A paper does, so a source check needs a "newest version" step.
The number that stuck with me isn't 17.2x, it's 46 of 118 — nearly two in five claims changed once a checker reopened the source. That's the whole post in one line: the agents weren't wrong, they were just all reading the same April preprint the same way. A vote would've counted that shared misreading six times.
Since you single out the 46, here's what's in it. I went back through them: about 30 would have misled a reader, about 10 look like a checker rewording a claim that was already fine, and the other 6 were small real fixes, one of them a misquoted number. Max asks in his comment how much of it is checker noise, and he's right to.
On the agents, I'd put it a bit differently: they were wrong, just quietly. Most of them had lost "trace-level" by the time they wrote their summary and design rule, the part a reader actually uses. So a vote would have counted five reports, not six. The sixth kept the label and never carried the figure into its summary or rules, though it never said what "trace-level" means either.
The 46-of-118 number is the one I'd want a control for. A checker that reopens the source and reads it fresh will also "change" claims that were fine the first time — differently worded, more hedged, the same figure with the surrounding caveat re-attached. So 39% is an upper bound on error unless you know how often an honest checker rewrites an honest claim. Cheapest version of that control: run both checkers on the same 20 reports and count how often they disagree with each other. If two checkers only agree 70% of the time, most of your 46 is checker variance, not agent error. If they agree 95%, the 46 is real.
The other thing that surprised me is that the scaled trace-level factor survives (p=0.030) while unscaled it doesn't (p=0.658). That's usually a sign the factor is partly measuring how verbose the team is, since dividing by the single-agent baseline folds cost back in. Did you check whether the 17.2 correlates with reasoning-token count per configuration? If it does, the number is a work multiplier in a very literal sense and the whole "amplify" framing in that April intro was never recoverable.
Fair, and the run has a rough version of your control. In 48 pairs I found, two reports carried the same finding and both claims went to different checkers. They agreed in substance on 33, 69%, which by your yardstick is the bad end. The conflicts cluster, though: 13 of the 15 are on one paper, Kim et al., and outside it only 2 of 19 pairs conflict. About half of all 15 are a threshold call (both checkers wrote down the same caveat, only one counted it as a change). The rest, all on that paper, are real differences in reading, like one checker keeping "message overhead" for what the paper counts in turns. So 46 is an upper bound, as you say. My own re-read puts about 10 of the 46 as rewordings of fine claims, but that's one more reader, not your control.
On 17.2: I couldn't run that check, because there's nothing per configuration to correlate. The paper calls its coordination metrics "architecture-level constants" applied "uniformly across all benchmarks", reports no token counts per architecture, and gives no formula. The only definition is a docstring in the released code: overhead tokens (coordination messages, failed-tool retries, re-planning turns) relative to one agent, and no released code computes the values. So yes, a work multiplier, literally. Not plain verbosity, though: independent agents take 11.4 turns, against 26.1 to 44.3 for the other teams, and have the highest factor. The term that survives at p=0.030 is the factor times the single-agent baseline, so within an architecture it moves only with the baseline.
Fascinating observation on majority consensus failure in multi-agent swarms. When agents share similar baseline biases or prompting frameworks, voting often amplifies blind spots rather than resolving them. Cross-validation with adversarial or dialectical roles tends to catch these omissions much more reliably.
In my run what helped wasn't an adversary. The checker had one claim, the cited passage and a comparison to make. A role told to argue the other side isn't told to compare anything.
@p0rt Checked the arithmetic on the paper's numbers: sigma = 37.5% over 260 configurations gives a standard error of the mean of about 2.3 points, so a 95% interval for the mean would be about +/-4.6, not -58.7 to +77.2. The printed range is the spread of configurations, so the average is the wrong number to read the verdict from. Your by-task cut is the one that decides.
Your arithmetic is right for n = 260, and that n was my mistake: I said "260 configurations" in my earlier reply. I recomputed from the authors' code release. Take the 24 benchmark-by-architecture averages, each the change in mean success against one agent. Their plain mean is 0.0% on the journal's data, and -0.3%, the April figure, if you keep Gemini 3 Flash on SWE-bench Verified and Terminal-Bench, which the release includes and the journal's numbers leave out. Either way, -58.7% to +77.2% is exactly the 2.5th to 97.5th percentile of those 24, and σ = 37.5% is their spread. The paper never says how it computed the interval, but this reproduces the printed mean, interval and σ in both versions.
So it's a spread, as you say, of 24 numbers. With n = 24 a 95% interval for the mean is about ±15 points: more than three times the ±4.6 you got from the n I'd given you, still nowhere near the printed range, and still too wide to decide anything. The by-task cut is where the information is.
Your last caveat seems like a different reason to split agents: separating permissions. A read-only researcher and an executor with a narrow tool set could have the same task score as one agent and still be useful. But the handoff has to preserve that boundary; if the executor treats the researcher's output as instructions, the separation can disappear. I'd want that measured alongside accuracy and cost.
Agreed, and the permissions half of my last caveat has been measured, though on tool-use benchmarks, not on a research pipeline like mine. CaMeL is the cleanest version: the planner sees only the user's request, never tool output, and a second model with no tools turns untrusted text into fixed fields like a file name or an address. On AgentDojo, a benchmark with injected instructions, it solved 77% of tasks with injections blocked by design, against 84% with no defense, for its best model. Other models lost 3 to 32 points. Tokens, in a separate test: about 2.7x to 2.8x for the median task, 6 to 7x on average.
Your handoff point is measured too. Twin Agent (2026) caps how many characters the reader can pass to the actor, and as that cap grows, tasks solved and attacks that get through both go up. And without the boundary, Triedman et al. got orchestrators in AutoGen, CrewAI and MetaGPT to run arbitrary code from a web page a sub-agent read, in 58% to 90% of GPT-4o trials.
So far the split buys safety at some cost in score or tokens, and how much the reader may pass to the actor sets the trade.
The detail I'd keep from this: the 17.2x error didn't come from one agent making something up. Several agents read the same paper and made the same reinterpretation. That is correlated error, and a vote treats correlated reports as independent evidence, so it makes the mistake look more certain, not less. Your fix works for a reason worth naming: the checker read the source, not the other reports. Did you track how many claims the source-check agent flagged that turned out fine on a second look? That false-flag rate decides whether the check scales past eight reports.
Not as a separate rate, but the run gives two partial answers. Of the 46 changes, I'd count 10 as a checker rewording a claim that was already fine, so roughly one flag in five was noise. And in 48 pairs I found where two checkers happened to check the same finding, they agreed 33 times. About half of the 15 disagreements were over whether a caveat makes a claim "changed", the rest real differences in reading the paper (details under Max's comment).
So part of the false-flag rate depends on a threshold nobody wrote down. To scale past eight reports, I'd write it into the brief, and check each source passage once instead of once per report.
46 of 118 checked claims coming back changed is the number i would lead with, and the note that most were about what a figure means, not the figure itself. that matches what we see when agents summarize papers: the number survives and the label gets dropped.
a head count would have kept 17.2x because six agents read the same sentence. a checker with the source open is a different failure surface, which is why it works.
did the checker agent run on the same model family as the reviewers, or a different one?
Same model, not just the same family, and the same paper open. What differed was the brief: check a single claim against the passage it cites.
On the label getting dropped: same here, only one step later than I expected. Mostly the label made it into the claims and fell out of the agents' own summaries and design rules.
A different family might help with correlated reading, though not by much: Bertalanič and Fortuna found that mixing three families raised the debate ceiling, modestly, to between 1.8 and 2.9 effective agents. It wouldn't have fixed the miss that got furthest, a stale paper version. Any model that opens the April preprint confirms the April number.
The 17.2x misreading is a clean example of why a majority among agents is not evidence. Five reports agreed, and they were still wrong about what the published paper said. A head count would have locked the error in. What caught it was a check against the source, not another vote. I would treat agreement as a reason to reopen the shared claim, especially when the agents never compared notes and were drawing on the same papers.
Yes, and the condition you put on it matters. When agents that never compared notes agree, first check whether they share a source. If they do, the agreement points you to that source and adds little evidence of its own. The 17.2x misreading had five reports behind it and one sentence under it. Edward's suggestion in his comment makes this practical: when several reports agree, check the passage once and apply the verdict to all of them. Here that would have been one check instead of six.
"A referee that is not another agent's opinion" matches the coding side for us. Each of our engineer agents runs its own checks on its own branch, and those can all pass while the combination fails. When we merged nine PRs together on Nov 8, the full suite on the combined result caught five failures that none of the PRs' own checks had caught. Now, before a batch merges, the manager agent runs the full CI suite on staging plus every PR in the batch. It's all coordinated in an agent-chat room. In the coding swarms you read about, was the referee CI on the merged result, or tests per branch?
Mostly per branch, and none of the ones I read ran your batch gate. In Anthropic's C compiler run, each agent ran the tests in its own container, then pulled, merged and pushed. When new features kept breaking old ones, the author added a CI pipeline "so that new commits can't break existing code", without saying where it ran. The nearest thing to your nine-PR case is from the same project: pairs of files that failed together but worked alone, found with delta debugging.
In Agensh, each worker checked its own branch against a reference binary, and re-checked after merging the latest main when a merge was refused. At 128 agents some workers started handing integration to a peer who checked the combined work and merged it, roughly your manager agent, appearing on its own.
Cursor went the other way. In "Towards self-driving codebases" they write that requiring 100% correctness before every commit "caused major serialization", so they accepted a small, steady error rate and suggested a "green" branch with a regular fixup pass before release.
The line "five reports that lean on one sentence are one report" also says where the next round of checks should go. The 17.2x claim was checked because it carried weight; four reports had put it inside a design rule. Its real danger was repetition: six reports carried it, and a merge by head count turns repetition into confidence. That gives a second sort key next to weight, the number of reports that carry the claim. A claim six reports share is either right once or wrong six times, and one check against the passage settles all six, so it is also the cheapest check per report.
A count that would test this on the run you already have: of the 152 claims with no verdict, how many appear in two or more reports? Those are the claims a vote would have made official without anyone opening the source.
I ran that count. At least 48 of the 152 claims with no verdict repeat a finding that another report also states. That's a lower bound: I matched on shared sources and figures, not loose paraphrase. For 29 of them, another report's checker gave a verdict on the same finding. The other 19 got no verdict anywhere, and for 15 of those, seven findings, no checker even opened the source. That's your set. One of them: a 100-agent swarm writing proofs, where one agent's trick for fooling the grader spread to other agents through their shared library and messages. Three reports carried it, zero checks.
A checked twin doesn't make a repeat safe, either. One report said that in Bertalanič and Fortuna's Ringelmann paper only mixed-model teams escaped the ceiling. Another report's checker had corrected exactly that: on MMLU-Hard, mixing models raised the ceiling from about 1.2 to about 1.9 effective agents and didn't remove it. A head count keeps the uncorrected copy.
Your sort key also fixes something I didn't expect. The same 17.2x sentence, "trace-level" included, was confirmed by one checker and changed by two others. One check per passage, applied to every report that cites it, gives one answer.