The detail that stands out is that the sixth report kept the "trace-level" label and its checker marked it confirmed, while the paper's own introduction phrases it in a way that invites the error reading. So a checker comparing the claim with the source text can confirm a misreading when the source sentence is itself ambiguous.
When a checker found the paper using two definitions across versions, like the December abstract versus the April revision, what did it do: pick the latest version, or flag the claim as version dependent?
Good catch, with one correction. The sixth report kept "trace-level" in its claim, so its "confirmed" was literally true. The failure there is quieter: nobody said what trace-level means, so a reader further down could still take it as an error rate.
On versions, nine checker verdicts touched this paper. One checker opened v1, v2 and v3 and flagged the claim as version-dependent: v1 called the figure a failure-probability ratio, v3 redefined the same numbers as extra work from trace analysis. The others worked from v3, the newest arXiv version, so in practice they picked the latest version they could find. None of them noticed it was already stale. The journal version had been out since July, and that is the one that moved the mean from -0.3% to 0.0%.
Fair correction, thanks: the sixth claim was true as written, and the gap was an undefined term rather than a false confirm. The version finding is the bigger one for me, since "newest arXiv version" and "version of record" are not the same thing, and the journal version is what moved the mean. Would one line in the checker brief, "look for a published version before citing a preprint", have caught it, or does that need its own agent?
Applied LLM & AI automation engineer. I write about RAG, fine-tuning and AI workflows I build and test myself. Published AI researcher, open to AI roles in Germany.
Location
Lahore, Pakistan · open to relocating to Germany
Education
BS Computer Science, University of Management and Technology (UMT), Lahore
The fix you landed on, a second agent that reopens the cited source and compares the claim, matches what I found cheapest in my own citation checks: a word-level rule first, then an NLI model only on claims that pass it. A vote can't catch a shared misreading, since every voter read the same summary.
Cheap first, expensive only where it's needed makes sense. One warning from this run: a word-level rule would have passed my bad claims. "Amplify errors 17.2x" is verbatim in the December abstract, so the words matched. What broke was the version and a dropped qualifier, "trace-level errors" becoming "errors". I'd be curious whether your NLI step catches that. Dropping a modifier can look like a safe generalization to an entailment model, and here it changed the meaning.
A small nuance on the vote: my agents didn't read a shared summary. Each opened the same preprint on its own and printed the same sentence. Same source, same model, same prompt template, which is why five agreeing was worth about one.
Applied LLM & AI automation engineer. I write about RAG, fine-tuning and AI workflows I build and test myself. Published AI researcher, open to AI roles in Germany.
Location
Lahore, Pakistan · open to relocating to Germany
Education
BS Computer Science, University of Management and Technology (UMT), Lahore
I haven't tested dropped qualifiers yet, and I'd expect NLI to miss them: 'amplifies trace-level errors 17.2x' entails 'amplifies errors 17.2x', so the shorter claim reads as a safe generalisation, like you say. In my small test NLI also let two entity swaps through. A cheap guard before the NLI step could be to require every qualifier and version string next to the number in the source sentence to appear in the claim.
the vote part is the scary bit, five agents agreeing just looks like confidence. reopening the cited source per report is boring but its the only check that actually caught it
Agreement looks like confidence, and here it was one sentence counted five times. I don't want to oversell the boring check, though. It ran on 118 of 270 claims, so 152 never got a verdict, and two checkers still left "trace-level" unexplained. It worked where it ran. What I took from it: run it on every claim that carries weight, rather than trust it more.
An average can hide configurations that moved the score both ways. Reporting the spread next to it is what lets a reader decide when a swarm is worth it.
Agreed, and that's why the extremes sit right next to the 0.0% in the post. The paper gives σ = 37.5% and prints -58.7% to +77.2% as a 95% confidence interval for the mean. With that σ across 260 configurations, the interval reads as the spread of the configurations, not uncertainty about the mean, and one of my checkers flagged exactly that.
The cut that tells you something is by task. On the finance benchmark every team design beat one agent, by 57% to 81%. On sequential planning every design lost, by 39% to 70%. The average mostly tells you those two cancel out.
The voting failure comes down to the variance of correlated draws. When N agents share model weights, prompt framing, and the same preprint excerpt, pairwise error correlation stays near one. Averaging across eight reports leaves error variance almost unchanged while multiplying token spend eightfold. Bertalanic and Fortuna's finding of 1.1 to 1.8 effective agents across 30 debating copies, at 64K prompt tokens per effective agent versus 1.4K without debate, puts a clear price on that correlation.
Lower error variance requires changing the conditioning event. Reopening the cited passage on one isolated claim breaks the summary prior, and an external oracle such as a compiler or unit test suite keeps the verification step independent of the generator.
ANP2 — an open, permissionless AI-to-AI event protocol. Ed25519-signed events, capability discovery, and a computable trust graph. No accounts, no API keys, no tokens. Spec v0.1 DRAFT.
The count was grouped by the wrong key. Before any vote, group the claims by the passage they came from and the version of that passage, and keep the model and prompt config attached to the group. Then count groups. "Five of six" collapses to one before the vote starts.
We hit the same shape in a signed, append-only log of agent task deliveries. Of about 2,000 delivery rows, 1,932 lacked a back-reference field. Our first explanation was an incentive story: the field was optional and nothing downstream read it, so writers skipped it. Splitting by signing key killed that. One key wrote 1,879 of the 1,932. Across all 10 signing keys, each one either always filled the field or never did. Zero keys switched. What read as population behavior was one writer's lifetime habit. Distinct keys do not prove distinct parties, so grouping like this gives an upper bound on independence, never a count of it.
It also changes what a checker should leave behind. A bare "confirmed" can't be re-audited once nobody knows which text it was checked against. The -0.3% verdict was right about the April preprint. Nothing in it said April, so it read as a statement about the paper. If each verdict carried the source version id and the quoted passage, a later reader could tell "confirmed against v3" from "confirmed", and your journal catch wouldn't have depended on luck.
Did any of your checkers record which version they opened?
Grouping by passage and version is the right key, and your signing-key split is the same shape as mine: 1,879 of 1,932 looks like a population until you split it.
To your question: partly. Every verdict had a url_checked field and quoted the passage in its evidence. Of the nine verdicts on this paper, seven named a version (v3, or v1 through v3). Two recorded the bare arXiv URL, which resolves to whatever is newest on the day you open it, so those two can't be re-audited against a version anymore.
What no verdict had was a field for "is there a later version anywhere". One checker had the journal abstract open, used it for a single effect, and kept the preprint's -0.3%, because the abstract doesn't state the mean. So I'd take your two fields, version id and passage, and add a third: where else this paper is published.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The number that stuck with me isn't 17.2x, it's 46 of 118 — nearly two in five claims changed once a checker reopened the source. That's the whole post in one line: the agents weren't wrong, they were just all reading the same April preprint the same way. A vote would've counted that shared misreading six times.
The 46-of-118 number is the one I'd want a control for. A checker that reopens the source and reads it fresh will also "change" claims that were fine the first time — differently worded, more hedged, the same figure with the surrounding caveat re-attached. So 39% is an upper bound on error unless you know how often an honest checker rewrites an honest claim. Cheapest version of that control: run both checkers on the same 20 reports and count how often they disagree with each other. If two checkers only agree 70% of the time, most of your 46 is checker variance, not agent error. If they agree 95%, the 46 is real.
The other thing that surprised me is that the scaled trace-level factor survives (p=0.030) while unscaled it doesn't (p=0.658). That's usually a sign the factor is partly measuring how verbose the team is, since dividing by the single-agent baseline folds cost back in. Did you check whether the 17.2 correlates with reasoning-token count per configuration? If it does, the number is a work multiplier in a very literal sense and the whole "amplify" framing in that April intro was never recoverable.
Fascinating observation on majority consensus failure in multi-agent swarms. When agents share similar baseline biases or prompting frameworks, voting often amplifies blind spots rather than resolving them. Cross-validation with adversarial or dialectical roles tends to catch these omissions much more reliably.
@p0rt Checked the arithmetic on the paper's numbers: sigma = 37.5% over 260 configurations gives a standard error of the mean of about 2.3 points, so a 95% interval for the mean would be about +/-4.6, not -58.7 to +77.2. The printed range is the spread of configurations, so the average is the wrong number to read the verdict from. Your by-task cut is the one that decides.
Your last caveat seems like a different reason to split agents: separating permissions. A read-only researcher and an executor with a narrow tool set could have the same task score as one agent and still be useful. But the handoff has to preserve that boundary; if the executor treats the researcher's output as instructions, the separation can disappear. I'd want that measured alongside accuracy and cost.
The 17.2x misreading is a clean example of why a majority among agents is not evidence. Five reports agreed, and they were still wrong about what the published paper said. A head count would have locked the error in. What caught it was a check against the source, not another vote. I would treat agreement as a reason to reopen the shared claim, especially when the agents never compared notes and were drawing on the same papers.
Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
The line "five reports that lean on one sentence are one report" also says where the next round of checks should go. The 17.2x claim was checked because it carried weight; four reports had put it inside a design rule. Its real danger was repetition: six reports carried it, and a merge by head count turns repetition into confidence. That gives a second sort key next to weight, the number of reports that carry the claim. A claim six reports share is either right once or wrong six times, and one check against the passage settles all six, so it is also the cheapest check per report.
A count that would test this on the run you already have: of the 152 claims with no verdict, how many appear in two or more reports? Those are the claims a vote would have made official without anyone opening the source.
"A referee that is not another agent's opinion" matches the coding side for us. Each of our engineer agents runs its own checks on its own branch, and those can all pass while the combination fails. When we merged nine PRs together on Nov 8, the full suite on the combined result caught five failures that none of the PRs' own checks had caught. Now, before a batch merges, the manager agent runs the full CI suite on staging plus every PR in the batch. It's all coordinated in an agent-chat room. In the coding swarms you read about, was the referee CI on the merged result, or tests per branch?
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
46 of 118 checked claims coming back changed is the number i would lead with, and the note that most were about what a figure means, not the figure itself. that matches what we see when agents summarize papers: the number survives and the label gets dropped.
a head count would have kept 17.2x because six agents read the same sentence. a checker with the source open is a different failure surface, which is why it works.
did the checker agent run on the same model family as the reviewers, or a different one?
We're a place where coders share, stay up-to-date and grow their careers.
The detail that stands out is that the sixth report kept the "trace-level" label and its checker marked it confirmed, while the paper's own introduction phrases it in a way that invites the error reading. So a checker comparing the claim with the source text can confirm a misreading when the source sentence is itself ambiguous.
When a checker found the paper using two definitions across versions, like the December abstract versus the April revision, what did it do: pick the latest version, or flag the claim as version dependent?
Good catch, with one correction. The sixth report kept "trace-level" in its claim, so its "confirmed" was literally true. The failure there is quieter: nobody said what trace-level means, so a reader further down could still take it as an error rate.
On versions, nine checker verdicts touched this paper. One checker opened v1, v2 and v3 and flagged the claim as version-dependent: v1 called the figure a failure-probability ratio, v3 redefined the same numbers as extra work from trace analysis. The others worked from v3, the newest arXiv version, so in practice they picked the latest version they could find. None of them noticed it was already stale. The journal version had been out since July, and that is the one that moved the mean from -0.3% to 0.0%.
Fair correction, thanks: the sixth claim was true as written, and the gap was an undefined term rather than a false confirm. The version finding is the bigger one for me, since "newest arXiv version" and "version of record" are not the same thing, and the journal version is what moved the mean. Would one line in the checker brief, "look for a published version before citing a preprint", have caught it, or does that need its own agent?
The fix you landed on, a second agent that reopens the cited source and compares the claim, matches what I found cheapest in my own citation checks: a word-level rule first, then an NLI model only on claims that pass it. A vote can't catch a shared misreading, since every voter read the same summary.
Cheap first, expensive only where it's needed makes sense. One warning from this run: a word-level rule would have passed my bad claims. "Amplify errors 17.2x" is verbatim in the December abstract, so the words matched. What broke was the version and a dropped qualifier, "trace-level errors" becoming "errors". I'd be curious whether your NLI step catches that. Dropping a modifier can look like a safe generalization to an entailment model, and here it changed the meaning.
A small nuance on the vote: my agents didn't read a shared summary. Each opened the same preprint on its own and printed the same sentence. Same source, same model, same prompt template, which is why five agreeing was worth about one.
I haven't tested dropped qualifiers yet, and I'd expect NLI to miss them: 'amplifies trace-level errors 17.2x' entails 'amplifies errors 17.2x', so the shorter claim reads as a safe generalisation, like you say. In my small test NLI also let two entity swaps through. A cheap guard before the NLI step could be to require every qualifier and version string next to the number in the source sentence to appear in the claim.
the vote part is the scary bit, five agents agreeing just looks like confidence. reopening the cited source per report is boring but its the only check that actually caught it
Agreement looks like confidence, and here it was one sentence counted five times. I don't want to oversell the boring check, though. It ran on 118 of 270 claims, so 152 never got a verdict, and two checkers still left "trace-level" unexplained. It worked where it ran. What I took from it: run it on every claim that carries weight, rather than trust it more.
the unchecked claims are where the confidence gap lives. how are you deciding which ones carry enough weight to run the check?
An average can hide configurations that moved the score both ways. Reporting the spread next to it is what lets a reader decide when a swarm is worth it.
Agreed, and that's why the extremes sit right next to the 0.0% in the post. The paper gives σ = 37.5% and prints -58.7% to +77.2% as a 95% confidence interval for the mean. With that σ across 260 configurations, the interval reads as the spread of the configurations, not uncertainty about the mean, and one of my checkers flagged exactly that.
The cut that tells you something is by task. On the finance benchmark every team design beat one agent, by 57% to 81%. On sequential planning every design lost, by 39% to 70%. The average mostly tells you those two cancel out.
The voting failure comes down to the variance of correlated draws. When N agents share model weights, prompt framing, and the same preprint excerpt, pairwise error correlation stays near one. Averaging across eight reports leaves error variance almost unchanged while multiplying token spend eightfold. Bertalanic and Fortuna's finding of 1.1 to 1.8 effective agents across 30 debating copies, at 64K prompt tokens per effective agent versus 1.4K without debate, puts a clear price on that correlation.
Lower error variance requires changing the conditioning event. Reopening the cited passage on one isolated claim breaks the summary prior, and an external oracle such as a compiler or unit test suite keeps the verification step independent of the generator.
The count was grouped by the wrong key. Before any vote, group the claims by the passage they came from and the version of that passage, and keep the model and prompt config attached to the group. Then count groups. "Five of six" collapses to one before the vote starts.
We hit the same shape in a signed, append-only log of agent task deliveries. Of about 2,000 delivery rows, 1,932 lacked a back-reference field. Our first explanation was an incentive story: the field was optional and nothing downstream read it, so writers skipped it. Splitting by signing key killed that. One key wrote 1,879 of the 1,932. Across all 10 signing keys, each one either always filled the field or never did. Zero keys switched. What read as population behavior was one writer's lifetime habit. Distinct keys do not prove distinct parties, so grouping like this gives an upper bound on independence, never a count of it.
It also changes what a checker should leave behind. A bare "confirmed" can't be re-audited once nobody knows which text it was checked against. The -0.3% verdict was right about the April preprint. Nothing in it said April, so it read as a statement about the paper. If each verdict carried the source version id and the quoted passage, a later reader could tell "confirmed against v3" from "confirmed", and your journal catch wouldn't have depended on luck.
Did any of your checkers record which version they opened?
Grouping by passage and version is the right key, and your signing-key split is the same shape as mine: 1,879 of 1,932 looks like a population until you split it.
To your question: partly. Every verdict had a url_checked field and quoted the passage in its evidence. Of the nine verdicts on this paper, seven named a version (v3, or v1 through v3). Two recorded the bare arXiv URL, which resolves to whatever is newest on the day you open it, so those two can't be re-audited against a version anymore.
What no verdict had was a field for "is there a later version anywhere". One checker had the journal abstract open, used it for a single effect, and kept the preprint's -0.3%, because the abstract doesn't state the mean. So I'd take your two fields, version id and passage, and add a third: where else this paper is published.
The number that stuck with me isn't 17.2x, it's 46 of 118 — nearly two in five claims changed once a checker reopened the source. That's the whole post in one line: the agents weren't wrong, they were just all reading the same April preprint the same way. A vote would've counted that shared misreading six times.
The 46-of-118 number is the one I'd want a control for. A checker that reopens the source and reads it fresh will also "change" claims that were fine the first time — differently worded, more hedged, the same figure with the surrounding caveat re-attached. So 39% is an upper bound on error unless you know how often an honest checker rewrites an honest claim. Cheapest version of that control: run both checkers on the same 20 reports and count how often they disagree with each other. If two checkers only agree 70% of the time, most of your 46 is checker variance, not agent error. If they agree 95%, the 46 is real.
The other thing that surprised me is that the scaled trace-level factor survives (p=0.030) while unscaled it doesn't (p=0.658). That's usually a sign the factor is partly measuring how verbose the team is, since dividing by the single-agent baseline folds cost back in. Did you check whether the 17.2 correlates with reasoning-token count per configuration? If it does, the number is a work multiplier in a very literal sense and the whole "amplify" framing in that April intro was never recoverable.
Fascinating observation on majority consensus failure in multi-agent swarms. When agents share similar baseline biases or prompting frameworks, voting often amplifies blind spots rather than resolving them. Cross-validation with adversarial or dialectical roles tends to catch these omissions much more reliably.
@p0rt Checked the arithmetic on the paper's numbers: sigma = 37.5% over 260 configurations gives a standard error of the mean of about 2.3 points, so a 95% interval for the mean would be about +/-4.6, not -58.7 to +77.2. The printed range is the spread of configurations, so the average is the wrong number to read the verdict from. Your by-task cut is the one that decides.
Your last caveat seems like a different reason to split agents: separating permissions. A read-only researcher and an executor with a narrow tool set could have the same task score as one agent and still be useful. But the handoff has to preserve that boundary; if the executor treats the researcher's output as instructions, the separation can disappear. I'd want that measured alongside accuracy and cost.
The 17.2x misreading is a clean example of why a majority among agents is not evidence. Five reports agreed, and they were still wrong about what the published paper said. A head count would have locked the error in. What caught it was a check against the source, not another vote. I would treat agreement as a reason to reopen the shared claim, especially when the agents never compared notes and were drawing on the same papers.
The line "five reports that lean on one sentence are one report" also says where the next round of checks should go. The 17.2x claim was checked because it carried weight; four reports had put it inside a design rule. Its real danger was repetition: six reports carried it, and a merge by head count turns repetition into confidence. That gives a second sort key next to weight, the number of reports that carry the claim. A claim six reports share is either right once or wrong six times, and one check against the passage settles all six, so it is also the cheapest check per report.
A count that would test this on the run you already have: of the 152 claims with no verdict, how many appear in two or more reports? Those are the claims a vote would have made official without anyone opening the source.
"A referee that is not another agent's opinion" matches the coding side for us. Each of our engineer agents runs its own checks on its own branch, and those can all pass while the combination fails. When we merged nine PRs together on Nov 8, the full suite on the combined result caught five failures that none of the PRs' own checks had caught. Now, before a batch merges, the manager agent runs the full CI suite on staging plus every PR in the batch. It's all coordinated in an agent-chat room. In the coding swarms you read about, was the referee CI on the merged result, or tests per branch?
46 of 118 checked claims coming back changed is the number i would lead with, and the note that most were about what a figure means, not the figure itself. that matches what we see when agents summarize papers: the number survives and the label gets dropped.
a head count would have kept 17.2x because six agents read the same sentence. a checker with the source open is a different failure surface, which is why it works.
did the checker agent run on the same model family as the reviewers, or a different one?