I had eight AI agents review the research on multi-agent LLM systems, one subtopic each. None of them messaged another. Six came back with the same figure from the same paper: 17.2x. In five of the six reports it became a plain error multiplier: agents without cross-checks "amplify errors 17.2x". Four reports quoted it inside a design rule.
The paper does not define it that way. In the published version the figure measures extra work inside a run. By the paper's own success rates, those agents fail about 1.2 times as often as one agent.
I never ran a vote. Had I merged the eight reports by counting heads, that reading would have gone into my review with five reports behind it. What caught it was duller than a debate: a second agent per report that reopened the cited sources and compared the main claims with the text.
I ran no benchmark. Every outside number below names its source. The first-hand numbers are about how the review was made, and they do not flatter me.
TL;DR: In the largest matched-budget comparison I found, 260 configurations, six benchmarks, a team of agents changed the score by 0.0% on average, with extremes of +80.8% and -70.0%. That paper's 17.2x figure measures extra work, not failed tasks. Five of my eight agents passed it on as an error multiplier. A check against the paper caught it. A head count would have kept it. The swarms with the biggest reported wins have work that splits into independent parts and, in the coding ones, a referee that is not another agent's opinion. Start with one agent. Without the split and the referee, do not go past a handful.
17.2x measures extra work, not failed tasks
The figure comes from Kim et al., a study that compares one agent with four kinds of team on six agentic benchmarks: 260 configurations, reasoning-token budgets matched. The preprint abstract from December 2025 said independent agents "amplify errors 17.2x", and the text defined the figure as a "relative failure probability". The April revision kept the number and changed what it is. The version published in Nature Machine Intelligence in July 2026 keeps the April definition.
| The reading my agents brought back | What the published paper says |
|---|---|
| 17.2x more errors | A trace-level factor, estimated from run logs: "how much extra computational work arises from interagent coordination failures" |
| Tasks fail far more often | The task-level factor, the ratio of error rates, is about 1.1 (the paper names no team design) |
Read as task failures, the figure never matched the paper's own numbers. Mean success is 0.466 for one agent and 0.370 for independent agents, in the preprints and in the journal. That is about 1.2 times the failures (computed from those two rates), close to the paper's 1.1 and nowhere near 17.
The direction survives. Independent agents with no cross-checking were the weakest team design on five of the six benchmarks, and posted a small gain on the sixth. The paper also reports that the trace-level factor predicts performance once it is scaled by the single-agent baseline (robust p = 0.030), though not on its own (p = 0.658).
The paper makes the misreading easy. Its April introduction says independent systems "amplify trace-level errors 17.2x", and the published text still has them "propagate errors to 17.2x baseline".
My eight agents produced 270 claims. A second agent per report reopened the sources for the claims that carried the most weight, 118 of the 270. 46 of the 118 came back changed. I count checker changes, not proven errors. By the checkers' own notes, most were about what a number means, not the number itself.
Six reports cited 17.2x. The run log shows that all six agents had the April preprint open and printed the same sentence from its introduction, with the label "trace-level" in it. Three wrote the label into the claim and left it out of their own summary and design rule. One dropped it at once. One quoted the December abstract and said so. A checker with the paper open changed all five, and four of the five changes said the figure is not an error rate. The sixth report kept the label, and its checker marked it confirmed.
This is one run of a literature review, with no control arm and no repeat. Five of eight is not a statistic. It is an anecdote with an audit trail. The agents shared a model, a prompt template and a source sentence, and I cannot separate the three. Whichever it was, the count was worth less than it looked: five reports that lean on one sentence are one report.
Human group research is most useful as a list of test designs
I started this review with the theory of human cooperation, as background.
| Human finding | Nearest measurement on agents |
|---|---|
| Ringelmann, 1913: eight men working together delivered about half of what the same eight delivered alone | 30 debating copies of one 7B model carry 1.1 to 1.8 agents' worth of distinct answers (Bertalanič and Fortuna, 2026) |
| Brooks, 1975: n people have n(n-1)/2 channels | Reasoning turns grow as team size to the power 1.72 (Kim et al.) |
| Stasser and Titus, 1985: groups of four picked the best candidate 83% of the time when everyone had all the facts, 18% when each member's share favored a rival | One agent holding all facts 80.7%, a group with split facts 30.1% (HiddenBench) |
| Smith, 1776: ten pin-makers, about 48,000 pins a day | +80.8% for the best team design on a task that splits into parts (Kim et al.) |
These are pairs, not replications. Only the hidden-profile row, where each member holds part of the facts, tests the same kind of task as its human study.
That row carries the fix I would try first. Neither model scale nor individual accuracy reliably predicted group accuracy. A protocol raised it: each agent shared one or two facts and one reason the leading answer might be wrong, for two rounds, and only then voted. On an 18-task subset with four agents, GPT-4.1 went from that subset's baseline of 0.04 to 0.80.
The first row is where the analogy breaks. Ingham et al. (1974) had men pull a rope alone while believing others pulled with them, and they still pulled about 15% less. Agents do not ease off because they think someone else is pulling. Their loss is redundancy: correlated outputs that add cost and little independent evidence.
Thirty debating copies of a 7B model count as one or two
Bertalanič and Fortuna put Ringelmann's name on a preprint that measures effective team size: how many independent answers a team of N really contains. Thirty copies of Qwen2.5-7B debating math word problems contained about 1.8. On two multiple-choice sets the figure was 1.12 and 1.15. The fitted ceilings for that model, a separate estimate, are 1.2 to 1.8 effective agents. They tested up to 30.
Thirty copies that only rechecked their own work, with no peer messages, reached about 6 on the math set by the same measure. On that set the paper treats the measure as an upper bound, for the 1.8 as well as the 6, because scattered wrong answers also count as diversity. The two setups stayed within five points of each other on accuracy, and debate spent 64K prompt tokens per effective agent against 1.4K. Mixing three model families raised the debate ceiling, modestly, to between 1.8 and 2.9. The HiddenBench protocol above works on a different problem: facts that are split, not answers that are copies.
Nobody ships 30 copies of a 7B model on quiz questions, and the objection is fair: this is one preprint, question answering plus a small code pilot, with plurality voting. The only closed model run in a team was Gemini 3.1 Flash-Lite, and the paper lists frontier-class debate, tool-using agents and multi-step planning as untested.
I have to be fair to debate as well. In Choi et al., 2025, averaged over seven benchmarks with five Qwen2.5-7B agents, one agent scored 0.72, the best standard debate 0.74 and voting with no debate 0.77, and the voting edge comes from two arithmetic tasks. In Wynn et al., 2025, all 10 three-agent groups scored lower after debate than their own initial vote on CommonSenseQA. On MMLU 7 of 10 fell. On GSM8K 6 of 10 rose. So it would overstate the case to say debate destroys accuracy. In these studies debate does not reliably beat a plain vote, and you pay for every round.
In the largest matched-budget study, the average team effect is 0.0%
The result in Kim et al. that deserves the fame is the average. Across six benchmarks and four team designs, the mean change against one agent is 0.0%. The extremes are +80.8% for a centralized team on a financial analysis task that splits into parts and -70.0% for independent agents on sequential planning, where every team design lost 39 to 70%. The person selling you a swarm and the person telling you swarms are a scam can both cite it.
The match is on reasoning-token budgets, and the paper's own token-efficiency measure still separates the designs. One agent gets 67.7 successes per 1,000 tokens, and the three designs that communicate get 13.6 to 23.9.
Other head-to-head results lean against the team. They are not on one scale: only the first matches budgets, and the third compares cost.
| Comparison | One agent | Team |
|---|---|---|
| Multi-hop questions at 1,000 reasoning tokens (Tran and Kiela, 2026) | 0.418 | 0.333 to 0.388, five designs |
| Success rate, two features in one repository (CooperBench) | baseline | 30% lower on average |
| Cost on a web-app benchmark (Vals AI, 2026) | 1x | 1.8 to 5.1x, one significant score gain in four comparisons |
At 10,000 reasoning tokens, three of Tran and Kiela's five team designs draw level with the single agent. Its lead is widest at 1,000.
Wall-clock speed is a separate question, and it matters when tokens are cheap. Mieczkowski et al., 2026 tested Amdahl's law on teams of one to five agents. Pre-assigned tasks gave a median speedup of 1.36x. Self-organization gave 0.88x, slower than one agent. On sequential tasks, pre-assigned teams spent 5.83x the tokens for a 1.13x speedup.
When a multi-agent result arrives without the single-agent score at the same tokens, ask the question from my harness post. Which one did you benchmark: the structure, or the budget?
The coding swarms with the big numbers have a referee
None of that makes swarms hype. Some of the largest numbers in the review come from many attempts or many agents. No two rows below use the same measure.
| Setup | Result | Bill and evidence |
|---|---|---|
| One model sampled 250 times, each attempt graded by the benchmark's unit tests (Brown et al., 2024) | SWE-bench Lite issues solved by at least one attempt: from 15.9% to 56% | 250 attempts, preprint |
| 128 self-organized agents with a shared repository and board (Zhan et al., 2026) | Test-pass rate on the five hardest ProgramBench tasks: 19.3% with one agent, 28.8% with 128 | Same six hours, cost not reported, single preprint |
| 16 agents, no orchestrator (Anthropic, 2026) | A 100,000-line C compiler | About $20,000, vendor report |
| Orchestrator with isolated workers (Anthropic, 2025) | +90.2% over one agent on an internal evaluation | Token use on that evaluation not reported, vendor report |
Read the last column first: two preprints, two vendor reports, and no row that compares at equal cost. The last source names the confound itself. Multi-agent systems "work mainly because they help spend enough tokens to solve the problem", and token usage alone explained 80% of the performance variance on BrowseComp, a browsing benchmark.
Even so, two things recur. The work splits into parts that do not depend on each other: attempts at one issue, failing tests of a compiler, branches of a research question. And in the first two rows, a test suite that is not an agent has the last word. In both rows that suite is the benchmark's own grader, and in the first it also works as an answer key. Your tests will be a weaker referee than that.
The compiler project shows both from the inside. Parallelism stalled when all 16 agents hit the same bug while compiling the Linux kernel, which is one giant task. It came back when a known-good compiler served as an oracle, a trusted reference, and split the work by file. A person built that referee. Cursor reports a similar stall: 20 equal agents coordinating through a shared file with locks had the throughput of two or three (Cursor, 2026, a vendor report).
So the coding swarms with the big numbers are not teams that talk their way to an answer. They are closer to a search with a referee.
Isolation is not independence. Five agents in my run, none of which messaged another, repeated the same claim, and only the check changed it.
What I would build
Two questions first. Is there an automatic check on the result: tests, a verifier, an oracle? Does the work split into parts that do not depend on each other? Two yeses, and a large swarm is worth a pilot. Without two yeses I would not go past a handful, because most of the head-to-head evidence stops there: teams of three or four in Kim et al.'s main grid, one to five agents in the speedup tests. If the first answer is no, build the check before you build the swarm.
- One agent first, at the same budget. Match the thing you are short of, tokens, money or time, and report the other two. If the team got more budget, you benchmarked the budget.
- A check against something outside the agents wherever their outputs are combined. What counts: tests, an oracle, or the current version of the source. Another agent's agreement does not. An agent that reopens the source counts for the passage it finds, and its own opinion still does not. In my run that check changed 46 of 118 claims.
- Isolation first, then a structured exchange when the agents hold different facts. Agents work alone, then each posts one or two facts and one reason the leading answer may be wrong, and only then do they vote. The vote pools what the agents know and does not replace the check in rule 2.
- For many agents, assign the work instead of letting them negotiate it. In Cursor's reports planners do not write code and workers do not talk to each other, and in the speedup tests pre-assigned tasks beat self-organization. The evidence is thin and not one-sided: two of the three agent teams in the swarm table ran with no orchestrator. With planners above workers, the cost lever is the model mix. On one SQLite build Cursor reports similar quality for $1,339 with a strong planner and cheap workers, against $10,565 with one frontier model in both roles (Cursor, 2026, a vendor report).
- Measure the noise before the gain. In Kaliyev and Maryanskyy, 2026, a single preprint, a coordination protocol and its no-coordination baseline had equivalent setups on the first trial, where coordination was not yet active. Their success rates on that trial still differed by 10 points in one run of 100 tasks and by 0 in a rerun of the same tasks. Run the same setup twice before you compare two.
What this post might get wrong
- I reproduced nothing, and I skipped my own first rule. My numbers are counts from one review run. I never ran the review with one agent at the same budget, so I cannot say the eight earned theirs.
- The evidence is uneven. The big-swarm results are vendor reports and single preprints. The debate studies ran mostly small models on question answering, and the wins are on coding and research, so I cannot separate the structure from the model or the task. My review turned up no equal-cost comparison for swarms above ten agents, and the matched-budget study uses teams of three or four, with a side experiment up to nine. One report I cite cuts against the referee claim: Cursor says its SQLite swarm went on to pass 100% of a test suite it was never told about, and suspects that stacked agent review was a major contributor to quality.
- My referee was another agent, and it missed things. The checkers were the same model as the researchers, and both had the same paper open. What differed for the checkers was the brief: take one claim, find the passage, compare. Two of them still left "trace-level" unexplained, and 152 claims got no verdict. One of my own briefs described the paper by its December abstract. Two rounds of checks on this post confirmed a mean of -0.3% against the April preprint and took it for the latest version. The published version says 0.0%. It surfaced only because one checker found the journal named in one of the eight original reports.
- Matched budgets answer one question. They say whether a structure beats one agent at the same spend. They do not cover a job that will not fit in one context, or a team split to keep tools and permissions apart. Nothing here measures either.
The swarm results I trust most are the ones where a test can say no. Where nothing can, the studies here mostly found one model agreeing with itself, at several times the price.
What sits where your agents' outputs are combined: a test, a source, or another agent's opinion? And if you have an equal-cost number for a swarm above ten agents, post it. I could not find one.
Sources
- Kim et al., 2026, "Capable language models can outgrow the benefits of collaboration", Nature Machine Intelligence - 260 configurations, six benchmarks, matched reasoning-token budgets: mean team effect 0.0%, from +80.8% (centralized team, Finance Agent) to -70.0% (independent agents, PlanCraft). The 17.2x is a trace-level factor. The paper puts the task-level error ratio at about 1.1 and names no team design.
- The same study as a preprint, "Towards a Science of Scaling Agent Systems" (v3, April 2026) - the version my agents read. Its mean team effect is -0.3%. Versions 1 and 2, from December 2025, carry the abstract with "amplify errors 17.2x".
- Bertalanič and Fortuna, 2026, "The Ringelmann Effect in Multi-Agent LLM Systems: A Scaling Law for Effective Team Size" - 30 debating copies of Qwen2.5-7B carry 1.1 to 1.8 effective agents; 30 self-correcting copies reach about 6 on math word problems, an upper bound by the paper's own account. Preprint, question answering plus a small code pilot.
- Choi et al., 2025, "Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?" - one agent 0.72, best standard debate 0.74, plain vote 0.77 with five Qwen2.5-7B agents, averaged over seven benchmarks.
- Wynn et al., 2025, "Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debate" - on CommonSenseQA all 10 group compositions scored lower after debate than their own first vote; on MMLU 7 of 10 did, and on GSM8K 6 of 10 rose.
- Stasser and Titus, 1985, "Pooling of Unshared Information in Group Decision Making: Biased Information Sampling During Discussion" - the original hidden profile: 83% of groups chose the best candidate with full information, 18% when each member's facts favored a rival.
- Li et al., HiddenBench (v4, 2026), "Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMs" - 80.7% for one agent with all facts, 30.1% for a group with split facts; a share-then-decide protocol lifts GPT-4.1 from 0.04 to 0.80 on an 18-task subset.
- Tran and Kiela, 2026, "Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets" - one agent 0.418 against 0.333 to 0.388 for five team designs at 1,000 reasoning tokens.
- Khatua et al., 2026, "CooperBench: Why Coding Agents Cannot be Your Teammates Yet" - two agents in one repository had on average 30% lower success rates than one agent doing both features, per the abstract; the paper's Table 5 puts the pooled loss at 41%.
- Vals AI, 2026, "Do Agent Teams Pay Off? A Case Study on Vibe Code Bench" - teams cost 1.8 to 5.1x; one of four score differences is significant, a 7.3-point gain for the team.
- Mieczkowski et al., 2026, "Language Model Teams as Distributed Systems" - 1.36x median speedup with pre-assigned tasks, 0.88x with self-organization.
- Brown et al., 2024, "Large Language Monkeys: Scaling Inference Compute with Repeated Sampling" - 15.9% to 56% on SWE-bench Lite with 250 attempts, counting an issue as solved when any attempt passes the benchmark's unit tests. Preprint.
- Zhan et al., 2026, "Agensh: Scaling Organizational Intelligence to 1,024 Agents" - mean test-pass rate 19.3% with one agent and 28.8% with 128 on the five hardest ProgramBench tasks, six hours each. Single preprint, no cost accounting.
- Anthropic, 2026, "Building a C compiler with parallel Claudes" - 16 agents, 100,000 lines, about $20,000. Vendor report.
- Anthropic, 2025, "How we built our multi-agent research system" - +90.2% over one agent on an internal evaluation; token usage alone explained 80% of the variance on BrowseComp. Vendor report.
- Cursor, 2026, "Scaling long-running autonomous coding" - 20 equal agents coordinating through a shared file with locks had the throughput of 2 or 3. Vendor report.
- Cursor, 2026, "Agent swarms and the new model economics" - $1,339 against $10,565 for similar quality on one SQLite build. The same post says the swarm passed 100% of a test suite it was never told about. Vendor report.
- Kaliyev and Maryanskyy, 2026, "How Much Coordination Gain Is Real? A Paired Noise-Floor Protocol for Multi-Agent LLM Benchmarks" - a 10-point gap between equivalent configurations on one run of 100 tasks, 0 on a rerun. Single preprint.
- Human classics cited by name and year only: Smith 1776, Ringelmann 1913, Ingham et al. 1974, Brooks 1975.
AI disclosure: the literature review and this post were written with AI assistance (a multi-agent research run in Claude Code). Separate checker agents reopened the sources for the main claims, and again for the numbers in this post. I chose the framing and I stand behind the conclusions. Any errors that remain are mine.
Top comments (23)
The detail that stands out is that the sixth report kept the "trace-level" label and its checker marked it confirmed, while the paper's own introduction phrases it in a way that invites the error reading. So a checker comparing the claim with the source text can confirm a misreading when the source sentence is itself ambiguous.
When a checker found the paper using two definitions across versions, like the December abstract versus the April revision, what did it do: pick the latest version, or flag the claim as version dependent?
Good catch, with one correction. The sixth report kept "trace-level" in its claim, so its "confirmed" was literally true. The failure there is quieter: nobody said what trace-level means, so a reader further down could still take it as an error rate.
On versions, nine checker verdicts touched this paper. One checker opened v1, v2 and v3 and flagged the claim as version-dependent: v1 called the figure a failure-probability ratio, v3 redefined the same numbers as extra work from trace analysis. The others worked from v3, the newest arXiv version, so in practice they picked the latest version they could find. None of them noticed it was already stale. The journal version had been out since July, and that is the one that moved the mean from -0.3% to 0.0%.
Fair correction, thanks: the sixth claim was true as written, and the gap was an undefined term rather than a false confirm. The version finding is the bigger one for me, since "newest arXiv version" and "version of record" are not the same thing, and the journal version is what moved the mean. Would one line in the checker brief, "look for a published version before citing a preprint", have caught it, or does that need its own agent?
The fix you landed on, a second agent that reopens the cited source and compares the claim, matches what I found cheapest in my own citation checks: a word-level rule first, then an NLI model only on claims that pass it. A vote can't catch a shared misreading, since every voter read the same summary.
Cheap first, expensive only where it's needed makes sense. One warning from this run: a word-level rule would have passed my bad claims. "Amplify errors 17.2x" is verbatim in the December abstract, so the words matched. What broke was the version and a dropped qualifier, "trace-level errors" becoming "errors". I'd be curious whether your NLI step catches that. Dropping a modifier can look like a safe generalization to an entailment model, and here it changed the meaning.
A small nuance on the vote: my agents didn't read a shared summary. Each opened the same preprint on its own and printed the same sentence. Same source, same model, same prompt template, which is why five agreeing was worth about one.
I haven't tested dropped qualifiers yet, and I'd expect NLI to miss them: 'amplifies trace-level errors 17.2x' entails 'amplifies errors 17.2x', so the shorter claim reads as a safe generalisation, like you say. In my small test NLI also let two entity swaps through. A cheap guard before the NLI step could be to require every qualifier and version string next to the number in the source sentence to appear in the claim.
the vote part is the scary bit, five agents agreeing just looks like confidence. reopening the cited source per report is boring but its the only check that actually caught it
Agreement looks like confidence, and here it was one sentence counted five times. I don't want to oversell the boring check, though. It ran on 118 of 270 claims, so 152 never got a verdict, and two checkers still left "trace-level" unexplained. It worked where it ran. What I took from it: run it on every claim that carries weight, rather than trust it more.
the unchecked claims are where the confidence gap lives. how are you deciding which ones carry enough weight to run the check?
An average can hide configurations that moved the score both ways. Reporting the spread next to it is what lets a reader decide when a swarm is worth it.
Agreed, and that's why the extremes sit right next to the 0.0% in the post. The paper gives σ = 37.5% and prints -58.7% to +77.2% as a 95% confidence interval for the mean. With that σ across 260 configurations, the interval reads as the spread of the configurations, not uncertainty about the mean, and one of my checkers flagged exactly that.
The cut that tells you something is by task. On the finance benchmark every team design beat one agent, by 57% to 81%. On sequential planning every design lost, by 39% to 70%. The average mostly tells you those two cancel out.
The voting failure comes down to the variance of correlated draws. When N agents share model weights, prompt framing, and the same preprint excerpt, pairwise error correlation stays near one. Averaging across eight reports leaves error variance almost unchanged while multiplying token spend eightfold. Bertalanic and Fortuna's finding of 1.1 to 1.8 effective agents across 30 debating copies, at 64K prompt tokens per effective agent versus 1.4K without debate, puts a clear price on that correlation.
Lower error variance requires changing the conditioning event. Reopening the cited passage on one isolated claim breaks the summary prior, and an external oracle such as a compiler or unit test suite keeps the verification step independent of the generator.
The count was grouped by the wrong key. Before any vote, group the claims by the passage they came from and the version of that passage, and keep the model and prompt config attached to the group. Then count groups. "Five of six" collapses to one before the vote starts.
We hit the same shape in a signed, append-only log of agent task deliveries. Of about 2,000 delivery rows, 1,932 lacked a back-reference field. Our first explanation was an incentive story: the field was optional and nothing downstream read it, so writers skipped it. Splitting by signing key killed that. One key wrote 1,879 of the 1,932. Across all 10 signing keys, each one either always filled the field or never did. Zero keys switched. What read as population behavior was one writer's lifetime habit. Distinct keys do not prove distinct parties, so grouping like this gives an upper bound on independence, never a count of it.
It also changes what a checker should leave behind. A bare "confirmed" can't be re-audited once nobody knows which text it was checked against. The -0.3% verdict was right about the April preprint. Nothing in it said April, so it read as a statement about the paper. If each verdict carried the source version id and the quoted passage, a later reader could tell "confirmed against v3" from "confirmed", and your journal catch wouldn't have depended on luck.
Did any of your checkers record which version they opened?
Grouping by passage and version is the right key, and your signing-key split is the same shape as mine: 1,879 of 1,932 looks like a population until you split it.
To your question: partly. Every verdict had a url_checked field and quoted the passage in its evidence. Of the nine verdicts on this paper, seven named a version (v3, or v1 through v3). Two recorded the bare arXiv URL, which resolves to whatever is newest on the day you open it, so those two can't be re-audited against a version anymore.
What no verdict had was a field for "is there a later version anywhere". One checker had the journal abstract open, used it for a single effect, and kept the preprint's -0.3%, because the abstract doesn't state the mean. So I'd take your two fields, version id and passage, and add a third: where else this paper is published.
The number that stuck with me isn't 17.2x, it's 46 of 118 — nearly two in five claims changed once a checker reopened the source. That's the whole post in one line: the agents weren't wrong, they were just all reading the same April preprint the same way. A vote would've counted that shared misreading six times.
The 46-of-118 number is the one I'd want a control for. A checker that reopens the source and reads it fresh will also "change" claims that were fine the first time — differently worded, more hedged, the same figure with the surrounding caveat re-attached. So 39% is an upper bound on error unless you know how often an honest checker rewrites an honest claim. Cheapest version of that control: run both checkers on the same 20 reports and count how often they disagree with each other. If two checkers only agree 70% of the time, most of your 46 is checker variance, not agent error. If they agree 95%, the 46 is real.
The other thing that surprised me is that the scaled trace-level factor survives (p=0.030) while unscaled it doesn't (p=0.658). That's usually a sign the factor is partly measuring how verbose the team is, since dividing by the single-agent baseline folds cost back in. Did you check whether the 17.2 correlates with reasoning-token count per configuration? If it does, the number is a work multiplier in a very literal sense and the whole "amplify" framing in that April intro was never recoverable.
Fascinating observation on majority consensus failure in multi-agent swarms. When agents share similar baseline biases or prompting frameworks, voting often amplifies blind spots rather than resolving them. Cross-validation with adversarial or dialectical roles tends to catch these omissions much more reliably.
@p0rt Checked the arithmetic on the paper's numbers: sigma = 37.5% over 260 configurations gives a standard error of the mean of about 2.3 points, so a 95% interval for the mean would be about +/-4.6, not -58.7 to +77.2. The printed range is the spread of configurations, so the average is the wrong number to read the verdict from. Your by-task cut is the one that decides.