TL;DR
- Multi-agent wins because it spends more tokens — token spend alone explains 80% of the variance. It isn't collective intelligence.
- Read fan-out works. Write fan-out is the failure mode. Every source converges here.
- Same-model, same-session self-review is nearly useless. Mean blind-spot rate of 64.5%; on code it misses 31.7% of its own semantic drift.
- Reviewing with clean context works. Reviewing the same diff more times does not: F1 drops.
- "Clean context" is not a separate session. It's the absence of producer-authored text.
- There is one controlled experiment comparing review topologies. Single author, N=30, injected errors. That's the whole corpus.
- The obvious success metric — "accepted PR" — is circular if you approve according to the reviewer. The reviewer becomes instrument and treatment at once.
- Almost every "magic bullet" in circulation is a single arm compared against the memory of how things used to be. That's testimony, not an experiment.
Table of contents
- The provocation
- Minimum vocabulary
- What the evidence supports
- What "clean context" actually means
- The hole
- Magic bullets without empiricism
- How to measure it yourself
- Hypotheses and open questions
- References
1. The provocation
There's an active debate about how to organize LLM agents to write software. One agent. An orchestrator with sub-agents. Self-refinement loops. Swarms.
And in every technical timeline there's the report that "company X put agents in a loop and productivity exploded."
What barely exists is measurement comparing topologies.
I started this piece writing "it doesn't exist." I was wrong — there is one controlled experiment. Single author, preprint, 30 artifacts, injected errors. And it contradicts part of the telemetry vendors publish.
That's the entire situation. It's not that the evidence points one way. It's that it barely exists, and the little that does conflicts with the folklore.
This document catalogs what's supported, delimits what isn't, names the magic bullets that circulate without backing, and closes with a measurement protocol any team running a harness can already implement.
2. Minimum vocabulary (to read the rest)
Three terms borrowed from experimental statistics. They aren't decoration — they're the difference between an experiment and testimony.
Arm
One of the configurations you compare. From trial arm, as in clinical trials: the treatment arm gets the drug, the control arm gets placebo. Same protocol, one variable different between them.
Here, an arm is a pipeline configuration:
| Experiment | Arm A | Arm B |
|---|---|---|
| Fan-out | N workers in parallel | 1 worker, N× budget, serial |
| Reviewer | gets diff + spec | gets diff only |
The discipline the word enforces: the only thing that differs between arms is the variable of interest. Same model, same repo, same gate. Changed two things? The experiment says nothing — you can't tell which one caused the difference.
The sentence "we put agents in a loop and productivity jumped" has no control arm. Jumped relative to what? To the memory of how it used to be. That's testimony, not an experiment.
Upstream and downstream
A river metaphor. Upstream is what comes earlier in the flow; downstream, later.
spec → task split → writing → gate → review → decision → merge → production
└──────── upstream ────────┘ └──────── downstream ────────┘
▲
you are here
It matters because an upstream failure contaminates everything downstream. If the task split is wrong (upstream), no clever downstream orchestrator fixes it. That's why MAST attributes ~42% of failures to specification: they're born at the top and propagate.
And it matters for measurement: a metric collected downstream of the reviewer's decision is contaminated by it. To measure the reviewer without circularity, you go upstream (finding level) or far downstream (what happens in production, weeks later). More in §7.
Arm, upstream, downstream — together
3. What the evidence supports
Six findings. None is about topology. All are about mechanisms that topology exploits or violates.
3.1 · Multi-agent wins because it spends tokens
Anthropic reported that its multi-agent research system (Opus lead + Sonnet sub-agents) beat a single-agent Opus by 90.2% on its internal eval.
The number that matters is three paragraphs down:
| Factor | Performance variance explained |
|---|---|
| Token usage | 80% |
| # of tool calls | ~10% |
| Model choice | ~5% |
Cost: multi-agent burns ~15× more tokens than a chat.
Analogy. You didn't hire a team that thinks together. You bought a bigger bookshelf — aggregated context that didn't fit in one window. If the task doesn't split into independent directions, you pay for the shelf and store one book on it.
⚠️ The consequence that haunts the rest of this article: every fan-out gain has the null hypothesis "it spent more tokens." If your experiment doesn't pair the budget, it measures budget, not topology.
📄 Anthropic — multi-agent research system
3.2 · Write fan-out is the failure mode
Cognition, "Don't Build Multi-Agents" (Jun 2025): actions carry implicit decisions. Two agents writing in parallel each decide, on their own, about style and edge cases. The decisions conflict. A stage is born whose only job is to reconcile the divergence the architecture created.
Ten months later, "Multi-Agents: What's Actually Working" (Apr 2026) — a narrowing, not a retraction:
It works when writes stay single-threaded and the extra agents contribute intelligence, not actions.
Rule of thumb: parallelize exploration, serialize decision.
📄 Don't Build Multi-Agents · What's Actually Working
3.3 · Self-review without an external signal degrades
Academic evidence, consistent, large effect.
Huang et al. (ICLR 2024) — intrinsic self-correction on GSM8K:
| GPT-3.5 | |
|---|---|
| wrong → corrected | 7.6% |
| correct → broken | 8.8% |
| net | negative |
Tsui (2025), *Self-Correction Bench* — 14 models, errors injected at 3 levels. Mean blind-spot rate of 64.5%: the model reliably corrects externally supplied errors and fails to correct identical ones in its own outputs. Almost comic detail: appending the word "Wait" cuts the blind spot by 89.3% — the capability exists, it's just asleep.
Xu et al. (ACL 2024), *Pride and Prejudice* — the self-refine pipeline improves fluency and amplifies self-bias. Working mitigation: larger model + accurate external feedback.
Reddy, Lolla & Sanku (AWS, 2026), *Articulate but Wrong* — 1,980 calls, 11 models, a real behavioral oracle. The producer judging its own output misses 31.7% of its own semantic drift. There are verbatim cases where it describes the behavioral difference and, in the next sentence, declares "behavior preserved."
Why it happens:
- Panickssery et al. (NeurIPS 2024): self-preference is linearly correlated with the model's ability to recognize its own text.
- Wataoka et al. (2024): the root is perplexity. Models prefer the familiar. Their own output is the most familiar thing there is.
Analogy. Asking a model to review its own code is asking an author to proofread their own manuscript from memory. They read what they meant to write, not what's on the page.
🔎 Honest counterpoint: a Jun 2026 preprint shows self-preference shrinks when an objective, verifiable criterion exists. Which reinforces the thesis: the enemy is the absence of an oracle, not authorship itself.
📄 2310.01798 · 2507.02778 · 2402.11436 · 2605.21537 · 2404.13076 · 2410.21819 · 2606.20093
3.4 · The one controlled topology experiment — and it contradicts the folklore
Song (Mar 2026), *Cross-Context Review* — arXiv:2603.12123. 30 artifacts, 150 injected errors, 4 conditions, 360 reviews.
| Condition | What it is | F1 | vs. CCR |
|---|---|---|---|
| CCR | fresh session, no production history | 28.6% | — |
| SR | self-review, same session | 24.6% | p=0.008 · d=0.52 |
| SA | subagent, with context | 23.8% | p=0.004 · d=0.57 |
| SR2 | repeated self-review, same session | 21.7% | p<0.001 · d=0.72 |
Larger gains on critical errors (+11pp) and code (+4.7 F1).
What makes the paper worth it: SR2 is the control. Reviewing twice in the same session does not improve over once (p=0.11). So CCR's gain comes from context separation, not from reviewing more.
And its sibling: Song (2026b), *More Rounds, More Noise* — arXiv:2603.16244. Extending CCR to multiple rounds hurts: F1 drops −0.073 (p<0.001). Extra rounds generate false positives faster than they find bugs. Optimal number of rounds = 1.
The conflict with Cognition
Cognition reports that Devin Review finds ~2 bugs/PR (~58% severe) and runs multiple cycles, finding new bugs each time.
- Vendor: "the loop finds new bugs each cycle."
- Experiment: "extra rounds generate false positives faster than discovery."
Both true. The reconciliation is the most important design rule in this piece:
🎯 What legitimizes a new review isn't the cycle. It's the diff having changed.
- Reviewing the same diff 2× → noise (that's D-CCR, F1 drops).
- Reviewing a new diff, produced by rework that passed the gate → legitimate review, different artifact.
Devin works because there's rework between passes. D-CCR fails because there isn't.
📄 2603.12123 · 2603.16244
3.5 · Raw loops (Ralph) are repeated sampling
Ralph (Huntley): an agent in an infinite loop, same prompt, fresh context each iteration, filesystem + git as memory.
Theoretical basis: Large Language Monkeys (Stanford, 2024). Coverage (fraction solved by some sample) scales log-linearly with the number of samples, across 4 orders of magnitude:
| SWE-bench Lite, DeepSeek-Coder-V2 | |
|---|---|
| 1 sample | 15.9% |
| 250 samples | 56% |
| single-shot SOTA at the time | 43% |
The caveat, in the same paper: without an automatic verifier, majority voting and reward models saturate after a few hundred samples. Coverage rises, performance doesn't. The needle is in the haystack and nobody finds it.
That's why every spectacular Ralph demo shares the same property: a simple, automatic oracle. A browser. A compiler. Compiles, passes, runs faster.
💡 Ralph is fuzzing. It's worth exactly what the oracle is worth. Without an oracle, it's a haystack machine.
3.6 · The best evidence/cost ratio available today
Advisor strategy (Anthropic, Apr 2026): the cheap model drives, escalates to the expensive one only when stuck.
| Config | Benchmark | Score | Cost/task |
|---|---|---|---|
| Sonnet solo | SWE-bench Multilingual | 72.1% | — |
| Sonnet + Opus advisor | same | 74.8% | −11.9% |
| Haiku solo | BrowseComp | 19.7% | — |
| Haiku + Opus advisor | same | 41.2% | — |
Better and cheaper — rare. Mechanism: the advisor produces 400–700 tokens per query and reduces total turns, because the executor errs less.
Cognition tried it with a weak primary and was honest: it didn't work — the primary didn't know when to escalate. It worked between frontier models: a capability router, not a difficulty escalator.
4. What "clean context" actually means
The point that confuses almost everyone (me included, in the first draft): "separate session" and "clean context" are not the same thing. The session is the container. Contamination travels in the content.
The axis that matters isn't session. It's authorship.
| Enters the reviewer | Authored by | Contaminates? | Why |
|---|---|---|---|
| Coder transcript | producer | 🔴 | Exactly what CCR removes |
| PR description / commit msg | producer | 🔴 | Compressed transcript |
| Implementation plan | producer | 🔴 | Carries the producer's decisions |
Inline comment (// safe because X) |
producer | 🔴 | The reviewer believes the claim |
| Spec / design doc | architect | 🟡 | See below |
| Acceptance criteria | architect | 🟡 | Same |
| Diff | — | 🟢 | It's the object |
| Repo at base commit | — | 🟢 | Necessary context |
| Test / build output | oracle | 🟢 | External signal |
There are two distinct problems hiding under the "clean context" label. A fresh session solves only one.
Problem 1 — authorship contamination ✅ solved
The reviewer reads what the coder wrote about the code and starts reasoning inside the coder's frame. That's Tsui's blind spot, Panickssery's self-preference.
A fresh session solves it. But check three silent leaks:
- PR body — if the Developer wrote the description and the reviewer reads it via the API, the transcript came back through the side door. Generate the description mechanically or don't pass it.
-
Commits —
fix: handle null safelyis a producer assertion. - Code comments — you can't remove them (they're the artifact). Instruct the reviewer to treat them as claims to verify, not evidence.
Problem 2 — spec anchoring 🟡 unsolved
A reviewer who reads "the spec asks for X" runs "does this do X?" → confirms and stops. Conformance checking.
A reviewer who reads only the diff is forced to reconstruct intent backwards → "is this wrong?" → the edge cases nobody specified surface. Which, by definition, are the ones the spec doesn't mention.
The spec is ground truth for conformance and a confirmation anchor for defects. Same input, opposite effect.
Hence the split into two reviewers:
diff + spec + AC → conformance → "is it what was asked?"
diff (only) → defect → "is it wrong?"
⚠️ Here the evidence runs out. CCR measured Problem 1. Nobody measured Problem 2 — spec-in vs. spec-out came from a Cognition blog post, no control. I recommend it with lukewarm conviction. And it's testable (§7).
Operational definition, then:
Clean context = the absence of any producer-authored text, at any stage. A fresh session is the mechanism. Authorship is the criterion.
5. The hole
5.1 · The one serious RCT isn't about agents
METR — the only rigorous randomized trial on AI and software productivity. 16 experienced devs, 246 tasks in their own repos, each task randomized to allow/forbid AI.
devs' prediction ............ 24% faster
post-task estimate .......... 20% faster
ACTUAL MEASUREMENT .......... 19% SLOWER
─────────────────
perception × reality gap .... 39 percentage points
Economists predicted +39%. ML experts, +38%. Everyone got the sign wrong.
Feb 2026 correction: the new cohort suffered severe selection bias — 30–50% of invited devs refused to work without AI, skewing toward those who benefit least. Revised: −4%, CI [−15%, +9%]. The methodological lesson beats the substantive one: the design doesn't survive mass adoption of the very thing it tries to measure.
And: humans using Cursor are not agents in a loop. It says nothing about topology.
📄 METR RCT · correction
5.2 · The best multi-agent paper is observational
MAST (Berkeley, NeurIPS 2025): 1,600+ traces, 7 frameworks, 14 failure modes, κ=0.88.
| Category | Share |
|---|---|
| Specification / design | ~42% |
| Inter-agent misalignment | ~37% |
| Task verification | ~21% |
Two sentences from the abstract: MAS gains over single-agent are often minimal; failures live in coordination, not cognition — a better model doesn't fix them.
But it's observational. It says why things break. It doesn't say which architecture breaks less.
5.3 · The benchmark everyone cites is broken
- SWE-bench+ (2024): 32.67% of "successful" patches had the solution leaked in the issue itself.
- OpenAI audit: of 138 hard problems, 59.4% had material test/statement flaws. OpenAI stopped reporting the benchmark.
- Successor: a later audit estimates ~30% of SWE-Bench Pro is broken.
You can't validate an architecture designed to please a benchmark using that same contaminated benchmark.
📄 OpenAI – why we no longer evaluate · signal vs noise · 2410.06992
5.4 · The hole, enumerated
| # | Question | Study? |
|---|---|---|
| 1 | Does fan-out of N beat 1 worker with N× budget? | ❌ |
| 2 | Optimal cap on rework↔review cycles (with rework between them)? | ❌ |
| 3 | Does cross-family reviewer beat same-family with clean context? | ❌ |
| 4 | Rate of semantic conflict between parallel PRs? | ❌ |
| 5 | How much of the cost is reconciliation tax? | ❌ |
| 6 | Does spec-first improve accepted PRs or just the feeling? | ❌ |
| 7 | Does a raw loop beat a structured pipeline on legacy? | ❌ |
| 8 | Escape rate — bug that passes and gets reverted later? | ❌ |
| 9 | Does clean-context review beat same-session? | ✅¹ |
¹ 1 preprint, single author, N=30, injected errors, F1 < 30% in all conditions — even the best one misses the majority.
One out of nine. The other eight are answerable with what's already in GitHub.
6. Magic bullets without empiricism
The catalog of things that circulate as truth and have no control arm. They aren't necessarily false — they're untested. The difference matters.
| "Magic bullet" | What it promises | What the evidence says | Status |
|---|---|---|---|
| "Loop it and productivity explodes" | Ralph solves everything | Worth what the oracle is worth; without one, coverage without performance | 🔴 folklore |
| "More agents = more intelligence" | Swarm > individual | 80% of variance is just token spend; parallel writes collide | 🔴 refuted at the mechanism |
| "The model reviews its own code" | Self-refine closes the loop | Blind spot 64.5%; drift 31.7%; self-refine amplifies bias | 🔴 refuted |
| "Review again until it finds nothing" | Convergence by iteration | Same diff 2× → F1 drops, false positives rise | 🔴 refuted (D-CCR) |
| "A review subagent already fixes it" | Isolating in a subagent is enough | Subagent with context loses to a clean session | 🟡 partial |
| "A detailed spec guarantees quality" | More spec = fewer errors | Plausible (MAST: 42% is spec), but the effect on accepted PRs is unmeasured | 🟡 untested |
| "Fan-out is faster/better" | Parallelizing always wins | Might be just token spend; never paired by budget | 🟡 confounded |
| "Clean context = fresh session" | Just open another session | Contamination is about authorship, not session | 🟡 imprecise |
| "Our productivity went up X%" | Self-reported gain | METR: perception gets the sign wrong by 39pp | 🔴 broken instrument |
| Clean context for the reviewer | Reviewer without history finds more | ✅ the only one with a controlled experiment | 🟢 supported |
| Advisor: cheap drives, expensive advises | Better and cheaper | ✅ direct comparison, +2.7pp and −11.9% | 🟢 supported (1 pair) |
Legend: 🔴 refuted or folklore · 🟡 plausible, untested · 🟢 supported by experiment
The 🟡 row is the most dangerous. Not because it's wrong — because it sounds right and nobody checked. That's where the next paper lives, and probably half your architectural assumptions too.
7. How to measure it yourself
The barrier is low: an orchestrator that already emits events, an NDJSON artifact, a DuckDB, a draw in the dispatcher.
7.0 · The metric you run BEFORE everything
If you approve according to the reviewer, "accepted PR" doesn't measure the reviewer — it's an echo of it. The reviewer becomes treatment and instrument at once. That's circular.
First thing to instrument:
override_rate = (PRs where you disagreed with the reviewer) ÷ (PRs reviewed)
If it's ≈ 0, your pipeline has no quality control. It has an opinion, repeated twice. Your entire
yieldis the reviewer's opinion of itself.
7.1 · The golden rule of the non-circular metric
Every metric downstream of the reviewer's decision is contaminated by it. To measure without circularity, go:
- upstream → finding level, before the decision, or
- far downstream → what happens in production, which the reviewer can't influence.
7.2 · Reviewer experiment (spec-in vs. spec-out) — paired design ⭐
Reviews are cheap to duplicate. Don't randomize — run both arms on the same diff and judge blind to the arm.
Unit of analysis: the finding, not the PR. You're still the oracle — but blind to the arm. A judge, not a rubber stamp.
| Metric | Answers |
|---|---|
| Precision per arm | valid findings ÷ emitted |
| True positives unique to B | what only the spec-less one caught |
| True positives unique to A | what only the spec one caught |
| Overlap A∩B | how redundant the second pass is |
Each PR is its own control → ~20–30 PRs already give power. Cost: 2× reviews. Latency: immediate.
7.3 · Fan-out experiment — Bernoulli per issue
Here the reviewer is the same in both arms (constant apparatus, not treatment), so randomizing is fine:
// in the dispatcher
const arm = Math.random() < 0.5 ? "fanout" : "sequential";
emit({ ...event, arm }); // write to NDJSON
Per issue, never per period — otherwise the model's tide (new release, big refactor) becomes a confounder correlated with the arm.
Compare with metrics not contaminated by the reviewer:
| Metric | Circular? | Definition |
|---|---|---|
| Semantic conflict ⭐ | 🟢 no |
tests@PR green and tests@integration red |
| Escape rate | 🟢 no | revert / hotfix / bisect within N days |
| Tokens per accepted PR | 🟢 no | pure accounting |
| Yield | 🟡 partial | but constant across arms, so comparable |
Semantic conflict is the fan-out's deterministic oracle: it measures exactly what fan-out is supposed to avoid. More semantic conflict + same yield = fan-out lost.
7.4 · The gold pairing against token spend
Recall §3.1: fan-out might win just because it spends more. The correct control is not "parallel vs. sequential." It's:
N workers in fan-out × 1 worker with N× the budget, with paired token budget.
With a paired budget, the residual gain is the real value of the topology. Without it, you're measuring the ticket price and calling it the destination.
7.5 · Defining "successful PR" without circularity
The honest ground truth is external and in the future:
- revert within N days, or
- hotfix touching ≥1 line from that PR, or
- incident/issue linked to the commit, or
- a regression test that, via
git bisect, points to that PR.
The last is elegant: when someone writes a test for a bug, you get a retroactive oracle for every earlier PR.
Cost: weeks of latency + large N (escapes are rare). It's the truth, but it's slow.
7.6 · Event schema
{
"run_id": "9f2c…", "issue": 142, "wave": 2, "task": "auth-mw",
"arm": "fanout",
"agent": "developer", "model": "claude-sonnet-4-6", "effort": "high",
"input_tokens": 84213, "output_tokens": 6120, "cache_read": 61004,
"duration_ms": 214000,
"files_touched": ["src/auth/mw.ts", "src/auth/mw.test.ts"],
"gate": { "tests": "pass", "types": "pass", "lint": "pass" },
"findings": [ { "severity": "major", "hash": "a91f…", "accepted": true } ],
"human_override": false,
"cycle": 1
}
One line per step, workflow artifact, aggregate in DuckDB. Deterministic data out of a deterministic system.
7.7 · Decision tree
8. Hypotheses and open questions
Framed as testable hypotheses — each with the experiment that decides it. Run any one to the end and you have publishable material, because literally nobody has run it.
Hypotheses with a predicted direction
H1 — Fan-out doesn't beat sequential with a paired budget.
Prediction: with equal token budget, yield and escape_rate tie; fan-out only wins latency.
Test: §7.4. Falsified if: fan-out wins with a paired budget → the topology has intrinsic value.
H2 — Omitting the spec in the defect pass increases edge-case capture.
Prediction: the spec-out arm has more unique true positives on unspecified errors.
Test: §7.2. Falsified if: spec-in dominates → the anchor helps more than it hurts.
H3 — A cross-family reviewer beats same-family with clean context.
Prediction: Claude writes, GPT reviews > Claude reviews Claude, even with a fresh session.
Test: three arms (same-family clean / cross-family / same-session) on the same diff.
Why it matters: it would separate "clean context" from "different authorship" — today conflated.
H4 — Semantic conflict grows super-linearly with the number of parallel workers.
Prediction: 2 workers ≈ fine; 4+ → conflict spikes.
Test: vary N in the fan-out arm, measure semantic conflict (§7.3).
Consequence: there'd be a small optimal N, and large swarms would always be bad.
Genuinely open questions (no obvious direction)
Q1 — What's the optimal cap on rework↔review cycles?
D-CCR says reviewing the same diff 2× is bad. It doesn't say how many times rework + new diff + review is worth it. The tail of cycles-to-green is where the cost lives, and nobody has characterized it.
Q2 — Does a raw loop beat a structured pipeline on legacy?
Every Ralph demo is greenfield with an oracle. Zero evidence on legacy without tests — which is most of the software that exists. My bet is Ralph collapses there, but it's a bet.
Q3 — Does spec-first improve accepted PRs or just the sense of control?
MAST says 42% of failures are spec-related. It doesn't say that more spec reduces failure — there may be a plateau, or negative returns from over-specification. Nobody measured the curve.
Q4 — Is there transfer between "finding injected bugs" and "finding real bugs"?
The entire review corpus (Song included) uses injected errors. A synthetic bug has a synthetic signature. If a reviewer good at injected bugs isn't better at real bugs, half this literature doesn't transfer — and we don't know.
The meta-problem
None of these hypotheses needs a lab. They all fit in a production harness with a draw in the dispatcher and an NDJSON. That they remain open isn't a tooling gap. It's that measuring is work and testimony is free.
Conclusion
The state of the art is:
- six mechanisms reasonably established,
- one controlled topology experiment (preprint, single author, F1 < 30%),
- one reference benchmark that its own biggest user abandoned,
- and an entire ecosystem deciding architecture by blog post and intuition — the intuition METR measured getting the sign wrong by 39 points.
This isn't an argument against building agent pipelines. It's an argument against building them without making them, themselves, instruments that measure themselves.
You already have the data. What's missing is the draw in the dispatcher and the nerve to find out fan-out maybe wasn't paying.
Meanwhile, "I looped it and productivity jumped" is the same epistemic class as "I felt 20% faster" — and that sentence, when measured, was 39 points off.
9. References
Productivity (experimental)
- Becker, Rush, Barnes & Rein — Measuring the Impact of Early-2025 AI on Experienced OS Developer Productivity. METR, 2025 · blog · arXiv:2507.09089
- Becker et al. — Changing our Developer Productivity Experiment Design. METR, 2026 · blog
Topology and self-review (experimental)
- Song, T.-E. — Cross-Context Review. 2026 · arXiv:2603.12123
- Song, T.-E. — More Rounds, More Noise. 2026 · arXiv:2603.16244
- Huang et al. — LLMs Cannot Self-Correct Reasoning Yet. ICLR 2024 · arXiv:2310.01798
- Tsui, K. — Self-Correction Bench. 2025 · arXiv:2507.02778
- Xu et al. — Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement. ACL 2024 · arXiv:2402.11436 · ACL
- Reddy, Lolla & Sanku (AWS) — Articulate but Wrong. 2026 · arXiv:2605.21537
- Panickssery, Bowman & Feng — LLM Evaluators Recognize and Favor Their Own Generations. NeurIPS 2024 · arXiv:2404.13076
- Wataoka, Takahashi & Ri — Self-Preference Bias in LLM-as-a-Judge. 2024 · arXiv:2410.21819
- Self-Preference Is Weak or Absent in Verifiable Instruction-Following Revision. 2026 · arXiv:2606.20093
Observational and scale
- Cemri, Pan, Yang et al. — Why Do Multi-Agent LLM Systems Fail? NeurIPS 2025 · arXiv:2503.13657
- Brown et al. — Large Language Monkeys. 2024 · arXiv:2407.21787
- Aleithan et al. — SWE-Bench+. 2024 · arXiv:2410.06992
- OpenAI — Why SWE-bench Verified no longer measures frontier coding capabilities · openai.com
- OpenAI — Separating signal from noise in coding evaluations · openai.com
Vendor telemetry
- Anthropic — How we built our multi-agent research system. 2025 · anthropic.com
- Anthropic — The advisor strategy. 2026 · claude.com
- Yan, W. (Cognition) — Don't Build Multi-Agents. 2025 · cognition.com
- Yan, W. (Cognition) — Multi-Agents: What's Actually Working. 2026 · cognition.com
- Huntley, G. — Ralph. 2025 · coverage The Register
Appendix · Strength of evidence
| Source | Type | Control? | Answers topology? |
|---|---|---|---|
| METR RCT | randomized | ✅ | ❌ humans with IDE |
| Song, CCR | controlled | ✅ | ✅ review topologies |
| Song, More Rounds | controlled | ✅ | ✅ # of rounds |
| Large Language Monkeys | benchmark | ✅ | ◐ sampling |
| Huang et al. | benchmark | ✅ | ❌ self-correction |
| Tsui | injected error | ✅ | ❌ blind spot |
| Xu et al. | experimental | ✅ | ◐ self-refine |
| Articulate but Wrong | behavioral oracle | ✅ | ❌ self-review |
| MAST | observational | ❌ | ◐ failures |
| Anthropic multi-agent | internal eval | ❌ | ◐ no paired budget |
| Anthropic advisor | internal eval | ✅ | ✅ 1 topology pair |
| Cognition (both) | telemetry | ❌ | ❌ |
Two ✅✅ rows. Song (N=30, single author, injected errors) and the advisor (internal eval, 1 pair). That's the whole corpus with both control and a topology answer. And the two disagree with half of what gets repeated out there.
Top comments (0)