I run a lot of code review through agent fan-outs. Dozens of agents, each with a different lens, structured output, adversarial verification passes. The kind of pipeline that looks rigorous in a diagram.
Last week it returned a confident, unanimous, wrong answer. I kept the transcripts, so I can tell you exactly what it cost and exactly what fixed it.
The numbers are from one week of runs: 158 agents across 18 workflows, 139 MB of transcript, roughly 28 million tokens. About a third of that was avoidable.
The failure
A panel of three agents reviewed a technical claim and had to decide whether it was real and worth acting on. All three said no. Each wrote a detailed, specific, well-argued rejection.
Their reasoning turned on a published standard that governed the decision. None of them could open it. The document sat behind a login, and the agents correctly refused to authenticate, which is the behavior I want. So they reasoned from the title, from search snippets, and from what a document with that title probably says.
It says close to the opposite.
I read it myself through an authenticated browser session. One tool call. The actual text answered, in its own worked example, the two specific objections all three agents had built their rejections on.
That read cost about 2,000 tokens. It overturned about 1.2 million.
A 600 to 1 asymmetry, and it is the single most useful thing I did that week.
The part that is actually uncomfortable
I re-ran the panel with the real document text. This time all three agents reversed and said yes.
Good result, except for one thing: I wrote both briefings. The first one told them to be hostile and assume the claim was wrong. The second one quoted the standard and noted the previous panel had misread it.
Same question. Same model. Same evidence. Opposite framings, opposite verdicts.
So the second run was not a correction. It was me getting the answer I had already decided on, and three agents handing it back with citations. I caught it only because I happened to reread my own prompt and noticed I had written "read that last paragraph twice" above the paragraph I wanted them to weight.
I killed that run and rewrote it: quote the source with no commentary, strip the helps and hurts labels off the evidence before passing it on, and add a required field asking each agent where the briefing is wrong.
That version also said yes. But this time two of the three told me my briefing had overstated one constraint and buried a counterargument I should have surfaced. One pointed out I had handed them a favorable argument while omitting an unfavorable clause from the same document.
That is what a panel is for. Agreement is cheap. The value is in the disagreement, and you only get it if you leave room for it.
The failure mode you are not watching for
Partway through the week I hit a usage limit. Four agents died mid-run.
That cost me nothing. The runtime printed a failures block. A one-line check over the journal confirmed it independently:
for d in */; do python3 -c "
import json;ev=[json.loads(l) for l in open('$d/journal.jsonl') if l.strip()]
print('$d',sum(1 for x in ev if x.get('type')=='started'),
sum(1 for x in ev if x.get('type')=='result'))"; done
Started versus returned. If those numbers differ, something died. Loud failures are cheap because they are visible.
The expensive failure was the agent that succeeded. It completed normally, returned a well-structured result, and reported that it could not reach the document it needed. Then it reasoned from snippets anyway, and the agents downstream treated its output as settled. Nothing flagged it, because nothing had gone wrong in any way a harness can detect.
An agent that dies tells you it died. An agent that guesses tells you nothing.
The fix, which is nearly free
I added one required field to every judging agent's output schema:
provenance: for each load-bearing claim, state READ (and what),
EXECUTED (and what you ran), INFERRED, or BLOCKED.
A verdict whose key premise is INFERRED must say so.
Both models I tested comply with this well, and it costs almost nothing. It changes what you get back from confident to checkable.
The best example came from a run where an agent marked a login-gated page BLOCKED, listed precisely which figures it had taken on trust from an earlier agent without re-running them, and separated those from the facts it had verified at source itself. That agent's verdict was worth more than the three confident ones, because I could see where to push.
The clean negative that was the most thorough run
After the framing result I wanted a harder measurement, so I set up a blind test. This is the same method I use for grading eval suites: take material where the answer is already established, hide the answer, and see what the instrument actually detects. A detection rate you measure beats a detection rate you assert.
I locked six agents to a source tree, put every file containing prior analysis out of reach, and gave all six an identical prompt that never said what to look for.
Three of the six surfaced the underlying mechanism. The thing that actually mattered was two separate observations that only become significant once you put them together.
My first pass at scoring said none of the six made that connection. That was wrong, and it was wrong for an embarrassing reason: I graded their output with a keyword search instead of reading it. When I read it, two of the three had made the connection explicitly. One wrote that a step in its third finding was "the exact precondition finding 1 needs." Another wrote that the same step "is what finding 1 and point (a) above both exploit."
Then each of them filed that observation as a sub-clause of a minor finding and moved on.
So the failure was not detection, and it was not even composition. It was re-scoring. Both agents found the link, wrote it down in plain language, and then let the combined thing inherit the low severity of its parts instead of asking what the combination was worth on its own. The interesting result is not that agents miss connections. It is that they can find a connection and still not update on it.
My scoring error is the same shape: a check that returned a confident answer for the wrong reason, and only reading the underlying material caught it. That is three times in one week, which is roughly the point of this whole piece.
The run I keep thinking about returned nothing at all.
That agent did the most careful work of the six. It built the project, ran the existing 391 tests, extracted every reference document in the repository, and checked each of its candidates against them. It correctly matched the pieces it found to issues that were already documented and accepted. Then it concluded, reasonably, that nothing new was there.
Its own notes include a sentence describing the exact lines the real problem lives on, and recording them as safe.
It was not careless. It was following my instruction to treat the list of already-known issues as a hard filter, and it followed that instruction more faithfully than the agents that got closer. Each documented issue described one mechanism. Each carried an exception for a case its own mechanism never reaches. The combination reaches it.
So my filter was wrong, not the agent. "Does this match something already known" discards the combination that escapes what is known. The question has to be "does this match, and does any combination of these candidates reach the exception that the known item itself states."
The general version: when you give agents a filter, you are also telling them what to throw away, and a careful agent will throw away more.
What I changed
Four rules, in the order they run:
Read the governing documents yourself, before any agent starts. Whatever the decision actually turns on: the spec, the standard, the policy, the config. Hand one facts file to every agent so none of them re-derive it, and none of them guess. If it is behind a login, that is a human's job, and it is one tool call.
Cheap checks before expensive ones. My pipeline built full reproductions and then discovered, late, that two candidates died on questions costing one lookup each. Now the cheap disqualifying questions run first, on everything, before anything expensive runs on anything.
Never brief a panel toward an answer. Quote sources without commentary. Strip verdict-shaped labels off evidence before passing it downstream. Require a disagreements field.
Cap exploratory work on elapsed time or error rate, not on completion. One probe ran 99 minutes and accumulated 465 errors before dying on rate limits, chasing something an earlier probe had already ruled out structurally. Nothing stopped it, because nothing was watching anything except whether it had finished.
Add a composition pass, and re-score what it finds from scratch. One agent, not a fan-out, asking of the whole candidate list: does any combination here add up to something none of them is alone, and what is that combination worth judged on its own rather than inherited from its parts. The second half is the one that failed.
Audit your filters. For every filter you hand an agent, write down what that filter throws away, then go and look at the discard pile once.
Do not grade agents with a regex. I scored a detection test with a keyword search and got the headline result backwards. Read the output.
The honest limits
One week. One pipeline. One domain. I measured transcript volume and token counts, not a controlled experiment, and the framing result is n equals 3 on a single question.
What I am confident about is narrower than it sounds: when a panel of agents agrees, and one person wrote all the briefings, the agreement is evidence about the briefing. Everything else above is a consequence of that.
The deeper version of this is not new. A check that cannot fail is not a check. We know this about CI gates, where a required job satisfied by a skipped step passes review for years because nobody asked it to prove it could go red. We know it about test suites. It is just as true of a review panel, and it is harder to see, because the output is fluent, structured, and arrives in triplicate.
If you run agent fan-outs, go and read your own prompts. Look for the sentence where you told them what you wanted to hear.
I found mine.
Top comments (0)