Alberta published a case study last week about scanning 466 million lines of code across 1,280 applications using around 50 specialized Claude agents - red-team agents mapping exploit paths, blue-team agents writing remediation plans, code-quality agents, content-clarity agents - all running in parallel. The scan that would have taken six years traditionally took 20 hours.
Everyone shared the 466 million number. The throughput is impressive.
But that's not the number I kept thinking about.
The number I kept thinking about: how many of those 50 agents needed to trust the output of another agent before they could do their job?
And the question nobody asked: who owned that handoff?
A Fleet Is Not One Agent Scaled Up
When you run one agent, you supervise it directly. You read the output, spot the gap, course-correct. The failure mode is visible. The loop closes fast.
When you run a fleet of specialized agents, something changes. You're not supervising each agent anymore. You're designing how they relate to each other. You're thinking in interfaces, not prompts.
In Alberta's setup, a red-team agent identifies a vulnerability. That finding becomes the input to a blue-team agent that writes a remediation plan. Two specialized agents, two distinct roles, one handoff between them.
The first time I chained two agents in my own setup, the second one confidently built on the first one's mistake - because I never defined what "done" meant at the handoff. The first agent's output looked complete. It wasn't. The second agent had no way to know.
That's the asymmetry a fleet creates. Individual agents fail loudly. Handoffs fail quietly.
The Handoff Is Where the Risk Moved
When agents work in sequence, one agent's output becomes the next agent's input. On the surface this looks like efficiency - specialization, parallelism, faster throughput. Underneath, it creates a compounding problem.
Each agent is good at its slice. The red-team agent knows how to map exploit paths. The blue-team agent knows how to write remediation plans. Neither agent is responsible for checking whether the transition between them worked.
Errors introduced in step one get treated as ground truth in step two. By the time you're looking at step three's output, the mistake is three layers deep and doesn't look like a mistake anymore - it looks like a conclusion.
I've watched this happen with much smaller chains. An agent summarizing a document slightly overstates a conclusion. The next agent builds a recommendation on that summary. By the time a human looks at the recommendation, the original overstatement is invisible. The recommendation looks well-reasoned.
The throughput is fast. The error is silent. Those two things together are the actual risk profile of a fleet.
Output Qualification Is the Missing Job
If the handoff is where the risk lives, then output qualification is the job that doesn't exist yet in most agent setups.
Output qualification isn't the same as error-checking. It's not about catching mistakes after the fact. It's about defining - upfront, at design time - what "good enough to proceed" looks like at each handoff point.
In the Alberta setup, this might look like: before the blue-team agent acts on the red-team agent's exploit mapping, something checks that the mapping includes the required fields, crosses a confidence threshold, and flags ambiguous cases for human review. That "something" can be an evaluation agent, a structured schema check, or a human gate. The form matters less than the fact that it exists.
Three things I've started doing when designing agent chains:
Define acceptance criteria at the handoff, not just the output. It's easy to write a prompt that says "produce a detailed analysis." It's harder - and more useful - to write criteria that say "this analysis is ready for the next agent when it includes X, cites Y, and has a confidence score above Z."
Build the qualification layer before you scale. One agent failing is a debugging problem. Five agents downstream from one failure is an incident. The cost of adding a qualification check at the handoff stays roughly constant as your fleet grows. The cost of not having one compounds.
Name the specialized roles explicitly. In Alberta's setup, "red-team agent" and "blue-team agent" are clear role definitions with implied scope boundaries. Each agent knows what it's responsible for. Most internal setups skip this. You end up with agents that have overlapping scope, no clean handoffs, and failures that fall in the gaps between them.
The Number Nobody Asked About
The 466 million lines in 20 hours is the headline number. It's the right number for a press release.
The more interesting number is how many of those 50 agents were receiving output from another agent rather than from a human - and how many qualification checks existed at those handoffs.
A fleet isn't one agent scaled up. The throughput lives in the parallelism. The risk lives in the seams.
When you design a fleet, you're not writing prompts. You're designing an org chart - role definitions, handoff policies, and qualification gates at every point where one agent's output becomes another agent's input.
If you've chained agents: what broke first?
Top comments (7)
"Output qualification is the missing job" names something I've been building toward from the tooling side without having a clean phrase for it. The part that lands hardest: individual agents fail loudly, handoffs fail quietly. That asymmetry is why a fleet's real failure mode is never a crash, it's a plausible-looking conclusion three layers downstream of an overstatement nobody flagged. Your three practices (acceptance criteria at the handoff, qualification layer before scaling, explicit role boundaries) are the right shape, but I'd push on the qualification layer itself: what checks the qualifier? If it's another LLM evaluating "does this mapping include required fields and cross a confidence threshold," you've just added a fourth trust layer instead of removing one. The version of this I'd trust more is a qualifier that's mechanically checkable, schema conformance, required-field presence, confidence above a hard number, not judgment calls dressed as gates. Curious whether Alberta's setup published anything on what their qualification checks actually looked like mechanically.
Pushing back a little: making qualification a job risks making it a bottleneck. If one role owns this, agents start optimizing to pass the reviewer, not to actually qualify output. A protocol baked into each handoff beats a gate one person holds.
We're on the same side and your pushback is my point with a sharper edge on it. "Agents start optimizing to pass the reviewer, not to actually qualify output" is just Goodhart aimed at the qualifier, and it's the exact reason I said the qualifier can't be another judgment call, because a judgment call is the most gameable target there is. So far, agreed all the way.
Where I'd hold the line: protocol-baked-into-the-handoff has the same failure unless the protocol checks receipts instead of assertions. If the handoff contract is "attach a confidence score and cite what you considered," an agent optimizing to pass emits a plausible confidence and plausible citations, and you've distributed the rubber stamp across every handoff instead of concentrating it in one reviewer. That's worse, because now the stamp is everywhere and nobody's watching any single instance of it. A protocol beats a gate-one-person-holds only when what it checks is un-gameable at the phrasing level: did the command run, does the cited artifact exist, does the exit code match the claim. The real axis isn't gate versus protocol, it's judgment versus receipt. Move the check to receipts and it doesn't matter whether one role holds it or every handoff carries it, the agent can't phrase its way through either. Leave it as judgment and both designs rot the same way, one just rots in more places at once.
Right — and the escape hatch I keep coming back to is making qualification a contract check rather than a judgment call. Schema violations fail mechanically; reviewer opinions get gamed. Goodhart applies to the qualifier the same way it applies to the agent, so the qualifier itself has to be un-gameable by design.
Un-gameable by design is the right bar, and it's worth pushing past schema-conformance to get there. Shape-correct isn't the same as content-correct: an agent optimizing to pass a contract check will happily emit a citation field that type-checks and is semantically empty, three plausible-looking references that don't actually bear on the claim. Schema catches "did you fill in the field," not "does the field mean anything." A contract check that stops at shape has the same hole a judgment-call gate has, just moved one layer down and harder to notice because it looks mechanical.
The receipt-versus-judgment axis you named still holds the fix: the contract needs content predicates that are themselves receipts, minimum citation count against artifacts that actually exist, not just present, is one. Shape gets you past the cheapest gaming move. Content-as-receipt is what closes the rest.
The handoff point is more important than the number of agents. Once one model’s output becomes another model’s assumed truth, the system needs the same controls we would expect between human teams: a defined contract, required evidence, validation and a clear owner who can reject the output.
I would be cautious about relying on self-reported confidence though, as models can be confidently wrong. I've seen this way too many times. For higher-risk work, structured checks against the original source or a human approval point seem safer than asking another agent whether the first one was correct.
the assumed truth problem is the one that got us. built a handoff where agent B was supposed to validate agent A’s output and it took two weeks to notice it wasn’t actually rejecting anything - just passing everything through with a slightly different format.
adding rejection criteria to the handoff spec helped more than I expected. framing it as "reject if" rather than "pass if" changed the behavior noticeably. and yeah, scrubbing the confidence score was the right call - ours was anchoring on it instead of running independent checks.