I had a three-agent setup: a planner, a researcher, and a critic passing work between them. I graded each one carefully. The planner made clean plans. The researcher cited its sources correctly. The critic caught weak claims. Every agent scored around 0.9 on its own tests, and I was happy.
The team was still wrong about a third of the time.
The plans were good. The research was good. The critiques were good. But somewhere between one agent and the next, things fell apart. The planner would say "research scaling laws, but skip this one paper, the user already has it," and the researcher would cite that exact paper two turns later. The constraint just vanished in the handoff. Nobody's individual turn failed a test, and the final answer was wrong anyway.
That is when it clicked: I was grading the agents when I should have been grading what happens between them.
Multi-agent is not single-agent times three
Here is the trap. A single agent is easy to reason about: it takes an input, does its thing, gives an output, and you score that output. So with three agents, the natural instinct is to score all three the same way and call it done.
But a team is not three separate functions. It is a chain, where one agent's output becomes the next agent's input. And the receiver never sees everything the sender knew. It sees the previous turn and whatever context got passed along. The team succeeds or fails based on how well those handoffs preserve what matters.
Do the math on it. If every agent is 95 percent good on its own turn but each handoff drops one small thing, a three-agent chain can be wrong a third of the time while every individual score stays green. The failures do not live in the agents. They live in the seams, and per-agent tests are blind to the seams by design.
Grade the handoff, not the agent
The shift that fixed this was changing what I treated as the thing under test. Not the agent's turn. The handoff: the moment one agent passes work to the next.
For every handoff, I now ask three plain questions:
- Did the receiver keep what the sender said? Every constraint, every decision, every open question. This is where my missing-paper bug lived. The researcher was clean on its own, but it dropped a constraint the planner had set.
- Did each agent stay in its lane? The planner should plan, the researcher should research, the critic should critique. When an agent starts doing someone else's job, the whole reason you split them up falls apart.
- Does the final answer still agree with the earlier turns? A correct fact from turn two should not get quietly contradicted by the summary at the end.
Same three agents, completely different lens. Instead of "is this turn good," I am asking "did the thing survive the trip to the next agent."
The three ways handoffs break
Almost every team failure I have seen falls into one of three buckets.
A constraint gets dropped or misread. The sender said something specific and the receiver lost it, or paraphrased it with a number flipped, or invented context that was never there ("as we agreed earlier" pointing at a turn that never happened). My skip-this-paper bug was this one.
An agent drifts out of its role. The critic starts proposing plans. The researcher starts writing summaries. Now the test you wrote for that agent is measuring the wrong job, and the division of labor you set up has quietly collapsed.
The team contradicts itself. The researcher gets a citation right. Two turns later the critic misreads it and reports it wrong. The planner builds the final answer around the critic's mistake. Every single turn looks fine on its own, and the team ships a wrong answer stitched together from correct-looking parts.
Seen as three separate agents, none of these show up. Seen as handoffs, each one has an obvious home and an obvious fix.
The one that surprised me: role drift after a model update
This is the failure I would not have predicted. When I upgraded the model behind the agents, nothing in the final-answer quality moved at first. What moved was roles. The critic, on the newer and more eager-to-help model, started drafting plans instead of just critiquing.
Role drift turned out to be the earliest warning sign of a model change going sideways, well before cost or quality budged. So now I pin the model versions and re-check role behavior every time I bump them, instead of trusting that a newer model behaves like the old one. It usually does not.
The mistakes that hide all of this
- Scoring only the final answer. It misses every handoff problem that did not quite ruin the final string, which is most of them.
- One test for all the agents. Each agent has a different job. A single shared rubric blurs role-specific failures into a meaningless average.
- No record of the handoff itself. If your traces only show the final conversation, you cannot see which agent pair dropped the ball. You need the sender turn and the receiver turn side by side.
- Treating a model upgrade as a non-event. New models drift roles first and answers second. Re-run your checks on every version bump.
The lesson I keep coming back to is that in a multi-agent system, the agents were never really the hard part. The hard part is the space between them, and that is the one place I was not looking.
If you want the deeper version, with the exact rubrics for scoring each handoff and how to capture the handoff in your traces, this piece goes through it properly.
If you run multi-agent teams, I would love to hear which handoff broke on you. Mine is almost always a constraint from the planner that the next agent quietly forgot.
Top comments (0)