Cross-post of Insights #12 — canonical: https://sheikhwasim.com/insights/handoff-note-agents-fail-at-the-seams/
Every agent in the pipeline passed its own evals. The claim still came out wrong.
When you split work across agents (a planner, an extractor, a reviewer), you add something no single-agent eval covers: the handoff. Each agent does its part well, and the system still loses the one fact that mattered, because nobody owned the space between them.
Most multi-agent bugs I review aren't model bugs. They're seam bugs. The fix is boring and it works: every agent passes a Handoff Note, a small typed record of what it knows, what it was told not to do, what's still open, and where each fact came from. The next agent reads the note first, and the system checks the note at every seam.
Quick use case
Situation. A regional insurer's claims intake team runs three agents. An intake agent reads the customer's email and attachments. An extractor pulls policy number, loss date, amount, and coverage type into the claims system. A reviewer agent checks the result and either approves it for an adjuster or kicks it back. Each agent has its own eval set, and each one scores well.
What broke. A customer wrote, "the water damage started before I renewed, but the agent on the phone said the old policy covers it." The intake agent caught it and summarized the case in a free-text paragraph. The extractor only took structured fields from the attachments, so it filed the claim under the new policy's dates. The reviewer re-read the attachments from scratch, saw consistent fields, and approved. The phone-call exception never reached the adjuster, and the claim was denied on a date technicality that took two weeks and an escalation to unwind.
No agent was "wrong" on its own test. The fact died in a paragraph that the next agent never had to read.
The fix (Handoff Note).
- Typed note, not prose. Each agent writes four lists: facts, constraints, open questions, and sources.
- Read the note before the raw input. The extractor starts from the intake note, so "possible prior-policy coverage" arrives as an open question it has to resolve or carry forward.
- Open questions can't disappear. A question is either answered with a source or passed on. The reviewer can't approve while one is still open.
- Check the seam. A cheap validator compares the note going in with the note coming out and flags any fact or question that vanished.
- Log the notes with the trace. When a claim goes wrong, you can see exactly which seam dropped what.
Same agents, same models. The exception now shows up on the adjuster's screen as an open question with the customer's own words attached.
Why per-agent evals aren't enough
Per-agent evals test each agent on its own input. In production, an agent's real input is the previous agent's output, and that output drifts as prompts, models, and upstream agents change. An extractor that's 98% accurate on clean attachments tells you nothing about whether it respects a caveat the intake agent raised.
Free-text summaries make this worse. They read well to humans and pass vibe checks, but they let the next agent pick what it pays attention to. Anything without a slot is optional, and optional facts get dropped under pressure.
The Handoff Note (five layers)
1. Four slots, always
Facts (with values), constraints ("don't file under the new policy until coverage is confirmed"), open questions, and sources (document, line, message ID). Keep it short. If a slot is empty, it says so on purpose.
2. Note first, raw input second
The receiving agent's prompt starts with the note and treats it as the brief. It can still read the raw documents, but it has to reconcile anything that conflicts with the note instead of quietly overwriting it.
3. Questions are conserved
An open question leaves a seam in one of two states: answered, with a source, or still open. "Gone" isn't a state. This one rule catches most dropped-exception bugs.
4. A boring validator at every seam
Not another LLM judge, just code. Diff the incoming and outgoing notes. Missing facts, missing questions, or new facts with no source fail the handoff and route to a human or a retry. This is the Boundary Contract idea applied between agents instead of between an agent and a tool.
5. Notes live in the trace
Store every note with the run, stamped with the Release Pin. When something goes wrong, you can replay the seams instead of guessing which agent "probably" missed it, and you can turn the bad run into a Golden Trace for the whole pipeline.
How this maps to what you already have
- 5-layer agent stack. The note is the contract between Orchestration and each agent's Memory, and it's recorded in Traces.
- Golden Trace. Add pipeline-level golden runs that assert which facts and questions survive end to end.
- Shadow Path. When you swap one agent in the chain, shadow it and compare its outgoing notes, not just its final labels.
- Ship Gate. Add one check: no seam in the pipeline accepts free-text-only handoffs.
A test you can run this week
Take ten recent runs of any multi-agent flow. For each one, list every caveat, exception, or "but" in the original input. Then check whether each one is visible in the final output or explicitly resolved somewhere along the way.
If even one disappeared without a trace, you have a seam bug. Add the four-slot note at that seam before you touch a prompt or a model.
Closing
Agents don't usually fail alone. They fail at the handoff, where a fact one agent knew never became something the next agent had to deal with. Give every seam a typed note, conserve open questions, check the diff with plain code, and keep the notes in the trace. Make the handoff the part you can audit.
I'm Wasim Sheikh, AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept, production.
Follow for practical AI architecture that ships.
Connect on LinkedIn: Wasim Sheikh · Site: sheikhwasim.com · Notes: Wasim Sheikh on Substack (practicalainotes.substack.com) · X: @anciwasim
Top comments (0)