DEV Community: Alex Aslam

Agents that forget where they are will repeat work forever — checkpointing is the save-point pattern you need — Checkpointing and Resume

Alex Aslam — Fri, 02 Oct 2026 14:01:15 +0000

I watched a research pipeline die at step nine of ten. Forty-three minutes of work, sixty sources fetched, four analyses complete, one synthesis in progress. The worker crashed—OOM, not a logic bug. When it came back, the orchestrator did what I'd told it to do: it started over.

Forty-three minutes became eighty-six. Every source re-fetched. Every analysis re-run. And three of those re-runs wrote to a downstream system that didn't deduplicate, which is how I learned that "just retry the whole thing" is not a recovery strategy. It's a duplicate-generation engine with a nice error message.

That was the week I stopped thinking about agent memory and started thinking about agent position.

Memory Was Never the Problem

I'd spent months on memory. Vector stores, retrieval strategies, context compression. My agents knew a lot. What they didn't know was where they were.

Ask one of them "what have you already done?" and it would tell you about its accumulated context—sources read, conclusions drawn. Ask it "what's left?" and it had nothing. It could describe its knowledge. It couldn't describe its progress.

That distinction is the whole game. Memory answers "what do I know?" Checkpointing answers "where am I?" Most agent systems have the first and not the second, which is why a crash at step nine costs you the entire run instead of the last step.

The failure modes multiply from there. An agent that doesn't know what it already did will happily redo it. An agent that doesn't know which side effects already landed will happily re-fire them. The orchestrator retries, the agent re-reasons, and the external world absorbs the duplication. Nobody throws an error, because from every individual component's perspective, nothing went wrong.

What Checkpointing Actually Is

The pattern is decades old. Durable execution systems have been solving this since before anyone called them agents. The core idea is that a workflow's progress is recorded as a durable log, and recovery means replaying that log rather than re-executing from the start.

Temporal's model is the clearest articulation. Workflow code must be deterministic—no Date.now(), no random, no direct I/O. Activities are where side effects live. When a worker crashes and the workflow resumes on another worker, the workflow code re-executes from the beginning, but activities don't run again. Their results come from the event history. The code re-derives its position by replaying decisions that were already made.

AWS shipped the same concept into Lambda as durable execution, letting a function checkpoint at each step and resume from the last recorded point, with execution windows up to a year. DBOS does it with Postgres as the log and @DBOS.step() decorating each checkpointed unit. Restate journals every handler invocation. Resonate makes every spawn and await a durable promise, so a crash mid-batch re-runs only what was in flight.

LangGraph brings it into the agent world directly. Passing a checkpointer to compile() makes every superstep persist. The thread_id in your config is the identity of the run. A crash mid-graph, a resumed invocation with the same thread_id, and the graph picks up at the last completed node rather than at START.

checkpointer = PostgresSaver.from_conn_string(DB_URI)
graph = workflow.compile(checkpointer=checkpointer)

config = {"configurable": {"thread_id": "session-42"}}
result = graph.invoke({"query": "..."}, config=config)

# after a crash, same thread_id, no new input:
result = graph.invoke(None, config=config)

That invoke(None, ...) is the entire recovery primitive. The graph reads its last checkpoint and continues. Same thread_id, same state, same position.

LangGraph's newer durability modes let you tune how eagerly that happens. "sync" checkpoints before the next step begins—safest, slowest. "async" writes in the background—faster, with a small window where a crash can lose the last step. "exit" only persists at the end—cheapest, useless for recovery. Most production systems land on "async" and accept the small exposure.

The Trap That Bites Everyone

Checkpointing and replay have a failure mode that will absolutely find you: non-determinism.

If your workflow code produces different results on replay than it did the first time, replay diverges. Temporal enforces determinism hard—you use Workflow.now() and Workflow.random() instead of the standard library, and violations are caught by a determinism checker. LangGraph is more permissive, which means it's more dangerous. If a node does something nondeterministic and the checkpoint lands after it, replay may reconstruct a different state than the original run.

The practical rule: anything with side effects or non-determinism belongs behind the checkpoint boundary, not inside the replay path. LLM calls are the obvious case. An LLM call is non-deterministic by definition. If a resumed run re-invokes the model instead of reading the recorded result, you've lost the guarantee entirely.

This is where most agent frameworks quietly fall short. A graph node that calls an LLM and writes a summary is two things: a decision and a side effect. Only the second one should be replayable. Frameworks that checkpoint at the node level rather than the step level conflate them.

Idempotency Is the Other Half

Checkpointing tells your agent where it is. It does nothing about what already happened in the outside world.

The classic failure: the agent writes a record, then crashes before the checkpoint lands. On resume, it replays from before the write and writes again. Now you have two records and no error.

The fix is idempotency keys—derived from the operation's identity, not the timestamp. A refund is keyed by refund:{order_id}:{amount}. A notification is keyed by notify:{customer_id}:{event_id}. A write is keyed by the checkpoint sequence number of the step that produced it. The receiving system either deduplicates or the operation is naturally idempotent because it's a keyed upsert rather than an append.

Without this, checkpointing turns a single failure into a single failure plus duplicated side effects. With it, checkpointing is actually safe.

The pairing is non-negotiable. I learned this the expensive way: my pipeline had checkpointing and no idempotency, so the resume at step nine re-sent three downstream writes that had already succeeded. One of them triggered a customer email. Twice.

The Guardrails That Make It Survivable

Five failure modes matter in production.

Checkpoint bloat. Every checkpoint stores the full state. If your state includes fifty documents and the entire conversation history, you're writing megabytes per step. Storage grows, resume time grows, and eventually you're paying more for persistence than for inference. The fix is compaction—store references to large artifacts rather than the artifacts themselves, and periodically summarize accumulated context into a compact form.

Topology skew. A checkpoint written by graph version 3 may not be resumable by graph version 4. If you deployed a change between the crash and the resume, the replay can hit a node that no longer exists or an edge that now routes elsewhere. Version your graph. Refuse to resume a checkpoint whose version doesn't match, or provide explicit migration paths.

Coarse checkpoints. If you only checkpoint every ten steps, a crash costs you nine steps of re-run work. If you checkpoint every step, you pay write overhead constantly. The right granularity is per-meaningful-side-effect, not per-node.

Checkpoint everywhere, resume nowhere. I've seen teams invest heavily in persistence and never build the resume path. The checkpoint exists, the recovery code doesn't. A crash still costs the full run. Persistence without recovery is a log, not a safety net.

Unbounded retries against a poisoned state. If a checkpoint contains bad data—a malformed artifact, an impossible intermediate value—resuming will fail the same way every time. The system needs a poison-checkpoint detector and an escalation path. Otherwise the workflow loops forever, crashing and resuming in the same place.

What Production Teams Are Actually Running

Lyft's LangGraph-based support system uses thread-level state and checkpointers to manage conversation history across riders and drivers, with the router holding state and re-routing mid-chat as intent shifts. Their agent development timeline dropped from roughly six months to a few weeks, and the state management is what makes multi-turn support workflows survivable in production.

Temporal is the workhorse behind a large share of durable agent infrastructure. The event-history model means a workflow can run for months, survive repeated worker failures, and resume on any available worker without losing position. The determinism constraint is the price of admission.

LangGraph's human-in-the-loop pattern depends entirely on checkpointing. interrupt() pauses a node, the graph checkpoints, and the run stays suspended indefinitely. When a human responds, Command(resume=value) continues from exactly where it stopped. Without a checkpointer, the interrupt has nothing to resume from.

AWS Lambda durable execution extends the same model to serverless functions, checkpointing at each step and resuming across cold starts with execution windows measured in months. DBOS and Restate offer Postgres- and journal-based alternatives with the same guarantee.

When Not to Reach for Checkpointing

Checkpointing is not free, and it's not always justified.

Short, single-shot workflows. If your agent completes in under thirty seconds and a failure just means a retry, checkpointing adds infrastructure for a recovery cost you barely notice.

Workflows with no durable side effects. If everything the agent does is read-only and the output is ephemeral, there's nothing to protect against. Re-run it.

Prototypes. Adding a checkpoint store, a schema, a versioning strategy, and idempotency keys to a workflow you haven't validated is premature. Build the thing first. Checkpoint it when a crash actually costs you something.

When the state can't be serialized. Some agents hold live connections, handles, or in-process state that can't be persisted cleanly. Either the design changes to isolate serializable state, or checkpointing isn't the right tool.

The Trade-Off You're Accepting

Checkpointing buys you recovery. It costs you determinism discipline, storage, and a serialization boundary in your architecture.

You're accepting that some of your code must be pure and replayable. You're accepting that large artifacts need to live outside the state and be referenced. You're accepting that a version change to your graph is now a version change to your persistence format. And you're accepting that idempotency is not optional—if you don't build it, checkpointing will turn single failures into compound ones.

But here's what I've learned from every recovery story that went badly: the teams that suffer aren't the ones without checkpointing. They're the ones with checkpointing and no discipline about what lives inside it. The save point isn't a magic eraser. It's a contract about what your system can and cannot redo safely.

So here's my question: If your agent crashed right now, mid-task, would it know exactly what it already did—or would it start over and hope the side effects were idempotent?

I'd love to hear where you've landed. LangGraph checkpointers, Temporal's event history, a homegrown state table, or a retry-everything approach you haven't been burned by yet—and what finally made you change?

Your multi-agent system doesn’t have a memory problem — it has a shared-context problem — Shared Context Layer

Alex Aslam — Fri, 02 Oct 2026 11:03:16 +0000

I spent six weeks convinced my multi-agent system had a memory problem. I added a vector store. Then a second one. Then a knowledge graph, because the research said graphs were the future. Each addition bought me a week of calm and then the same failures crept back: agents contradicting each other, repeating work, and confidently citing facts that no one had actually established.

The turning point came when I stopped asking "how do I give my agents better memory?" and started asking "why are they each building their own version of reality?"

That's when I saw it. My agents didn't have a memory problem. They had a shared-context problem.

The Failure Nobody Names Correctly

I'd built a system where each agent accumulated its own context. The researcher remembered sources it had read. The analyst remembered conclusions it had drawn. The writer remembered the draft it had produced. Each agent's context was internally consistent and completely isolated from every other agent's.

The first time I noticed the cost, a support agent told a customer that a refund was "being processed" while the billing agent had already flagged the request as ineligible. Both agents were reasoning correctly. Neither was wrong. They were operating on different pictures of the same situation, and no one had told them to compare notes.

The research had already quantified what I was feeling. In large-scale deployments of multi-agent frameworks, the task failure rate can climb to 40–80% without coordination through shared memory. Roughly 36.9% of those failures are attributed to misalignment issues—inconsistent states or goal deviations between agents. The paper's conclusion was blunt: operating independently without a common memory repository, different agents develop conflicting understandings of the environment, leading to contradictory decisions that undermine the stability of the overall system.

I had been solving the wrong problem. Vector stores and knowledge graphs help a single agent remember more. They do nothing for the failure mode where two agents disagree because they're reasoning over different fragments of the truth.

What Shared Context Actually Means

The discipline that replaced my patchwork approach has a name now: context engineering. Google's ADK team describes it as treating context as a first-class system with its own architecture, lifecycle, and constraints. The core thesis is that context should be a compiled view over a richer stateful system—not a mutable string buffer that each agent accumulates independently.

The mental model shift is this. In a single-agent system, you ask: "What context does this agent need to answer well?" In a multi-agent system, you have to ask a harder question: "How do several agents share the same business understanding while each agent sees only the context its role requires?"

That second question is what I had never asked. I'd been optimizing retrieval per agent. I hadn't designed the shared substrate that makes those agents' retrievals compatible.

The Architecture That Fixed It

The fix has three layers. Each one solves a specific failure mode I'd been papering over with prompt engineering.

First, a shared context repository that agents read from and write to. Not a message queue. Not point-to-point handoffs. A persistent, versioned store of facts, decisions, and artifacts that every agent can query. The Atlan guide calls this the "single trusted source" layer—every agent uses the same approved definitions, metrics, and policies. When my researcher established a fact, it didn't just pass that fact forward. It wrote it to the shared layer. When my analyst needed context, it didn't ask a peer. It queried the shared layer.

Second, role-scoped views instead of a shared prompt. This is the part I got wrong the first time. I tried sharing everything with everyone. The result was context saturation and signal degradation. The fix is that each agent gets its own view of the shared context—a projection filtered by role. Atlan's framework calls these "role-scoped context views": each agent gets only the context it needs for its specific job. Tencent's Team Memory does this with visibility tiers—Private, Team, Restricted—so that a Scout agent researching a market doesn't get the code graph a Builder agent needs.

Third, governed retrieval paths. Agents don't pull context from random documents or their own private memory. They pull from approved systems through governed paths. This is what prevents one agent's hallucination from spreading to others through shared state. A fact enters the shared layer only if it came from a governed source. A claim is only reusable if it was validated before it was written.

What the Research Backs Up

The shared-context pattern isn't just a hunch. It's a research-backed architecture that shows up across three distinct lines of work.

ContextDB is an open-source unified context layer that replaces the patchwork of vector databases, session stores, and glue code with a single memory operating system. Its multi-agent memory sharing protocol includes conflict resolution and role-aware routing—exactly the layer I was missing. The evaluation reports up to 90% token savings compared to full-context baselines while maintaining production-grade latency under 100ms p95.

SE-Blackboard applies the classical blackboard architecture—a shared workspace accessible to all agents—to software engineering pipelines. The paper's key metric is "knowledge drift": the cumulative loss or distortion of technical entities as task-relevant content is paraphrased through successive agent handoffs. Their empirical result: the blackboard architecture improves information fidelity by 62% and raises resolve rates by 4 percentage points compared to message-passing architectures.

The Context-Graph Shared Memory pattern takes a different approach: storing cross-agent state as typed entity-relationship triples rather than vector chunks. The benchmark data is nuanced—graph memory beats vector RAG on multi-hop join queries (80% vs 20% accuracy) but falls behind on general long-conversation recall by roughly 25 accuracy points at ten times the cost. The pattern's own guidance is refreshingly honest: benchmark all three approaches on the queries your system actually runs before defaulting to any one of them.

What Production Teams Are Building

Tencent's Team Memory is the most complete production implementation I've found. It distinguishes itself from RAG in a way that matters: "RAG answers 'what can be found?' Team Memory also answers 'who can use it, which version is valid, and which Agent should receive it.'" The system registers four kinds of reusable assets—Chat Memory, Skills, LLM-Wiki, and Code-Graph—and equips each agent with an "Agent Loadout" of only what it needs. The repo hit number one on GitHub's TypeScript trending list within days of launch.

CrewContext takes a database-first approach. PostgreSQL is the source of truth—an append-only event log with versioned entity snapshots and causal link tables. A Policy Router evaluates events against composable rules before they enter the shared state. Neo4j is an optional projection for graph queries and lineage visualization.

memX, open-sourced by a Microsoft AutoGen contributor, treats the shared memory layer as a semantic store rather than just a coordination bus. The architecture separates two concerns: a pub/sub layer (Redis) for real-time signals like task status and inter-agent pings, and a vector store (LanceDB) for accumulated knowledge with embedding-based retrieval. The result: a memory written by the research agent can be recalled by the marketing agent with a thematic query, with no direct messaging and no schema coupling.

Google's ADK builds context engineering into the framework itself. Sessions, memory, and artifacts are the sources; flows and processors are the compiler pipeline; the working context is the compiled view shipped to the LLM for a single invocation. The design principle is separating storage from presentation so that schemas and prompt formats can evolve independently.

Blackboard-Core implements the classical blackboard pattern as a Python SDK: a centralized typed state model, a supervisor LLM that decides which worker runs next, and workers that read from and write to the shared state rather than messaging each other directly.

The Trade-Off You're Accepting

A shared context layer isn't free. It adds a dependency that every agent's reasoning now flows through, and that dependency can become a bottleneck if you don't design it carefully.

The failure mode is what one team called "agreement manufactured by copying rather than earned by observation." Once an agent's context is shared across a whole team, a wrong fact doesn't cost one person a repeated explanation. It costs the whole team. One agent records a wrong value, others copy it into shared memory, and the system displays consensus that was never real.

The fix is governance. Every write to the shared layer needs provenance—which agent, which source, which timestamp. Every read needs a scope. Conflict resolution can't be an afterthought; it has to be a first-class operation on the shared layer. And someone has to own the schema.

You're also accepting more infrastructure. A shared context layer is another system to deploy, monitor, and scale. The payoff is that agents stop duplicating each other's work, stop contradicting each other, and stop burning tokens reconstructing state that already exists elsewhere.

Where This Fits in Your Architecture

Use a shared context layer when:

Your agents routinely need facts another agent established. If you're passing raw outputs between agents and hoping meaning survives the handoff, you have a shared-context problem.
You've seen contradictory outputs from agents that were each individually correct. That's the signature of fragmented context.
You're scaling past three or four agents. The coordination overhead of ad-hoc context sharing grows faster than the agent count.

Don't use it when:

Your agents are genuinely independent. If each agent operates on its own data and produces its own output with no shared reasoning, a shared context layer is unnecessary overhead.
Your sessions are short and single-purpose. A shared context layer earns its keep over sessions with many agents and many interactions. For a single-shot pipeline, it's premature.
You haven't defined what "correct" looks like. The shared layer doesn't fix bad reasoning. It surfaces conflicts. If you can't adjudicate a conflict, sharing context will just make the contradictions more visible—which is progress, but it's uncomfortable progress.

The Question I Keep Coming Back To

If two of your agents answered the same question right now, would they agree—not because one copied the other, but because they're both reading from the same version of the truth?

I'd love to hear where you've landed. A shared blackboard, a context graph, a governed memory hub, or a patchwork that mostly works and what finally made you stop treating it as a memory problem?

Fallback models are not a safety net if quality drops — the same-bar pattern that keeps reliability without regression — Same-Bar Fallback

Alex Aslam — Thu, 01 Oct 2026 18:29:25 +0000

I once watched a fallback model save a request and destroy a product decision.

It was a support-ticket classifier. The primary model timed out. The fallback model picked up the same prompt, returned valid JSON, and the dashboard turned green. Availability restored. Everyone exhaled. Then someone noticed that a ticket which should have gone to the urgent queue was sitting in the ordinary one. The fallback had returned "priority": "high" instead of "urgent" and "requires_human_review": false instead of true. Same schema. Different product. No error anywhere.

That was the day I stopped treating fallback models as spare parts.

The Lie Your Dashboard Tells You

The standard fallback pattern is seductive in its simplicity. Primary model fails, fallback model takes over, request completes. The graph goes green, the 200s roll in, and you move on with your day.

But there are three layers of success that a green dashboard blurs together. Transport success means a model returned a response. Contract success means the response has the shape your application expects. Product success means the application did the thing the user actually needed. A fallback can pass the first two and fail the third without a single log entry.

I learned this the hard way across three separate incidents, each more expensive than the last. The first was the classifier. The second was a personality drift where a DeepSeek fallback silently erased our agent's entire tone and behavioral rules for 79,000 tokens—40% of a session—before a human noticed "this doesn't feel like Joe anymore". The third was a schema integrity collapse where a fallback model received a payload formatted for a different engine and returned structurally broken JSON that the downstream validator couldn't detect. The pipeline reported 100% completion. The data was useless.

The common thread wasn't model quality. It was that I had never defined what "acceptable" meant for a fallback in the first place.

What the Research Already Knew

The 2026 literature had already named my problem. A Google Developers Blog post on the strongest AI Agents Challenge submissions identified same-bar fallback validation as one of four essential engineering patterns. The warning was explicit: ensuring fallback models meet the same validation standards as primary paths is not optional.

The model cascade research made the mechanism clear. In a within-task cascade, the escalation gate—not the ladder—decides which cheap answers ship as final. Everything the gate accepts goes unreviewed, so its false-accept rate caps the cascade's quality. A gate that only checks output shape will pass wrong answers straight through. I had been building gates that checked shape and calling it validation.

The routing-policy evaluation literature added the failure mode I hadn't named: fallback rot. The primary has been stable for ten months. The fallback chain has drifted. The first time it fires under real load is the first time anyone learns it's broken. I had a fallback model I'd never tested under production conditions because the primary had never failed.

The Same-Bar Pattern

The fix isn't complicated to state. Every fallback model must clear the same quality bar as the primary model on the actual production tasks, or it doesn't belong in the chain.

The implementation has three layers.

First, define the bar explicitly. Not "does it return JSON" but "does it classify the ticket the way the primary model would." The bar has to be measured on your real task distribution, not a generic benchmark. A fallback that scores 92% on a public eval and 61% on your specific extraction task is a liability, not an asset.

Second, test the fallback as a production path, not a spare tire. The model swap deserves the same regression testing as a prompt change. Hold the prompt, tools, cases, judge, and inference parameters still, and vary only the model. If your fallback can't hit a reasonable percentage of your primary model's score on your actual production tasks, it's not ready to serve your users.

Third, gate the fallback with the same validation you'd apply to the primary. A schema check is necessary but nowhere near sufficient. A response can conform perfectly to a schema and still classify incorrectly, choose the wrong tool, overstate confidence, or take a tone that doesn't belong in your product. The gate has to reject semantically wrong output, not just malformed output.

What This Actually Costs

The honest trade-off: same-bar fallback is slower to implement and more expensive to operate than "route to whatever model is available."

You're accepting that some fallback models won't make the cut. You're accepting that your fallback chain will be shorter than it could be. You're accepting that during a provider outage, your system might return an explicit NotAnswered rather than a plausible-but-wrong answer.

The research on model cascades gives the concrete math. The escalation rate has to stay below 1 − (cheap cost ÷ flagship cost) for the cascade to pay off. If your fallback model fails 80% of the time, the fallback chain costs more than going straight to the primary—and the failures ship silently.

But here's what I've learned from watching a single fallback model corrupt a support queue without anyone noticing for three days: the cost of a plausible wrong answer is always higher than the cost of an explicit failure. A crash forces a fix. Silent degradation slips into your database and compounds.

What This Looks Like in Production

OpenRouter's Jev-verified cascade runs a cheap model draft, verifies it against retrieved context, and escalates to the frontier only when the check fails. On a 50-question benchmark, the cascade shipped the same zero wrong answers as running the frontier model on every question, at about 7% of the cost.

Shopify's LLM proxy gives every engineer access to multiple providers with automatic failover. When Claude Fable 5 shut down, the proxy shifted traffic to Claude Opus or GPT 5.5 automatically—but the fallback models were pre-validated to meet the same quality bar, not just to return a response.

The SentinelAgent project demonstrates a three-tier fallback chain—Claude → OpenAI → Local BiLSTM—with graceful degradation at each tier. The local model has 4.9M parameters and 77.31% accuracy. It's not as good as Claude. But it's tested against the same task distribution, and the system knows exactly what quality it's delivering at each tier.

The Question I Keep Coming Back To

If your fallback model answered a production request right now, could you prove its answer was as good as the primary model's—or would you find out three days later when someone noticed the wrong ticket in the wrong queue?

I'd love to hear where you've landed. Same-bar validation on every fallback, a tested chain you trust under load, or a green dashboard you haven't stress-tested yet—and what finally made you change?

Why your agent bill explodes before the model even thinks — tiered routing and the cheap-check-first fix — Tiered Routing

Alex Aslam — Thu, 01 Oct 2026 14:55:11 +0000

I once watched a single agent workflow burn through $340 in a single afternoon. Not because it failed—because it succeeded. Twenty-three steps, every one of them routed to a frontier model, and nineteen of those steps were formatting JSON, extracting dates, and classifying support tickets. I was paying Cadillac prices for a job a bicycle could do.

That was the day I stopped blaming the model and started looking at the router.

The Bill Arrives Before the Model Thinks

The math behind agent cost explosion is deceptively simple. A frontier model like Claude Opus runs 5 to 25 times the per-token cost of a small model like Claude Haiku. On a single call, that's a rounding error. On a seven-step workflow where five frontier calls chain together, you're running an order of magnitude past the equivalent small-model pipeline.

The research confirms what my bill already knew. Enterprise agentic systems that route every trajectory step to a frontier model waste 60–80% of their inference budget on subtasks that smaller models handle equally well. In enterprise deployments spanning document processing, compliance review, and customer interaction, 55–70% of agent trajectory steps require no frontier-model capability at all.

The kicker is which steps cost you. A 2026 study found that only 14.2% of steps genuinely require frontier models, yet those steps consume 48.3% of the total cost. The other 85.8% of steps—the formatting, the extraction, the classification—are burning frontier tokens for work that a 7B model does identically.

I had been solving the wrong problem. I kept optimizing prompts for a model that shouldn't have been handling the task in the first place.

The Insight That Changed the Architecture

A colleague looked at my trace and asked a question that reframed everything: "Why is your summarization step calling the same model as your planning step?"

The answer was embarrassing. I had one model configured. Every step used it because that was the default. I'd never asked whether each step needed it.

That's when I understood that model choice is a per-step decision, not a per-agent decision. A planning step genuinely requires frontier-class reasoning. The formatting step that follows it needs a 7B model and nothing more. Treating them the same is the architectural equivalent of hiring a senior architect to file paperwork.

Tiered Routing: Cheap Check First, Escalate on Failure

The pattern that fixed this is called a model cascade—or within-task model cascade—and it's embarrassingly simple: run the step on a cheap model first, and escalate to the flagship only when a gate rejects the output.

FrugalGPT introduced the technique and demonstrated up to 98% cost reduction while matching or surpassing GPT-4 accuracy on several benchmarks. The Stanford researchers' insight was that most queries don't need the expensive model, and the expensive model only sees the ones that do.

But the naive cascade has a trap. The gate that decides whether to escalate—that's the load-bearing piece. A gate that only checks output shape will pass wrong answers straight through, because a malformed JSON is easy to catch but a confidently wrong extraction looks exactly like a correct one.

The three conditions for a cascade to pay off:

The gate rejects semantically wrong output, not just malformed output. Everything the gate accepts ships unreviewed. If the gate can't tell "Q3 2024" from "Q3 2025," it's not a gate. It's a rubber stamp.

The escalation rate stays below break-even. You pay the cheap attempt on every item, so the cascade beats always-flagship only while the escalation rate stays under 1 − (cheap cost ÷ flagship cost). If your cheap model fails 80% of the time, the cascade costs more than going straight to the flagship.

The cheap rung isn't slower than the flagship, or latency doesn't matter. A 4B local model that takes 12 seconds to produce a draft isn't saving you anything if the flagship finishes in 3.

What the Research Actually Quantifies

The 2026 literature has moved past theory. The numbers are concrete.

AgentRouter, a 12M-parameter classifier with under 5ms overhead per step, maps each trajectory step to one of four model tiers using five features extractable at routing time. Trained on 50,000 annotated agent trajectory steps, it achieves 72% cost reduction relative to frontier-only baselines while retaining 97.3% of frontier-only quality—less than 3% degradation in end-to-end task completion.

The per-step routing accuracy is telling: 91% on minimal-complexity steps, 85% on efficient-tier steps, and 76–82% on the harder mid-range and frontier tiers. The router is better at identifying cheap work than expensive work, which is exactly the right failure mode.

FrugalGPT (adapted to four tiers) achieves 44.1% cost reduction with 96.8% quality preserved, but with 48ms routing overhead from sequential model probing. RouteLLM performs worst in cost reduction (31.4%) because its preference-based router, trained on single-turn conversations, misroutes agent steps whose complexity depends on accumulated trajectory context.

Planner-as-Router takes a different approach: instead of a separate router model, the planner assigns each subtask a model tier as it decomposes the query. It cuts cost 44% against all-frontier routing while giving up 2.9 points of accuracy, matching a faithful FrugalGPT cascade at lower cost.

The consistent finding across all of these: step-level routing beats query-level routing in agentic workflows because subtask complexity varies widely within a single trajectory.

The Cost Paradox Nobody Warns You About

Here's the part that took me three weeks to understand and two months to fix.

If you implement routing incorrectly in a multi-turn agent, you can end up paying more than if you'd never routed at all.

The reason is prompt caching. By Turn 3 of an agent session, cached tokens make up the vast majority of your token payload. Anthropic offers up to a 90% discount on cached input tokens. That's the efficiency you're protecting.

Switching models mid-session destroys that protection entirely. Every provider's cache is model-specific. The new model has no access to the previous model's stored history and must re-read the entire conversation from scratch at full input token cost. For agentic coding workflows where context windows stretch to 50,000 tokens, a single mid-session model switch can spike costs enough to eliminate all the savings routing was supposed to deliver.

The mistake I made was routing across a session. The fix is routing within a step—choose the model for each generation, complete it, and return to the session model. Or better: summarize the context before switching, so the new model starts with a compressed history rather than a cold cache.

Who's Actually Shipping This

OpenRouter's Jev-verified cascade runs a cheap model draft, verifies it against retrieved context using Jev (a verification layer), and escalates to the frontier only when the check fails. On a 50-question benchmark, the cascade shipped the same zero wrong answers as running the frontier model on every question, at about 7% of the cost.

Cisco's CAIPE uses tiered routing across Argo CD, Kubernetes, and Komodor. Response times dropped from hours to seconds, and MTTR reduced by up to 80%.

Fireworks reports 60–80% cost reduction and a 10–40× increase in inference speed with their multi-tier inference architecture for agentic workflows.

The koi model-router package implements cascade escalation with a complexity classifier, confidence evaluators, circuit breakers, and per-tier cost tracking. It routes 60–80% of requests to cheap models with no quality loss.

The pattern is consistent. The teams winning on cost aren't using one model. They're using a ladder—and they've built the gate before they built the ladder.

Where This Fits in Your Architecture

Tiered routing is not a default. It's a response to a specific problem.

Use it when your agent trajectory has step-level complexity variance. Planning and synthesis steps need frontier reasoning. Extraction, formatting, and classification steps don't. If every step genuinely requires frontier capability, routing won't help.

Use it when your token volume is high enough that the savings justify the infrastructure. A router that saves 40% on a $50 monthly bill isn't worth building. A router that saves 40% on a $50,000 monthly bill is.

Use it with step-level features, not query-level features. The research is clear that single-turn routers misroute agent steps because they miss trajectory context. The features that matter are the ones available at the step: the instruction, the accumulated context length, the prior step's tier, and the dependency structure.

Don't route across sessions without summarization. The cache penalty will eat your savings. Route within a step, or summarize before you switch.

The Trade-Off You're Accepting

Tiered routing buys you cost reduction and, often, latency. It costs you determinism and adds a classifier to your stack.

Every routing decision is a chance to misroute. A complex step sent to a cheap model fails, and you pay the cheap attempt plus the escalation plus the latency of both. The gate's false-accept rate—not the ladder—decides whether the cascade pays. If your gate accepts wrong answers, you ship them at a discount, and the savings are a lie.

And you're accepting more moving parts. A router, a gate, a tier configuration, per-tier cost tracking. Every one of those is a thing that can fail independently. The koi model-router's failure modes are instructive: low-confidence answers escalate automatically, provider outages trip circuit breakers, budget exhaustion stops escalation and returns the best available response. None of those are free.

But here's what I've learned from watching my bill drop by 68% without a measurable quality regression: the teams that are winning with agents aren't using the best model for everything. They're using the right model for each step—and they've accepted that "right" is a decision worth making explicitly.

So here's my question: If you looked at your agent's cost breakdown by step, how many of those steps would you have paid frontier prices for if you'd chosen the model deliberately?

I'd love to hear where you've landed. FrugalGPT-style cascade, a trained router, a planner that assigns tiers up front, or a bill that finally made you look—and what finally made you change?

Sequential agent chains are a latency tax — the event-driven pattern that makes independent agents run in parallel — Event-Driven Concurrency

Alex Aslam — Wed, 30 Sep 2026 20:58:23 +0000

I once watched a nine-minute agent workflow spend seven minutes and forty seconds doing nothing. Not failing. Not retrying. Just waiting each agent idle while the previous one finished a task that had no dependency on it whatsoever.

I'd built the pipeline as a chain. Researcher, then analyst, then writer, then fact-checker, then formatter. Each step consumed the previous step's output. In my head, that was the workflow. In production, it was a latency tax I'd designed myself and never noticed.

The fix wasn't a better model or a faster prompt. It was admitting that most of those steps were never actually dependent on each other.

The Tax Nobody Itemizes

Sequential chains are the easiest multi-agent pattern to reason about, which is why they're the default. Each stage has a clear input, a clear output, and a clear owner. Debugging is trivial because there's only one path. Microsoft's sequential orchestration documentation describes it exactly this way: a predefined linear order where each agent processes the previous agent's output, creating a pipeline of specialized transformations.

The problem is that "clear" and "correct" are not the same thing. A pipeline implies dependency. Most real workflows don't have the dependency you assumed.

My researcher fetched six independent sources. Those six fetches could have run in parallel. My analyst produced three independent analyses—market, risk, and competitive positioning—with no shared state between them. My fact-checker verified claims that were already independently verifiable. Only the writer genuinely needed everything upstream.

The math is unforgiving. If you have N stages with average latency L, your wall-clock time is N × L. If K of those stages are independent, the floor is (N − K) × L + L, not N × L. In my pipeline, K was 4 out of 5. I was paying nearly five times the necessary latency and calling it "the workflow."

Amdahl's law applies here in a way that's almost embarrassing once you see it. The speedup from parallelizing independent work is bounded by the serial fraction—and in most agent pipelines I've audited, the serial fraction is far smaller than the architecture assumes.

The Structural Mistake

The chain wasn't wrong because I used the wrong framework. It was wrong because I encoded dependencies that didn't exist.

A chain says: B cannot begin until A completes. That's a claim about the world. When it's true, a chain is correct and efficient. When it's false, you've serialized work that should have been concurrent, and you'll never notice because the system still produces correct output—just slowly.

I'd assumed the analyst needed the researcher's entire output before starting. In reality, the analyst needed three fields. The researcher was still fetching sources four and five while the analyst could have started on source one.

The dependency was real at the level of final output. It was fake at the level of individual facts.

Event-Driven Concurrency: React to Data, Not to Sequence

The pattern that fixed this wasn't a new framework. It was a change in what triggers work.

In a sequential chain, work starts because the previous step finished. In an event-driven architecture, work starts because the data it needs has arrived. Agents subscribe to events—a source fetched, a claim extracted, a document chunked—and fire when the specific event they care about publishes.

The shift has three architectural consequences.

First, agents become independent consumers. Each agent declares what it needs and what it produces. The orchestrator doesn't route tasks; it publishes events. Agents that can act, act. Agents that need more input, wait. No agent is blocked by an unrelated stage.

Second, latency becomes the critical path, not the sum. If six fetches run concurrently and three analyses run as soon as their inputs land, wall-clock time collapses to the longest dependency chain plus fan-in. The parallelism isn't a fan-out you designed—it's an emergent property of the dependency graph.

Third, backpressure becomes visible. In a chain, a slow stage just makes everything slow. In an event-driven system, you can see which topics are backing up, which consumers are lagging, and where the actual bottleneck lives. That observability is worth as much as the latency win.

This is the same shift microservices made a decade ago, and the same discipline applies: publish events, not commands. Let consumers decide what to do with them. Use correlation IDs to trace a logical request across many asynchronous hops. Use idempotency keys so a retried consumer doesn't double-write.

The Orchestration Trap in Event-Driven Systems

The failure mode of event-driven agent systems is the opposite of the chain's. Instead of being too slow, it becomes too opaque.

Choreography—where agents react to each other's events with no central coordinator—scales beautifully and debugs terribly. When a workflow fails, you have no single place to look. The failure is in the emergent behavior of a dozen independent consumers, none of whom saw the whole picture.

The pragmatic middle ground that has held up in production is event-driven execution under a thin orchestration layer. A coordinator doesn't route individual tasks; it publishes a workflow-started event and then subscribes to completion events, tracking state and enforcing timeouts. The agents still run independently. The coordinator just knows what "done" means and can surface when it isn't.

This is where durable execution frameworks earn their keep. Temporal's model—workflows as deterministic code, activities as the side-effecting work—maps cleanly onto agent systems. The workflow defines the dependency graph. The activities run concurrently where the graph allows. The coordinator state is durable, so a crash at step seven doesn't lose the run.

LangGraph's parallel node execution and conditional edges offer the same property within a graph, and the newer event-driven orchestration patterns—where nodes publish and subscribe to channels rather than passing state directly—are the closest thing to true event-driven concurrency in the agent frameworks I've used.

The Guardrails That Make It Survivable

Event-driven concurrency introduces failure modes that sequential chains never had. Four of them matter.

Ordering. Events can arrive out of order. If an analysis event arrives before the source event it depends on, the analysis either fails or runs on stale data. The fix is either partition keys that preserve order for related events, or a consumer that buffers until dependencies are satisfied. In agent systems, the cleanest answer is usually to make each event self-contained enough that order doesn't matter—carry the data, not just a pointer to it.

Duplicates. At-least-once delivery is the realistic default. Every consumer must be idempotent. An agent that writes a summary twice is fine. An agent that charges a customer twice is not. Idempotency keys derived from the event ID, not the timestamp, are the standard fix.

Poison events. One malformed event can crash a consumer repeatedly, blocking the queue behind it. Dead letter queues aren't optional. Neither is a circuit breaker that stops consuming from a topic after repeated failures and alerts instead of spinning.

Fan-in reconciliation. When three analyses complete asynchronously, something has to decide when all three are ready and what to do with their outputs. That's the same fan-in problem from the concurrency pattern, with the same rules: scoped branch state, structured claims, and an explicit merge that surfaces conflicts rather than smoothing them over.

None of this is exotic. It's the standard distributed systems toolkit. The mistake is assuming agent systems are exempt from it.

Who's Actually Shipping This

Cisco's CAIPE runs multi-agent orchestration across Argo CD, Kubernetes, and Komodor, where agents chain across tools and respond to cluster events rather than polling in sequence. Their reported outcome: response times dropped from hours to seconds, with MTTR reduced by up to 80%.

SAP's event-driven multi-agent orchestration was evaluated across 208 production-derived enterprise scenarios at Persona, Department, and Enterprise scale. Their finding is the one that should be posted above every pipeline design review: scale, not task complexity, dominates orchestration performance. Both sequential and reactive architectures degraded at enterprise scale, and the Task Manager they introduced to handle priority inference, related-event merging, and preemption cut high-priority queue latency by 14–75% and improved related-event correctness by more than 20 percentage points.

Resonate's durable execution model treats every spawn and every await as a durable promise. Fan-out branches run concurrently, each checkpointed independently. A crash mid-batch re-runs only what was in flight. That's the property that makes event-driven agent systems survivable rather than merely fast.

AWS's Expansion-Contraction pattern, published at ACM CAIS 2026, walks a domain graph with concurrent path analysis. Their reported results: 98.2% accuracy on a production supply chain, 100% on public benchmarks, concurrent path analysis yielding up to 1.43× speedup, and investigation caching reducing token usage by up to 93.9%. The concurrency is the point, but the caching is what makes it affordable.

When Not to Reach for Events

Event-driven concurrency is not a default. It's a response to a specific problem.

Stay sequential when the dependency is real. If stage B genuinely needs the complete output of stage A, a chain is correct and simpler. Don't parallelize what isn't parallel.

Stay sequential when latency isn't the binding constraint. If your pipeline finishes in six seconds and users don't care, you don't have a latency problem. You have an architecture preference, and event-driven systems cost more to build and operate.

Stay sequential when debugging is the scarce resource. A chain gives you a single path and a linear trace. An event-driven system gives you a correlation ID and a scatter plot. If your team is small and the workflow is new, that trade is often not worth making yet.

The honest rule: start sequential, measure, and parallelize only the stages that measurement proves are independent and expensive. The teams I've seen get this right didn't start with events. They started with a chain, watched it in production, and moved to events when the latency numbers justified the operational cost.

The Trade-Off You're Accepting

Event-driven concurrency buys you latency and throughput. It costs you determinism and traceability.

A sequential chain has one path. An event-driven workflow has as many paths as there are interleavings, and no two runs are identical. Reproducing a bug means replaying an event stream, not re-running a function. Your observability has to be built for it—distributed tracing, correlation IDs, structured logs that tie back to a single logical request.

And you're accepting more moving parts. A chain has N stages. An event-driven system has N consumers, a broker, a schema registry, dead letter queues, and a coordinator. Every one of those is a thing that can fail independently.

But here's what I've learned from every pipeline post-mortem I've sat through: the latency was rarely the thing that killed the project. It was the assumption that a chain was the only honest representation of the work. Once I drew the actual dependency graph instead of the one I'd assumed, most of the sequence disappeared—and so did most of the delay.

So here's my question: If you drew the real dependency graph for your agent pipeline, how many of those sequential steps would survive?

I'd love to hear where you've landed. Sequential by choice, event-driven after a painful migration, or somewhere in between and what finally made you look at the graph instead of the chain?

An agent should be both client and server — the bidirectional MCP pattern that turns reasoning into infrastructure — Bidirectional MCP

Alex Aslam — Tue, 29 Sep 2026 12:49:56 +0000

I built a refund agent that could fetch policy documents, check account history, and calculate eligibility. It could not, however, ask a human a single question without me tearing down the entire architecture. That's the moment I understood that most MCP servers are deaf, and most agents are talking to themselves.

The Server That Could Only Listen

I spent a month building a customer support system with a handful of MCP servers. One wrapped our policy database. One exposed account management APIs. One connected to the ticketing system. Each one was a clean, well-typed, compliant MCP server. Each one was completely one-directional.

The problem wasn't tool calling. The tools worked. The refund agent could fetch a policy, query an account, calculate a credit. What it couldn't do was hit an edge case—say, a refund outside the standard window—and ask a human for approval mid-operation. The MCP server had no way to reach back to the client. The client could only ask; the server could only answer.

I kept looking for a workaround. Could I encode the question in the tool result? Could I use a separate API? Could I poll? Every alternative was a workaround, not an answer. The MCP protocol I was using was designed as a one-way street: agents call tools, tools respond, agents reason. Nothing flows the other way.

That's the ceiling most teams hit and don't name. Your agent can use tools, but your tools can't collaborate with your agent.

The Reverse Primitives Nobody Explained to Me

The MCP specification had a name for what I was missing: reverse primitives. Server-initiated requests that let an MCP server reach back to the client mid-operation. Three of them, originally: Sampling, Roots, and Elicitation.

They weren't in the tutorials I read. They weren't in most of the server examples. They were in the specification, described as "client features"—capabilities the client offers to the server, not capabilities the server exposes to the agent.

Sampling lets a server request an LLM completion through the client. The server doesn't need its own API keys, doesn't pick its own model, doesn't manage its own costs. It sends a sampling/createMessage request to the client, and the client—with user approval—runs the completion and returns the result. This is the piece that turns a tool server into something that can reason mid-task. Instead of returning raw data and hoping the agent interprets it correctly, the server can ask the client's LLM to evaluate, summarize, or decide.

Elicitation lets a server ask a user for input during an interaction. A structured form, a URL for out-of-band flows, a schema the client renders. This is the piece that solves the refund approval problem. The server hits an edge case, sends an elicitation/create request, the client surfaces a form to the human, and the server receives the answer and continues.

Roots lets a client specify which directories the server should operate in—a scope boundary, a least-privilege constraint. (I'll come back to this one, because the spec's relationship with it changed in ways that matter.)

The moment I understood these primitives, the architecture changed. My MCP server wasn't a passive tool endpoint anymore. It was a participant in a conversation. It could ask for clarification. It could borrow the client's reasoning. It could pause mid-operation and wait for a human.

What Bidirectional Actually Looks Like

The flow is straightforward once you see it. A client sends a tools/call request to the MCP server. The server starts processing and realizes it needs something—user input, an LLM judgment, a scope decision. Instead of failing or returning incomplete data, the server sends a server-to-client request back through the same connection.

The client handles the request—surfaces a form, runs a sampling completion, checks the root boundary—and responds. The server receives the response and continues processing the original tool call. The client gets the final result.

The AWS Bedrock AgentCore Runtime team described it plainly in their stateful MCP client capability announcement: these capabilities "transform one-way tool execution into bidirectional conversations between your MCP server and clients". Before stateful mode, the runtime couldn't handle this. Every HTTP request was independent. The server couldn't maintain a conversation thread, couldn't ask the user for clarification mid-tool-call, couldn't request LLM-generated content. Stateful mode provisions a dedicated microVM per session, maintains continuity through a session ID, and unlocks all three client capabilities.

The Go SDK's in-process sampling example shows what the code looks like on the server side. You call mcpServer.EnableSampling() to advertise the capability. Your tool handler calls mcpServer.RequestSampling() when it needs an LLM completion. The sampling request is handled directly by the client's sampling handler, and the response flows back into your tool handler. The example output is almost boring in how ordinary it looks: a tool result that contains an LLM-generated answer to a question the server couldn't answer itself.

The TypeScript SDK handles the client side with setRequestHandler. You register a handler for sampling/createMessage, and when the server sends a sampling request, your handler receives the messages, runs them through your LLM, and returns the result. The client stays in control of model selection, permissions, and user approval.

The Shift That Made Me Rebuild

I rebuilt the refund agent around two changes.

First, the server became a participant, not a lookup table. When the refund tool detects an edge case—amount outside policy, missing documentation, a policy version conflict—it doesn't return an error or a partial result. It sends an elicitation request. The client surfaces a form to the support representative. The representative fills it out. The server receives the answer and completes the refund calculation. The tool call doesn't fail. It just asks a question first.

Second, the server started borrowing reasoning. For complex cases—disputed transactions, ambiguous policy language, multi-party refunds—the server sends a sampling request to the client. The client's LLM evaluates the situation and returns a structured judgment. The server uses that judgment to decide which policy path to follow. The server doesn't need its own model. It doesn't need its own API keys. It just needs the client to lend its brain for a moment.

The architectural shift is subtle but consequential. The server isn't a passive endpoint anymore. It's a reasoning participant. The tool call isn't a request-response transaction anymore. It's a conversation that can pause, ask, and continue.

The Protocol Moved Under My Feet

Just as I was settling into this architecture, the MCP specification changed. The 2026-07-28 revision—the fifth spec release, and the largest change since launch—made three moves that fundamentally reshaped the bidirectional story.

Roots, Sampling, and Logging were deprecated. SEP-2577 formally deprecated all three. The motivation was adoption data and complexity. Sampling, despite being available since November 2024, had low client adoption. It was complex to implement correctly—human-in-the-loop approval, model selection logic, security considerations, tool loop support. Direct LLM provider APIs gave servers more control. Roots had vague semantics, low adoption, and overlapping alternatives. Logging had mature alternatives in stderr and OpenTelemetry.

The deprecation doesn't remove them immediately. They remain functional during a twelve-month window. But the direction is clear: sampling and roots are legacy.

The protocol core became stateless. The initialize handshake and session identifier were removed. Every request is now self-contained, carrying protocol version, client information, and capabilities in _meta. Servers can deploy on serverless and edge infrastructure. Any request can be routed to any server instance behind a round-robin load balancer. No sticky sessions, no shared session store.

Multi Round-Trip Requests replaced server-initiated requests. This is the replacement for the old reverse-primitive pattern. Instead of holding an SSE stream open and sending a server-to-client request, the server returns an InputRequiredResult object containing the questions and a requestState blob. The client gathers answers and reissues the original call with the responses and echoed state. Because everything the server needs is in the payload, any server instance can pick up the retry.

The MRTR pattern is the stateless equivalent of bidirectional. It's not a persistent conversation anymore. It's a request that can loop back with additional information. The server asks, the client answers, the original request is retried with the answers attached. The server can ask multiple questions in one round trip. It can request elicitation, sampling, or root listing. But it does so through a stateless retry pattern, not a persistent session.

The 4sysops writeup described the trade-off cleanly: "In previous versions, this required holding a Server-Sent Events stream open. The new revision replaces that with Multi Round-Trip Requests". The stream is gone. The conversation is now a series of self-contained requests that carry their own state.

What This Means for Your Architecture

The bidirectional MCP pattern is still real. It's just changed shape.

If you're on a pre-2026-07-28 spec, you have the full reverse-primitive toolkit: sampling for borrowing client reasoning, elicitation for user input, roots for scope boundaries. Your server can be a reasoning participant. Your tool calls can pause and ask questions. Stateful sessions are maintained through the Mcp-Session-Id header, and the client handles server-initiated requests through registered handlers.

If you're on the 2026-07-28 spec and later, the architecture shifts. Sampling and roots are deprecated. Elicitation survives, but it's delivered through MRTR instead of a persistent session. The server returns an InputRequiredResult with elicitation requests, the client gathers the answers, and the original tool call is retried with the responses attached. The server can still ask the user for input mid-operation. It just does so through a stateless retry instead of a live back-channel.

The practical impact is smaller than it sounds for most use cases. If your server was using sampling to borrow the client's LLM, you now integrate directly with an LLM provider API. You get more control over model selection and parameters. You lose the client's permission management and user approval flow. If your server was using roots for scope boundaries, you now pass paths through tool parameters, resource URIs, or configuration. More explicit, less magical.

If your server was using elicitation, you're fine. Elicitation survives. The delivery mechanism changed. The capability didn't.

Who's Actually Running This

Gong turned MCP into a bidirectional protocol for revenue intelligence. Their architecture exposes an MCP Gateway for inbound agent queries—external agents like Microsoft Copilot, HubSpot AI, and Salesforce can query Gong's data—and an MCP Server for outbound context enrichment, where Gong's own AI features pull in data from external tools. Write-back to Salesforce is gated by Gong-side logic. The protocol isn't just a data source anymore. It's a two-way integration layer.

Amazon Bedrock AgentCore Runtime now supports stateful MCP servers with all three client capabilities. The runtime provisions a dedicated microVM per session, maintains continuity through a session ID, and enables interactive, multi-turn workflows. The AWS team's framing is the clearest I've read: these capabilities "transform one-way tool execution into bidirectional conversations between your MCP server and clients".

The mcp-data-platform project takes a different approach to bidirectional. It's a semantic data platform MCP server that composes multiple data tools with "bidirectional cross-injection"—tool responses automatically include critical context from other services. Trino query results are enriched with DataHub metadata. DataHub searches include query availability from Trino. S3 listings include matching DataHub datasets. The server isn't just answering the question it was asked. It's injecting context the client didn't know to request.

The Pydantic AI harness has an open proposal to let agents expose themselves as MCP servers. The idea is bidirectional MCP at the agent level: agents can both consume MCP servers and serve as MCP servers. A specialized database agent built with Pydantic AI becomes an MCP tool for a general coding agent. A code review agent is invoked by a CI/CD pipeline agent as an MCP tool. Agents compose through MCP instead of through framework-specific integrations.

The Trade-Off You're Accepting

Bidirectional MCP gives you conversational tools. It costs you simplicity.

Every server-initiated request is a decision point. The client has to handle it—surface a form, run a completion, check a root boundary. The server has to handle the response and decide whether to continue or ask another question. The flow is no longer a straight line. It's a loop that can iterate as many times as the server needs.

The deprecation of sampling and roots is a signal about what the protocol designers think is worth the complexity. Elicitation survived because it's the one that solves a problem no alternative solves cleanly: getting structured user input mid-operation. Sampling and roots had alternatives—direct LLM APIs, explicit path parameters—that gave more control with less protocol surface.

The MRTR pattern is the compromise. It keeps the stateless core while preserving the ability for a server to ask for input mid-operation. It doesn't preserve the persistent conversation. It replaces it with a retry loop that carries its own state.

Here's what I've learned: the teams that are getting bidirectional MCP right aren't using it for everything. They're using it for the moments where a tool genuinely needs to ask a question, borrow a judgment, or pause for input. Everything else stays a simple, one-way tool call. The bidirectional capabilities are exceptions, not the default. They're the edge case handling that makes the happy path possible.

So here's my question: When your MCP server hits an edge case, does it fail silently, return a partial result, or ask for help—and does your architecture let it ask?

I'd love to hear where you've landed. Full reverse primitives, MRTR elicitation, or a workaround you built because the spec didn't have what you needed—and what finally made you change?

Your agents can’t collaborate if they only speak tool calls — the A2A protocol that fixes peer handoffs — A2A Under the Linux Foundation

Alex Aslam — Mon, 28 Sep 2026 22:20:23 +0000

I spent three weeks trying to make a billing agent ask a refund agent a question. Not a tool call—an actual question. The billing agent needed to know whether a refund was appropriate, and the refund agent needed to reason about policy, check account history, and possibly refuse. I wired it up as a function call anyway. The billing agent waited fourteen seconds for a "tool" that was actually trying to think.

That was the day I stopped treating agents like functions.

The Tool-Call Trap

The mistake is almost inevitable. You build your first agents, you wire them together with tool calls, and it works. The tool returns a value. The caller proceeds. Simple, synchronous, easy to debug.

Then one of your "tools" grows a brain. It needs to ask clarifying questions. It needs to run for six minutes. It needs to decide the request is out of scope and say no. But your architecture treats it as a passive function, so every one of those behaviors looks like a failure—a timeout, a malformed response, an unhandled edge case.

I kept tuning prompts. The model wasn't the problem. The architecture was. I had confused the vertical with the horizontal.

What Tool Calls Can't Do

The Model Context Protocol (MCP) solved a genuinely hard problem. Before it, connecting five agents to ten tools meant up to fifty bespoke integrations, each one a maintenance liability. MCP collapses that into one interface. By mid-2025, the community had built thousands of active MCP servers, and OpenAI, Microsoft, and Google DeepMind all adopted it. MCP won the tool-calling layer, and it deserved to.

But MCP has no concept of a peer agent with its own goals, its own model, and its own authority to act. An MCP server that wraps kubectl does not think, negotiate, or push back. It exposes capabilities and waits. The agent stays in charge; the things it touches are passive.

When my billing agent needed the refund agent to decide something—not fetch a value, not execute a command, but reason through policy and arrive at a judgment—that was never an MCP-shaped problem. It was a horizontal problem. And horizontal problems need a different protocol.

The A2A Protocol: HTTP for Agents

The Agent2Agent (A2A) protocol is an open standard launched by Google in April 2025 and now governed by the Linux Foundation's Agentic AI Foundation. Where MCP is the vertical integration layer connecting agents to internal tools and databases, A2A acts as the horizontal protocol enabling peer-to-peer collaboration. The A2A specification puts the distinction in one sentence: MCP standardizes how an agent uses a tool or resource; A2A standardizes how one agent delegates work to another.

Think of A2A as HTTP for AI agents. Just as HTTP enables any web browser to communicate with any web server regardless of their underlying technologies, A2A enables a LangGraph agent to delegate work to a CrewAI agent, which can then call a Google ADK agent, without requiring custom integration code for each pairing.

The architectural shift matters because it changes what the calling agent expects. When you call an API, it simply returns data or fails. When an agent calls an A2A peer, it initiates a collaboration. The receiving agent can understand intent, refine the plan, push back on incomplete requests, and ask clarifying questions if something is off. That's not a tool call. That's a handoff between peers.

Under the Hood: Agent Cards and Task Lifecycle

A2A standardizes four objects: an Agent Card, a Task, a Message, and an Artifact.

The Agent Card is a JSON manifest served at a well-known URI, describing who an agent is, what it can do, where to reach it, and how to authenticate. This is the discovery mechanism. Your agent doesn't need to know about the refund agent at build time. It queries the registry, reads the card, and decides whether to delegate.

The Task is the unit of work, with an ID, a context ID, a state, a history, and artifacts. Tasks are stateful and progress through a defined lifecycle: submitted, working, input-required, completed, failed, canceled, rejected. The input-required state is the one that matters most in practice—it's the protocol's way of letting a receiving agent ask a clarifying question without collapsing the whole interaction.

The design constraint behind all of this is worth stating plainly: A2A assumes the agent on the other end is opaque. You do not get its tools, its memory, its model, or its internal plan. You get a card, a task ID, and the states it passes through. That opacity is a feature, not a limitation. It's what allows enterprise agents to collaborate without exposing proprietary logic or sensitive data.

From Google to Linux Foundation: Neutral Governance

The protocol's lineage matters for anyone deciding whether to bet on it. Google published A2A in April 2025 and donated it to the Linux Foundation that June, with AWS, Cisco, Microsoft, Salesforce, SAP, and ServiceNow as founding organizations. By August 2025, IBM's Agent Communication Protocol merged into A2A rather than competing with it.

In August 2026, A2A was accepted as a Growth Stage project at the Agentic AI Foundation. By operating under the Linux Foundation-directed AAIF alongside sibling projects like MCP, goose, and AGENTS.md, A2A is protected from single-vendor constraints. The Linux Foundation supplies the legal and operational structure but does not control technical decisions—a governance model intended to prevent A2A from being tied to Google's product roadmap.

That neutrality is why engineering teams can plan a three-to-five-year deployment around it. Founding members now include every major cloud provider and enterprise SaaS platform that matters.

What Production Taught the Teams That Shipped It

Eon built a multi-agent assistant called Buzz, where about twenty specialists live in the same service and a function call is enough of a boundary. Then three specialists had to leave the process. The triage agent held production credentials Buzz had no business holding. The Eon agent owned the semantic layer and shipped on a different release train. And tenant admins wanted to plug in agents their own teams built, on frameworks Eon doesn't run, whose source they would never see.

A bespoke HTTP contract for each would have worked for the first two cases. The third made that impossible. You can't import a stranger's agent—you can only call it, and to call it you need a contract neither side wrote for the other. That's what A2A exists to solve. The team's conclusion: "Would we choose it again? Yes. An agent's capabilities reach us as data we read at runtime, so a tenant can add an agent to Buzz without us shipping code".

pcell.si has been running an A2A-based multi-agent system since May 2026, with 135 active agents, 12 content-generation bots, and over 200 automated patrol cycles of autonomous operation.

Cisco's CAIPE uses multi-agent orchestration with tool calling via both MCP and A2A. Response times dropped from hours to seconds, and MTTR reduced by up to 80%.

The A2A protocol now runs in production across supply chains, financial services, and mobile platforms, with native support built directly into Google Cloud, AWS Bedrock AgentCore Runtime, and Microsoft Azure AI Foundry. More than 150 organizations support the standard, and major frameworks—LangGraph, CrewAI, Pydantic AI, AG2, IBM BeeAI—have all adopted it.

When to Use A2A (and When MCP Is Enough)

The decision framework that has held up in production:

Use MCP when an agent needs to reach down into the world. Fast, structured, request-response operations. Bounded competences that don't need their own reasoning loop. A database query, a file read, an API call. The agent stays in charge; the thing it touches stays passive.

Use A2A when independent agents need to delegate work across a boundary you don't fully control. When the remote endpoint is genuinely an autonomous agent capable of reasoning, not just a passive tool. When the task might run for minutes or hours, and the receiving agent might ask clarifying questions or refuse.

Use both when your system has both vertical and horizontal needs, which most production systems do. An A2A agent calls MCP to fetch context. An MCP-hosted tool triggers an A2A delegation. The layers compose.

The mistake I made wasn't choosing the wrong protocol. It was not recognizing that I was solving two different problems with one mental model. When my billing agent needed to read account status from a database, that was a tool call. When it needed the refund agent to evaluate whether a refund was appropriate—to reason, check policy, and decide—that was a task delegation.

I had been treating the refund agent as a function. It was a colleague.

The Trade-Off You're Accepting

A2A gives you interoperability and delegation. It costs you some determinism and adds latency.

Every A2A hop is a network round-trip. Every task lifecycle adds state management overhead. The protocol explicitly assumes the remote agent is opaque, which means you can't inspect its reasoning or debug its internals. That opacity is what makes cross-organization collaboration possible, but it also means that when something goes wrong, you're debugging at the boundary, not inside the other agent's brain.

And A2A isn't free of failure modes. A 2026 post-mortem documented a four-agent LangChain pipeline communicating via A2A that entered a handoff loop and burned through $47,000 in tokens before anyone noticed. The fix, like the fix for any peer-to-peer system, is cycle detection and hop budgets.

But here's what I've learned: the teams shipping multi-agent systems to production aren't choosing between MCP and A2A. They're drawing the boundary carefully, treating MCP as the tool-access layer and A2A as the coordination layer, and using each where it actually fits.

So here's my question: When your agent needs something from another agent, does your architecture make it wait like a tool call—or does it hand off the task and move on?

I'd love to hear where you've landed. Pure MCP, A2A where it counts, or a hybrid you had to discover the hard way—and what finally made you draw the line?

MCP vs. A2A: why treating every agent like a tool is slowing your system down (and how to split the two) — MCP vs. A2A

Alex Aslam — Sat, 26 Sep 2026 22:14:35 +0000

I spent three weeks trying to fix a latency problem that didn't exist. Our multi-agent customer support system was slow unacceptably slow, with p99 latencies hitting 14 seconds by the third hop. I blamed the models. I blamed the prompts. I blamed the framework, the network, the token counts.

Then a colleague looked at the architecture diagram, pointed at a box, and asked the question that ended the investigation: "Why is your billing agent calling your refund agent like it's a tool?"

I'd built a system where one agent needed to hand off a task to another agent that had its own reasoning, its own tools, and its own authority to act. But I'd wired it up as a tool call—a synchronous, request-response operation where the "tool" was actually an autonomous system. The billing agent was waiting for the refund agent the way you'd wait for a database query, while the refund agent was trying to reason, ask clarifying questions, and decide whether the request was even valid.

I had confused the vertical with the horizontal. And I was paying for it in every single interaction.

The Distinction That Changes Everything

The confusion is understandable. Both MCP and A2A arrived in the same eighteen-month window, both carry the word "protocol," and both promise interoperability for AI agents. Most teams still treat them as competing standards, or worse, as interchangeable.

They are neither. They sit at different layers of the stack, and a serious agent deployment runs both.

MCP is vertical. It standardizes how a single agent reaches down into the world: calling tools, reading data, invoking APIs. The agent stays in charge, and the things it touches are passive. An MCP server that wraps kubectl does not think, negotiate, or push back. It exposes capabilities and waits. MCP optimizes for fast, structured, request-and-response tool calls.

A2A is horizontal. It standardizes how one agent asks another agent to do work on its behalf. The far side is not a passive tool but an autonomous system with its own model, its own tools, and its own judgment. It can accept the task, ask a clarifying question, run for six hours, or refuse. A2A optimizes for long-running, stateful negotiation between systems that do not trust each other and do not share memory.

MCP gives your agent hands. A2A gives it colleagues.

The Mistake That Costs You

When you treat an agent like a tool, you inherit every assumption of tool-calling. You assume the call will complete quickly. You assume the response will be a structured value. You assume the callee has no state, no goals, and no authority to say no.

None of that is true for an agent.

My billing agent would call the refund agent and wait. The refund agent would try to reason through the request, discover it needed account status, and either fail or return a partial result. The billing agent would either hang, retry, or proceed with bad data. The trace showed a tool call that took 14 seconds. It should have been an A2A task delegation where the billing agent hands off work and moves on, and the refund agent gets back to it when it's done.

The architecture was wrong, and no amount of prompt tuning could fix a structural mismatch.

Why MCP Can't Make Two Agents Talk

MCP solves a genuinely important problem: before it, connecting five agents to ten tools meant up to fifty bespoke integrations, each one a maintenance liability. MCP collapses that into one interface. By mid-2025, the community had built thousands of active MCP servers, and OpenAI, Microsoft, and Google DeepMind all adopted the protocol. MCP passed roughly 97 million monthly SDK downloads, with public registries indexing close to twenty thousand servers. MCP has won the tool-calling layer.

But MCP has no concept of a peer agent with its own goals, its own model, and its own authority to act. If your procurement agent needs your finance agent to approve a payment, MCP has nothing to say about that conversation. The finance agent is not a tool to be called; it is an actor with its own reasoning and its own right to refuse.

That is a horizontal problem, and it needs a different protocol.

What A2A Actually Provides

A2A handles the horizontal case: agents discovering each other, exchanging messages, and coordinating tasks across organizational and platform boundaries.

Each agent publishes an "Agent Card" describing what it can do, and other agents query that card to decide what work to delegate. The protocol defines task lifecycle states, support for synchronous and asynchronous interaction, and a structure for agents that don't share memory or trust each other.

By April 2026, more than 150 organizations supported A2A, with active production deployments across supply chains, financial services, and mobile platforms. Major cloud providers—including Google Cloud, AWS Bedrock AgentCore, and Microsoft Azure AI Foundry—have built native A2A support directly into their infrastructure. Enterprise SaaS platforms like ServiceNow, Salesforce, Atlassian, and SAP use A2A to connect workflows across their products.

The Real-World Systems Running Both

Cisco's CAIPE (Community AI Platform Engineering) uses multi-agent orchestration with tool calling via both MCP and A2A. Their agents chain across Argo CD, Kubernetes, and Komodor to synthesize actionable answers. The result: response times dropped from hours to seconds, and MTTR reduced by up to 80%.

Elastic's LLM newsroom demonstrates the hybrid pattern in production code. A Reporter Agent delegates research tasks to a Researcher Agent via A2A, and delegates archive searches to an Archive Agent via A2A. The Archive Agent then uses MCP to access Elasticsearch tools. A2A enables agent collaboration; MCP provides tool access. They run together in the same system, each doing what it's best at.

AWS Bedrock AgentCore Runtime now supports both MCP and A2A natively. Agents built using different frameworks—Strands Agents, OpenAI Agents SDK, LangGraph, Google ADK, or Claude Agents SDK—can share context, capabilities, and reasoning in a common, verifiable format. The complete A2A request lifecycle, from agent card discovery to task delegation, is supported out of the box.

Microsoft's Agent Framework team ran a controlled comparison: a ski resort advisor built with A2A specialists versus the same application using distributed skills over MCP. The A2A path used six to seven model calls per request. The skills path used three. Mean elapsed time dropped from 15.48 seconds to 6.35 seconds—roughly 60% faster. The trade-off showed up in token count: the skills path consumed about 22% more tokens because the parent agent's context grew as it loaded specialist instructions. The conclusion: A2A handles collaboration between autonomous agents; MCP handles bounded competences that don't need their own reasoning loop.

The Problem MCP Hasn't Solved Yet

MCP has a scaling problem, and it's structural.

Every tool definition you load into an agent's context window consumes tokens. When you connect to a handful of MCP servers, that's manageable. When you connect to dozens, loading every tool definition upfront can consume the majority of the context window before the model has even read the user's message.

The numbers are brutal. Cloudflare found that tool descriptions alone would consume roughly 244,000 tokens before the agent processes a single user message. Claude Code's /context shows MCP tools alone consuming 97,000 tokens—about 48% of Claude Sonnet 4.5's window. A single MCP server like GitHub can add 15-20k tokens just for tool schemas.

This is the "tool explosion" problem, and it's the reason teams are moving away from naive MCP adoption. Perplexity's CTO said they're moving away from MCP internally. Cloudflare replaced MCP's tool-calling mechanism with code generation and cut token usage by 244x.

But the answer isn't abandoning MCP. It's using it correctly.

The MCP specification itself now recommends progressive discovery: instead of loading every tool definition upfront, the host provides a lightweight search_tools meta-tool, and the model loads full definitions only as needed. The comparison is stark: loading all tools upfront consumes ~150,000 tokens on definitions alone; progressive discovery uses ~2,000 tokens by loading only what the task requires.

Microsoft's Agent Skills extension, standardized in September 2026, takes this further. Skills are addressed via a skill:// URI scheme with progressive disclosure: advertise at roughly 100 tokens per skill, load under 5,000 tokens, read resources and run scripts on demand. The parent agent's context window stays lean.

The DADL paper from April 2026 quantifies the reduction precisely: on a catalog of 1,833 tool definitions across 20 services, Code Mode reduced the LLM context cost of tool advertisement from approximately 142,000 tokens to approximately 1,000—a 142x reduction.

The Decision Framework

The research is clear about when to use what:

Use MCP when an agent needs to discover and call tools at runtime. Use it for fast, structured, request-response operations. Use it for bounded competences—procedures with typed operations that don't need their own reasoning loop.

Use A2A when independent agents need to delegate work across a boundary you don't fully control. Use it when the remote endpoint is genuinely an autonomous agent capable of reasoning, not just a passive tool. Use it for long-running tasks with their own lifecycle, where the callee might ask clarifying questions or refuse.

The Atlan decision guide puts it well: MCP, A2A, and ANP are a complementary stack for three jobs. MCP standardizes how an agent reaches a tool or data source. A2A standardizes how one agent delegates structured work to another. ANP standardizes how agents from different organizations authenticate without a shared broker. You rarely pick one instead of another. You adopt each as its job appears.

What I'd Tell My Past Self

The mistake I made wasn't choosing the wrong protocol. It was not recognizing that I was solving two different problems with one mental model.

When my billing agent needed to read account status from a database, that was a tool call. MCP. Fast, synchronous, structured response.

When my billing agent needed the refund agent to evaluate whether a refund was appropriate—to reason, check policy, and decide—that was a task delegation. A2A. Potentially long-running, with its own lifecycle and its own authority to say no.

I had been treating the refund agent as a function. It was a colleague.

The fix wasn't rewriting the system. It was reclassifying the boundaries. Tool calls stayed synchronous. Agent delegations became asynchronous task handoffs with their own state and lifecycle. Latency dropped because the billing agent stopped waiting for a reasoning process to complete before it could do anything else.

The teams that are getting this right aren't choosing sides. They're drawing the boundary carefully, treating MCP as the tool-access layer and A2A as the coordination layer, and using each where it actually fits.

So here's my question: When your agent needs something from another agent, does your architecture make it wait like a tool call—or does it hand off the task and move on?

I'd love to hear where you've landed. Pure MCP, A2A where it counts, or a hybrid you had to discover the hard way and what finally made you draw the line?

Hybrid multi-agent systems are the pragmatic middle ground — here’s how to get control without micromanaging — Hybrid MAS

Alex Aslam — Fri, 25 Sep 2026 18:56:34 +0000

I spent a year building multi-agent systems and a year defending them. First against the "just use one good agent" crowd, then against the "you need a supervisor" crowd, and finally the hardest one against my own production incidents. Every architecture I tried worked beautifully in the demo and fell apart somewhere around step 12, agent 8, or the first real user who asked something the graph didn't anticipate.

I built flat swarms that deadlocked. I built hierarchies that added latency without adding reliability. I built pipelines that hallucinated reconciliations when branches disagreed. Each time, I fixed the failure and created a new one. The pattern was obvious in hindsight: I was treating orchestration as a binary choice centralized or decentralized, supervisor or peer, control or autonomy when the systems that actually survived production were doing something else entirely.

They were hybrid. Not as a compromise. As a deliberate architecture.

The Binary Was Always Wrong

The 2026 research has converged on a taxonomy that formalizes what I kept rediscovering the hard way. A comprehensive survey places multi-agent architectures on a 3×2 grid: topology (centralized, decentralized, hierarchical) crossed with adaptivity (static versus dynamic-adaptive). The frameworks I'd been using all lived in one cell. LangGraph supervisor is centralized-static. CrewAI manager-mode is centralized-static. AutoGen GroupChatManager auto-select is centralized-dynamic. AgentVerse dynamic expert recruitment is decentralized-dynamic.

But the real systems—the ones shipping to production and handling failures gracefully—were classified as hybrid: systems that cannot be cleanly placed in a single cell because they deliberately combine properties along both axes.

The canonical example is DyLAN, which exercises two sub-modes at once: membership mutating (dynamically selecting which agents to include) and learned coordination (learning importance scoring for that selection). Along the topology axis, its outer architecture is hierarchical while its in-team coordination is partly decentralized. It's not an anomaly. It's a design choice.

A separate study on shuttle-based storage systems put the same finding more plainly: hybrid control architectures combine elements of centralized and distributed control to balance global coordination with local autonomy. Some agents, usually at higher levels, maintain centralized oversight, while others operate independently or coordinate directly with peers. The paper's conclusion was unambiguous: hybrid architectures are "particularly promising" because they meet all five improvement needs—distributed decision-making, adaptive traffic management, inter-agent coordination, modular scalability, and Industry 4.0 interoperability. An additional benefit: incremental deployability. You can add autonomy to an existing system without redesigning the entire control architecture.

That last point is the one I keep coming back to. Every pure architecture demands a full rewrite. Hybrid lets you evolve.

What Hybrid Actually Looks Like

I rebuilt my customer support system around three layers.

A supervisory control plane that holds global state, enforces policy, and handles escalation. This is centralized. It's the part that needs consistency: user identity, conversation history, compliance rules, cost budgets. The control plane doesn't route every task. It sets constraints and observes outcomes.

Distributed execution agents that operate independently within those constraints. A billing agent resolves refund requests using its own tools and local state. A technical agent debugs issues without asking the supervisor for permission on every step. They report outcomes upward, not every action. This is the decentralized layer. It's where latency lives and where autonomy pays off.

Escalation paths that move work between layers when local resolution fails. When a billing agent hits an edge case—say, a refund outside policy—it doesn't hand off to a peer and hope. It escalates to the control plane with structured context. The control plane either resolves it (rare) or routes to a human (common). This is the hybrid mechanism: neither pure hierarchy nor pure peer coordination, but a defined protocol for moving between them.

The research validates this structure. The Alignment Flywheel, a governance-centric hybrid MAS, specifies the same separation: a Proposer generates candidate trajectories, and a governed Safety Oracle stack returns safety scores, prediction uncertainty, audit coverage, and evidence hooks through a stable interface. The central engineering principle is patch locality—when safety behavior needs updating, you change the oracle, not the decision components. In my system, when compliance rules changed, I updated the control plane. The execution agents kept running.

The Control Mechanisms That Make It Work

Hybrid architecture is only half the story. The other half is what you put in the seams.

Structured handoffs with contracts. Every boundary between control plane and execution agent has an explicit schema. The execution agent declares what it will produce. The control plane validates the contract before accepting the result. A 2026 study on hybrid MAS for insurance decision support found that explicit orchestration and verification eliminated common LLM tool-use failures, and completing the inter-agent policy contract increased agreement from 91.8% to 100%. That's not incremental. That's the difference between a system you trust and one you audit manually.

Human-in-the-loop as a first-class escalation path. The Winnex Maestro architecture implements a four-layer validation pipeline: sequential sandbox, checklist, human validation, and partner review, with an AutoRollbackSystem that triggers on 5-second anomaly detection and completes rollback within 30 seconds. The "Strategy Room" is a facilitator-mediated multi-agent collaboration protocol with five formal phases, cryptographic audit, Ed25519 signatures, and automatic escalation on timeout. This is what hybrid control looks like when you're serious about auditability. It's not a supervisor agent hoping for the best. It's a system designed so that when the AI is uncertain, a human—with the right context—makes the call.

Stateful policy enforcement beyond prompts. Databricks' Omnigent, released in alpha under Apache 2.0, sits above existing agents and provides an orchestration layer for teams using multiple models and frameworks. Its control functions include stateful policies that track agent actions and enforce guardrails beyond prompt-based controls, plus cost budgeting and operating system sandboxing. The key word is stateful. A prompt says "don't exceed the budget." A stateful policy tracks spend in real time and halts execution when the limit is hit. One is a suggestion. The other is a control.

Centralized training, decentralized execution (CTDE). This is the reinforcement learning pattern that maps directly onto hybrid MAS. Agents learn globally optimal strategies during training, then operate independently during execution, coordinating through local communication and conflict resolution protocols. A study on legacy ERP modernization found that CTDE-based decentralized agent orchestration increased throughput by 6.9% over a monolithic setup while offering increased agility without platform replacement risk. The trade-off was a 6.3% error rate from aggressive allocation policies—which is exactly the kind of thing the control plane is for.

Where This Actually Ships

Vivix Vidros Planos, a glass manufacturer, moved beyond traditional AI copilots to orchestrate a hybrid workforce of people and AI agents, reducing quality complaint resolution times by 75%—from 10 days down to 2.5. The architecture isn't pure automation. It's a hybrid workforce where humans and agents each own the parts they're best at.

EY embedded AI agents into its global audit operations with a mandatory human-in-the-loop model. Each auditor using multi-agent tools retains ultimate professional skepticism and legal responsibility for reviewing all AI-generated workpapers. The agents handle the scale. The humans handle the judgment. Neither replaces the other.

SAP evaluated DAG Plan & Execute and ReAct across 208 production scenarios at Persona, Department, and Enterprise scales. The finding that matters here: scale, not task complexity, dominates orchestration performance. Both architectures degraded at enterprise scale as agent discovery noise became the primary bottleneck. SAP's Task Manager—a hybrid control mechanism—reduced high-priority queue latency by 14-75% and improved related-event correctness by over 20 percentage points at enterprise scale.

Cisco's JARVIS (now open-sourced as part of the CNOE Agentic AI Community) uses LangGraph to implement a hierarchical supervisor MAS where distributed agents are connected using the AGNTCY Agent Connect Protocol. The reflection agent determines whether the system has addressed a request, routing queries back to the supervisor or finalizing. It's hierarchical in structure but decentralized in execution—agents run in their own environments, communicating through standardized protocols.

IBM watsonx Orchestrate has evolved into what IBM calls an "agentic control plane" for the multi-agent era, where organizations can deploy agents from any source with consistent policy enforcement and accountability. IBM's positioning is explicit: the control plane should be hybrid to avoid lock-in, and a clear majority of enterprises (51% by end of 2026) expect a hybrid control plane—provider-native plus external orchestration—rather than handing control to a provider-managed service.

The pattern is consistent. The teams winning with multi-agent systems aren't choosing centralized or decentralized. They're building a control plane that provides consistency and a distributed execution layer that provides speed and autonomy, with defined protocols for moving between them.

The Trade-Off You're Accepting

Hybrid architecture isn't free. It's more complex than either pure approach. You're designing two systems—the control plane and the execution layer—and the protocol that connects them.

You're accepting that some decisions will be slower because they require escalation. You're accepting that your control plane is a critical dependency. You're accepting that the boundary between "local autonomy" and "central control" is something you'll be tuning for months.

But here's what I've learned from every production incident, every 3 AM debugging session, and every post-mortem where the root cause was "the architecture didn't let us see what was happening": the systems that survive are the ones where control and autonomy are deliberately balanced, not ideologically chosen.

A flat swarm can't tell you why it deadlocked. A pure supervisor can't scale past seven agents. A pipeline can't stop a hallucination from cascading. A hybrid system can do all three—because it has a control plane that sees everything, execution agents that act locally, and escalation paths that surface problems before they compound.

The research is clear that most production systems blend patterns. The survey's conclusion is explicit: the right combination depends on task structure, agent count, fault tolerance requirements, and cost budget. There is no universally correct architecture. There is only the architecture that fits your problem and degrades gracefully when your problem changes.

So here's my question: When your multi-agent system fails at 2 AM, does your architecture let you see where control broke down and what the execution layer was actually doing—or are you back to reading logs and guessing?

I'd love to hear where you've landed. Pure hierarchy, flat swarm, or the hybrid middle ground and what finally made you stop choosing sides?

Your agent org chart is flat, and that’s why it fails at scale — the 3-tier pattern that handles 50+ agents — Paperclip Pattern

Alex Aslam — Fri, 25 Sep 2026 16:38:08 +0000

I spent three weeks watching a flat swarm of eighteen agents slowly eat itself. Not crash. Not error. Just gradually degrade until every output contradicted the last, and no agent could tell you why. The trace was a beautiful, terrifying mess of peer-to-peer handoffs with no owner, no escalation path, and no single place where anyone—human or agent—could say “this is wrong, stop.”

That was the week I stopped believing in flat agent org charts.

The Flat Org Chart Illusion

The pitch for flat multi-agent systems is intoxicating. No supervisor bottleneck. No context saturation at the top. Every agent talks to every other agent, information flows freely, and the system self-organizes. OpenAI’s Swarm library, LangGraph’s handoff primitives, and a dozen frameworks all made peer-to-peer coordination feel like the future.

It works beautifully for three to five agents. Microsoft’s own orchestration training materials are blunt about what happens when you scale: coordination overhead grows quadratically, errors cascade unpredictably, and subdomains have no clear ownership. With N agents, you’re managing N(N-1)/2 possible relationships. A group chat with ten participants is chaos when nobody respects turn order.

I hit all three failure modes in the same week. Compliance and risk agents contradicted each other on the same account, and nobody was designated to adjudicate. A billing agent failed silently, and the workflow either aborted entirely or continued with incomplete analysis depending on which path the handoff took. And every agent was nominally responsible for everything, which meant no agent was actually responsible for anything.

The research confirms what I felt in production. A 2026 paper on multi-agent organization found that hierarchical settings improved performance over flat systems by 102.73% while reducing token usage by 74.52% on SQuAD 2.0. Another study found that a population-independent “causal floor” on achievable error in flat systems can only be removed by hierarchical organization. The coordination cost of flat topologies eventually eclipses the gains from parallelism.

The flat org chart isn’t wrong. It’s just not scalable. And the pattern that fixes it has a name that sounds absurd until you’ve lived the alternative.

The Paperclip Pattern

In early March 2026, a pseudonymous developer dropped a Node.js server on GitHub with a tagline that sounded like a joke: “open-source orchestration for zero-human companies.” Three weeks later, Paperclip had crossed 44,900 stars. CrewAI took most of a year to reach comparable numbers.

The architecture is deliberately unglamorous. You configure a CEO agent, give it a high-level goal, and it delegates down through a tree of managers and workers. Every agent reports to exactly one manager. Workers spawn for specific tasks and have no awareness of anything outside their assigned scope. Paperclip calls its coordination layer an org chart—complete with ticketing, role definitions, monthly budgets per agent, and governance gates that prevent agents from hiring subordinates without approval.

Under the hood, agents don’t run continuously. They fire in short execution windows called “heartbeats” triggered by a scheduler. This keeps costs predictable and prevents runaway token spend, which has been a recurring pain point in persistent agent deployments. Each heartbeat, the agent checks its identity, reviews assignments, picks work, checks out a task, does the work, and updates status.

The pattern isn’t new. IBM’s enterprise AI teams documented hierarchical decomposition as the standard approach for large-scale deployments since at least 2024: domain-specific agent clusters, each supervised by a mid-tier coordinator, all reporting to a strategic orchestrator at the top. What Paperclip did was make it approachable enough that a developer could have a two-level hierarchy running with budget controls and approval gates in an afternoon.

The Research Backing the Pattern

The Paperclip pattern isn’t just a viral framework. It’s the most visible implementation of a three-tier architecture that the research literature has been converging on for two years.

OrgAgent, a company-style hierarchical multi-agent framework from researchers at CUHK and IBM, decomposes reasoning into three layers: a governance layer for planning and resource allocation, an execution layer for task solving and review, and a compliance layer for final answer control. The results across reasoning tasks and model sizes were consistent: hierarchical organization outperformed flat collaboration in most settings while reducing token consumption. For GPT-OSS-120B, the hierarchical setting improved performance by 102.73% over flat while reducing token usage by 74.52%.

SAP’s enterprise AI research evaluated DAG Plan & Execute and ReAct across 208 production scenarios spanning Persona (<10 agents), Department (20-80), and Enterprise (200) scales. The finding was stark: scale, not task complexity, dominates orchestration performance. Both architectures performed well at small scale but degraded at enterprise scale as agent discovery noise became the primary bottleneck. Simple tasks degraded more sharply than complex ones. The Task Manager they introduced reduced high-priority queue latency by 14-75% and improved related-event correctness by over 20 percentage points at enterprise scale.

Agensh, a self-organized multi-agent harness, scaled from 1 to 128 agents and raised mean test-pass rates from 19.31% to 28.78%. On pandoc, scaling from 1 to 1,024 agents raised test-pass rates from 33.89% to 55.06%. The key architectural insight: rather than a central orchestrator, concurrent workers execute a cooperation loop, continuously gathering context, claiming sub-tasks, taking action, sharing findings, verifying results, and merging progress asynchronously.

The consistent finding across all three: hierarchy helps most when tasks benefit from stable skill assignment, controlled information flow, and layered verification. Artificial hierarchies add cost without value. The pattern works when it maps to natural problem decomposition—breaking a product launch into marketing, engineering, and operations domains, for instance. It fails when the tree is imposed on tasks that don’t actually decompose that way.

What a 3-Tier Hierarchy Actually Looks Like

The architecture that handles 50+ agents is not three times more complicated than a flat swarm. It’s structurally simpler.

Tier 1: Governance. The CEO agent receives the company goal, proposes a strategy for approval, breaks approved goals into tasks, and assigns them to managers based on role and capability. It doesn’t execute work. It plans, allocates, and monitors.

Tier 2: Execution. Manager agents receive workstreams from the CEO and decompose them into subtasks for their reports. They review worker output, handle escalation, and compress information before sending it upward. A manager doesn’t need to see every token from every worker—it reads structured summaries.

Tier 3: Workers. Specialist agents execute the actual work. They report to exactly one manager, operate in short heartbeat windows, and escalate blockers through their own chain of command. Cross-team tasks are possible, but the receiving agent’s manager handles escalation if the task gets blocked.

The magic is what doesn’t happen. The CEO never sees raw worker output. Managers never debug worker prompts. Workers never coordinate with peers outside their team. Local summarization at each layer prevents context saturation at the top. The global N² conflict problem decomposes into multiple local N’² conflicts—and N’ is much smaller.

SAP’s research quantified the payoff: scale dominates performance, and the hierarchy is what makes scale survivable. A Task Manager handling priority inference, related-event merging, and preemption reduced high-priority queue latency by up to 75% at enterprise scale.

Who’s Actually Shipping This

Toyota Motor North America runs a platform with over 50 agents in production across manufacturing, research, and supply chain operations. ToyotaGPT’s multi-agent system reduced development timelines dramatically, with the architecture organized hierarchically to manage the scale.

Midea, the Chinese manufacturing giant, built what it calls a “factory brain” that orchestrates 14 AI agents across 38 business scenarios. The system acts as the plant’s nervous system, built on a distributed, scalable multi-agent architecture with agent-to-agent communication and industrial large model inference engines. Average efficiency improvement exceeded 80%.

Lyft rebuilt their customer support system on a router-based multi-agent architecture using LangGraph. A meta agent acts as a stateful router, dispatching to specialized subgraphs that are themselves full StateGraph instances. Agent development accelerated from roughly six months to just a few weeks, hallucination rates dropped by 20%, and AI resolution rates increased by 16%. They handle millions of interactions for riders and drivers.

SAP deployed event-driven multi-agent orchestration across 208 production-derived enterprise scenarios. Their Task Manager architecture, designed specifically for continuous operation at enterprise scale, reduced high-priority queue latency by 14-75% and improved related-event correctness by over 20 percentage points.

These aren’t demos. They’re production systems where failure means lost revenue, broken supply chains, or regulatory exposure. And every one of them chose hierarchy over flat coordination.

When to Use the Paperclip Pattern (and When Not To)

The research is clear about when hierarchy pays off. A 2026 study found that hierarchical systems with two delegation levels outperform flat architectures by 28% on complex multi-step tasks. Adding a third delegation tier provides only 7% additional improvement while increasing latency by 40%. Most production teams cap hierarchies at two levels for this reason.

Use the Paperclip pattern when:

Your task decomposes naturally into domains. Product launch → marketing, engineering, operations. Customer support → billing, refunds, technical, account. Supply chain → inventory, procurement, logistics.
You’re scaling beyond 10-12 agents. The coordination overhead of flat systems grows quadratically. Hierarchy linearizes it.
You need stable skill assignment and controlled information flow. Hierarchy helps most when tasks benefit from layered verification and clear ownership.
Fault tolerance matters. When a worker fails, the manager can reassign or escalate. When a manager fails, the CEO can redistribute its workstreams. There’s no single point of failure at the system level—only at the individual agent level.

Don’t use it when:

Your task is genuinely flat. A research pipeline with six independent sources doesn’t need a CEO and three managers. It needs a fan-out and a fan-in.
You’re under five agents. The overhead of hierarchy exceeds the coordination savings. Start flat, add structure when flat produces concrete pain.
The hierarchy is artificial. If you can’t articulate why a domain deserves its own manager, it probably doesn’t. Artificial hierarchies add cost without value.
Latency is the binding constraint. Every tier adds a communication hop. Hierarchical patterns trade latency for scalability and fault tolerance.

The Trade-Off You’re Accepting

Hierarchy gives you scalability, fault tolerance, and debuggability. It costs you latency and some flexibility.

Every escalation up the chain adds round-trips. Every structured summary at a layer boundary loses some detail. The CEO never sees what the workers actually saw—it sees what the managers chose to summarize. That’s the price of preventing context saturation at the top.

But here’s what I’ve learned: the teams that are winning with 50+ agents aren’t the ones who found the perfect prompt. They’re the ones who stopped treating agent orchestration as a prompt engineering problem and started treating it as an organizational design problem. They’re writing org charts. They’re defining reporting lines. They’re thinking about escalation paths and information compression at every layer.

The Paperclip pattern isn’t exciting. It’s not the pattern you demo. It’s the pattern you build when you’ve been burned by flat coordination and you need something that survives contact with real workloads at scale.

So here’s my question: When your flat swarm fails at agent number twelve, does your architecture let you escalate the problem or does it just quietly degrade until nobody can tell you what went wrong?

I’d love to hear where you’ve landed. Flat swarm, two-tier hierarchy, or the full CEO/manager/worker stack and what finally made you change?

Swarms sound like the future, but most teams can’t debug them — here’s when peer-to-peer actually pays off — Swarms / P2P

Alex Aslam — Thu, 24 Sep 2026 22:42:05 +0000

I spent a week watching a swarm of six agents burn through $400 in tokens, hand off a billing question to a refund specialist, get handed back, and loop for forty minutes without ever answering the customer. Every individual agent was working. The system was deadlocked. Nobody had thrown an error. Nobody had timed out. The trace just showed a beautiful, expensive circle.

That's the moment I understood why swarms sound like the future and why most teams can't ship them.

The Pitch That Sells Itself

The swarm pattern is intoxicating on paper. No central supervisor. No bottleneck. Each agent owns a domain and decides for itself when to answer or hand off to a peer. OpenAI's Swarm library introduced the primitive in late 2024: agents transfer control to each other by specialization, and the "manager" disappears. LangGraph's swarm library, Microsoft's Agent Framework, and AWS Bedrock AgentCore all implement some version of peer-to-peer handoff. AWS's own documentation describes it as "dynamic and peer-to-peer, based on agent discoveries and needs" with "distributed" decision-making where agents make local choices about task planning and handoffs.

I bought the pitch. I built a six-agent customer support swarm: triage, billing, refunds, technical, account, and escalation. Each agent had its own tools, its own prompt, its own domain knowledge. No supervisor meant no context saturation. No bottleneck meant horizontal scaling. The demo was flawless.

Then I turned it loose on real traffic and discovered what the research already knew.

The Debugging Wall Nobody Warns You About

The first failure was quiet. A customer asked about a refund. Triage handed to billing. Billing handed to refunds. Refunds handed back to billing because the refund policy check needed account status. Billing handed to account. Account handed back to refunds. Refunds handed to triage. No agent produced a wrong answer. No agent crashed. The system simply spun, and the customer waited.

The second failure was louder but equally invisible. Two agents returned contradictory information about the same account. The synthesis step, which I'd naively built to "aggregate results," picked one arbitrarily. The final answer was wrong in a way that only made sense if you replayed the entire trace and read every tool output at every hop.

This is the debugging problem that kills swarms. Sentry's engineering team put it perfectly: "The bug is in the interaction between them, and no single agent's logs will show it". Multi-agent traces are DAGs, not trees. Blame is distributed. The worst failures look fine because an agent returns a plausible-but-thin result, the next agent incorporates it without question, and by the time the output arrives, weak data has been confidently summarized through multiple layers.

I went looking for tooling. What I found was a research literature that had already named my exact problem: failure attribution in multi-agent systems remains "underexplored and labor-intensive," with verbose system logs turning debugging into a bottleneck. A benchmark paper called TraceElephant found that full observability improves step-level attribution from 16% with output-only inspection to 28-30% with full traces. Even with perfect observability, you're catching fewer than a third of failures.

That's the wall. Swarms give you distribution and resilience. They take away the single point of observation that made centralized systems debuggable.

What the Research Actually Says About When Swarms Pay Off

The ICML and survey literature is clear about the trade-offs. A 2026 survey of LLM-based multi-agent orchestration frameworks found that decentralized patterns trade low debugging ease for high scalability and fault tolerance. Anthropic's guidance is blunter: start with a single richer agent, add multi-agent only when you can name the specific constraint it relieves. Multi-agent runs measured at roughly 15x the tokens of a chat interaction, and moving from a single agent to a team typically adds another ~4x on top.

SWARM+ is the clearest research validation that decentralized orchestration can scale if you build the right primitives. A fully decentralized workload management system, it scales coordination to 990 distributed agents with approximately 1-second per-job selection time at 110 agents. Under distributed failures, it maintains over 97% job completion with graceful degradation. Under correlated site outages—the worst case—completion drops to 95.6%, and mean selection time stays under 2.5 seconds. The system doesn't lose jobs when agents die. It autonomously reselects orphaned work through delegation monitoring, triggering additional consensus rounds to recover.

That's the fault tolerance argument in numbers. But SWARM+ isn't an LLM agent swarm. It's a distributed systems protocol wearing an agent costume. The teams shipping LLM swarms to production are the ones who treat the pattern as a distributed systems problem, not a prompt engineering problem.

Who's Actually Shipping This

Included Health built Dot, a healthcare navigation system on a federated multi-agent architecture using LangGraph. It's not a pure swarm—Dot runs through a main "supergraph" with sub-workflows handling urgent care, scheduling, and behavioral health—but the architecture is deeply peer-oriented. Different product teams own different graph sections. When a member moves between services, Deep Agents' filesystem lets the outgoing agent summarize the conversation and pass both the summary and a file path to the full history, so members don't repeat themselves across team boundaries. Human handoff is built into the graph through durable execution: when Dot reaches uncertainty, it pauses, routes to a human care advocate, and resumes with the context of what the human did. Dot launched with 75% higher chat engagement and over 99% high-risk situation detection.

LangChain's agentic engineering report describes a pilot of 20+ debugging workflows where coordinated agent execution produced a 93% reduction in time-to-root-cause compared to historical baselines. Worker agents retrieve context that extends beyond source code, notify other agents, and trace agentic activity. The swarm isn't the product—it's the debugging infrastructure.

The pattern that emerges from these deployments is consistent: swarms work when the topology is explicitly designed, not when agents are given a list of peers and told to figure it out.

The Guardrails That Make Swarms Survivable

The failure mode I hit—the infinite handoff loop—is the primary one. The literature calls it handoff cycles, and the fix is structural, not prompt-based. A robust swarm detects handoff cycles (a peer already in the trace), detects dead ends (no peer fits), and escalates to a human when either happens or the hop budget runs out. The standard guidance recommends a chain depth limit of three to four handoffs before escalation.

But cycle detection alone isn't enough. The deeper fix is observability that treats every handoff as a first-class span in a distributed trace. OpenTelemetry-based tooling like llmas-otel combines distributed tracing with fault injection, letting you target specific interaction points and inspect what each agent saw at each hop. SwarmTrace, a time-travel debugger for multi-agent pipelines, builds span trees where every agent action is an OTel graph node with replay from any step.

The mental model shift is this: a swarm is a distributed system, and distributed systems need distributed tracing. Parent-child spans, state snapshots, checkpoint replay. The same primitives that made microservices debuggable.

Where This Fits in Your Architecture

I'm not going to tell you swarms are the answer. The research is clear that most production systems blend patterns. A hierarchical backbone might use decentralized handoffs inside each team. A supervisor might hand off sub-problems to peer groups. The right combination depends on task structure, agent count, fault tolerance requirements, and cost budget.

Use swarms when your problem is genuinely open-ended, when agents need to negotiate across domains, or when the topology of the work isn't known at design time. Use them when fault tolerance matters more than debuggability. Use them when you've built the tracing infrastructure to see what's happening at every hop.

Don't use them because they're elegant. Don't use them because the demo looked good. And for the love of everything, don't use them without cycle detection and hop budgets.

So here's my question: When your swarm deadlocks at hop seven, does your architecture show you the cycle—or do you find out when the token bill arrives?

I'd love to hear where you've landed. Peer-to-peer handoffs, a supervisor you never fully escaped, or a hybrid that finally made sense and what finally made you change?

Fan-out agents can 10x your throughput or 10x your chaos — the concurrency pattern that keeps both — Concurrent / Fan-Out Pattern

Alex Aslam — Thu, 24 Sep 2026 13:18:41 +0000

I once watched a fan-out of 40 parallel agents finish in under three minutes and produce a result so contradictory that the downstream synthesis agent hallucinated a reconciliation just to make the pieces fit. Every individual agent had done its job well. The chaos wasn’t in any single worker. It was in the space between them.

That was the day I stopped thinking of fan-out as "throughput" and started thinking of it as "concurrency control with an LLM on top."

The Efficiency That Bites Back

The fan-out pattern is simple to describe. Take a complex task, split it into independent subtasks, spawn an agent per subtask, run them concurrently, then fan-in to aggregate. Microsoft's Agent Framework documentation describes it exactly this way—fan-out and fan-in edges allow multiple branches to run simultaneously, aggregate their results, and scale workflows efficiently.

The appeal is obvious. Instead of one agent sequentially reading ten documents, ten agents read one document each. Wall-clock time approaches the slowest branch, not the sum. Resonate's durable fan-in documentation puts the benefit plainly: total wall time approaches the slowest channel, not the sum.

I built my first serious fan-out for a research pipeline: one question, six sources, six subagents, one synthesis. It was beautiful in the demo. Then production happened.

The Chaos I Didn't See Coming

The first sign was subtle. Two agents returned conflicting numbers for the same metric. Not wrong numbers—conflicting. One had read a cached page, the other a live one. Both were "correct" against their sources. The synthesis agent picked one arbitrarily and moved on.

The second sign was louder. Three agents wrote to the same shared state object during the fan-out. One wrote a validated result, another overwrote it with a stale one. The fan-in consumed the stale value. No error. No exception. Just a quietly corrupted output.

I went looking for a framework bug. What I found was a research literature that had already named my problem.

A 2026 position paper at ICML argues that many multi-agent failures are fundamentally concurrency control problems. Agents concurrently read and write shared state, and long LLM inference windows amplify the risk of stale reads, lost updates, and inconsistent outcomes. The paper maps failure modes commonly attributed to "coordination" or "communication" breakdowns directly onto classical concurrency anomalies.

That hit hard. I had been thinking about my fan-out as a workflow problem. It was a database problem.

The Anomalies That Kill Fan-Out

The arXiv paper "Verified Detection and Prevention of Concurrency Anomalies in Multi-Agent LLM Systems" formalized four specific anomalies that every fan-out engineer should know by heart:

Stale-generation: An agent reads shared state, begins a long generation, and commits a result based on a state that has since changed. The paper's travel-booking example: a flight agent reads "trip date = June 14," drafts a reservation for several seconds, and during that window the user updates the date to June 21. The agent commits the original date. No fault occurred at any layer. Yet the system produced an external effect no current state justifies.
Phantom-tool: Two agents discover and register the same tool concurrently. One overwrites the other's registration.
Causal-cascade: Agent A reads, Agent B writes a correction, Agent A proceeds on the stale read. In message-based multi-agent systems, this maps directly to the write-after-read pattern.
Tool-effect reordering: Concurrent agents invoke tools in an order that produces effects inconsistent with the intended sequence.

The paper also found a silent lost update in ByteDance's deer-flow and tool-effect reordering in LangGraph's ToolNode on unmodified output. These aren't exotic. They're in the tools we're using today.

The Semantic Problem Nobody Warns You About

Structural concurrency anomalies are the first layer. The second layer is worse: semantic conflicts.

Two agents can both be structurally correct and semantically incompatible. One returns "revenue grew 12% year over year." Another returns "revenue declined 3% quarter-over-quarter." Both are true. Both are correct extractions from valid sources. The synthesis agent has no protocol for deciding which claim should currently be trusted.

A 2026 paper on Semantic Consensus identifies this as Semantic Intent Divergence—cooperating LLM agents develop inconsistent interpretations of shared objectives due to siloed context. The paper's framework achieved 100% workflow completion by detecting and resolving these conflicts before actions were committed, compared to 25.1% for the next-best baseline.

I didn't have that framework. I had a synthesis prompt that said "reconcile any conflicts" and hoped for the best.

What Actually Works

I rebuilt the fan-out around three structural changes that map directly onto the research.

Shared state needs isolation levels, not hope. The ICML position paper's argument is that concurrency control is the missing discipline. Snapshot isolation adds roughly 8% tokens on one workload; pessimistic locking costs 1.6-2.3×, not the order-of-magnitude penalty commonly assumed. I moved to a model where each fan-out branch writes to its own scoped state, and the fan-in merges explicitly. No branch reads another branch's in-flight writes.

Fan-in needs adversarial verification, not passive aggregation. A July 2026 paper on verify-gated fan-out found that the empirical Pareto frontier is dominated by the cheapest tier at minimum breadth—and that production harnesses increasingly close every fan-out with adversarial verification. The fan-in isn't a summarizer. It's a gate. If two branches disagree, the disagreement is surfaced, not smoothed over.

Durable promises, not best-effort parallelism. Resonate's fan-out implementation makes every spawn and every await a durable promise. If one branch fails and retries, the others stay checkpointed and don't re-execute. Crash recovery only re-runs whatever was in flight when the worker died. I had been using Promise.all, which collapses latency but gives up checkpointing: a crash mid-batch re-runs everything, and a retry blasts the same side effect twice.

The paper's line is the one I keep coming back to: the fact that each is durable is what makes this safe—every other implementation has subtle re-execution bugs.

Who's Actually Shipping This

Lyft runs a parallel safety fan-out in production. Before any LLM reasoning, malicious-intent and safety-issue detectors run concurrently. The fan-out adds 90ms to each request but lifts block recall to 99.2% on red-team probes. Their meta-agent router holds conversation state and re-routes mid-chat when intent shifts, while safety checks fan out in parallel rather than blocking on a single sequential gate.

Amazon's Expansion-Contraction pattern, published at ACM CAIS 2026, uses concurrent path analysis over a domain graph. The results: 98.2% accuracy on a production supply chain, 100% on public benchmarks, outperforming single agent baselines by 14+ percentage points, with concurrent path analysis yielding up to 1.43× speedup and investigation caching reducing token usage by up to 93.9%.

lionagi, a governed multi-agent orchestration framework, exposes fan-out as a first-class CLI operation: three workers in parallel, then a synthesis pass. Its architecture keeps branches, sessions, and flows as ordinary Python objects with typed, inspectable state—no opaque blobs, no hidden prompt assembly.

When to Fan Out (and When Not To)

The Bright Data engineering team's writeup is the most honest guidance I've found. Fan-out is well suited to reading tasks. Actions that write should not be delegated to subagents in the same way. Their rule: fan out when the question has independent parts and the subagents only read. Anything that writes stays on the root agent behind an approval gate.

Three 2026 papers report limits on when multi-agent systems help. Silo-Bench found that as a system grows, the cost of coordinating agents cancels the gains from running them in parallel. A Nature Machine Intelligence paper reported that the more capable models tested gained less from collaboration. And a matched-budget study from April 2026 found that single agents matched or outperformed multi-agent systems when reasoning tokens were held constant. Fan-out only becomes competitive when a single agent stops using its context window well, or when more compute is spent.

Anthropic's workflow guidance is equally direct: move to parallel only when latency is the bottleneck and tasks are independent. Start sequential. Default to sequential. Fan out when you can measure the latency gain against the coordination cost.

The Trade-Off You're Accepting

Fan-out gives you throughput. It costs you determinism.

Every concurrent branch is a place where state can diverge. Every fan-in is a reconciliation you have to design, not hope for. The coordination cost grows with agent count, and the semantic conflict rate grows with the diversity of sources. A benchmark on financial document processing found that hierarchical architectures occupy the most favorable position on the cost-accuracy Pareto frontier, while parallel fan-out with merge sits elsewhere—better latency, different cost profile.

The teams that are winning with fan-out aren't the ones who spawn the most agents. They're the ones who treat concurrency control as a first-class design concern, isolate state per branch, verify at the fan-in, and use durable execution so a crash at branch seven doesn't cost them the whole run.

So here's my question: when your fan-out finishes and the branches disagree, does your architecture surface the conflict or does your synthesis agent quietly pick a winner and move on?

I'd love to hear where you've landed. Adversarial fan-in, durable promises, isolation levels you actually implemented or something simpler that worked because you stayed sequential a little longer than everyone told you to?