TL;DR — Most agent failures get blamed on bad reasoning, but the real culprit is usually the orchestration loop's treatment of the chat transcript as ground truth. Transcripts are lossy summaries, and re-planning from a decaying summary compounds errors geometrically across iterations. The fix is architectural: separate memory from belief, checkpoint verified state, and bound how long a loop can run on unverified assumptions.
When an agent goes off the rails — books the wrong flight, deletes the wrong file, loops forever calling a tool that already succeeded — the postmortem almost always blames the model. "It hallucinated." "It didn't reason carefully." "We need a smarter base model." This diagnosis is comforting because it's someone else's problem to fix. It's also usually wrong.
The actual failure, in a large fraction of production agent incidents, lives in the orchestration loop, not the model's reasoning. Specifically: the loop treats its own transcript as the source of truth about the world, when the transcript is a lossy, human-readable compression of what actually happened. Every re-plan step re-derives intent and state from that compression. Errors introduced early don't stay constant — they compound.
The transcript is not the state
Almost every agent framework — ReAct-style loops, plan-execute-reflect patterns, multi-agent orchestrators — uses the same underlying mechanism: append an observation to a running context, feed the whole thing back to the model, ask it to decide the next action. This is elegant and it's why the pattern spread everywhere. It's also a category error.
The transcript records what the agent said happened, filtered through however the tool output got summarized, truncated, or paraphrased on the way in. It is not a database. It has no schema, no invariants, no way to distinguish "the API call returned this exact JSON" from "the model's gloss of that JSON three turns ago." Once a tool result gets summarized to fit a context budget, the summary becomes the new ground truth for every subsequent decision — even though nobody verified the summary was accurate.
This is fine for one turn. It is not fine across ten. A file listing that gets condensed to "the directory has a few config files" is a reasonable compression once. By the fifth re-plan cycle, an action gets taken on the assumption that only "a few config files" exist, when the real count matters and was never re-checked.
Why orchestration frameworks make this worse, not better
The tooling built to make agents more capable often amplifies this exact problem. Frameworks that add "memory" typically mean one of two things: a longer context window, or a vector store that retrieves semantically similar past snippets. Neither one restores ground truth. A longer window just delays the point where summarization becomes necessary. A retrieval layer returns the most relevant fragment of a past observation, not a verified current state — it's memory of memory.
Multi-agent orchestration compounds this differently. When a planner agent hands a subtask to a worker agent and receives back a natural-language report of what the worker did, the planner is once again operating on a summary — this time authored by another model with its own incentive to sound confident and complete. Chain three or four agents together and you get a game of telephone with tool calls attached. Each hop re-derives belief from the previous hop's prose, and nobody along the chain is checking the prose against the actual system state.
The failure mode isn't "the model got confused." It's that the architecture never gave any component a way to ask "is this still true?" before acting on it.
Errors compound geometrically, not linearly
This matters because it changes how you should think about reliability engineering for agents. A single tool call with a 95% success rate sounds fine in isolation. Chain eight of those calls through a planning loop where each step's plan depends on the accuracy of the previous step's summarized state, and you're not looking at 0.95, you're looking at something closer to 0.95 to the eighth power for the parts that are independent — and worse for the parts that aren't, because a bad summary early in the loop doesn't just fail once, it corrupts the premise every subsequent step reasons from.
This is the deep reason agent demos look great and agent production systems disappoint. A three-step demo rarely hits the compounding zone. A twenty-step agent running unattended for an hour lives entirely inside it. The failure isn't rare edge cases piling up — it's the geometry of re-planning on unverified state becoming the dominant term in the reliability equation.
What actually fixes this
The fix is not a smarter model. It's separating two things that agent frameworks conflate: memory of what happened, and belief about the current world state. Memory can be lossy — it's for narrative continuity, for the model to explain its own actions coherently. Belief cannot be lossy, because every action gets taken against it.
Concretely, that means:
Structured state, not prose state. Maintain an explicit state object — file counts, record IDs, confirmation tokens, account balances — updated only by verified tool output, never by model paraphrase. The model reads this object each turn instead of re-deriving it from context.
Re-fetch before consequential actions. Any action with real-world side effects (a write, a payment, a delete) should trigger a fresh read of the relevant state immediately beforehand, not rely on a read from several turns ago. This is the agent equivalent of optimistic locking.
Bound loop depth against unverified assumptions. If a plan has been running for N iterations without a ground-truth checkpoint, force a re-grounding step or escalate to a human rather than letting the loop continue extrapolating from its own earlier guesses.
Treat summarization as a named, testable operation. If you're compressing tool output to save context, that compression step deserves the same scrutiny as a retrieval step in RAG — write tests that check whether the summary preserves the facts the downstream plan will depend on, not just whether it reads fluently.
None of this requires a better model. It requires admitting that "agent reliability" is mostly a state-management problem wearing a reasoning-problem costume. The industry's instinct is to throw a bigger context window or a more capable base model at flaky agents, and that sometimes helps at the margins. But a model with perfect reasoning, fed a stale or paraphrased picture of the world, will still confidently take the wrong action — because from where it's sitting, the paraphrase is the world.
The uncomfortable implication
If this thesis is right, then a lot of agent benchmarking is measuring the wrong thing. Single-turn tool-use accuracy tells you almost nothing about whether a loop will hold up over twenty iterations, because the failure mode only shows up when state has had time to drift from belief. The benchmarks that matter for production agents are the ones that run long enough for compounding to kick in, and that specifically inject state changes mid-loop to see whether the agent notices its picture of the world is out of date. Most current evaluations don't do this, which is part of why so many agents that look great in a demo fail quietly in week two of running unattended.
Top comments (0)