The best models in 2026 hallucinate on about 1% of grounded, single-shot tasks. Hand one a document and ask for a summary and it almost never invents anything.
Now let that same model drive itself: plan a task, call a tool, read the result, decide the next step, update its memory, repeat for twenty steps with no human in the loop. It will now go wrong on most runs. Same model, same weights. The thing that changed is autonomy, and autonomy is where hallucination is actually manufactured.
This is the part almost nobody states plainly: for a self-driving agent, the dominant cause of "making things up" is not the model's factual accuracy. It's the structure of running unattended across many steps. A better model barely moves it. Below is why, with the numbers, and what actually does.
TL;DR — Agent errors compound multiplicatively: at 95% per-step accuracy, a 10-step task succeeds ~60% of the time, a 20-step task ~36%. Even a fictional 99%-per-step agent fails ~18% of 20-step tasks. And it's worse than the clean math, because agents self-condition on their own earlier mistakes, so errors accelerate rather than just accumulate. This is structural, not a model-quality problem: no checkpoint fixes p^n. What fixes it is architecture, shorter chains, per-step validation, scoped context, and escalation, plus not reaching for a reasoning model, which for grounded steps often hallucinates 3–4× more.
The one equation that governs autonomous agents
Reliability engineering has a law for chained systems (Lusser's law): the reliability of a sequence is the product of the step reliabilities, not the average. For an agent taking n independent steps, each succeeding with probability p, end-to-end success is p^n.
That exponent is the entire story of why autonomy is hard. Watch what it does:
| Per-step accuracy | 5 steps | 10 steps | 20 steps |
|---|---|---|---|
| 99% (fantasy) | 95% | 90% | 82% |
| 95% (excellent) | 77% | 60% | 36% |
| 90% (good) | 59% | 35% | 12% |
| 85% (a strong result) | 44% | 20% | 4% |
Read the 95% row again, because 95% per step sounds great. An agent that gets each step right 95% of the time succeeds on a 20-step task 36% of the time. It fails most of them. And 85% per step, which is a genuinely strong result on a complex reasoning action, collapses to 4% over 20 steps: one success in twenty-five.
Demis Hassabis called compounding agent errors "compound interest in reverse," and it's exact. The same multiplicative machinery that makes compound interest miraculous over time makes autonomous error catastrophic over steps. Demos hide it because a demo is 2–3 steps. Production is 5+ steps over messy inputs, which is precisely where p^n bites.
The load-bearing consequence: no amount of model improvement fully solves this. Going from 95% to 97% per step helps, but you're still fighting an exponent. The gap between "impressive demo" and "reliable product" is not a capability gap you close with a better model. It's a structural property of chaining a non-deterministic component, and you engineer around it or you ship a coin flip.
Why it's actually worse than p^n: self-conditioning
The table above assumes each step's error is independent. In real agents, it isn't, and the dependence runs the wrong way.
Researchers documented a self-conditioning effect in multi-step LLM pipelines in early 2026: once an agent has produced an error and that error is sitting in its context, the model conditions on it. It sees its own earlier mistake as established fact and reasons from it, which makes the next mistake more likely, not less. Errors don't just accumulate, they compound on themselves. The real failure curve is steeper than p^n predicts.
This is the mechanism underneath "the agent went off the rails around step 6." It didn't hit a hard wall. It made one small wrong turn at step 4, wrote that into its own working context, and every subsequent step treated the wrong turn as ground truth. By step 8 it's confidently building on a fact it invented four steps ago. The hallucination isn't fresh at step 8, it's laundered from step 4 through the agent's own memory.
And this points straight at one of the most effective fixes, which is counterintuitive: scoped context. A step that doesn't know about the errors in steps 2 and 3 can't condition on them. Passing only the relevant slice of context forward, instead of the full accumulated history, breaks the self-conditioning loop. Less memory, more reliability, the opposite of the instinct to give the agent everything.
The autonomy-specific amplifiers
On top of the raw compounding, self-driving specifically adds failure sources a single-shot call never has:
Lossy memory compaction. When the agent's history won't fit in context, it summarizes, and repeated summarization is brutal. A 2026 rate-distortion study measured it: an agent archiving raw records and retrieving on demand held ~95% recall; an agent overwriting memory with LLM summaries dropped to 33–56% recall, worst when it compacted most often. At equal budget, lossy self-summarization loses roughly half the facts over a long run, and a single-turn benchmark can't see it because the loss only appears when compaction repeats. The agent then confidently fills the holes it created.
Context drift. ~65% of enterprise agent failures in 2025 were attributed to context drift or memory loss during multi-step reasoning, not to running out of context. As the window fills, attention de-prioritizes the original task; early tool outputs get overwritten by later ones; the agent quietly diverges before it ever hits the limit. Chroma's context-rot work shows degradation accelerating past ~30k tokens even in models with far larger windows, capacity is not fidelity.
Multi-agent contamination. Split the work across agents for reliability and you can make it worse: naive full-broadcast state sharing increased hallucination by 34% in a controlled 2026 study, because one agent's error propagates to all of them. Chaining five agents at 95% each yields 77% end-to-end, not 95%, the same p^n exponent, now spread laterally.
Every one of these is a property of running autonomously, not of the model's factual accuracy. A single API call to the same model has none of them.
Where the model does matter: don't make per-step accuracy worse
The model isn't irrelevant, it sets your p. But the counterintuitive part is that the "smarter" model many teams reach for to fix agent reliability often lowers p on exactly the grounded steps agents are made of.
Reasoning models hallucinate more on grounded tasks. Same base model: DeepSeek-V3 hallucinates at 3.9% on grounded summarization; its reasoning sibling R1 at 14.3%, ~4× worse. It's a pattern, not a one-off: on Vectara's 2026 dataset, every reasoning model exceeded 10%, while a non-reasoning model (Gemini Flash-Lite) led at 3.3%. The mechanism is the same self-conditioning drift, reasoning models generate long internal chains that wander from the source. So "let me upgrade to a reasoning model so my agent thinks more carefully" can lower your per-step accuracy and, through the exponent, wreck end-to-end reliability. For the grounded steps that make up most agent loops, a fast non-reasoning model is often the higher-p choice.
"Lowest hallucination" can mean "abstains most." Claude 4.1 Opus posts 0% on a knowledge benchmark, but by declining when unsure, which for an agent is often exactly right (a confident wrong step corrupts everything downstream; an "I'm not sure" step can be escalated). Calibration, knowing what it doesn't know, matters more for p than raw accuracy, because a self-conditioning agent punishes confident-wrong far more than honest-uncertain.
The takeaway on models: pick for high, calibrated per-step accuracy on your actual step type (usually grounded → non-reasoning), not for leaderboard position. But understand that even a great p is fighting p^n, so the model is necessary and nowhere near sufficient.
What actually fixes autonomous hallucination
None of these is a model upgrade. All of them attack the exponent or the self-conditioning.
- Shorter chains. The single highest-leverage move. Cutting a 20-step task to 8 steps does more for reliability than any accuracy bump, because you're reducing the exponent, not the base. If a task can be decomposed into shorter independent sub-tasks with checkpoints between, do that.
- Validation between steps. Schema-validate each step's output before passing it forward. A 95%-accurate step that's checked can't silently corrupt everything downstream, you catch the error at step 4 instead of discovering it at step 20. This directly interrupts self-conditioning.
- Scoped context, not full history. Pass only what the next step needs. A step that can't see earlier errors can't condition on them. Less context is more reliability here.
- Escalation and circuit breakers. The agent must know when it's uncertain and hand off. The dangerous failure isn't the agent that stops and reports, it's the one that fails silently and continues, laundering a wrong turn through twelve more steps. Test explicitly: does it recognize its own error, signal it, and halt rather than compound?
- Selective autonomy, not full autonomy. Aggressive automation for low-risk, reversible steps; hard human gates on high-stakes, irreversible ones. Cap the blast radius. The goal was never "fully autonomous," it's "autonomous where it's safe, gated where it isn't."
The thing to internalize
A single model on a single task is a solved-ish problem, ~1% on grounded work. An autonomous agent is a different object: it multiplies per-step reliability across many steps, and it conditions on its own mistakes, so a 95%-per-step agent is mathematically guaranteed to fail most long tasks, and the real curve is steeper still. That is not a model-quality problem and no checkpoint fixes it.
The teams whose agents are still running in 2028 won't be the ones who bought the most capable model. They'll be the ones who treated compound failure as a design constraint from day one, shorter chains, validation between steps, scoped context, escalation, and the discipline to leave a step out of autonomy when it can't be made safe.
When your self-driving agent hallucinates, don't reach for a smarter model first. Count the steps, check whether it's conditioning on its own earlier output, and ask which of those steps ever needed to be autonomous at all. The model isn't lying. Autonomy handed it its own earlier mistake and asked it to keep going.
What's your real per-step accuracy once you measure the whole chain, not the demo, and where did shortening the chain buy you more than any model swap? I'd bet most "the model hallucinates" complaints are compound error and self-conditioning wearing a model-shaped mask.
Sources & further reading
- "The Compounding Errors Problem: Why Multi-Agent Systems Fail", Zartis (2026) — p^n across accuracy levels; why model improvement can't fully solve it.
- "The Math That's Killing Your AI Agent", Towards Data Science (2026) — the compound table; fail-detectably-and-gracefully test.
- "The Compound Error Problem: Why 95% Accurate AI Agents Still Fail", Highland Edge (2026) — self-conditioning; "compound interest in reverse"; scoped context passing.
- "The Math Behind Why Multi-Step AI Agents Fail in Production", Flavius Dinu (2026) — Lusser's law; shorter chains + verification + guardrails.
- "Multi-Agent Reliability Math: Chaining 5 Agents Drops to 77%", MindStudio (2026) — lateral compounding; 85–90% realistic per-agent rates.
- "What to Keep, What to Forget: A Rate–Distortion View of Memory Compaction" (2026) — reversible ~0.95 vs irreversible 0.33–0.56 recall.
- "Hallucination as Context Drift" (2026) — +34% from naive multi-agent sync.
- Reasoning-tax and abstention figures: Vectara HHEM 2026; Suprmind AI hallucination benchmarks (DeepSeek V3 3.9% vs R1 14.3%; Claude 4.1 Opus 0% via abstention).
Top comments (6)
This resonates deeply — especially the shift from 'knowledge hallucination' to 'intent hallucination.' I've been seeing this in the wild while building AgentShare, an MCP server for Solana DeFi data.
Agents often execute perfectly on the wrong goal because the communication layer between the model, the tool, and the environment was never fully specified. The model isn't hallucinating facts — it's hallucinating the contract of the task.
The framing of 'autonomy as a bug' is useful. In our paper trading system (OpenClaw-based), we've found that explicit guardrails — like daily loss caps, risk tier filters, and mandatory confirmation loops — reduce behavioral drift far more effectively than prompt engineering alone.
It makes me wonder: as M2M (Machine-to-Machine) commerce scales, will the most valuable infrastructure be the one that enforces clear semantic contracts between agents? That's where I see the real opportunity — not just better models, but better coordination protocols.
"the model isn't hallucinating facts, it's hallucinating the contract of the task" is your line, not mine, and it's better than anything in the post. worth flagging because it points at a gap in my framing.
my whole p^n argument assumes the goal is fixed and asks what fraction of steps survive. intent hallucination is orthogonal to that: an agent can hit p = 1.0 on every single step and still be completely wrong, because it executed the wrong contract flawlessly. compound math doesn't catch it at all. and it's arguably worse than compound error, because compound error produces something visibly noisy or incomplete, while intent hallucination produces a clean, confident, fully-executed result for a task nobody asked for. it passes per-step validation trivially, since every step is correct relative to the wrong goal.
which explains why your guardrails work better than prompt engineering, and i think it's because they operate on a different layer than the fixes in my post. step validation catches compound error. loss caps, risk tiers, and confirmation loops are outcome bounds, they don't check whether a step was correct, they check whether the result is allowed to exist. those are the only thing that catches a perfectly-executed wrong intent, because the intent error is invisible at every step and only shows up in the outcome.
on semantic contracts between agents: yes, but with one condition that i'd say decides whether it works. a contract only helps if the receiving side can verify it, not just read it. if agent A declares "this is a price quote, freshness 30s, source X" and agent B has no way to check that independently, you've added a self-reported field, which is the same failure one layer up. the contract has to be enforced by something neither agent can author. that's the difference between a protocol and a convention, and under M2M scale i'd bet convention breaks first.
so agreed on coordination protocols over better models, with the caveat that the protocol needs teeth, not just a schema.
Appreciate the deep expansion on this! You hit the nail on the head regarding 'outcome bounds' vs 'step validation.' That’s exactly why prompt engineering feels like treating the symptoms rather than the disease when it comes to autonomy.
Your point about the protocol needing 'teeth' rather than just a schema is profound. Self-reported metadata is a house of cards at M2M scale. This is precisely why we are looking into programmatic verification layers—where the financial/execution contract is bound by cryptographic or on-chain primitives (like what we see emerging around the x402 protocol and programmatic settlement).
The receiving agent shouldn't have to trust the sending agent's word; the infrastructure itself should enforce the cost, state, and validity of the exchange.
Truly glad this sparked a gap-fill in the framing. Looking forward to your next pieces on this!
on-chain settlement is the right primitive for the part it covers, and worth naming precisely which part that is: it makes the transaction unforgeable (what was paid, to whom, under what terms). it doesn't make the claim unforgeable. an agent can settle honestly on-chain for a quote that was stale, or for a tool result it narrated instead of ran. the money is verifiable, the fact underneath it isn't.
which means you probably need two layers with different enforcement: settlement for the economic contract (cryptographic, on-chain, solved), and provenance for the epistemic one (was this observed or asserted, how fresh, which runtime produced it). the second one has no chain to lean on, so it needs a runtime that can sign what it actually executed, with the agent structurally unable to hold that key.
for your DeFi case the epistemic layer is probably the sharper risk anyway, since a settled trade on a stale price is a perfectly valid transaction and a completely wrong decision.
good thread, thanks for pushing on it.
This is exactly the distinction I've been circling — settlement vs provenance. Thank you for articulating it so clearly.
I think you're right that the epistemic layer is the sharper risk for DeFi. A stale price settlement is a valid transaction and a wrong decision — that's the hardest kind of failure to debug because nothing 'breaks' in the traditional sense.
What you're describing with 'a runtime that can sign what it actually executed' is essentially what we're building toward with AgentShare's MCP server — not just delivering data, but delivering attestable data that includes freshness, source, and execution provenance. The agent should be able to prove not just that it paid, but that the data it acted on was valid at the time of execution.
The challenge is: how do you build that provenance layer without introducing centralization? If the runtime signs the data, who holds the key? It's a hard problem, but I think it's the next frontier for agent infrastructure.
Thanks for pushing this conversation forward — it's rare to find someone thinking about these layers in the same depth. Looking forward to more.
The p^n law explains a design decision I made early on and could not have articulated this clearly at the time. In Opportunity Skill, human outreach is explicitly excluded from recurring scheduled execution. Discovery and lead engagement can run autonomously on a daily schedule, but the moment of contact always requires human confirmation. That is exactly the selective autonomy prescription at the end of your article. Keep the autonomous chain short where the stakes are irreversible, and automate only the steps where a wrong answer is cheap to discard.