The best models in 2026 hallucinate on about 1% of grounded, single-shot tasks. Hand one a document and ask for a summary and it almost never invents anything.
Now let that same model drive itself: plan a task, call a tool, read the result, decide the next step, update its memory, repeat for twenty steps with no human in the loop. It will now go wrong on most runs. Same model, same weights. The thing that changed is autonomy, and autonomy is where hallucination is actually manufactured.
This is the part almost nobody states plainly: for a self-driving agent, the dominant cause of "making things up" is not the model's factual accuracy. It's the structure of running unattended across many steps. A better model barely moves it. Below is why, with the numbers, and what actually does.
TL;DR — Agent errors compound multiplicatively: at 95% per-step accuracy, a 10-step task succeeds ~60% of the time, a 20-step task ~36%. Even a fictional 99%-per-step agent fails ~18% of 20-step tasks. And it's worse than the clean math, because agents self-condition on their own earlier mistakes, so errors accelerate rather than just accumulate. This is structural, not a model-quality problem: no checkpoint fixes p^n. What fixes it is architecture, shorter chains, per-step validation, scoped context, and escalation, plus not reaching for a reasoning model, which for grounded steps often hallucinates 3–4× more.
The one equation that governs autonomous agents
Reliability engineering has a law for chained systems (Lusser's law): the reliability of a sequence is the product of the step reliabilities, not the average. For an agent taking n independent steps, each succeeding with probability p, end-to-end success is p^n.
That exponent is the entire story of why autonomy is hard. Watch what it does:
| Per-step accuracy | 5 steps | 10 steps | 20 steps |
|---|---|---|---|
| 99% (fantasy) | 95% | 90% | 82% |
| 95% (excellent) | 77% | 60% | 36% |
| 90% (good) | 59% | 35% | 12% |
| 85% (a strong result) | 44% | 20% | 4% |
Read the 95% row again, because 95% per step sounds great. An agent that gets each step right 95% of the time succeeds on a 20-step task 36% of the time. It fails most of them. And 85% per step, which is a genuinely strong result on a complex reasoning action, collapses to 4% over 20 steps: one success in twenty-five.
Demis Hassabis called compounding agent errors "compound interest in reverse," and it's exact. The same multiplicative machinery that makes compound interest miraculous over time makes autonomous error catastrophic over steps. Demos hide it because a demo is 2–3 steps. Production is 5+ steps over messy inputs, which is precisely where p^n bites.
The load-bearing consequence: no amount of model improvement fully solves this. Going from 95% to 97% per step helps, but you're still fighting an exponent. The gap between "impressive demo" and "reliable product" is not a capability gap you close with a better model. It's a structural property of chaining a non-deterministic component, and you engineer around it or you ship a coin flip.
Why it's actually worse than p^n: self-conditioning
The table above assumes each step's error is independent. In real agents, it isn't, and the dependence runs the wrong way.
Researchers documented a self-conditioning effect in multi-step LLM pipelines in early 2026: once an agent has produced an error and that error is sitting in its context, the model conditions on it. It sees its own earlier mistake as established fact and reasons from it, which makes the next mistake more likely, not less. Errors don't just accumulate, they compound on themselves. The real failure curve is steeper than p^n predicts.
This is the mechanism underneath "the agent went off the rails around step 6." It didn't hit a hard wall. It made one small wrong turn at step 4, wrote that into its own working context, and every subsequent step treated the wrong turn as ground truth. By step 8 it's confidently building on a fact it invented four steps ago. The hallucination isn't fresh at step 8, it's laundered from step 4 through the agent's own memory.
And this points straight at one of the most effective fixes, which is counterintuitive: scoped context. A step that doesn't know about the errors in steps 2 and 3 can't condition on them. Passing only the relevant slice of context forward, instead of the full accumulated history, breaks the self-conditioning loop. Less memory, more reliability, the opposite of the instinct to give the agent everything.
The autonomy-specific amplifiers
On top of the raw compounding, self-driving specifically adds failure sources a single-shot call never has:
Lossy memory compaction. When the agent's history won't fit in context, it summarizes, and repeated summarization is brutal. A 2026 rate-distortion study measured it: an agent archiving raw records and retrieving on demand held ~95% recall; an agent overwriting memory with LLM summaries dropped to 33–56% recall, worst when it compacted most often. At equal budget, lossy self-summarization loses roughly half the facts over a long run, and a single-turn benchmark can't see it because the loss only appears when compaction repeats. The agent then confidently fills the holes it created.
Context drift. ~65% of enterprise agent failures in 2025 were attributed to context drift or memory loss during multi-step reasoning, not to running out of context. As the window fills, attention de-prioritizes the original task; early tool outputs get overwritten by later ones; the agent quietly diverges before it ever hits the limit. Chroma's context-rot work shows degradation accelerating past ~30k tokens even in models with far larger windows, capacity is not fidelity.
Multi-agent contamination. Split the work across agents for reliability and you can make it worse: naive full-broadcast state sharing increased hallucination by 34% in a controlled 2026 study, because one agent's error propagates to all of them. Chaining five agents at 95% each yields 77% end-to-end, not 95%, the same p^n exponent, now spread laterally.
Every one of these is a property of running autonomously, not of the model's factual accuracy. A single API call to the same model has none of them.
Where the model does matter: don't make per-step accuracy worse
The model isn't irrelevant, it sets your p. But the counterintuitive part is that the "smarter" model many teams reach for to fix agent reliability often lowers p on exactly the grounded steps agents are made of.
Reasoning models hallucinate more on grounded tasks. Same base model: DeepSeek-V3 hallucinates at 3.9% on grounded summarization; its reasoning sibling R1 at 14.3%, ~4× worse. It's a pattern, not a one-off: on Vectara's 2026 dataset, every reasoning model exceeded 10%, while a non-reasoning model (Gemini Flash-Lite) led at 3.3%. The mechanism is the same self-conditioning drift, reasoning models generate long internal chains that wander from the source. So "let me upgrade to a reasoning model so my agent thinks more carefully" can lower your per-step accuracy and, through the exponent, wreck end-to-end reliability. For the grounded steps that make up most agent loops, a fast non-reasoning model is often the higher-p choice.
"Lowest hallucination" can mean "abstains most." Claude 4.1 Opus posts 0% on a knowledge benchmark, but by declining when unsure, which for an agent is often exactly right (a confident wrong step corrupts everything downstream; an "I'm not sure" step can be escalated). Calibration, knowing what it doesn't know, matters more for p than raw accuracy, because a self-conditioning agent punishes confident-wrong far more than honest-uncertain.
The takeaway on models: pick for high, calibrated per-step accuracy on your actual step type (usually grounded → non-reasoning), not for leaderboard position. But understand that even a great p is fighting p^n, so the model is necessary and nowhere near sufficient.
What actually fixes autonomous hallucination
None of these is a model upgrade. All of them attack the exponent or the self-conditioning.
- Shorter chains. The single highest-leverage move. Cutting a 20-step task to 8 steps does more for reliability than any accuracy bump, because you're reducing the exponent, not the base. If a task can be decomposed into shorter independent sub-tasks with checkpoints between, do that.
- Validation between steps. Schema-validate each step's output before passing it forward. A 95%-accurate step that's checked can't silently corrupt everything downstream, you catch the error at step 4 instead of discovering it at step 20. This directly interrupts self-conditioning.
- Scoped context, not full history. Pass only what the next step needs. A step that can't see earlier errors can't condition on them. Less context is more reliability here.
- Escalation and circuit breakers. The agent must know when it's uncertain and hand off. The dangerous failure isn't the agent that stops and reports, it's the one that fails silently and continues, laundering a wrong turn through twelve more steps. Test explicitly: does it recognize its own error, signal it, and halt rather than compound?
- Selective autonomy, not full autonomy. Aggressive automation for low-risk, reversible steps; hard human gates on high-stakes, irreversible ones. Cap the blast radius. The goal was never "fully autonomous," it's "autonomous where it's safe, gated where it isn't."
The thing to internalize
A single model on a single task is a solved-ish problem, ~1% on grounded work. An autonomous agent is a different object: it multiplies per-step reliability across many steps, and it conditions on its own mistakes, so a 95%-per-step agent is mathematically guaranteed to fail most long tasks, and the real curve is steeper still. That is not a model-quality problem and no checkpoint fixes it.
The teams whose agents are still running in 2028 won't be the ones who bought the most capable model. They'll be the ones who treated compound failure as a design constraint from day one, shorter chains, validation between steps, scoped context, escalation, and the discipline to leave a step out of autonomy when it can't be made safe.
When your self-driving agent hallucinates, don't reach for a smarter model first. Count the steps, check whether it's conditioning on its own earlier output, and ask which of those steps ever needed to be autonomous at all. The model isn't lying. Autonomy handed it its own earlier mistake and asked it to keep going.
What's your real per-step accuracy once you measure the whole chain, not the demo, and where did shortening the chain buy you more than any model swap? I'd bet most "the model hallucinates" complaints are compound error and self-conditioning wearing a model-shaped mask.
Sources & further reading
- "The Compounding Errors Problem: Why Multi-Agent Systems Fail", Zartis (2026) — p^n across accuracy levels; why model improvement can't fully solve it.
- "The Math That's Killing Your AI Agent", Towards Data Science (2026) — the compound table; fail-detectably-and-gracefully test.
- "The Compound Error Problem: Why 95% Accurate AI Agents Still Fail", Highland Edge (2026) — self-conditioning; "compound interest in reverse"; scoped context passing.
- "The Math Behind Why Multi-Step AI Agents Fail in Production", Flavius Dinu (2026) — Lusser's law; shorter chains + verification + guardrails.
- "Multi-Agent Reliability Math: Chaining 5 Agents Drops to 77%", MindStudio (2026) — lateral compounding; 85–90% realistic per-agent rates.
- "What to Keep, What to Forget: A Rate–Distortion View of Memory Compaction" (2026) — reversible ~0.95 vs irreversible 0.33–0.56 recall.
- "Hallucination as Context Drift" (2026) — +34% from naive multi-agent sync.
- Reasoning-tax and abstention figures: Vectara HHEM 2026; Suprmind AI hallucination benchmarks (DeepSeek V3 3.9% vs R1 14.3%; Claude 4.1 Opus 0% via abstention).
Top comments (0)