An AI agent should escalate to a human whenever it hits a pre-defined confidence boundary, touches money or an irreversible action, or sees an input pattern outside what it was built to handle. If you're not designing for that moment before you ship, you're finding out about it from an angry customer instead.
Why "just make it more accurate" isn't the fix
Most teams treat escalation as a failure mode to eliminate rather than a feature to build. The instinct is: if the agent is wrong sometimes, add more training examples, tune the prompt, add another retry. That works up to a point, then hits diminishing returns, because some fraction of real-world inputs are genuinely ambiguous. A refund request that references a policy exception. A support message in a dialect the model handles poorly. A sales lead whose intent doesn't match any of your qualification categories.
No amount of prompt engineering removes ambiguity from the world. What you can control is what the agent does when it encounters it: guess and hope, or stop and ask.
Guessing is cheap until it's wrong. And it compounds if the agent operates in a chain, since one bad guess several steps in can send everything downstream (see AI agents vs. workflows for why chains fail differently than single-call systems). An agent that escalates instead of guessing costs you a small amount of latency on the hard cases and nothing on the easy ones.
What actually triggers a good escalation
Three signals are worth building around, and they're mechanically different from each other:
- Confidence threshold. A confidence signal doesn't have to be a probability score the model reports about itself. It can be something you compute externally, like how many concrete facts the agent actually extracted before it tries to act. That's the exact mechanism behind the outreach-agent gate described below: not "is the model sure," but "does it have enough material to work with." Below whatever floor you set, stop and route to a human instead of returning the best guess.
- Stakes classification. Tag actions by blast radius before the agent runs, not after. Sending a templated email is low stakes. Issuing a refund on a request that cites a policy exception, or messaging a customer on something with legal implications, is high stakes. High-stakes actions get a lower autonomy threshold regardless of confidence, because being 90% sure isn't good enough when the 10% failure is expensive or irreversible.
- Out-of-distribution detection. Log what the agent has actually seen in production. A support message in a dialect the model wasn't tuned on, or a sales lead whose intent doesn't match any qualification category you built for, are both out-of-distribution in the same way: the input looks nothing like your training or eval set, and that's a signal independent of how confident the model claims to be about its own answer.
None of these require a research team. They require you to decide, up front, what "I don't know" looks like for your specific agent, and to build a path for it that isn't a generic error message.
Build the queue before you need it
The most common mistake we see is agencies and in-house teams building the happy path first and bolting escalation on after a bad outcome forces the issue. By then it's reactive: a human is triaging a mess instead of catching it at the decision point.
Build the escalation surface as part of the initial scope:
- A place for flagged cases to land. This can be as simple as a Slack channel with the full context: the raw input (the refund request with its policy citation, the dialect message, the off-taxonomy lead), what the agent tried, and why it flagged rather than acted. Don't make a human dig through logs to reconstruct what happened.
- A clear resolution path. Someone needs to own responding to escalations within a defined window, or the queue becomes a graveyard and the agent's flags become pointless. If nobody answers, the customer experience is worse than if the agent had just guessed.
- A feedback loop back into the agent. Every human resolution is training data. If the same category of case keeps escalating, like the same dialect or the same off-taxonomy lead type, that's a signal to either expand the agent's scope for that category or accept it'll always need a human and design the UX around that permanently.
A pattern from our own agent work
We run an outreach agent internally that scrapes each prospect's site and drafts one tailored email per company. Early versions tried to always produce a draft, even when the scraped page had almost no usable content, a thin "coming soon" site, or a page that was mostly navigation with no actual business description. Forcing a draft out of thin content produced generic, obviously-templated emails that hurt more than they helped.
The fix wasn't a better prompt. It was adding a check: if the extracted facts fell below a minimum threshold of specificity, the agent skips the draft and flags the company for a human to either research manually or drop from the list. That one gate improved the average quality of what actually got sent, because the agent stopped forcing output in cases where it had nothing good to say. The lesson generalizes: an agent that can say "I don't have enough to work with" is more useful than one that always produces something.
Where this fits in scoping a project
If you're evaluating a vendor or planning your own build, escalation design should show up in the initial architecture discussion, not as a change request after launch. Ask any agency pitching you an agent: what happens when it's wrong, and how does it know? If the answer is "we'll monitor it and fix issues as they come up," that's a maintenance cost you're signing up for indefinitely (see our breakdown of AI agent maintenance costs). If the answer includes a specific confidence mechanism and a defined human handoff, that's a team that's thought about failure, not just the demo.
It's also worth checking this alongside your broader risk posture. Escalation design overlaps with the same questions covered in an AI agent security checklist: what can this system do without a human in the loop, and who's accountable when it does something wrong.
The bottom line
Escalation isn't a fallback you add when things break. It's the mechanism that keeps things from breaking in the first place, by giving the agent a legitimate third option beyond "succeed" or "fail silently." Design it at the same time you design the happy path, tie it to concrete triggers (confidence, stakes, distribution shift), and staff the queue like it matters, because the cases that land there are, by definition, the ones your agent couldn't handle alone.
If you're scoping an agentic system and want a second opinion on where the escalation boundaries should sit, let's talk.
Originally published on the Pykero blog.
Top comments (0)