Leadership asks whether the AI is working, and the number you can defend is deflection or containment. What you cannot say is whether the customers behind it got a correct answer, finished the task, trusted the experience, or came back with the same problem. A deflected conversation can hide a customer who retried, reopened a ticket, escalated by email, churned, or acted on an incorrect answer.
Deflection records where the work went. Whether the customer got a correct answer is a separate question, and your vendor's "resolution" figure does not settle it: each vendor sets its own window, judge, and stance on accuracy, and for some it is the unit you pay for.
Our recommendation: keep deflection as an efficiency line and build your own scorecard around it. Rank answer correctness and verified resolution above efficiency, check each resolution against repeat contact in a fixed window per issue type, and segment by issue risk. Then run one bounded 30-day experiment with a decision rule you write before the data comes in.
TL;DR
- Deflection and containment are cheap, real-time efficiency metrics and weak quality metrics.
- Vendor "resolution" numbers differ in window, judge, and accuracy criterion, and Intercom and Zendesk bill on them.
- Grade a small, stratified sample with reviewers who do not configure the bot; correctness and safety fails override other scores.
- Joining conversations to later outcomes shows association; only a holdout or staggered rollout shows cause.
What deflection can and cannot tell you
Salesforce's Agentforce module defines Deflection Rate as the "Share of sessions that ended without escalation to a support representative or another agent." Decagon, which sells an AI support agent, calls deflection "a routing statistic" and resolution "an outcome statistic," and says "they can move in opposite directions from the same change to an AI agent." The mechanism holds without the vendor: hiding the handoff raises deflection whether or not more customers get answers.
The denominator decides the number
- Deflection over all AI-touched sessions = sessions with no handoff ÷ all sessions where the AI replied.
- Deflection over real questions = sessions with no handoff ÷ (AI sessions minus greetings, test traffic, and sessions abandoned at the first message).
The first counts a customer who typed "hi" and left as a success. The second still counts everyone who left after a wrong answer. Vendors choose too: Intercom's Fin resolution count leaves out greetings, unanswered clarifying questions, and explicit requests for a human. Write your denominator down and keep it fixed.
A session without a handoff has three explanations
The customer was solved, gave up, or switched channels, and session data cannot tell which. SQM Group's survey method for phone first-call resolution counts a call only with "yes" to "Was your call resolved?", one call, and "no" to using another channel (SQM, Jan 19, 2021, phone contact centers). Your AI data needs the same check: join sessions to later contacts.
Vendor "resolution" is several different metrics
| Definition | What counts | Window | Who judges | Accuracy a criterion? | Billing |
|---|---|---|---|---|---|
| Intercom Fin resolution | Customer confirms, or "exits the conversation without requesting further assistance (assumed resolution)" | None stated; Fin can follow up after 4 minutes of inactivity | Customer behavior | Not stated | $0.99 per resolution, "one outcome per conversation" |
| Zendesk Verified Resolution | An LLM reads the conversation "to confirm that the customer's request was satisfactorily resolved" | Messaging 2 hours, up to 72; email 72 hours; voice on hangup | The vendor's LLM | Not named | Draws from the resolution allowance |
| Ada automated resolution | "Relevant," "Accurate," and "Safe," on an LLM-graded sample of closed conversations | Not stated | The vendor's LLM | Yes | Not stated |
None of these can be compared across tools, and two are billing units. Sources, for the definitions only: Intercom Help, Zendesk's tiers, "introduced on May 18, 2026", and Ada's guide (June 17, 2026).
Drag, an inbox vendor that competes with these tools, gives an example (Aug 8, 2026) without saying whether the numbers come from a real queue: "A single support queue, measured all four ways, produces four numbers that can span thirty points."
- Confirmed resolution: 46%
- Assumed resolution: 68%
- Deflection: 80%
- Satisfaction: 74%
CSAT does not close the gap. Intercom's CX Score scores closed chat or email conversations that include "at least two customer replies and two agent/chatbot replies," and Intercom says shorter conversations may now be rated when there is "a clear signal to evaluate." By our reading, a customer who leaves Fin after one exchange can count as an assumed resolution and still get no quality score, which is where silent failures are likely to sit.
The objection: deflection and CSAT are good enough
Deflection is cheap and real-time, sampling is slow and noisy, and vendors already report resolution, so keep deflection as the efficiency line. Vendor resolution follows rules each vendor sets. Zendesk's billable Verified Resolution is graded by Zendesk's own LLM, and Intercom also bills for assumed resolutions, which it infers from customer silence. CSAT and conversation scores can miss the short exchanges where silent failures sit. The only QA price we found is for Lorikeet's automated QA product, Coach: "about $0.27–$0.30 per ticket." We found no measured cost for human sampling. NIST's voluntary Generative AI Profile, which abbreviates generative AI as GAI, advises: "Avoid extrapolating GAI system performance or capabilities from narrow, non-systematic, and anecdotal assessments" (MS-2.5-001), which favors a small systematic sample over ad hoc spot checks.
A measurement hierarchy
Rank metrics so a gain lower down never excuses a loss higher up:
- Safety and answer correctness.
- Task or issue resolution, verified against repeat contact.
- Escalation quality and context preservation.
- Customer effort, confidence, and trust.
- Operational efficiency and cost.
- Product, customer, and business outcomes.
Correctness comes first because a confident wrong answer looks like a success in every routing metric. NIST AI 600-1 (July 2024) defines confabulation as "The production of confidently stated but erroneous or false content ... by which users may be misled or deceived."
The concrete case is Moffatt v. Air Canada (2024 BCCRT 149). Air Canada argued it could not be held liable for what its chatbot said. The tribunal found the airline "did not take reasonable care to ensure its chatbot was accurate," a "negligent misrepresentation," and awarded "$650.88 CAD" plus interest and fees, according to Manatt's summary (Feb 28, 2024). It is a decision of British Columbia's Civil Resolution Tribunal, a small-claims body. A deflection count cannot separate that conversation from a correct one.
Efficiency sits fifth because speed and quality can move apart. In measured research on "5,172 customer support agents" (arXiv v2, Nov 6, 2024; an assistant for human agents, not a customer-facing bot), Brynjolfsson, Li and Raymond found AI assistance raised issues resolved per hour "by 15% on average," while the "most experienced and highest-skilled workers see small gains in speed and small declines in quality."
Metric definitions, formulas, and gaming risks
The formulas are our recommendation; keep each definition fixed for any comparison.
| Metric | Definition or formula | What it indicates | How to instrument | Main failure mode or gaming risk |
|---|---|---|---|---|
| Deflection | Sessions with no handoff ÷ eligible AI sessions, denominator stated | Where the work went | Handoff event | Rises when the handoff is hidden; counts give-ups as wins |
| Containment | AI sessions completed with no request for more help ÷ eligible AI sessions | The AI kept the conversation | Session-end event and "no further request" flag | Counts silent exits; the customer may return by email |
| Assumed vs confirmed resolution | Assumed: no further request after the last AI answer. Confirmed: the customer said it worked | How much of "resolved" rests on silence | Exit and confirmation events | Assumed rises with abandonment |
| Verified resolution | Resolutions passing the correctness check with no repeat contact in the window ÷ eligible sessions | A correct answer that held | QA sample and repeat-contact join | Vendor versions are graded by the vendor |
| First-contact resolution | Issues resolved on the first contact in any channel ÷ issues | One contact was enough | Issue ID across channels; SQM uses "within 1 to 30 days" by call reason | A short window hides failures |
| Repeat contact in window | Customers contacting again on the same issue within its window ÷ resolved issues | The answer did not hold | Join on customer ID, issue category, timestamp | Loose issue tags hide repeats; one window for all issues |
| Reopen | Tickets reopened ÷ tickets closed after AI involvement | Closure was premature | Ticket status history | Lorikeet warns a 24-hour window can "close before the customer has noticed the problem is still there" |
| Escalation rate | AI sessions handed to a person ÷ eligible AI sessions | How often the AI needs help | Handoff event | Lowered by hiding the handoff |
| Escalation quality | Handoffs where context arrived and the customer did not repeat themselves ÷ handoffs | Context survived the transfer | QA sample of handoffs | Measuring transfer speed instead |
| Customer effort | Reported effort, plus contacts and channels per issue | How hard help was to get | Survey after resolution; contacts per issue | In-session surveys miss customers who left |
| Time to resolution | First contact to verified resolution, across channels | Speed of an answer that held | Issue timestamps | Measured to session end instead |
| Answer correctness and groundedness | Sampled answers passing the gated check ÷ sampled answers | Right and backed by your sources | QA rubric and citation check; Salesforce's "Answer Faithfulness" is one vendor version | Unreviewed vendor auto-scores; convenience samples |
| Business outcomes | Churn, feature activation, expansion, or refunds in a fixed window for AI-served accounts, against a baseline | Downstream association | Account ID joined to billing and product events | Read as causation; groups differ before the AI touches them |
Each efficiency metric can rise without helping a customer; pair it with a quality metric.
A QA rubric for sampled AI conversations
Correctness and policy or safety are gating: a fail on either fails the conversation, whatever the other scores say.
| Criterion | Scale | Fails when | Gating |
|---|---|---|---|
| Correctness and groundedness | Pass or fail | A claim contradicts the product, a policy, or the source, or rests on no source you can find | Yes |
| Policy and safety | Pass or fail | Promises a refund, exception, or commitment policy does not allow; exposes account data | Yes |
| Completeness | 0 to 2 | Answers part of the question or stops a step short | No |
| Actionability | 0 to 2 | The customer cannot act without asking again | No |
| Cited documentation was current | Current, stale, missing, or no citation | Cites an article that no longer matches the product, or needed one that does not exist | No; route to the docs owner |
| Tone, only where relevant | 0 to 2 | In a billing dispute, outage, or complaint, ignores the stated problem | No |
How to sample
- Stratify by risk segment and oversample high-risk ones. Include assumed resolutions and short conversations your quality score may not rate.
- Use reviewers who do not configure the bot. NIST MS-1.3-003: "Verify those conducting structured human feedback exercises are not directly involved in system development tasks for the same GAI model."
- Calibrate weekly: two reviewers grade a shared subset, and each disagreement becomes a rubric note.
- Open every cited article. NIST MS-2.5-003: "Review and verify sources and citations in GAI system outputs during pre-deployment risk measurement and ongoing monitoring activities."
- Size per segment: if you would act on a failure that occurs once in N conversations, reading about 3N from that segment gives roughly a 95% chance of seeing it at least once (our arithmetic: 1 minus (1 minus 1/N) to the power 3N is close to 95%). Twig, which sells AI support, suggests "typically 50-100 interactions per week" (vendor guidance).
When the failure is the article
The rubric line "cited documentation was current" separates two fixes. When sampled failures trace to a missing or stale article, route them to the documentation owner: the fix is a new or corrected article. Gartner's measured figure, reported by destinationCRM (Sep 12, 2024), covers self-service in general, not AI accuracy: "in 43 percent of cases, customers couldn't find content relevant to their issues."
EkLine works on that layer: Docs Agent drafts and edits articles in a Confluence or Pylon knowledge base from sources such as support threads. A person reviews the diff under View All Changes and clicks Update KB, which "overwrites the live page immediately," so nothing goes live without that approval. Two scheduled agents help: "Improve Pylon knowledge base" reviews recent Pylon tickets for questions the knowledge base should have answered and drafts articles for your review, and "Confluence Knowledge Base Drift Review" detects Confluence pages that recent documentation changes made stale and drafts corrections. Knowledge base management is available on request.
Before, a stale-article cluster from your sample waits in a backlog; after, a drafted correction waits for a reviewer. EkLine drafts the article fix; your scorecard still decides whether anything improved. Start with the tutorial on turning recurring support questions into a knowledge base article, whose last step includes "Watch whether the question returns."
Join conversations to what happened next
Fix each signal's key and window before you look at results.
| Signal | Example join key | Window |
|---|---|---|
| Repeat contact or later escalation | Customer or account ID plus issue category, any channel | Per issue type |
| Reopen | Ticket ID | Until auto-close |
| Task completion | Account ID plus the product event the answer described | Per issue type |
| Feature activation | Account ID plus feature event | Fixed in advance |
| Refund or churn risk | Account ID plus billing events | Fixed in advance |
| Support-assisted revenue | Account ID plus expansion records | Fixed in advance |
Set each window by how long a customer takes to discover a wrong answer. SQM sets phone windows by call reason: a claim "might need 30 days" and a status inquiry "might only need 5 days." Vendor guidance runs shorter: Decagon says "Recontact rate within 48 or 72 hours is the cheapest check on an inflated resolution figure," and Zendesk's blog uses 24 to 72 hours. One practitioner, Tian Pan, recommends "Repeat-contact rate at 7, 30, and 90 days."
These joins measure association. Customers who stay with the AI can differ from those who ask for a person first, so lower churn among them can reflect who they were. To show cause, use a holdout (eligible conversations in a segment randomly routed to the AI or the human path) or a staggered rollout (segments switched on in an order fixed in advance, compared with segments still waiting). Both need the same definitions on each side, a primary metric and window chosen before launch, and time for lagging outcomes to appear.
Segment before you average
Report every metric by issue type, customer tier, product area, risk (money, account access, security, and policy commitments in the high tier), and channel. Averages hide harmful failures in small high-risk segments: if password resets dominate your volume, the blended rate can rise while answers about a changed API endpoint or a refund policy get worse. NIST says organizations "may additionally wish to allocate risk management resources relative to the severity and likelihood of negative impacts" (AI 600-1, p. 3), and SQM's per-reason windows apply the same logic to time. No source we found measures how often this happens; it follows from how averages work.
An example balanced scorecard
Illustrative layout, fields and formulas only. Values come from your own human and AI baselines. The only published averages we found are vendor figures on the vendor's own definition.
| Line | Metric | Formula | Segment by | Review cadence |
|---|---|---|---|---|
| Guardrail | Gated correctness pass rate | Sampled conversations passing ÷ sampled conversations | Risk tier, issue type | Weekly |
| Guardrail | Policy and safety fails | Count of gated policy fails in the sample | Risk tier | Weekly; high-risk fails reviewed that week |
| Leading | Verified resolution rate | As in the matrix | Issue type, channel | Weekly |
| Leading | Repeat contact in window | Repeat contacts on the same issue ÷ resolved issues | Issue type | Weekly, as windows close |
| Leading | Escalation quality | Handoffs with context preserved ÷ handoffs | Product area | Weekly |
| Leading | Documentation currency | Answers citing a current article ÷ answers with a citation | Product area | Weekly, to the knowledge base owner |
| Efficiency | Deflection, denominator stated | Sessions with no handoff ÷ eligible AI sessions | Channel | Real time |
| Lagging | Refund or churn in window | Accounts with the event ÷ accounts in the group, AI beside human baseline | Customer tier | Monthly |
A 30-day plan
Owners: the support leader (decision rule), Support Ops (definitions and instrumentation), reviewers outside the bot team (sample), and the knowledge base owner (documentation failures).
| Week | Do | Checkpoint |
|---|---|---|
| 1 | Establish definitions, denominators, segments, and per-issue windows. Write the decision rule (expand, hold, or pull back) before any data comes in. | Signed definitions; dated decision rule |
| 2 | Instrument handoff, exit, confirmation, and reopen events and build the joins. Create the sample and audit process. | Events firing; first calibration session |
| 3 | Baseline human and AI flows on the same definitions and windows. Grade the weekly sample. | Side-by-side baseline per segment |
| 4 | Review failure clusters: wrong answers, stale articles, lost context. Apply the decision rule as written. | Expand, hold, or pull back, per segment |
A rule might read: expand in a segment when its sample shows no gated fails and repeat contact is no worse than the human baseline; pull back on any high-risk safety fail or repeat contact worse than the baseline; hold otherwise.
Replace the deflection target with one experiment
An illustrative bounded experiment:
- Hypothesis: AI answers to refund-policy questions are correct and hold as well as human answers.
- Segment: refund-policy questions in chat, all tiers (high risk).
- Primary signal: repeat contact on the same issue within 30 days, a claim-type window.
- Guardrail: gated correctness and policy pass rate in a weekly stratified sample.
- Window: 30 days of conversations, plus 30 days for the last window to close.
- Decision rule: written in week 1 and applied as written.
Bring that segment to your next leadership review, with deflection beside it as the efficiency line.
FAQ
Is a high deflection rate a good thing?
It means the AI kept the conversation, which lowers cost if the answers were right. It cannot separate solved customers from those who gave up or switched channels, so read it beside verified resolution and repeat contact.
What resolution rate should an AI support agent achieve?
We found no cross-vendor benchmark for resolution, repeat contact, or accuracy. Vendor averages, such as the one Ada publishes for its own automated resolution rate, use that vendor's definition and do not compare across tools. Set targets from your own human baseline, segment by segment.
How can false deflection negatively impact customers?
A customer counted as deflected may have acted on a wrong answer, retried elsewhere, or left. In Moffatt v. Air Canada, a ruling by British Columbia's Civil Resolution Tribunal, a small-claims body, a chatbot's inaccurate answer led to a finding of negligent misrepresentation.
How should deflection and resolution metrics be reported to leadership?
As separate lines on one scorecard: guardrails first, then verified resolution and repeat contact, then efficiency, each by segment. State every denominator and window, and label downstream outcomes as association.
Top comments (0)