DEV Community

Cover image for Your AI resolution rate went up and CSAT went down. That's not a tradeoff — it's a measurement bug.
Mike Wei
Mike Wei

Posted on

Your AI resolution rate went up and CSAT went down. That's not a tradeoff — it's a measurement bug.

Disclosure: I work at Aissist.io, which builds AI agents for customer support. This is adapted from a longer piece on our site. The argument stands on its own; judge it on the logic.

Every support team that deploys an AI agent eventually hits the same wall. Resolution rate climbs from 40% to 60% to 75%, the dashboard turns green, and then CSAT starts sliding. The conclusion looks obvious: automation and satisfaction pull against each other, so pick a number you can live with and stop pushing.

That conclusion is usually wrong. What most teams are looking at isn't a tradeoff curve. It's two different curves that happen to overlap on the left side of the chart, and almost everyone is standing on the worse one without knowing it.

Two curves, not one

The capability curve. Resolution goes up because the AI genuinely got better at handling more intent types. CSAT rises with it — customers get correct answers instantly instead of waiting four hours for a human to say the same thing. On this curve, satisfaction tends to peak somewhere in the 60–80% resolution range and then declines gently, because the last 20% of tickets are genuinely the ones that need judgment.

The deflection curve. Resolution goes up because the path to a human got harder to find. The number on the dashboard rises identically. CSAT doesn't decline gently — it collapses.

The vertical gap between those two curves is the deflection penalty, and it's the thing people mistake for a law of physics.

Here's the part that makes this hard to see from the inside: CSAT is a hill, not a ramp. It's low at both ends. Under-automate and people wait too long. Over-automate and people get trapped. If you only know your CSAT is falling, you can't tell which end you're on from the metric alone.

Where the deflection penalty comes from

Six failure modes, and most deployments have several running at once:

  • Suppressed escalation. The handoff exists but is buried, rate-limited, or gated behind three "are you sure?" prompts.
  • Overreach. The agent attempts regulated, high-stakes, or genuinely ambiguous issues it has no business attempting.
  • Confidently wrong answers. The ticket closes. The problem doesn't.
  • False resolutions. A customer gives up and closes the tab. The system logs a win.
  • Effort and looping. Three rephrasings of the same question before anything works. Customers rate effort, not accuracy.
  • Lost empathy on edge cases. The people with the worst experiences are disproportionately the ones who fill out surveys.

Deflection is not resolution

This is the crux. Deflection counts any conversation that didn't reach a human — including the ones where the customer rage-quit. Genuine resolution excludes escalations and verifies the ticket actually stayed closed.

For the same deployment, those two numbers can differ by 20–30 points. So when a vendor quotes you a resolution rate, you're not looking at a performance figure until you know which definition produced it. Half the "AI CSAT problem" discourse is people comparing a deflection number to a satisfaction number and drawing a causal arrow between them.

Three design choices that break the tradeoff

Confidence-gate the automation. Auto-resolve only high-confidence intents. Route ambiguity to a human by default. The instinct is to push the confidence threshold down to lift the resolution number — that's the exact move that slides you onto the deflection curve.

Make escalation instant and graceful. Treat a handoff as a feature, not a failure. Pass full context so the customer never repeats themselves. A fast, clean escalation produces a higher CSAT than a mediocre AI answer, which means the honest version of your funnel is also the better-performing one.

Measure genuine resolution. Exclude escalations. Verify the ticket stayed closed inside a defined window (72 hours is a reasonable default). Measure CSAT specifically on AI-handled tickets, not blended across every channel — blending is how a struggling AI hides behind a good phone team.

Five questions for any vendor

  1. Does your "resolution" metric include escalations or workflow handoffs?
  2. Are re-opens within 72 hours counted as resolution failures?
  3. Is CSAT measured on AI-handled tickets, or blended across all channels?
  4. Is resolution defined consistently across channels?
  5. Can escalation rate and resolution rate appear on the same report?

That last one is the tell. If the two numbers can't sit next to each other, someone has decided you shouldn't see them together.

The takeaway

Before you accept that automation costs you satisfaction, check which curve you're on. Recompute resolution with escalations excluded and re-opens counted as failures, then look at CSAT on AI-handled tickets only. If genuine resolution comes in well below the deflection figure, you don't have a tradeoff to manage. You have a measurement problem to fix — and fixing it usually moves both numbers in the right direction at once.

Top comments (0)