I wouldn't trust an AI customer support dashboard that claims 90% automation without asking one question: 90% of what?
Conversations handled? Tickets closed? Or customer problems actually solved?
Those numbers can tell very different stories.
An AI chatbot might answer 900 out of 1,000 customer requests, yet still leave hundreds of people without a working solution. The dashboard looks impressive, but the support team continues dealing with repeat tickets, failed refunds, and unresolved account issues.
That's what caught my attention while reading YourGPT's AI customer support benchmark. It looks beyond automation percentages and considers verified resolutions, human involvement, and the total cost of support.
I wanted to examine the numbers from a developer's perspective. What should count as a successful AI resolution? How do we measure it? And when does automating support actually save money?
Problem With a 90% Automation Rate
Let's say a company receives 1,000 support requests.
Its AI chatbot handles 900 conversations without immediately transferring them to a human.
The company reports a 90% automation rate.
But what happened to those 900 customers?
Some probably received the information they needed. Others may have abandoned the conversation, received an incorrect answer, or returned later with the same problem.
An automation rate alone doesn't tell us.
These metrics often get mixed together:
| Metric | What it tells us |
|---|---|
| Automation rate | How much work or interaction is handled automatically |
| Containment rate | How many conversations avoid human takeover |
| Resolution rate | How many customer issues are successfully completed |
| AI-only resolution | How many issues AI resolves without human involvement |
| Escalation rate | How often ownership transfers to human support |
Here's a simple example.
A customer asks an AI assistant to cancel a subscription.
The chatbot explains how to cancel it and ends the conversation.
Technically, the AI handled the request. But if the customer expected the chatbot to cancel the subscription, the job isn't finished.
Now imagine the chatbot says, "Your subscription has been cancelled," but the billing API actually failed.
That's worse. The customer thinks the problem is fixed when it isn't.
For me, the useful number is how many issues reach a verified outcome, not how many conversations end without a human agent.
What Should Count as a Successful AI Resolution?
I'd start with a simple rule: the customer's requested outcome needs to happen, and there should be evidence that it happened.
For an AI agent connected to external systems, that means checking more than the conversation transcript.
Consider these examples:
| Customer request | Evidence of resolution |
|---|---|
| Cancel my subscription | Billing system confirms cancellation |
| Where is my order? | Correct tracking information is retrieved and provided |
| Update my account details | Authorized changes appear in the account record |
| Process my refund | Payment system confirms the required refund outcome |
| Fix my login issue | Customer regains access or the agreed recovery process succeeds |
Not every request requires an API call. Sometimes a correct, useful answer is the entire task.
But when the AI promises to change something, that change must be verified.
I'd also track whether customers return with the same unresolved issue within a defined period.
Otherwise, a chatbot could mark a ticket as resolved today even though the customer comes back tomorrow because nothing changed.
This becomes especially important when building AI agents that can take actions instead of simply answering questions.
A 1,000-Ticket Benchmark Shows Why the Numbers Matter
Here's a worked example adapted from the YourGPT benchmark. These figures are illustrative, not results from a new independent test.
A company receives 1,000 valid customer support issues.
Of these, 600 are eligible for AI automation. The remaining 400 require human handling or fall outside the AI's approved scope.
After the evaluation period, the final outcomes look like this:
| Outcome | Issues | Share |
|---|---|---|
| AI resolved independently | 480 | 48% |
| AI resolved with human input | 60 | 6% |
| Human agents resolved | 380 | 38% |
| Unresolved | 60 | 6% |
| Unknown outcome | 20 | 2% |
| Total | 1,000 | 100% |
Now let's calculate the results.
AI-only resolution across all customer issues:
480 / 1,000 × 100 = 48%
Total verified resolution rate:
(480 + 60 + 380) / 1,000 × 100 = 92%
AI-owned resolution among eligible issues:
(480 + 60) / 600 × 100 = 90%
That last figure is interesting.
The system achieved 90% AI-owned resolution among eligible requests. Yet only 48% of the company's entire support queue was resolved without human involvement.
Also, the 90% figure includes 60 cases where AI retained ownership but received human input. It isn't a fully autonomous resolution rate.
Neither number is wrong.
The problem starts when someone presents the 90% figure without explaining which requests were included.
If I were reviewing this system, I'd ask for three numbers together:
- How many incoming issues were eligible for AI?
- How many were resolved without human help?
- How many were resolved across the entire support operation?
Without that context, comparing two AI support platforms is difficult.
AI Can Handle More Than FAQs, but Not Every Task Is Equal

AI support is often discussed as though answering a shipping question and resolving a billing dispute require the same capabilities.
They don't.
A chatbot answering product questions mainly needs accurate, current information.
An AI agent cancelling an order needs the correct customer identity, appropriate permissions, access to the order system, and confirmation that the action succeeded.
More complicated requests introduce more ways for things to go wrong.
For example, AI may be well suited to:
- Order status and delivery questions
- Product information
- Standard troubleshooting
- Policy explanations
- Routine account requests
Billing disputes, security issues, unusual refunds, and complex technical failures need closer evaluation.
This doesn't mean those workflows must always go to humans. With the right integrations and controls, some can be automated.
But I wouldn't combine all these tasks into one resolution percentage and assume it represents the system's capabilities equally well.
A 95% resolution rate on FAQs tells us very little about how an agent handles payment failures.
Benchmarks become more useful when performance is measured separately for each workflow.
Human handoffs also deserve more attention.
If an AI agent recognizes that a refund requires approval, gathers the necessary details, and transfers the case correctly, that's useful work.
A successful handoff should preserve context so the customer doesn't need to repeat everything.
The mistake is treating every escalation as a failure. Sometimes passing the request to a person is exactly what the system should do.
What Does an AI Customer Support Resolution Actually Cost?
This is the part I find more useful than model pricing comparisons.
It's easy to calculate the price of an LLM request. It's much harder to calculate the cost of completing a customer support issue.
Model tokens are only one expense.
A production support system may also need human agents, quality reviews, API integrations, knowledge base updates, monitoring, and correction of failed actions.
Let's use the illustrative cost model from the benchmark.
| Cost component | Amount |
|---|---|
| AI platform and API usage | $850 |
| Human support and approvals | $2,600 |
| Quality review and maintenance | $500 |
| Allocated setup costs | $400 |
| Error remediation | $250 |
| Total cost | $4,600 |
The operation handled 1,000 customer issues and achieved 920 verified resolutions.
So:
Cost per verified resolution
= Total operating cost / Verified resolutions
= $4,600 / 920
= $5.00
AI platform and API usage account for about 18.5% of the total modeled cost.
Human support and approvals account for approximately 56.5%.
That surprised me more than the $5 figure.
It shows why comparing chatbot subscription prices or token costs doesn't tell us much about the economics of an entire support operation.
The remaining human workload can still represent the largest expense.
And if the AI creates additional work through incorrect answers or failed actions, those costs need to be counted too.
AI vs Human Support: Is Automation Actually Cheaper?
Let's compare the AI-assisted operation with a human-led baseline handling the same volume of requests.
Again, these are illustrative numbers, not industry averages or independently measured results.
| Metric | AI-assisted support | Human-led support |
|---|---|---|
| Customer issues | 1,000 | 1,000 |
| Verified resolutions | 920 | 940 |
| Resolution rate | 92% | 94% |
| Total cost | $4,600 | $6,000 |
| Cost per resolution | $5.00 | $6.38 |
| Cost per 100 issues | $460 | $600 |
The AI-assisted operation costs $1,400 less, reducing total modeled operating costs by approximately 23.3%.
It also produces a lower cost per verified resolution.
But it successfully resolves 20 fewer customer issues.
Would I choose the AI-assisted system?
Not based on this table alone.
If those extra unresolved issues involve simple product questions, the lower operating costs might justify the difference.
If they involve failed payments, account access, or serious customer complaints, the consequences could be much greater.
And if the business had set a minimum 93% resolution requirement before the test, the AI-assisted operation wouldn't qualify.
This is why I think benchmarks should establish acceptable resolution quality before comparing costs.
The cheapest option isn't automatically the most valuable.
A Cheap AI Response Can Create an Expensive Support Ticket
Consider a common engineering problem.
An AI agent sends a refund request to a payment API.
The payment provider processes the request, but the connection times out before the agent receives confirmation.
The AI assumes the operation failed and retries.
If the workflow doesn't use appropriate idempotency safeguards or check the transaction state, the retry might produce a duplicate operation.
The original AI request might have cost pennies. Fixing the resulting issue could require human investigation, payment reconciliation, and another customer conversation.
There's another technical detail worth mentioning: an HTTP 200 response doesn't always mean the customer's task is complete.
An API might accept a request for asynchronous processing while the actual operation remains pending. Even a successful request response needs to be interpreted according to the API's contract.
For action-based AI support, I'd want developers to test:
- What happens when an API times out?
- Can repeated requests cause duplicate actions?
- Does the agent confirm the final operation state?
- What happens when customer authorization is missing?
- Can the system recover or escalate when an operation fails?
These aren't minor implementation details.
They determine whether an AI agent can safely complete the work it claims to have done.
How I'd Benchmark an AI Support System Before Using It
I wouldn't begin by throwing hundreds of random questions at a chatbot and counting how many answers sound reasonable.
I'd use a smaller, representative set of real support workflows and evaluate the outcomes.
Here's the process I'd follow.
Start with actual customer issues. Include FAQs, subscriptions, billing problems, technical troubleshooting, and account-related requests. Don't choose only the easiest tickets.
Define success before testing. Decide what completed means for every workflow. A correct answer may be enough for an FAQ, but an account change needs confirmation from the relevant system.
Test under comparable conditions. If comparing two platforms, give them equivalent approved knowledge, integrations, and permissions wherever practical.
Record the final outcome. Keep AI-only resolutions, AI-assisted resolutions, human resolutions, unresolved issues, and unknown outcomes separate.
Measure human work. Count the time spent approving actions, reviewing answers, correcting failures, and completing escalations.
Track the full cost. Include platform usage, human labor, infrastructure, maintenance, and error remediation without double-counting.
Review results by workflow. An AI system that performs well on order tracking might still struggle with account modifications.
I'd keep the final reporting dashboard focused on these metrics:
| Metric | Why I'd track it |
|---|---|
| AI-only resolution rate | Shows how much work AI completes independently |
| Overall verified resolution | Measures customer issues successfully completed |
| Repeat contact rate | Helps identify incomplete resolutions |
| Human handling time | Reveals the remaining workload |
| Time to resolution | Measures time until the issue is actually completed |
| Action failure rate | Shows how reliably integrations perform |
| Cost per verified resolution | Measures cost per successful outcome |
| Cost per 100 issues | Shows cost against a fixed amount of incoming demand |
For sensitive workflows, I'd add strict checks for unauthorized actions and critical errors.
I'd also repeat the benchmark after major changes to the model, knowledge base, or integrations. A system that worked reliably with one API version might behave differently after that API changes.
What Is a Good AI Resolution Rate?
I don't think one percentage can answer that.
A chatbot resolving 95% of basic FAQs and an agent resolving 75% of approved subscription workflows are doing very different jobs.
Which is more valuable depends on what customers actually need.
There's useful evidence that AI assistance can improve human support performance, too.
A 2025 study published in the Quarterly Journal of Economics examined 5,172 customer support agents and found that access to an AI assistant increased issues resolved per agent-hour by 15% on average.
That's an important finding, but it measures human productivity with AI assistance, not autonomous AI resolution.
I wouldn't use it to claim that an AI chatbot can independently resolve 15% more customer requests.
The lesson is that the type of work and the measurement method matter as much as the reported percentage.
A good resolution rate is one that meets the business's quality requirements across the workflows the AI is expected to handle, with clear evidence of successful outcomes.
Conclusion
After looking at the numbers, I think the biggest problem with AI customer support benchmarking is how easily a technically correct percentage can tell an incomplete story.
A platform might report 90% resolution among eligible requests while independently completing less than half the total support queue. It might also reduce costs while leaving more customer issues unresolved.
Neither result should be hidden behind a single impressive number.
Before trusting an AI support dashboard, I'd ask three questions: How many customers needed help? How many problems were actually fixed? And how much did those resolutions cost?
If a platform can't answer those questions clearly, its automation rate doesn't tell me enough to judge its performance.


Top comments (1)
very well explained👏🏻