Most chatbot dashboards are theater. They show you total conversations, messages sent, and a satisfaction emoji score that nobody trusts. None of that tells you whether the bot is doing its job.
If you're building or running an AI chatbot for support or lead gen, you need metrics that map to outcomes: tickets deflected, pipeline created, humans spared. Here are the seven that actually matter, and how to instrument them.
1. Resolution Rate
The single most important number. What percentage of conversations ended with the user's problem actually solved - no human, no follow-up ticket?
Don't confuse this with "conversation ended." A user rage-quitting is not a resolution. You need a signal: an explicit thumbs-up, a ticket_closed event, or the absence of a re-open within 24 hours.
def resolution_rate(conversations):
resolved = [
c for c in conversations
if c["outcome"] in ("self_served", "confirmed_solved")
and not c["reopened_within_24h"]
]
return round(len(resolved) / len(conversations) * 100, 1)
Aim for 60%+ on support use cases. Below 40% and your bot is mostly a routing layer with extra steps.
2. Fallback Rate
How often does the bot say some version of "I didn't understand that" or punt to a human it shouldn't have? This is your failure signal.
Track every fallback with the triggering message attached. That log becomes your training backlog - the fastest way to improve a bot is to read the questions it choked on.
fallback_rate = fallback_events / total_user_turns * 100
# Anything over 15% means your intent coverage has real gaps
One nuance for LLM-based bots: fallback isn't always a clean "I don't know." A confident hallucination is worse than an honest fallback. Log low-confidence responses and grounding failures separately.
3. Escalation Rate (and Escalation Quality)
Escalation isn't inherently bad. A bot that never hands off is dangerous. The question is whether escalations are appropriate.
Split them:
- Warranted escalations - genuinely complex, high-value, or emotional cases.
- Avoidable escalations - things the bot should have handled.
The avoidable bucket is your improvement roadmap. If 30% of your escalations are password resets, that's a content problem, not an AI problem.
4. Containment Rate
Containment is the share of sessions handled end-to-end without a human ever touching them. It's related to resolution but not identical - you can contain a conversation and still leave the user unhappy.
Use both together. High containment + low resolution = you're trapping users in a bot that can't help. That's the worst quadrant, and it quietly torches your CSAT.
5. Pipeline Impact (for lead-gen bots)
If your chatbot exists to generate leads, conversation counts are meaningless. Follow the money.
Track the full chain: sessions → qualified leads → meetings booked → opportunities → closed revenue. Attribute each stage back to the bot with a source tag.
// Fire this on every bot-qualified lead
analytics.track('lead_qualified', {
source: 'chatbot',
intent: session.detectedIntent,
score: session.leadScore,
bookedMeeting: session.calendarBooked,
sessionId: session.id
});
The metric leadership cares about is cost per qualified lead versus your other channels. A bot that qualifies leads at a third of your SDR cost is a budget line that defends itself.
6. Time to Resolution
Speed is a feature. Measure the median (not the mean - outliers will lie to you) time from first message to resolution.
Compare bot-handled versus human-handled for the same intent categories. When the bot resolves a billing question in 40 seconds versus a 6-hour email queue, that gap is your value story.
Also watch the distribution's tail. If 10% of sessions run 20+ turns before resolving, you have a UX loop somewhere - the bot is confirming and re-confirming instead of acting.
7. Handoff Continuity
When the bot escalates, does the human get the context, or does the customer have to repeat everything? Repetition is the number one driver of post-escalation frustration.
Instrument it simply: log whether the transcript, detected intent, and collected fields were passed to the agent, and survey whether the customer had to re-explain. This is a binary you can drive toward 100%.
Putting It Together
Build a single dashboard with these seven and drop the vanity charts. A healthy support bot looks roughly like:
- Resolution rate above 60%
- Fallback under 15%
- Avoidable escalations trending down week over week
- Containment and resolution moving together, not apart
For lead-gen bots, one number rules: cost per qualified lead versus your existing channels.
The pattern here is consistent - every KPI ties back to a business outcome, not a bot activity. "5,000 conversations" tells you nothing. "3,000 tickets deflected at a quarter of the cost" tells you whether to expand the program or kill it.
Instrument for outcomes, review the fallback logs weekly, and let the numbers - not the demo - decide what you build next.
Originally published at getmichaelai.com
Top comments (0)