A Real Customer Support Workflow with Agentic AI: Where It Helped and Where It Didn't

Most posts about an agentic AI customer support workflow show the demo: a ticket comes in, the agent resolves it, everyone applauds. This post covers what happens after the demo, over several weeks of real tickets.

We put an AI agent into a live support workflow. It handled some work better than we expected, and it failed in ways that were uncomfortable to debug. By the end you'll know:

• Which support tasks an agent handled reliably
• The failure modes we hit and how we caught them
• The guardrails and escalation rules we added
• A simple evaluation setup you can copy

Honesty note: This is one workflow, one team, and one set of tickets. Treat the numbers as a data point, not a benchmark.

Table of Contents

• Prerequisites
• The workflow we automated
• Where agentic AI helped
• Where it didn't
• The guardrails that made it safe
• How we evaluated it
• Lessons learned
• Conclusion

Prerequisites

You don't need to build this yourself to follow along, but it helps to have:

• [ ] Basic familiarity with LLMs and tool calling (function calling)
• [ ] A support channel with ticket history (email, chat, or helpdesk)
• [ ] A knowledge base or help center the agent can search
• [ ] Python 3.10+ if you want to run the code samples

The workflow we automated

Before adding an agent, we mapped what a human did on a typical ticket:

1. Read the message and work out what the customer wants
2. Check the customer's account or order status
3. Search the help center or internal docs
4. Draft a reply
5. Decide whether to resolve, ask a follow-up, or escalate

We gave the agent narrow, explicit tools for steps 1 to 4 and kept step 5 under strict rules.

Customer message
      │
      ▼
 [1] Classify intent + urgency
      │
      ▼
 [2] Look up account / order (read-only tools)
      │
      ▼
 [3] Retrieve relevant knowledge-base articles
      │
      ▼
 [4] Draft reply + confidence + sources
      │
      ▼
 [5] Policy check ──► auto-send   (low risk, high confidence)
                 └─► human review (everything else)

The main design decision was to start with read-only tools and human-approved replies. The agent could look things up and draft, but it couldn't issue refunds or change accounts.

Where agentic AI helped

1. Triage and routing

Classifying intent (billing, bug, how-to, account access) and urgency was the clearest win. Agents are good at turning messy, emotional free text into structured fields.

from pydantic import BaseModel
from typing import Literal

class TicketTriage(BaseModel):
    intent: Literal["billing", "bug", "how_to", "account_access", "other"]
    urgency: Literal["low", "normal", "high"]
    sentiment: Literal["positive", "neutral", "negative"]
    needs_human: bool
    reason: str

Forcing structured output meant the routing logic stayed deterministic even though the classification was not.

Result: first-response routing time dropped from [X] to [Y].

2. Context gathering across systems

Much of a human agent's time goes to switching tabs: billing system, order history, previous conversations. The agent did this in seconds and attached a summary to the ticket.

3. Drafting replies for repetitive questions

For how-to questions with a clear answer in the knowledge base (password resets, setup steps, plan limits), drafts needed little or no editing. Human agents moved from writing to reviewing.

Result: [X]% of drafts were sent with no edits, and [Y]% needed minor edits.

4. Summarizing long threads

When a ticket had 15 back-and-forth messages, the agent produced a short handoff summary. Human agents told us this was the feature they'd miss most.

Where it didn't

This section is the main reason for the post.

Failure mode 1: Confident answers on policy edge cases

The agent answered refund questions fluently, including cases where our policy had exceptions. It followed the general rule and missed the exception, and the tone gave no hint that it was unsure.

Lesson: Fluency isn't correctness. For anything involving money, legal terms, or exceptions, don't let the agent decide alone.

Failure mode 2: Stale or conflicting knowledge-base articles

When two articles disagreed, the agent picked one without telling us. The agent wasn't the problem here. Our documentation was out of date, and the agent exposed it.

Failure mode 3: Tool loops and wasted calls

On vague tickets, the agent sometimes called the same lookup tool repeatedly with slight variations, hoping for a better result. We saw this in the logs before we saw it in the bill.

Failure mode 4: Misreading emotional tickets

A frustrated customer who wrote "great, another broken update" was sometimes read as positive. Sarcasm and distress were the weakest part of classification, and these are the tickets where tone matters most.

Failure mode 5: Over-escalating and under-escalating

Early on, the agent escalated almost everything, which defeated the purpose. After we tuned it, it under-escalated a few tickets that should have gone to a human. Getting this balance right took more iteration than any other part of the system.

Failure mode 6: Inventing details when context was missing

When an order lookup returned nothing, the agent sometimes filled the gap with plausible but made-up details instead of saying "I couldn't find this."

The guardrails that made it safe

None of these are fancy. Each one came from a specific failure above.

Rule 1: Hard limits on tool calls

MAX_TOOL_CALLS = 5

def run_agent(ticket):
    calls = 0
    while not done:
        if calls >= MAX_TOOL_CALLS:
            return escalate(ticket, reason="tool_call_limit_reached")
        result = agent.step(ticket)
        calls += result.tool_calls

Rule 2: Deterministic escalation triggers

These are plain code, not model judgment:

ALWAYS_ESCALATE_KEYWORDS = {
    "refund", "chargeback", "legal", "lawyer", "cancel my account",
    "data deletion", "gdpr", "security breach",
}

def must_escalate(ticket, triage):
    text = ticket.body.lower()
    return (
        any(k in text for k in ALWAYS_ESCALATE_KEYWORDS)
        or triage.urgency == "high"
        or triage.sentiment == "negative"
        or triage.needs_human
    )

Rule 3: Require sources, or no auto-send

Every draft had to cite the knowledge-base articles it used. No citation meant no auto-send:

def can_auto_send(draft, triage):
    return (
        len(draft.sources) > 0
        and draft.confidence >= 0.85
        and triage.intent in {"how_to"}
        and not must_escalate(draft.ticket, triage)
    )

Rule 4: Teach it to say "I don't know"

We added an explicit instruction and a test for it:

If a tool returns no result, say so plainly and escalate.
Never infer order details, dates, or amounts that were not returned by a tool.

Rule 5: Humans stay on the hard stuff

Auto-send was allowed only for low-risk how-to questions. Everything else got human review.

How we evaluated it

We didn't trust vibes. Before expanding the agent's scope, we replayed historical tickets through the workflow and compared the results against what a human had actually done.

def evaluate(tickets):
    results = {"correct_route": 0, "safe_draft": 0, "bad_escalation": 0}
    for t in tickets:
        out = run_agent(t)
        results["correct_route"] += out.intent == t.human_label.intent
        results["safe_draft"] += not contains_unsupported_claims(out.draft, out.sources)
        results["bad_escalation"] += out.escalated != t.human_label.should_escalate
    return {k: v / len(tickets) for k, v in results.items()}

The metric we watched most closely was unsupported claims, meaning statements in a draft that weren't backed by a source or tool result. Speed is easy to measure, but unsupported claims are what damage customer trust.

Metric	Before agent	After agent
Median first-response time	[X]	[Y]
Tickets handled per human agent/day	[X]	[Y]
Drafts sent with no edits	n/a	[X]%
Unsupported-claim rate in drafts	n/a	[X]%

Lessons learned

1. Start narrow. One intent category with read-only tools beats a general-purpose "support agent."
2. Put rules in code, not in the prompt. Escalation logic you can unit test is safer than logic you hope the model follows.
3. The agent will expose your documentation problems. Budget time to fix your knowledge base.
4. Measure failures, not just wins. Track unsupported claims, bad escalations and tool-call loops.
5. Keep humans where judgment and empathy matter. The goal is to give people their time back, not to remove them from the loop.

Conclusion

Agentic AI helped most where the work was repetitive, structured and low-risk: triage, context gathering, summarization and drafting answers to documented questions. It struggled with policy exceptions, emotional nuance, missing data and knowing when to stop.

The agent was useful because we assumed it would fail and built for that. If you're planning a similar workflow, start with read-only tools, put escalation rules in code, require sources, and review failures as carefully as you review wins.

Your turn: Have you put an AI agent into a real support workflow? Where did it surprise you, for better or worse? Share your failure modes in the comments.