Imagine a customer sends this message: “I was charged twice, but one of the charges looks different from the other. Can you figure out what's happening?”
There isn't an obvious workflow here. The system needs to understand what the customer means, look at their account, identify the transactions, check relevant policies, compare the charges, and explain what happened.
Now compare that with: “I want to cancel my subscription.” That's a completely different problem. The system already knows what needs to happen:
- Validate account
- Check subscription
- Check cancellation rules
- Cancel subscription
- Update status
- Send confirmation
I don't need an autonomous AI agent deciding how to execute that process. I'd use normal software. That's why if I had to build an AI customer-support system today, I wouldn't start with an agent. I'd start with a control boundary. Something closer to:
User
↓
Intent Classification
↓
Deterministic Workflow
↓
RAG
↓
LLM
↓
Tool Layer
↓
Human Escalation
The goal isn't to minimize the use of AI. It's to put AI where it actually provides value.
Not Every Support Request Needs Reasoning
Customer support contains a surprising amount of predictable work. A refund request might follow a known set of rules:
Validate Order
→ Check Refund Window
→ Calculate Refund
→ Issue Refund
→ Update Order
→ Notify Customer
These are business rules. The system knows what to do and what conditions must be satisfied. There's very little value in asking an LLM: “Do you think this customer deserves a refund?” The model might interpret the request correctly, but the actual decision should come from deterministic business logic. For example:
This code is boring. That's exactly why I like it. When the rules are known, boring software is often the better software.
Where the LLM Actually Helps
Now go back to the customer who says: “I was charged twice, but one of the charges looks different from the other. Can you figure out what's happening?”
The system doesn't have a single obvious path. Maybe one transaction is a duplicate authorization. Maybe one is a completed payment. Maybe there's a currency conversion. Maybe the customer is looking at a temporary authorization hold. Maybe there really was a duplicate charge.
The system first needs to understand the problem. That's where an LLM becomes useful. It can interpret the user's language, identify the likely intent, retrieve relevant information, reason over the evidence, and explain the situation in a way the customer can understand. A support architecture could therefore look something like:
User Message
↓
Intent Classification
↓
Known Request? ── Yes ──→ Deterministic Workflow
│
No
↓
RAG + LLM Reasoning
↓
Tool Request
↓
Validation
↓
Execution
The model handles the ambiguous part. The application controls the actual operation.
RAG Gives the Model the Right Context
Customer support systems often need access to policies, product documentation, account information, previous conversations, and troubleshooting procedures. That's where RAG becomes useful.
Instead of expecting the model to know company-specific information, the system retrieves the relevant evidence and gives it to the model.
For example, a customer asks: “Can I get a refund if I cancelled yesterday?”
The system can retrieve the current refund policy, subscription details, and relevant account information before asking the LLM to explain the answer. The model isn't inventing the policy. It's reasoning over information supplied by the system. That's an important boundary.
Tools Shouldn't Trust the Model
This is where I think many agent architectures go wrong. Suppose the LLM decides:
Should the backend simply execute it? Absolutely not. The model can request an operation. It shouldn't have final authority over whether that operation is allowed. The tool layer should still enforce:
- Authorization
- Input Validation
- Business Rules
- Execution For example:
Even if the model makes a mistake, the backend still has control. That's the boundary I want in a production system.
Sometimes the Best AI Decision Is to Stop
There's another important component that doesn't get enough attention: human escalation. What happens when the system isn't confident? What happens when the request involves a large amount of money? What happens when the customer disputes a charge and the available evidence is contradictory?
My answer wouldn't be: “Give the agent more autonomy.” I'd escalate. A good support system should know when it doesn't have enough evidence or authority to continue. That could be as simple as:
Low confidence
↓
Human escalation
↓
Agent reviews context
↓
Customer gets resolution
The AI doesn't have to solve every problem. Sometimes its job is to recognize that a human should take over.
AI for Ambiguity. Software for Operations.
This is the distinction I'd use when designing an AI customer-support system:
Ambiguity → LLM
Predictable operation → Code
If the customer describes a complicated problem in natural language, let the model interpret it. If the system needs to retrieve company knowledge, use RAG. If an operation has known business rules, enforce those rules in software. If the model needs to perform an action, let it request a tool, but keep authorization and validation outside the model. And if the situation is sensitive or uncertain, escalate to a human.
You don't need an agent controlling the entire customer-support system to make it intelligent. In fact, I'd argue that giving an agent control over everything often creates unnecessary complexity: more tool-selection decisions, more failure modes, more state to manage, and more difficult debugging.
I'd rather build a system where the model handles the parts that require interpretation and deterministic software handles the parts that require consistency. That's a much healthier boundary for production AI. AI should handle ambiguity. Software should handle operations.
Top comments (0)