Building an AI agent that answers a demo question takes an afternoon: a model, a prompt, one tool. Building one that handles thousands of real cases a month is a different project, and most of the work is outside the model. These are the seven steps we follow, taken from running agents on live customer service. The volumes are in our case study.
Step 1: pick one job and write down what "done" means
Not "automate support". One job: "reply to emails asking where an order is". Then define a finished case: the customer has the latest tracking status and the delivery time the policy states. If you cannot say what done means, you cannot test the agent.
Pick a job that repeats daily, has real volume, and has a system behind it to look things up in. How to choose is in AI agent use cases.
Step 2: connect tools, read-only first
List what a person looks up to do this job: the order, the tracking page, the policy document. Each becomes a tool the model can call. Start with tools that only read. An agent that can read everything and change nothing cannot damage your data while you learn how it behaves.
Design each tool to return a small, fixed answer. Fulfilment status should come back as one of a few fixed values. We once passed it as a sentence of description, and the code read every description as "shipped". Over three days, 25 address-change replies were held back, and 12 of those orders had not shipped.
Separate "there is nothing" from "the lookup failed". An empty result and an error need different handling.
Step 3: prepare the facts before the model sees them
This step decides the quality more than the choice of model. Do not hand the model the whole email thread, the whole order or the whole document. Turn them into the few facts this case needs: what the customer wants now, what was promised before, what has been done, what is missing.
We learned this from two cases. In a thread of 25 messages, the model did not notice that the company had already promised a full refund, and contradicted it. On an email about spare parts, sending only the order lines the email concerned cut the input from 37,604 tokens to 4,722, and all four parts in the reply were then correct.
Step 4: write the rules once
How the agent should think and speak is written in one text, and every part of the system refers to it. When the same rule exists in five places, the copies drift apart and the model follows whichever it read last. We found a reimbursement rule in five places that did not agree. Agents kept correcting the drafts until all five said the same thing.
Write principles, not ready-made sentences. A model copies an example sentence word for word into every reply.
Step 5: set the approval lines
Decide, action by action, what the agent does alone and what a person approves. Our starting lines: the agent answers questions of fact and published policy alone. Anything that costs money or sends goods is prepared by the agent and approved by a person. The detail is in who approves what.
Record every approved action from the version the person approved, not from the agent's first proposal.
Step 6: replay real cases before launch
Take real past cases and run the agent on them without sending anything. Compare its output with what your people did. Three details matter:
- Run each case twice. Model output varies between runs.
- Freeze anything that changes over time. If the agent can search internal messages, a replay today finds corrections posted after the original case.
- Make the replay unable to write or send. Check that by trying a write and seeing it fail.
A small set is enough to see the pattern. We look at five to ten cases before running more.
Step 7: launch with review, then measure what people change
At launch, a person reviews every output. Store the agent's draft and the version that was sent in separate fields. The difference between them is your measurement: in a review of 400 edited drafts, 53 percent of the edits removed a promise, 22 percent added stock status, and 10 percent replaced "we have no information" with an answer the knowledge base already held.
Each number named a fix. Only when a type of reply goes out unchanged, case after case, do you let the agent send that type alone. In the 30 days to 19 November 2026, our email agent sent 3,331 replies alone and people sent 23,120 after review.
What to do when it goes wrong
Remove before you add. Look for the repeated explanation, the two rules that contradict each other, the step that adds nothing. A new rule written for one case makes the next case worse. This is described in how an AI agent learns from agent edits.
Build it yourself or hire it out
The seven steps need someone who knows the job, someone who can connect the systems, and someone who watches the output every week. If you have all three, build. If not, the options are in build, open source or buy. Our free plan lays out the seven steps for your job within two working days.
Originally published at ai.dingboai.com. Dingbo AI builds and runs AI agents for customer service, paid per completed task.
Top comments (0)