TL;DR: a well-built support system handles 70-80% of tickets without a human. The 70-80% is not a model capability, it is a property of your ticket mix, so measure it before you build. The knowledge base is the product; the model is a component. And ship it drafting replies for a human before you let it answer anyone directly.
Support automation fails in a specific and predictable way. Someone connects a model to a help page, it answers confidently and wrongly, customers get angry, and the project is written off as a technology problem. It was not. It was an architecture problem, and the architecture is not complicated.
Step 1: count your actual tickets before you build anything
Pull the last few hundred tickets and sort them into three piles. Answerable from documented facts. Answerable but needs account data. Needs a human judgment call.
The first pile is your automation ceiling, and it is knowable in an afternoon. In most businesses it is 50-70%, and adding the second pile once you have integrated order and account lookups takes it to 70-85%. If your first pile is 20%, no tool will get you to 80% and anyone promising otherwise has not looked at your tickets.
This exercise has a second payoff that often exceeds the first. The most common question in the pile is usually something you could fix at the source. If four hundred people a month ask where their order is, the answer is better shipping notifications, not a faster way to answer the question.
Step 2: build a knowledge base, not a bot
The single biggest determinant of quality is the content the system answers from. A capable model over accurate, current, well-structured content is reliable. The same model over a stale help page produces confident nonsense, because that is what it was asked to do.
What goes in:
Policies as they are actually applied, including the exceptions your team makes in practice
Product facts: specifications, compatibility, what is and is not included
Process answers: shipping times by region, returns, warranty, how to change an order
The genuine answers to your top twenty questions, written the way your team would say them
Explicit statements of what the system must not answer, which matters as much as what it should
Write it as answers, not as marketing pages. And put a review date on it. The failure mode nine months in is not the model getting worse, it is a policy changing and nobody updating the source.
Step 3: retrieval, so answers come from your data
The model should not be answering from what it absorbed in training. It should be handed the relevant passages from your knowledge base and asked to answer from those, citing which it used. That is retrieval-augmented generation, and the reason it matters is not accuracy in the abstract, it is that you can trace any answer back to the source that produced it.
Two practical rules. If retrieval returns nothing relevant, the correct behaviour is to escalate, not to try. And answers should be traceable, so that when one is wrong you fix the source document rather than argue with a prompt. A system you cannot debug is one you will stop trusting.
Step 4: define escalation explicitly
Escalation must be rules you wrote down, not a judgment the model makes about its own confidence. Confidence is exactly the thing these systems are worst at reporting.
Escalate immediately, without attempting an answer:
Anything about refunds, chargebacks or money owed
Any message with an angry or distressed tone
Anything mentioning legal action, a regulator, or a public complaint
Anything where retrieval found nothing relevant
A repeat contact about an issue already raised: second touch goes to a person
Anything about safety, health, or a defective product
Any request the system has already failed once
The handover has to carry the whole conversation with it. Making an already-frustrated customer repeat themselves to a human undoes every minute the automation saved, and it is the detail most implementations get wrong.
Step 5: ship it as a draft writer first
This is the pattern that separates projects that land from projects that get switched off in week three. For the first few weeks the system does not talk to customers. It drafts a reply, an agent reads it, edits if needed, and sends.
Why this is worth the delay:
You find out what it gets wrong on real tickets while the cost of being wrong is zero
Your team gets faster immediately, so the project delivers value from week one
Every edit an agent makes is a defect report pointing at a knowledge base gap
The team ends up trusting it, because they watched it earn it rather than being told to
When the edit rate on a category drops to near zero, let that category answer directly. Promote category by category, never all at once. Categories with money or emotion in them may never graduate, and that is a correct outcome rather than a failure.
Step 6: channels and stack
Put it where your customers already are rather than where it is easiest to deploy. Email is the simplest to automate well and usually the highest volume. Live chat has the highest expectations and the least tolerance for a wrong answer. A help-centre search that actually answers is underrated and low risk. Social DMs are the hardest to do well because the tone is public.
The stack is unremarkable and should be: your helpdesk as the system of record, a retrieval layer over your knowledge base, a current model doing the writing, and Make or n8n as the connective tissue. None of these choices are where the project succeeds or fails.
Step 7: measure four things, not activity
Tickets deflected is a vanity number on its own. These four together tell you the truth.
Resolution rate without human touch, by category. One number for the whole queue hides everything useful.
Escalation rate, and whether escalations are arriving early or after a failed attempt. Late escalations are the expensive kind.
Reopen rate on automated answers. This is the honesty check: a high deflection rate with a high reopen rate means you moved work, not removed it.
Satisfaction on automated versus human replies, tracked separately. If the gap widens, stop promoting categories.
Maintenance, which is not optional
Budget a couple of hours a month and a named owner. The recurring work is small and specific: review escalations for patterns that should have been handled, review the answers agents edited, update the knowledge base when policy changes, and retire content for products you no longer sell.
The systems that decay are the ones with no owner. Nothing breaks loudly; the content just drifts out of date and the answers get quietly worse until someone notices in a review three months later.
What good looks like
Seventy to eighty percent of tickets resolved without a human, response times measured in seconds for that share, agents spending their day on the cases that actually need judgment, and a reopen rate no worse than it was before. Cost is typically $3,000-6,000 to build and $100-300/month to run, most of the running cost being model usage that scales with volume.
One client went from three to four hours a day on support to under thirty minutes. The team did not shrink. The person who had been answering the same ten questions started doing the work that had been waiting all year.
Not sure what share of your tickets is genuinely automatable? That is a counting exercise, not a sales call. The audit at 2pizza.team/audit takes two minutes and will tell you where to start.
Originally published at 2pizza.team. We build AI and automation systems for small teams - fixed price, two to six weeks. See the work.
Top comments (0)