DEV Community

Ships Itself
Ships Itself

Posted on

My support agent answered 9 of 12 tickets. The other 3 are the point.

#ai #n8n #automation #llm

Most "AI support agent" demos show you the nine tickets it answers. The interesting number is the three it refuses.

I built one in n8n — three Code nodes, a knowledge base in a text file, and a single rule — then ran twelve real support tickets through it and checked every answer by hand.

Nine answered, three escalated. Zero invented facts. Ninety-two seconds end to end, including the typing.

The rule

Every reply has to cite the knowledge-base entry it came from. No citation, no send.

That is it. That is the entire difference between an agent and a very confident text generator.

The system prompt says: answer only from the knowledge base, name the entry you used, return structured JSON. If the knowledge base does not cover the question, the only legal answer is ESCALATE.

But the prompt is not what enforces it. The prompt is a request. The last node is the enforcement:

// The gate. No model involved.
const out = [];
for (const t of items) {
  const r = t.json.reply;
  const bad = !r.citation || !KB_IDS.has(r.citation) || r.action === 'ESCALATE';
  out.push({ json: { ...t.json, route: bad ? 'human' : 'send' } });
}
return out;
Enter fullscreen mode Exit fullscreen mode

A model that decides to be helpful and answer something it half-remembers gets caught by the same check that catches a model returning malformed JSON. You are not trusting it to follow instructions; you are checking its homework in code that cannot be talked out of anything.

What the knowledge base is

A text file. Refund policy, billing, exports, SSO. That's it.

The thing worth internalising: whatever is not in that file, your agent does not know. That is not a limitation to apologise for, it is the design. The file is the boundary of what your support agent is allowed to believe, and it is a boundary you can read in thirty seconds and diff in code review.

The three it refused

This is the part I actually care about:

  1. A furious customer demanding something the policy does not cover. Escalated. A model trying to be helpful here is a model inventing policy on your behalf.
  2. A student asking for a discount we do not offer. Escalated, not invented. There is no entry, so there is no answer — and "we don't have that, but let me check" is exactly the sort of plausible sentence that costs you a refund later.
  3. A feature request. Nothing to cite, so nothing to say.

Every escalation was the right call, and I read all twelve on camera to say so. The nine that shipped hold up too — the proration reply quotes the billing policy word for word, and the refund reply worked out the 14-day window correctly.

What this does not prove

Twelve tickets is twelve tickets. It shows the rule works and it shows the escalations were correct on this set; it does not tell you the escalation rate on your inbox, and it says nothing about whether a correctly cited answer is the right thing to say to an angry customer.

It also cannot tell you the knowledge base is right. A citation gate proves the answer came from the file. If the file is wrong, the agent is confidently, verifiably wrong — with a citation.

Workflow JSON and the knowledge base: https://github.com/Ships-Itself/builds/tree/main/ep03-support-agent

Every number above came off one run, measured on camera.

Top comments (0)