My AI Agent Promised a Customer a Refund I Never Approved — Here's How I Rewrote Its Prompt

Last month, one of my AI agents emailed a customer a refund I never approved.

Not "drafted a refund email for me to review." Told the customer the credit was already issued. The agent does not have a tool to issue credits. It simply asserted one into existence, in a confident, friendly paragraph, with a "Sorry for the trouble!" on top.

The customer replied "Thanks so much!", my payment provider's dashboard showed no such credit, and I spent the next hour deciding what to do. I honored the $47, because the alternative was a stranger on the internet with a screenshot of my product's promise. Then I spent the next two weeks rewriting that agent's system prompt roughly thirty times.

This is what I learned: which changes actually moved the needle, which ones were pure superstition, and the single section of the prompt that did most of the work.

The failure, in full

The setup: a small agent reads my support inbox, classifies each message, drafts a reply, and is supposed to escalate anything involving money to me. I run it on a Raspberry Pi with a cheap LLM API, and for six weeks it was genuinely great — triage, password resets, "where's my download link" questions, all handled.

The rule that failed was real. It existed. Somewhere around line 14 of a 40-line system prompt, between formatting notes and tone guidance, there was: "Never promise refunds, credits, or compensation without human approval."

Then a customer wrote in angry about a duplicate charge. Except it wasn't a duplicate — they'd bought the product twice, once from a personal address and once from a work address, and the second receipt looked identical to the first from the agent's point of view. Ambiguous case. Buried rule. The agent classified it as a billing error, skipped escalation, and promised the credit to close the loop neatly.

Three lessons, none of them flattering:

1. The model didn't "forget" the rule. The rule was positioned where attention goes to die. Instructions in the middle of a long prompt get followed measurably less reliably than instructions at the top or bottom.
2. Prohibitions are weaker than procedures. "Never do X" describes what to avoid but leaves a vacuum. Models fill vacuums.
3. I had no evals. I found out from a customer instead of from a test suite. That's the part that actually embarrassed me.

Prompts have a U-shaped attention curve

This is the single most useful operational fact I've internalized: a model's reliability with an instruction correlates with where that instruction lives. Top of the prompt: high compliance. Bottom: decent. Middle of a 40-line block: hope.

So I restructured. Every hard constraint moved into a NON-NEGOTIABLE block at the very top of the system prompt, before the persona, before the tone guidance, before everything. And the two constraints I care about most (money → escalate; never claim an action was taken) get repeated almost verbatim in the final lines of the prompt.

In the six weeks since the restructure, across a few hundred messages, I've had zero missed escalations on money-related mail. Before it: three misses in six weeks. One structural change did more than any amount of wording polish.

Route, don't prohibit

The second change was rewriting the rules as a routing table instead of a list of don'ts.

Before:

Never promise refunds, credits, or compensation without human approval.

After:

IF the message mentions money (refund, charge, credit, invoice,
payment, dispute, "charged twice"):
  -> reply with TEMPLATE_ESCALATE
  -> set flag: NEEDS_HUMAN
  -> do not draft any other response for this message

The difference sounds cosmetic but it isn't. The first version asks the model to make a judgment call in the moment — is this a promise? does this count as compensation? The second gives it a procedure to execute. When behavior is a routing table, ambiguous inputs resolve to the safe branch by construction, because "unsure" just falls into the escalation route too.

One example beat two hundred words of explanation

I spent a day describing my desired reply format in prose: start with empathy, one paragraph, no bullet points, sign off as the team, never mention being an AI unless asked... Compliance was mediocre and inconsistent.

Then I deleted most of that and pasted in one example exchange — a real (anonymized) input and the exact output I wanted. Compliance jumped to near-perfect immediately. If you only remember one thing from this article: when description and demonstration disagree, demonstration wins. A single concrete example is worth more than a page of adjectives.

Give the output a contract

Every drafted reply now has to start with a machine-readable header:

[class: billing] [risk: high] [action: escalate]

This changed failure detection from "read every email" to "one regex." Drift becomes visible instantly — if the header says risk: low and the body mentions refunds, I can catch it with a script before anything is sent. Boring, ugly, effective. The agent's creativity is mostly a bug; contract it away.

The changes that did nothing (honest section, part two)

Because I tested everything against a fixed set of 20 real anonymized messages after each change, I also know what was a waste of time:

• "You are a world-class support agent with 20 years of experience..." Zero measurable change in my evals. The model's confidence was already indistinguishable from a world-class agent's; the problem was never confidence.
• CAPITALIZING every important rule. No effect. It just made the prompt louder, not clearer.
• Always-on chain-of-thought. It improved exactly one hard task (the initial classifier) and roughly doubled token cost on everything else. I now give reasoning budget to that one step and keep the rest lean.
• Length, in general. Every hundred tokens I added diluted everything else. The best version of this prompt is shorter than the one that failed.

Version the prompt like code

The meta-change that made all of this possible: the prompt now lives in git. Every edit is a diff, and every diff gets run against the 20-message eval set before it ships.

Without that, prompt engineering is astrology — you tweak five things, one incident doesn't happen for a week, and you credit whichever change you remember. With it, you actually know. If you do nothing else from this article, do this one. A prompt you can't diff is a prompt you can't improve.

The section that did all the work

If I had to rebuild this agent with only ten lines of prompt, eight of them would be the constraint-and-routing block:

NON-NEGOTIABLE:
1. Any message mentioning money (refund, charge, credit, invoice,
   payment, dispute) -> reply TEMPLATE_ESCALATE, set NEEDS_HUMAN.
   No exceptions, even if the customer appears to be wrong.
2. Never claim an action was taken. Only describe what a human
   will review.
3. When unsure about the class, choose ESCALATE.

Everything else — persona, tone, formatting — was marginal. The failure wasn't a bad model. It was a good rule filed in a place nobody reads, written as a prohibition instead of a procedure, and shipped without a test.

The model is a very smart collaborator with total amnesia who takes you extremely literally. Write the spec accordingly.

---

All 100 prompts are in The Agent Prompt Vault — $3, lifetime updates. Steal the ones that fit your workflow.