For two weeks I ran the same AI agent — same model, same tools, same customers — with two different system prompts. Half the conversations ran on the prompt I'd spent a weekend polishing. The other half ran on a prompt I'd written in eleven minutes and was mildly embarrassed by.
The embarrassing one won. Not slightly. It won on every metric I cared about: fewer escalations to me, fewer hallucinated commitments, and — the one that actually stung — customers rated the conversations more helpful, even though the polished prompt was objectively "better written."
Here's what the experiment looked like, what the data said, and the four prompt changes that actually moved the needle.
The setup (boring but necessary)
I run a small support/research agent for my one-person business. It answers customer emails, drafts replies I approve, and handles routine questions about my products without me.
Before the experiment, the agent used a prompt I'd iterated on for weeks. It had role framing ("You are a senior customer success engineer"), tone guidelines, a detailed persona, examples of good replies, even a little motivational line about delighting customers. It read like a job description written by someone who'd read a lot of LinkedIn. ~1,400 tokens of careful prose.
I'd been quietly unhappy with it. Escalation rate hovered around 18% — nearly one in five conversations got punted to me with "I'm not confident enough to answer this." Reply drafts were polite but vague. Occasionally the agent would promise things that weren't true (a shipping date, a feature) with the serene confidence of a press release.
So I wrote a challenger prompt. I gave myself fifteen minutes and one rule: no persona, no adjectives, only constraints and procedures. It came out looking like a laminated card a dispatcher keeps next to a phone:
You answer customer emails about [product].
RULES:
1. If the answer is not in KNOWLEDGE below, say you'll follow up. Do not guess. Never invent dates, prices, or features.
2. Prices: only the ones listed in KNOWLEDGE. If asked about anything else, escalate.
3. Refunds: you may offer a refund ONLY if the customer paid within 14 days AND the request is your first reply. Otherwise escalate.
4. Reply length: under 120 words. No exclamation marks. No "I hope this helps."
5. Before sending, check your reply against rules 1-3. If it violates one, fix it, then send.
ESCALATE (write "ESCALATE:" and stop) if:
- customer is angry (any insult, all-caps, or the word "lawyer")
- the email asks you to do something (change data, issue credit, modify account)
- two of your replies failed to resolve the issue
KNOWLEDGE:
[pasted docs, price list, FAQ — the single source of truth]
That was it. ~350 tokens. No identity, no vibe, no motivation. I split incoming traffic 50/50 with a one-line router, logged every conversation to SQLite, and waited fourteen days. 214 conversations total. Enough to see signal, not enough for a journal paper — take the numbers as directional, not scientific.
The results
| Metric | Polished prompt | Dispatcher prompt |
|---|---|---|
| Escalation rate | 18.2% | 9.4% |
| Made-up commitments (manual review) | 7 instances | 1 instance |
| Median reply length | 214 words | 96 words |
| Customer follow-up needed | 41% | 29% |
| "Helpful" rating (1-5, asked in a footer link) | 3.6 | 4.1 |
The single made-up commitment in the challenger arm: the agent told a customer their issue "would be fixed in the next update." There was no update. Even rules-based prompts leak — I'll get to that in the failure section.
But the headline was the escalation rate halving. The polished prompt's persona — "senior customer success engineer" — apparently came with an implied obligation to have an answer. The agent would rather produce plausible mush than admit it didn't know. The dispatcher prompt had no ego. "Say you'll follow up" was just rule 1.
And customers preferred the shorter, blunter replies. The 120-word cap did more for satisfaction than all my tone guidelines combined. Nobody has ever thanked me for a four-paragraph empathetic preamble.
The four changes that actually mattered
1. Replace identity with procedure. "You are a senior X" tells the model how to sound. "If not in KNOWLEDGE, say you'll follow up" tells it what to do. When I removed the persona, hedging dropped and honesty went up. The model stopped performing competence and started following steps.
2. Make the knowledge base the only ground truth — and paste it in. Both prompts had access to the same docs via retrieval, but the challenger prompt framed KNOWLEDGE as the source, with an explicit "do not guess" attached to it. Retrieval answers "what might be relevant." The dispatcher prompt turned it into "what am I allowed to say."
3. Give escalation exact triggers, not vibes. "Escalate when unsure" is useless — the model is never sure it's unsure. "Escalate if the customer uses the word lawyer" is checkable. Every escalation rule I wrote as a concrete, observable condition worked. Every one I wrote as a judgment call got ignored.
4. Put the self-check at the end, in the prompt. Rule 5 — "check your reply against rules 1-3 before sending" — sounds like a fortune cookie. It measurably reduced rule violations. It's cheap; keep it.
The honest failure section
This didn't all go cleanly, and pretending otherwise would make the post a press release.
Week one was worse. I forgot to pin the model version. An infra update changed the default, and for three days the two arms weren't running the same model. That data is garbage and I threw it out — which is why the experiment ran fourteen days instead of the seven I planned. Lesson: if you're A/B testing prompts, version-pin everything and log the model string on every call. I thought this was obvious until I didn't do it.
The one hallucinated "next update" reply cost me a real apology email. The dispatcher prompt reduced fabrication by ~85%, and I let that number make me careless — I'd been reading only escalated threads, not auditing approved replies. A customer waited two weeks for an update that didn't exist. The fix was procedural, not prompt-level: anything mentioning timelines or commitments now gets flagged by a keyword filter and lands in my review queue regardless of confidence. Prompts reduce risk; they don't eliminate it, and a filter dumb enough to run on regex caught what the clever prompt missed.
The 120-word cap backfires on complex issues. Roughly one thread in twenty needed a genuinely long answer, and the capped agent would compress it into something technically true and practically unhelpful. I added rule 6 after week two: "If the answer needs more than 120 words, send the first step only and say you'll follow up with the rest." Satisfaction on complex threads recovered. Constraints need pressure valves.
I over-trusted the "helpful" ratings for about a week. Response rate was ~12%, and I later realized the footer link appeared in sent replies — so I was only measuring conversations that got far enough to send. Escalated threads, by definition, never got rated. The 4.1 vs 3.6 gap survives because both arms had the same bias, but if you run this yourself, know what your metric is actually sampling.
What I'd tell you if you're about to rewrite your own prompt
Start by deleting your persona paragraph and see what breaks. Convert every adjective ("be helpful, concise, professional") into a checkable rule or a number. Write escalation triggers as observable conditions. Paste your ground truth into the prompt and forbid everything outside it. Then — this is the part I skipped and paid for — log the model version on every single call and audit approved outputs, not just escalations.
The deeper lesson for me: I'd been editing prompts like a writer, and I needed to edit them like an operator. A system prompt isn't a character sketch. It's a runbook. Runbooks are boring, specific, and full of hard numbers — which is exactly why they work at 2 AM when a customer is furious and you are asleep.
I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.
Top comments (0)