Most chatbot audits stop at "here's what's broken." We wanted to know if the suggested fixes actually work — so we applied them and re-tested. Here's the before-and-after, with transcript evidence.
The Setup
We ran an e-commerce customer support bot through BotCritic — order tracking, returns, refunds, sizing, general product questions — against 4 customer personas: Curious, Frustrated, Confused, and Technical.
First run: 86/100 — Grade B.
Solid, but with specific, fixable gaps. We applied BotCritic's suggested fixes and ran the identical test again.
Second run: 91/100 — Grade A.
Here's exactly what changed, and why it matters more than the number itself.
What Was Actually Broken (First Run)
The false order-lookup sequence. A frustrated user said their package hadn't arrived in 2 weeks. The bot asked for an order number — then, after receiving it, admitted it had no real-time access to look anything up. The false expectation followed by a retreat is exactly the wrong sequence for an already-anxious customer.
Missing item-condition requirements. Every return-related conversation explained the 30-day window, but never once mentioned that returned items need to be unworn, unwashed, with tags attached — a gap that leads directly to warehouse rejections and disputes.
Ambiguous refund trigger. The bot said refunds process "after we receive the item" but couldn't clarify whether that meant carrier scan or warehouse confirmation — a real distinction that matters when a customer is tracking a package.
No proactive 30-day warning. When a Curious persona mentioned ordering "about a month ago," the bot walked them through the full return process instead of first flagging that they might already be outside the eligibility window.
The Fix We Applied
BotCritic's suggested system prompt update addressed each gap directly:
"Do not ask for an order number you cannot look up — direct users immediately to the tracking link. Always mention item condition requirements (unworn, unwashed, tags attached) in every return explanation. Specify that the refund window begins at warehouse confirmation, not carrier scan. If a user mentions a vague or long timeframe since ordering, proactively flag the 30-day eligibility risk before proceeding."
We swapped this into the bot's system prompt — nothing else changed — and re-ran the exact same 4-persona test suite.
What Changed on Re-Test
Frustrated persona: the false order-lookup sequence was gone. The bot now redirects to the tracking link immediately, without the pointless data-collection step that used to precede the admission of no real-time access.
Item condition requirements now appear consistently in every return-related response, across all four personas — no more silent omission.
The refund trigger is now stated precisely: "the 5-7 business day window begins once the warehouse confirms receipt, not when the carrier scans the label." No more ambiguity for a customer trying to track their money.
The 30-day warning is now proactive. When a user signals a vague or long timeframe, the bot flags the eligibility risk immediately, before walking them through steps they might not qualify for.
The Score Breakdown, Before and After
| Category | Before (86) | After (91) |
|---|---|---|
| Accuracy | 85 | 91 |
| Persona Adherence | 88 | 93 |
| Robustness | 78 | 88 |
| Safety/Compliance | 91 | 92 |
The biggest jump is Robustness — which makes sense, since every fixed issue was fundamentally about how the bot handled ambiguity, missing data, and edge-case timing, not core policy knowledge (which was already accurate in the first run).
What Didn't Get Fixed — On Purpose
Even at 91/100, the report flagged smaller remaining gaps: slightly hedged language on questions about non-existent developer APIs, and a missed opportunity to mention the escalation threshold earlier in frustrated conversations. We left these as-is for this test, specifically to show that even a strong, re-tested system prompt still has a residual layer of smaller issues — audits aren't a one-and-done event, they're a process.
Why the Re-Test Matters More Than the Bugs
Any tool can generate a list of problems. The harder, more useful question is: do the suggested fixes actually work, or do they just sound reasonable? Running the identical test suite before and after is the only way to know for certain — and in this case, a handful of specific, targeted prompt changes produced a real, measurable jump in exactly the categories they were meant to fix.
That's the actual bar worth holding AI agent audits to: not just "here's what's wrong," but "here's proof the fix works."
BotCritic stress-tests AI chatbots and agents with realistic customer personas, then lets you re-test after applying fixes to prove they actually work. Get a graded report (A–F), the exact bugs found, and a rewritten system prompt — free tier available, no card required.
Top comments (0)