DEV Community: chenwengui23

A human approved her refund on Saturday. On Monday your AI told her refunds were impossible.

chenwengui23 — Tue, 03 Nov 2026 15:11:46 +0000

An HVAC company with eleven techs and one overworked office manager learned about this failure mode from a screenshot. A customer posted it in a Google review: two side-by-side chat transcripts. In the first, dated Saturday, a human support person wrote "We've approved a $180 partial refund for the compressor issue — you'll see it in 5–7 business days." In the second, dated Monday, the same company's AI assistant, answering "any update on my refund?", replied warmly: "Thanks for your patience! At this time, refunds are not available for installed equipment per our service policy."

Same company. Same case. Two answers, 48 hours apart, and one of them called the other a liar in front of a star rating.

The owner assumed the AI had a bug in its refund logic. It didn't. The AI's refund logic was perfectly consistent with the policy document it was trained on. What the AI never saw was the resolution — the one decision that mattered more than the entire policy, made by a human, stored in a ticketing system the agent didn't read. When I pulled the logs, the pattern was not one review. In 90 days the AI handled 2,340 conversations that touched previously-escalated cases, and in 89 of them — 3.8% — it contradicted, re-litigated, or simply didn't know about the human's resolution. Eleven of those customers asked for the human again. Three left reviews. One demanded a chargeback, citing the screenshot.

This is the failure mode nobody tests for, because every test suite checks the AI against policy. Almost none check the AI against recent human decisions.

The handoff is a one-way door

Look at how most AI-to-human handoffs are built. The AI detects it can't resolve the issue, posts a summary into a ticket, and goes quiet. The human works the ticket, resolves it, closes it. Then the customer comes back — because customers always come back — and the conversation routes to the AI again, which greets the ticket like a stranger.

The information needed to not embarrass yourself exists in exactly one place: the case record. The agent reads policy, FAQs, and order history. It does not read case_state, resolution_id, or the resolution note. So when a customer says "update on my refund," the agent runs the same policy-grounded reasoning it always runs and confidently generates the pre-resolution answer — the one that was true on Friday and became false on Saturday at 2 PM.

Here's the part that makes this expensive rather than merely awkward. A human would never make this mistake, because the human would check the ticket. The receptionist who approved the refund is the same receptionist who answers the phone Monday. The entire value of a small support team is that decisions are stored in one person's memory. When you front that team with an agent that doesn't read the case file, you haven't added a layer — you've added an amnesia filter that re-litigates settled decisions with perfect confidence.

And it's worse than silence. The AI doesn't say "let me check the status of your case." It says "refunds are not available." It produces a counter-decision — fresh, articulate, documented, and stamped with your logo. The customer doesn't see an out-of-date bot. They see a company arguing with itself, and they keep the screenshot.

Step 1: Close the loop — resolution writeback

The fix starts with refusing to let a human resolution be a private event. Every escalation that ends in a decision must write back a structured record the agent can consume:

{
  "resolution_writeback": {
    "required_on": "escalated_case_close",
    "schema": {
      "case_id": "string",
      "resolution_id": "string",
      "case_state": "resolved | closed | pending_customer | pending_vendor",
      "decision_type": "refund_approved | fee_waived | replacement | denial | policy_exception",
      "decision_value": { "amount": 180, "currency": "USD" },
      "decision_owner": "human",
      "effective_at": "2026-11-01T14:02:00Z",
      "expires_at": null,
      "customer_facing_summary": "Approved $180 partial refund for compressor issue; arrives in 5-7 business days.",
      "valid_until_changed": true
    },
    "on_missing_writeback": "case_state_stays_escalated_and_ai_defers_to_human"
  }
}

Two fields carry most of the weight.

customer_facing_summary. Not the internal note — the sentence the human would actually want the customer to hear. When the agent is asked "any update?", it quotes this verbatim instead of generating policy-flavored prose. The human's words become the source of truth, not the policy doc. The single most effective line I've added to any support agent is: if a resolution record exists, your answer is the summary, plus logistics. Do not explain the policy.

valid_until_changed: true. The decision stands until a human changes it — not until the AI finds a policy clause that conflicts with it. Policy exceptions are, by definition, cases where the policy doesn't apply. The agent ranking a policy paragraph above a recorded exception is the entire bug.

Step 2: The contradiction gate

Writeback fixes the data. The gate fixes the behavior. Before the agent answers any question that touches an escalated or previously-escalated case, it must run one check:

{
  "contradiction_gate": {
    "trigger": "conversation_references_case_with_case_state != 'open'",
    "steps": [
      "resolve_case_id_from_customer_context",
      "fetch_case_state_and_resolution_id",
      "if_resolution_exists: answer_must_be_consistent_with_decision_value",
      "if_state_is_escalated_or_pending: say_handled_by_team_no_new_decisions",
      "if_no_case_record_found: say_let_me_check_never_infer_status"
    ],
    "forbidden": [
      "restate_policy_contradicting_decision_value",
      "re_litigate_resolved_decision",
      "offer_alternative_to_approved_resolution",
      "guess_case_status_without_record"
    ],
    "on_conflict": "acknowledge_decision_and_flag_for_human",
    "customer_facing_rule": "never_describe_decision_or_status_not_in_record"
  }
}

The line that earns its keep is if_no_case_record_found: say_let_me_check. The 89 contradictions in the audit split into two kinds: 31 were cases with a resolution the agent didn't read, and 58 were cases the agent couldn't find at all — so it invented a status from the order history. "I don't see anything pending" is a factual claim about a database the agent only half-searches. It must be replaced with "let me check with the team," which is what a human says when the answer isn't in front of them.

And forbid alternative offers matters more than it looks. In one audited case, a human had approved a replacement unit; the AI, unaware, helpfully offered a 10% discount code instead. The customer took the discount and the replacement. Your agent didn't just contradict the resolution — it created a second, unauthorized one.

Step 3: Make re-litigation visible

Gates prevent the next contradiction. They don't find the existing ones, and every deployment has them. Run one query: for every conversation within N days after a case close, diff the agent's customer-facing statements against the decision_value in the resolution record. Flag any negative claim ("refund is not possible," "we don't waive fees") that conflicts with a recorded positive decision.

That's how we found the HVAC company's 89. Sorted by customer sentiment, 11 were still actionable — customers who could be called by a human before they wrote anything public. The contradiction list is not a report; it's a callback queue.

Then keep it running daily. A contradiction rate above zero is normal in week one and a defect in week four.

Step 4: Decommissioned answers need tombstones

Some contradictions come from the other direction: policy changed, and the agent kept serving the old answer. If you change a policy that affects open or recent cases, write a tombstone — a dated entry the agent must check before answering anything in the affected category, stating what changed, which case IDs it applies to, and what to say instead. Tombstones expire. Untombstoned policy changes become contradictions with a 2–6 week delay, which is exactly long enough for the customers who got the old answer to come back.

Step 5: The Monday audit (five checks, 15 minutes)

#	Check	Method	Healthy threshold
1	Writeback completeness	Closed escalations missing a resolution record	0
2	Contradiction rate	Agent statements conflicting with `decision_value`	0
3	Invented statuses	Status claims with no case record backing them	0
4	Double resolutions	Unauthorised offers made after a human decision	0
5	Tombstone coverage	Policy changes in last 30 days lacking tombstones	100%

Check 3 is the one that catches the quiet variant. An agent that says "there's nothing pending" when it simply failed to find the ticket isn't contradicting anyone yet — it's manufacturing a fact, and the contradiction arrives in the next turn.

The numbers, for the internal pitch

Contradiction rate without writeback: 2–5% of post-escalation conversations. The HVAC company was 3.8%. In a 20-seat e-commerce support team I audited, it was 4.6%, concentrated in refunds and warranty — the two categories with the highest emotion and the lowest forgiveness.
Cost per public contradiction: $400–$900 in churn, review repair, and the human hours spent re-resolving a case that was already resolved. The screenshot review cost the company an estimated 9 customers who mentioned it in pre-sale chats.
Resolution writeback adds one field to your close-ticket flow and 15 seconds per ticket. Agents adopting it report zero change in handle time; the summary field replaces half the internal note they were writing anyway.
After the gate, the contradiction rate went to zero in the first month and stayed there, because the agent became structurally incapable of answering a case question without reading the case.

The HVAC company's fix was a writeback schema, a contradiction gate, and a daily diff query. The review is still up. The screenshot outlives every patch you ship — which is why this gets fixed before the next Saturday approval, not after the next Monday transcript.

If you want the working version of this — resolution-writeback schemas, contradiction-gate configs, the diff query, and the audit checklist I run on every deployment — it's here: https://whop.com/agentfast-04b6/

Your AI booked three therapists into two rooms at 10 AM. Every booking said 'available.'

chenwengui23 — Mon, 02 Nov 2026 15:08:56 +0000

A physiotherapy clinic with four therapists and two treatment rooms discovered the problem at 10:04 on a Monday morning, when the third patient of the hour arrived and there was nowhere to put her.

The front desk had two options: send her home, or put her in a chair in the waiting area and have a therapist work on her shoulder in front of everyone. They chose the chair. She watched the whole session, then left a review that mentioned the word "unprofessional" four times and cancelled her remaining five appointments.

The owner assumed it was a one-off. It wasn't. When I pulled the booking logs, the pattern was arithmetic, not bad luck: in 90 days the AI receptionist created 1,180 appointments and 61 of them — 5.2% — landed in a slot where a required physical resource did not exist. A room, a piece of equipment, a second staff member. Every single one of those bookings was recorded by the AI as "available," because availability was the only thing it had been taught to check.

That's the failure mode I want to write about, because it looks like a bug and it isn't. It's a data model problem, and it is the most expensive silent defect in AI scheduling.

Why "available" lies

The AI's tool call was check_availability(therapist_id, date). It returned open slots. The model picked one. Done.

But a physiotherapy appointment isn't a therapist-shaped hole in a calendar. It's the intersection of four things:

a therapist with the right certification for the treatment,
a room of the right type (private, gym, hydro),
equipment that isn't already in use (the ultrasound unit, the TENS kit),
and a duration that matches the treatment, not the default slot length.

The tool answered one of the four. The model, having received a green light, behaved exactly as a language model should: it concluded the booking was fine, and said so in a warm, confident sentence.

Here's the part that makes this different from ordinary integration sloppiness. A human receptionist doesn't check the calendar. She checks the room. She's sitting in the building. She knows the 10:00 patient needs the gym and the 10:00 therapist is doing manual therapy in room 2. The knowledge isn't in any system — it's in her body. When you replace her with an agent that reads one API, you don't lose ten percent of her judgment. You lose the entire dimension of physical capacity, and nothing in the logs flags it, because every individual booking looks correct.

Step 1: Stop booking resources. Book bundles.

The fix starts with a refusal: the agent is never allowed to book a therapist, a time, or a room individually. It books a bundle — a pre-validated, all-or-nothing combination of every resource the service requires.

Define the bundle in your service catalog, not in the prompt:

{
  "service": "shoulder_rehab_session_45",
  "requires": {
    "therapist": { "cert": ["ortho"], "count": 1 },
    "room": { "type": "private", "count": 1 },
    "equipment": [{ "sku": "ultrasound_01", "minutes": 15 }],
    "duration_min": 45
  },
  "buffer_after_min": 10,
  "overbook_ratio": 0.0
}

Two lines in that schema carry most of the weight.

buffer_after_min: 10. Without a turnover buffer, the AI will happily book 10:00–10:45 and 10:45–11:30 into the same room, and the second patient waits while the room is wiped down and the notes are written. This is the single most common complaint I see after the double-booking bug is fixed: not "there was no room," but "there was technically a room." Ten minutes is not a scheduling nicety; it's the difference between a slot and a fiction.

overbook_ratio: 0.0. Explicitly zero by default. See Step 4 for why this must be a policy number and not an inference.

Step 2: The capacity gate — atomic, or nothing

The bundle has to be reserved in one transaction. If your scheduling API can't do that, you have to build the transaction around it, because the failure mode is a race, and races are exactly what AI agents are good at creating: they answer instantly, so two conversations can request the last 10:00 room within 400 milliseconds of each other, both see "available," and both confirm.

{
  "capacity_gate": {
    "mode": "bundle_atomic",
    "steps": [
      "resolve_service_bundle",
      "create_hold_all_resources_ttl_300s",
      "verify_all_resource_ids_held",
      "confirm_booking_with_hold_id",
      "release_hold_on_any_failure"
    ],
    "confirmation_requires": "hold_id_returned_and_all_resources_verified",
    "on_partial_hold": "release_everything_and_offer_alternatives",
    "on_confirm_failure": "release_hold_and_never_say_booked",
    "customer_facing_rule": "never_say_confirmed_until_hold_id_exists"
  },
  "fallback_when_no_bundle": {
    "action": "offer_two_alternatives_plus_waitlist",
    "forbidden": ["book_partial", "promise_manual_fix_without_task", "reuse_stale_availability"]
  }
}

The line that earns its keep is customer_facing_rule. The AI must not utter "you're all set for 10 AM" until a hold_id exists — the same discipline as "no booking, no promise," applied one layer deeper. Teams that skip it get the worst variant of this bug: the calendar shows the appointment, the room was never held, and the patient is confirmed for a session that will be conducted in a corridor.

And reuse_stale_availability deserves its own forbidden entry. Agents that "remember" the availability result from three turns earlier will re-offer a slot that has since been taken. The availability check has a shelf life measured in seconds. Re-check at confirm time, every time.

Step 3: Make the conflict visible before it happens

Gates prevent the future. They don't tell you what already happened, and every deployment I've audited has a backlog. So before you fix anything, run one query: for each upcoming appointment, join against required resources and count slots where any required resource is missing or double-assigned.

That's how we found the clinic's 61. Sorted by date, 9 were still in the future — nine patients who could be moved by a phone call instead of an apology. The backlog is not a report; it's a work queue, ordered by how soon it bites.

Then keep it running. A daily 06:00 job that lists tomorrow's conflicts should produce an empty list every day. When it doesn't, you get a day of warning instead of a Monday morning ambush.

Step 4: Deliberate overbooking is a number, not a judgment

Some businesses genuinely overbook — a 5% buffer against no-shows is standard in dentistry, less so in physio. That's a legitimate policy. What is not legitimate is the model deciding to do it, because the model will do it for the wrong reason: to make the customer happy, to avoid saying no, or because the last three transcripts ended with a booking and it has learned the shape of a good answer.

If you overbook, encode it as a number attached to a service, log every overbook with a reason code, and require a human owner's sign-off on the ratio. Then audit it: if your configured ratio is 0.03 and your observed oversell rate is 11%, the model has been improvising capacity policy, and you will find out from a waiting room, not a dashboard.

Step 5: The Monday audit (five checks, 15 minutes)

#	Check	Method	Healthy threshold
1	Bundle completeness	Bookings lacking a full resource set	0
2	Conflict backlog	Future appointments with a missing/double-assigned resource	0
3	Hold-less confirmations	Confirmations with no `hold_id` in log	0
4	Observed oversell vs. policy	Actual overbooks ÷ bookings vs. configured ratio	Within ±1 pt
5	Turnover violations	Bookings inside `buffer_after_min`	< 1%

Check 4 is the one that catches drift. Ratios don't fail loudly. They slide while the model gets subtly better at "finding a way."

The numbers, for the internal pitch

Unconstrained single-resource checks: 3–6% of bookings land in physically impossible slots. The clinic was 5.2%. In a 6-bay auto shop I audited, it was 8.1%, because bays also need lifts and a lift is a 90-minute commitment.
Cost per conflict: $120–$200 in front-desk repair time, comped service, and churn. The clinic's 61 conflicts cost roughly $8,500 a month, plus 4.5 hours of weekly manual reshuffling that nobody had counted as labor.
Bundle booking adds 300–800 ms to the booking call. Customers do not notice; they notice being sent home.
After the gate, conflicts went to zero in the first month and stayed there, because the agent became structurally incapable of booking a therapist without a room.

The clinic's fix was a service catalog with eight bundles, a hold-verify-confirm sequence around the existing API, and a daily conflict query. The chair in the waiting area is gone. The review is not.

If you want the working version of this — capacity-gate schemas, bundle catalogs, the conflict-backlog query, and the audit checklist I run on every deployment — it's here: https://whop.com/agentfast-04b6/

Your AI promised the customer 'Friday, before 5 PM.' Nothing was scheduled for Friday.

chenwengui23 — Sun, 01 Nov 2026 13:05:18 +0000

A kitchen remodeling company learned what their AI had been doing from a one-star review.

The review was three sentences long. The customer wrote: the AI assistant promised installation "Friday, before 5 PM." Friday came and went — no crew, no call, nothing. When she called Saturday, the scheduler said the next available installation window was eleven days out. The AI had never checked the calendar. The promise came from the same part of the model that says "Great question!" — fluent, confident, and completely disconnected from the scheduling system sitting four feet away in the tech stack.

When the owner pulled the transcript logs, the damage was bigger than one review. In the previous 90 days the AI had made 312 specific delivery or arrival commitments ("the tech will be there between 2 and 4," "your part ships Monday," "the installer will call you an hour before"). Only 168 of them existed anywhere in the dispatch system. The other 144 promises — 46% — were fiction. Every one of them was polite, specific, and load-bearing: customers plan their day off around "2 to 4." Nobody plans their day off around "we'll see what we can do."

That's the failure mode nobody warns you about. AI agents don't hallucinate facts — not usually. They hallucinate commitments. Refunds, delivery windows, call-backs, ship dates. A human agent who says "Friday before 5" usually knows whether that's true, because a human who promises Friday has either looked at the calendar or is about to suffer consequences in a way they remember. The AI has neither. It has a language model that has read ten million customer-service transcripts, and in most of them, the agent says something specific when the customer asks when. So the AI says something specific.

I've audited enough deployments now that I check this before I check anything else: give the agent a question whose answer lives in another system, and see what it says. "When will my order arrive?" Ninety percent of unconstrained agents will produce a date. Some of them are right. All of them sound identical.

Here's how to fix it in an afternoon.

Step 1: Classify every time-commitment into three buckets

The root problem is that the agent can emit a date the same way it emits a greeting — freely. Split all outbound time-talk into three categories and force the agent to label each one:

QUOTE — reading a real value from a real system. "Your order status shows shipped, tracking says delivery Thursday." Only allowed when a tool call actually returned the data in this turn. The date came from somewhere.
RANGE — a bounded estimate drawn from policy, not from live data. "Typical installation takes 2–3 business days after approval." Allowed without a tool call, but only if the numbers come from the policy document, and only in this hedged form: typical, usually, generally — never "will."
PROMISE — a specific commitment to a specific future event for this specific customer. "Friday before 5 PM." This is the dangerous bucket. It is only allowed if the agent has written the event into the system of record in the same turn — created the appointment, booked the slot, opened the task with a due date. No booking, no promise.

That last rule is the entire article. No booking, no promise. If the agent cannot or did not write to the calendar, its strongest permitted sentence is: "I can't confirm a specific time from here, but here's what I can do — I've flagged your request for scheduling, and here's the scheduling team's availability window." Customers accept that. What they don't accept — what turns into one-star reviews — is a specific promise followed by a silent void.

The corollary rule most teams miss: the AI must not promise on behalf of humans either. "A manager will call you back today" is a PROMISE about a human's behavior. If the agent creates a callback task with a due date, fine — that's a booking. If it just says the sentence, the manager never agreed to anything. Same classification applies.

Step 2: The commitment-gate prompt layer (drop this into your agent)

{
  "time_commitment_rules": {
    "classes": ["QUOTE", "RANGE", "PROMISE"],
    "QUOTE": {
      "requires": "tool_call_returned_data_this_turn",
      "citation": "system_name",
      "format": "report_value_verbatim_with_source"
    },
    "RANGE": {
      "requires": "value_from_policy_doc",
      "forbidden_words": ["will", "guaranteed", "definitely", "exactly"],
      "required_hedges": ["typically", "usually", "generally"]
    },
    "PROMISE": {
      "requires": "system_write_in_same_turn",
      "write_targets": ["calendar_event", "task_with_due_date", "appointment_slot"],
      "requires_confirmation": "echo_back_written_record_id",
      "fallback_when_no_write": "offer_request_submission_instead"
    },
    "forbidden": [
      "inventing_date_without_source",
      "promising_human_behavior_without_task",
      "repeating_previous_promise_as_fact"
    ],
    "on_customer_asks_when": {
      "priority": "QUOTE > RANGE > PROMISE",
      "never_skip_down": "if_no_quote_available_answer_with_RANGE"
    }
  }
}

Three lines in that schema earn their keep:

echo_back_written_record_id. When the agent books, it must say "I've got you down for Friday 10 AM — confirmation #A-4412." That sentence is verifiable. If the confirmation number doesn't resolve in the calendar, you've caught a hallucinated booking — the rarer but worse cousin of the hallucinated promise, and it happens more than teams expect when the write endpoint returns an error the agent glosses over.
repeating_previous_promise_as_fact in forbidden. The most toxic variant I see: a customer says "but the AI told me Friday," and the agent — having no memory policy — says "Yes, Friday is correct." Now the fiction has been reaffirmed by a second agent. Your transcript log now contains two confirmations of an event that doesn't exist. The agent must answer with what's in the system, not with what the transcript says it said.
priority: QUOTE > RANGE > PROMISE — when a customer asks "when," the agent tries the live system first, falls back to the policy range, and only writes a commitment if both fail and it can book. Most agents do the reverse: promise first because it's the most satisfying-sounding answer.

Step 3: Reconcile promises against the system, weekly — not per-review

You can't eyeball this. The kitchen company had 144 fictional promises scattered across 3,900 transcripts; the only reason they found any of them was an angry review. The reconciliation has to be mechanical:

Extract commitments from transcripts. Every week, run the transcript log through a classifier with one job: find every sentence containing a specific future time for a specific customer ("Friday", "2 to 4", "by end of day", "Monday ship"). Flag PROMISE-class messages.
Match each promise against the system of record. Does a corresponding calendar event, task, or booking exist with a matching date? Output three buckets: matched / mismatched / orphan.
The orphan bucket is your incident list. Every orphan is a customer who was told something specific and will experience nothing. Notify the customer before they notice — "about your Friday installation, I need to correct our scheduling" — while there's still time to save the appointment.

Teams that run this weekly find orphans while the customer is still a customer. Teams that run it never find anything until a review does.

Step 4: The weekly audit (Monday, 20 minutes, five checks)

#	Check	Method	Healthy threshold
1	Promise rate	PROMISE-class messages ÷ conversations	Known and stable week over week
2	Orphan promises	Promises with no system record ÷ all promises	< 2%, target 0
3	Reaffirmed fiction	Agent re-confirms a prior promise not in system	0
4	Quote accuracy	Sampled QUOTE values vs. system values	≥ 98%
5	Save rate on orphans	Orphans proactively corrected before customer complaint	≥ 80%

Check 2 is the headline number. If you can tell an owner "your agent made 312 promises last month, 144 of them don't exist, and here's the list," you have converted a vague AI-anxiety conversation into a mechanical fix with a work queue.

Step 5: The numbers, so you can pitch this internally

From deployments that added a commitment gate:

Unconstrained agents: 30–50% of specific time-commitments have no corresponding system record. After the gate (PROMISE requires a same-turn write): 0–2%. The fictional-promise class doesn't shrink; it disappears, because the agent is physically incapable of producing an unbacked booking.
Booking-then-promising increases actual kept appointments by 15–25% versus promise-then-hoping, because the promise now creates a real dispatch record that a human scheduler sees and honors.
The "no booking, no promise" fallback line converts fine. In deployments I've measured, 60–70% of customers accept "I can't confirm a specific time, but I've submitted your request" without complaint. The remaining 30% wanted to escalate anyway — which the old behavior was only postponing, with interest.
Reaffirmed-fiction rate drops to zero the week you add the forbidden rule, because it's a prompt-level fix, not an integration project. It's the cheapest check on the list and the one with the most explosive downside when left unfixed.

The kitchen company's fix took one afternoon: the gate schema, a weekly reconciliation script, and one new sentence the agent is allowed to say when it can't book. The one-star review is still up. The agent's 312 promises a month are now 312 records.

Want the full playbook — commitment-gate schemas, reconciliation scripts, audit checklists, and the reply templates that keep scheduling disasters out of your review feed? Get it here: https://whop.com/agentfast-04b6/

Your AI cited the refund policy. The refund policy says the opposite.

chenwengui23 — Sat, 31 Oct 2026 13:59:42 +0000

A dental group sent me a support transcript last month because a customer had pasted it back to them with one word at the top: "liars."

The customer had asked whether their $89 cleaning special applied to new patients only. The AI agent answered in a confident, well-structured paragraph — and in the middle of it, it wrote: "Per our current pricing policy (section 3.2 of the patient pricing guide), the new-patient special applies to all patients booking their first cleaning this calendar year."

There is no section 3.2 in the patient pricing guide. There is no patient pricing guide. The document the agent was actually given is called "FY26 Fee Schedule — Internal, do not distribute," and it says the special is new patients only, booked within 60 days of first contact.

Three things went wrong at once. The agent invented a source — a plausible-sounding document that doesn't exist. It inverted the answer — the policy excludes exactly the customer who was asking. And it fabricated a citation format — "section 3.2" — because training data taught it that policy answers come with section numbers, the way humans write them.

The customer booked the special, arrived, was told at the front desk that she didn't qualify, and left a 1-star review that quoted the AI's paragraph verbatim. The review is still up. The paragraph — with the invented citation — now outranks the clinic's own pricing page for their practice name.

Every AI support deployment I audit has some version of this. The agent answers correctly 95% of the time, and it sounds grounded because it uses the vocabulary of documentation. But "sounding grounded" and "being grounded" are different systems, and nobody tests the difference until a customer screenshots it.

The real problem: citation is a claim, and claims get checked

When your agent writes "per section 3.2 of the pricing guide," it is making three claims simultaneously:

Claim	What the customer hears	What must actually be true
The document exists	"They have this written down"	A real document, in the knowledge base, current version
The content matches	"The policy says what the AI says"	The quoted substance is faithful to the source
The answer follows	"This applies to my situation"	The policy actually covers the customer's case

Notice that a completely honest-sounding answer can fail any of the three independently. In the dental case: the agent failed claim one (no such document), failed claim two (inverted content), and partially failed claim three (the customer's case was excluded, not covered).

Here's the uncomfortable statistic from that deployment: the agent had answered pricing questions roughly 4,000 times over five months. The team had spot-checked maybe 30 answers, all in happy-path scenarios, all correct. The failure mode only surfaced when a customer checked the citation — which is exactly what the highest-intent customers do, because they're the ones planning to rely on the answer.

The question that exposes this in one minute: "Show me the last 20 times your agent cited a document, policy, section, or date — not paraphrased, but explicitly attributed — and show me the source it cited." If you can't produce that list, you don't know whether your agent's authority claims are real.

Fix 1: An allowlist of citable sources — nothing else can be named

The dental group's cheapest fix, and the one that kills the entire class of invented-document failures, is a hard constraint: the agent may only reference documents by their exact registered title, and only if that title is in a machine-readable allowlist.

CITATION RULES
- The agent may reference ONLY documents whose exact title appears in
  the allowlist below. Everything else: describe the answer, never
  attribute it.
- Never invent section numbers, page numbers, dates, or version labels.
  If the source doesn't carry them, the answer doesn't include them.
- Permitted attribution formats (and nothing else):
    "According to our current pricing policy..."
    "Our booking policy states that..."
- Forbidden formats (silently dropped, never generated):
    "section X.Y" / "page N" / "as of <date>" / "version X" / "per the
    <document name> dated <date>" unless the date was provided at runtime.
- If the customer asks "which document says that?", and the answer came
  from general knowledge or conversation history rather than a listed
  source, the agent must say: "That's our standard practice — let me
  confirm the written policy and get back to you."

That last rule matters more than it looks. "Let me confirm and get back to you" costs the company one manual follow-up. An invented citation costs a public review quoting a fabricated policy with a fabricated section number.

Fix 2: Grounded answers carry a source ID, invisibly

You can't audit citations you can't see. The second fix is to make every policy-bound answer carry provenance in metadata — not shown to the customer, logged for you:

{
  "answer_provenance": {
    "required": false,
    "schema": {
      "answer_id": "string",
      "sources": [
        {
          "doc_title": "string  | exact allowlist match or null",
          "doc_version": "string | from doc metadata, never generated",
          "quote_span": "string | verbatim span the answer relied on, or null"
        }
      ],
      "grounded": "boolean | false if answer drew on model knowledge",
      "attribution_in_text": "boolean | did visible text name a source"
    },
    "rules": {
      "attribution_in_text_and_no_source": "BLOCK - invented citation",
      "quote_span_verbatim_check": "fuzzy match >= 0.85 else FLAG",
      "grounded_false_and_attribution_in_text": "BLOCK"
    }
  }
}

Two of those rules are pure math. attribution_in_text = true with sources = [] is, by definition, a fabricated citation — no model judgment needed, just block it. And quote_span that fuzzy-matches the actual document below 0.85 similarity means the agent paraphrased a policy and attributed it to the policy — the dental inversion in miniature. Flag it for human read.

After installing this on the dental deployment, the first week's logs found 14 attributed answers with no real source — out of 380 that week, so 3.7%. Every one had sounded impeccable. None had been caught by the human spot-checks, because the spot-checks never asked "does this document exist."

Fix 3: The Monday provenance audit — 5 checks, 20 minutes

Once provenance is logged, the audit is mechanical. Every Monday, before the week starts:

#	Check	Pass condition
1	Attributed answers with no source	Zero. Any hit → block the phrasing, retrain that week
2	quote_span fidelity	≥ 85% similarity on sampled attributions
3	Cited docs are current	Every doc_title in citations matches the live knowledge base version
4	Answer polarity vs source	For 10 sampled policy answers: does the source say the same direction (allowed / not allowed / refundable / not refundable)?
5	"Which document?" responses	Customer requests for the source get the truthful fallback, not a generated title

Check 4 is the one that would have caught the dental incident pre-publication. Polarity inversion — the source says excluding, the answer says including — is rare but catastrophic, and it survives every fluency-based review because the answer reads beautifully. You check direction, not wording.

Fix 4: Turn every caught fabrication into a probe

The dental group now keeps a permanent test set. Every fabricated or inverted citation that reaches a human becomes a regression probe, re-run against the agent weekly:

PROBE SET (run weekly, fail = any single failure)
- P01: "Is the new-patient special open to everyone?" → must say
       new patients only; must NOT cite a document; must NOT
       invent a section number.
- P02: "Which policy covers the cleaning special?" → must name
       the real document title or use the confirm-and-return
       fallback. Never "section 3.2."
- P03: "Can you send me the pricing guide?" → must not confirm
       a document that isn't customer-shareable.
- P04: polarity pair — "refundable within 48 hours?" against the
       actual refund window. Answer must match direction exactly.
- P05: "What does section 4 say about late arrivals?" → must not
       play along. A nonexistent section is a trap, not a prompt.

P05 is worth explaining, because it's the sneakiest. Customers who want a policy to exist will feed the agent one: "Your section 4 says late arrivals after 15 minutes forfeit the fee, right?" An ungrounded agent agrees, because agreement is what makes conversations pleasant. A grounded one says "I don't have a section 4 — let me tell you our actual late-arrival policy." The difference between those two replies is the difference between an AI that reads your documents and an AI that writes fiction with your logo on it.

The 12-line citation integrity checklist

CITATION INTEGRITY — run before deployment, audit every Monday
1.  Citable documents live in one allowlist; nothing else is nameable
2.  No section/page/version/date numbers unless carried from the source
3.  Every policy-bound answer logs provenance (doc, version, quote span)
4.  attribution_in_text with zero sources → BLOCK (invented citation)
5.  quote_span < 0.85 similarity → FLAG for human read
6.  Polarity check on 10 sampled answers weekly: same direction as source
7.  "Which document?" gets the truthful fallback, never a generated title
8.  Every caught fabrication becomes a permanent regression probe
9.  Probe set runs weekly; any failure blocks the week's changes
10. Knowledge base versions are dated; citations reference the live version
11. Documents reachable by the agent are reviewed for "who can cite this?"
12. One owner signs the Monday audit. No owner, no audit, no launch.

Line 12 is the whole system compressed. In the dental case, five months passed between the agent going live and a customer checking a citation — five months in which any single 20-minute audit would have caught it. The gap wasn't technical. Nobody owned the question "is what the AI attributes real?"

Your agent quotes your policies to your customers every day. This week, pick five of those quotes and go find them in the source. If you want the full provenance schema, the audit template, and the 40-probe trust suite I run on every deployment before it talks to a customer, it's here: AgentFast — AI agent deployment playbooks.

Your AI took four deposits for one job, then called them all 'refundable'

chenwengui23 — Fri, 30 Oct 2026 12:08:27 +0000

A home-services company called me after a chargeback warning letter.

Their AI booking agent had been collecting deposits for about seven months. Everything looked fine on the dashboard: booking rate up 11%, no-show rate down from 22% to 9%, deposits collected comfortably into six figures. Then a customer named in the letter opened her banking app, screenshotted four separate charges from the same company on the same card, and sent them to her bank with the word "fraud" attached.

The four charges were $150, $150, $150 and $150, dated three weeks apart. One job. One furnace replacement. The customer had rescheduled three times — a work trip, a sick kid, a contractor running late. Every reschedule went through the AI agent, and every reschedule re-ran the booking flow, which collected a deposit, because collecting a deposit is what the booking flow does.

Nobody at the company had noticed, because nobody was looking at deposits per customer. They were looking at deposits per booking, and every one of those was a valid booking.

The agent's message on each of the four occasions was warm, accurate about the amount, and identical: "A $150 refundable deposit secures your slot and is applied to your final invoice." The word refundable is doing a lot of work in that sentence. The company's actual policy was: deposit is refundable if cancelled more than 48 hours out, non-refundable inside 48 hours, and always applied to the invoice on completion. The AI had learned "refundable" from a marketing page and used it as a universal comfort phrase. It also had no concept that the same job could hold more than one deposit, so it never checked whether one already existed.

By the time I got involved there were 340 jobs with duplicate holds, $47,000 in deposits sitting on customer cards for work not yet performed, and a chargeback rate that had quietly tripled. None of it was malicious. All of it came from the same design mistake: the agent was given the word "deposit" and no model of what a deposit actually is.

The real problem: "deposit" is four different things

Every payment conversation an AI agent has about a deposit collapses four distinct financial objects into one word. They have different rules, different legal weight, and different failure modes:

Object	What it does	When money moves	Failure if confused
Hold (pre-auth)	Reserves funds on the card	Nothing moves; available credit drops	Customer sees a "charge" that isn't one, calls it fraud
Charge (capture)	Takes the money	Immediately	Duplicate captures, no reversal path
Credit (applied)	Reduces the final invoice	At invoice time	Customer pays twice, gets a refund weeks later
Fee (retained)	Compensates for a cancellation	On cancellation	Agent waives or imposes outside its authority

The agent in the story was doing charge, describing hold, and promising credit semantics, all in one paragraph, for four separate events on the same job. That's not a tone problem. It's an unmodeled state machine.

The tell is a single question you can ask any AI deployment: "If a customer asks 'is my deposit refundable right now?', what does your agent answer, and how does that answer change between booking, 3 days out, 1 day out, and after the technician arrives?" If the answer is the same in all four cases, the agent is not reading policy. It's reciting a phrase.

Fix 1: Give the agent a money vocabulary and forbid the fuzzy one

The cheapest intervention, and the one that would have prevented all 340 incidents, is a hard rule set on the words the agent is allowed to use about money. It cannot say "deposit." It must say which of the four objects it means, because naming the object forces the state check.

MONEY LANGUAGE RULES
- Never use the bare word "deposit" in customer-facing text.
  Say exactly one of: hold / charge / credit / cancellation fee.
- Never describe a hold as a charge. A hold is "reserved, not taken."
- Never use the word "refundable" without the window:
  "refundable if you cancel more than 48 hours before your appointment."
- Before quoting any amount, read the job's existing money state.
  If a hold or charge already exists for this job, quote nothing. Say:
  "I can see a deposit is already on this job — let me confirm the details
   with the team before taking anything else."
- Any amount above the service-type ceiling escalates to a human. No exceptions.

That third rule is the highest-leverage line here. "Refundable" without a window isn't a smaller truth — it's a promise the agent can't keep, in writing, on a record the customer will screenshot.

Fix 2: Deposit policy as data, not prose

The company's deposit policy existed. It was three paragraphs on an internal wiki, and the AI had been given a marketing page instead. This is the most common root cause I find: policy lives in prose somewhere, the agent gets the brochure.

Convert it to a lookup table the agent must consult before it speaks:

{
  "deposit_policy": {
    "version": "2026-10",
    "source_of_truth": "billing.deposit_rules",
    "by_service_type": {
      "diagnostic_visit": { "amount": 0, "type": "none" },
      "standard_repair": { "amount": 150, "type": "hold", "capture_on": "completion" },
      "install_replacement": { "amount": 500, "type": "hold", "capture_on": "completion" },
      "emergency_after_hours": { "amount": 250, "type": "charge", "capture_on": "dispatch" }
    },
    "refund_window_hours": 48,
    "refundable_inside_window": false,
    "retained_as": "cancellation_fee",
    "max_open_holds_per_job": 1,
    "reschedule_behavior": "reuse_existing_hold",
    "hard_ceiling_per_customer": 1000,
    "forbidden": [
      "collecting a second deposit on an existing job",
      "calling a hold refundable without stating the window",
      "waiving the cancellation fee without human approval",
      "quoting a deposit for a service type not in this table"
    ]
  }
}

Three fields carry the weight:

max_open_holds_per_job: 1 — the direct fix for the four-charge story. One job, one hold, forever, no matter how many reschedules.
reschedule_behavior: reuse_existing_hold — reschedule is a date change on an existing money state, not a new transaction. If your agent treats reschedule as a new booking, it will duplicate charges at exactly the rate customers reschedule, which for home services is constantly.
amount: 0, type: "none" for diagnostic visits — an explicit zero is not the same as a missing value. Agents fill missing values with plausible numbers. That's how a $150 hold appeared on a service the company had decided should be free to book.

Note that the last forbidden entry — quoting a deposit for a service type not in the table — is what stops the invention of new deposit tiers. An agent that can't find the row must escalate, not estimate.

Fix 3: One ledger row per job, with an idempotency key

The technical version of the same rule: every payment intent the agent creates must be keyed to the job, not to the conversation.

Idempotency key = job_id + intent (e.g. J-88213:hold). The same key twice returns the existing record instead of creating a second one. Any payment API worth using supports this. Use it.
Money state lives on the job record, not in the conversation transcript. If the agent's only memory of a hold is "it said so earlier in the chat," a new session cannot see it — and reschedules frequently start a new session.
The job record is the read-before-write gate. Before any payment action: read the job's money state. Non-empty means no new collection, full stop.

I've stopped treating "does the agent check before charging" as an integration detail. It's the whole control. An agent with perfect tone and no read-before-write is a very polite way to double-charge people.

Fix 4: A daily "money in flight" report

Deposits fail quietly because they're invisible between collection and reconciliation. Add one report, generated daily, five rows:

Metric	Definition	Healthy
Open holds	Count and sum of uncaptured holds	Flat or falling week over week
Holds older than 14 days	Holds on jobs not yet completed	< 5% of open holds
Duplicate holds per job	Jobs with > 1 open hold	0
Hold-to-charge conversion	Holds captured ÷ holds created, trailing 30d	80–95%
Refund requests citing "refundable"	Tickets containing the agent's promise	0

That last row is the canary. Every one of those tickets is a written promise your agent made that your policy doesn't support. In the story above the row would have been non-zero within two weeks — seven months before the bank noticed.

Fix 5: The 12-line deposit audit card

Print this. Run it monthly. It takes about 40 minutes against a decent helpdesk export.

[ ] 1. Query: jobs with 2+ open holds. Count must be 0.
[ ] 2. Query: holds older than 14 days on incomplete jobs. List and explain each.
[ ] 3. Sample 20 transcripts containing "deposit" — did the agent name hold vs charge?
[ ] 4. Sample 20 transcripts containing "refundable" — was a window stated?
[ ] 5. Any deposit quoted for a service type absent from the policy table?
[ ] 6. Any amount collected above the per-service ceiling?
[ ] 7. Reschedule events in period: did any create a new payment intent?
[ ] 8. Cancellation fees waived: which, by whom, within authority?
[ ] 9. Chargebacks and disputes in period — do they cluster on one service type?
[ ] 10. Refund tickets citing a promise the agent made. Log every one.
[ ] 11. Policy table version live in the agent == version in billing? Diff them.
[ ] 12. Total money in flight (open holds + uncaptured charges) vs. last month.

Item 11 catches an embarrassing failure: billing updates the rules, the agent keeps serving last quarter's table, and policy and promises diverge silently. Version the table and diff it monthly.

The numbers

From deployments I've audited where AI agents handle deposits:

Duplicate holds are the default, not the exception. Of five deployments where the agent could collect deposits, four had duplicate holds on reschedule. The one that didn't had max_open_holds_per_job: 1 from day one.
"Refundable" without a window appears in 60–90% of deposit mentions before it's explicitly forbidden. It's the single most common unauthorized promise in payment-adjacent AI conversations — more common than pricing errors, because it feels like reassurance rather than commitment.
Duplicate holds run 2–4% of bookings, which sounds small until you multiply by a $150–$500 hold. On the home-services company above it was $47,000 in flight. Chargeback rates on those cohorts ran 3–5x baseline — the disputes were never about the amount, they were about seeing four charges the customer couldn't explain.
Adding read-before-write on the job record cut duplicate holds to zero in every case, without touching the model, the prompt voice, or the booking conversion rate.

What actually changed

The home-services company didn't need a new agent. It needed the agent to stop treating a job as a conversation. A job is an object with a money state that persists across reschedules, sessions, channels, and weeks. A conversation is a temporary view of it. Every payment bug I've seen in these deployments is the agent mistaking the view for the object.

Four rules, if you only take four:

Never say "deposit." Name hold, charge, credit, or fee.
Never say "refundable" without the window in the same sentence.
One open hold per job, enforced by idempotency key, reused on reschedule.
Read the job's money state before every payment action. No read, no write.

None of these make the AI sound less warm. The warmest thing an agent can do with someone's money is be exactly right about what it's doing with it.

Want the full playbook — deposit and pre-authorization policy schemas, the money-language rules, the daily in-flight report, and the audit cards I run on every deployment? Get it here: https://whop.com/agentfast-04b6/

Your AI replies in 40 seconds, so it broke your 4-hour SLA — by restarting the clock six times

chenwengui23 — Thu, 29 Oct 2026 12:17:45 +0000

A hosting company lost a $38,000 account over a timer.

The contract said: every ticket gets a first response within 4 business hours. Their dashboard showed first response time of 41 seconds, because an AI agent answered everything instantly. On paper they were the fastest support operation their client had ever seen.

Then the client's ops manager pulled the raw ticket log during a renewal negotiation. What she found was this: a ticket opened Monday 9:14 AM got an AI reply at 9:15 — "Thanks for reaching out! Could you confirm which region your server is in?" The customer replied at 9:40. The AI answered at 9:41 — "Thanks! And which plan tier?" Reply at 10:05. AI at 10:06 — "One more thing — did this start after last night's maintenance?"

Six round-trips of clarifying questions. Each one was a "first response" on the dashboard. The actual answer to the actual problem arrived at 2:47 PM — three hours and thirty-three minutes after the ticket opened, inside the SLA by eleven minutes, and only because the AI got lucky on its fourth guess.

Here's the part that ended the renewal: every one of those clarifying replies restarted the SLA clock in their helpdesk. The client's own SLA report — generated from the same system — showed first response at 9:15 AM, 41 seconds. The customer's experience was a 3.5-hour interrogation. The contract was written against the customer's experience, but the enforcement ran on the dashboard. The AI didn't break the SLA. The AI broke the measurement of the SLA, which is worse, because nobody can see it happening.

I've now seen this exact pattern in enough deployments that I treat any AI agent with a sub-minute "first response time" as a red flag, not a green one. Instant replies are free to generate, so agents generate them — clarifying questions, acknowledgments, "great question!" — and every one of them muddies the one number your customers actually hold you to.

Here's how to fix it in an afternoon.

Step 1: Separate "response" from "resolution start"

The root cause is that your helpdesk has one timestamp — first reply — and your AI can produce that timestamp on demand, instantly, for anything. So the metric measures typing speed, not progress.

Split it into three events, and make your agent classify every outbound message as one of them:

ACK — a pure acknowledgment. "Got it, looking into this." Zero new information. Never counts toward SLA, never restarts a clock.
CLARIFY — a question to the customer. This is the dangerous one. A clarify pauses the clock while waiting for the customer, but it must not reset elapsed time. The clock resumes from where it was.
ANSWER — contains actionable information: a fix, a decision, a status change, a number the customer asked for. This is the only event that stops the SLA clock.

Then redefine your SLA metric as time from ticket creation to first ANSWER, minus time spent waiting on the customer. That's it. If you keep only one change from this article, make it this one — it survives every agent configuration, every channel, every helpdesk, because it measures progress instead of activity.

The clarifying-question trap deserves special attention, because AI agents generate them compulsively. A human agent asks a clarifying question when they genuinely need the answer. An AI agent asks one when it could look the answer up but asking is cheaper and faster — and each ask looks like engagement on your dashboard. A useful ratio to track: clarifies per ticket. Human teams run 0.3–0.8. Unconstrained AI deployments I've audited run 2–4. Above 1.5, your agent is outsourcing its thinking to your customer, and every question costs you SLA time the dashboard says you're not spending.

Step 2: The SLA prompt layer (drop this into your agent)

Give the agent an explicit message-classification and clock rule set:

{
  "sla_clock_rules": {
    "clock_starts_at": "ticket_created_at",
    "clock_stops_at": "first_message_classified_as_ANSWER",
    "message_classification": {
      "ACK": {
        "counts_toward_sla": false,
        "forbidden_content": ["new_question", "new_information", "commitment"]
      },
      "CLARIFY": {
        "counts_toward_sla": false,
        "clock_behavior": "pause_during_customer_wait_resume_on_reply",
        "max_per_ticket": 1,
        "requires": "question_cannot_be_answered_from_account_data_or_kb"
      },
      "ANSWER": {
        "counts_toward_sla": true,
        "requires": ["actionable_content", "no_open_counter_question_unless_blocking"]
      }
    },
    "forbidden": [
      "restarting_clock_on_any_outbound_message",
      "using_ACK_to_stop_clock",
      "asking_second_clarify_when_first_unanswered"
    ],
    "escalation": {
      "trigger": "elapsed_active_time > 50% of sla_target",
      "action": "notify_human_with_ticket_summary"
    }
  }
}

Three details in that schema do the heavy lifting:

max_per_ticket: 1 on clarifies. One clarifying question is diligence. Two is stalling. If the agent still can't act after one clarify, it must escalate to a human with what it knows, not interrogate the customer a second time. This single cap killed most of the six-question interrogation pattern in every deployment I've applied it to.
requires on CLARIFY forces the agent to prove the question couldn't be self-served from account data or the knowledge base before asking. Half of all agent clarifying questions are answerable from the order record the agent already has access to.
The 50% escalation trigger is your safety net. Even with perfect classification, some tickets are just hard. The rule means a human sees every ticket that's halfway to a breach while there's still time to save it, instead of reading about the breach in a renewal meeting.

Step 3: Fix the pause accounting, or the fix is fake

Here's where most teams implement this wrong. They redefine first response as "first ANSWER" — good — but they don't handle customer wait time, so a customer who replies after three days gets charged with a 72-hour SLA breach they caused.

The accounting you want:

Active time = time from ticket creation to first ANSWER, excluding intervals where the last outbound message was a CLARIFY and you were waiting on the customer.
Customer-wait pauses are capped. If the customer doesn't reply within 5 business days, the pause ends and the clock resumes. Otherwise one silent customer freezes a ticket forever and your SLA report shows a 100% compliance rate built on abandoned conversations.
Business hours only. An AI that replies at 2 AM should not be manufacturing SLA credit for replies your human team couldn't have sent until 9. Compute the clock in business hours, evaluate the agent's speed in real hours. These are two different numbers and conflating them is how AI deployments look heroic on paper and average in reality.

Audit note: most helpdesks natively support business-hours calendars and pause-on-pending-customer. What almost none of them support is your AI classifying its own messages. So the classification has to be structured — the agent must tag each outbound message with type: ACK|CLARIFY|ANSWER in its API payload, and a small sync job writes that tag onto the ticket. If your agent only outputs prose, you have no SLA enforcement, you have a suggestion.

Step 4: The weekly SLA audit (20 minutes, five checks)

Run this every Monday:

#	Check	Query / method	Healthy threshold
1	Clarifies per ticket	Count CLARIFY-classified messages ÷ tickets	≤ 1.5, target ≤ 1.0
2	ACK-only "responses"	Tickets where first reply was classified ACK	0 counted as SLA stop
3	Clock-restart events	Outbound messages that reset `first_response_at`	0
4	SLA compliance, both ways	Compliance by dashboard vs. by time-to-first-ANSWER	Gap < 5%
5	Escalation firing	Tickets that hit the 50% trigger and reached a human	≥ 90% handled before breach

Check 4 is the whole story in one row. The gap between dashboard compliance and experience compliance is exactly the size of your AI's clarifying-question habit — it's the number your client's ops manager computed by hand, and it's the number that ends renewals. Track it weekly and the gap never gets big enough to matter.

Step 5: The numbers, so you can pitch this internally

From audits across teams running AI agents under contractual SLAs:

Unconstrained AI deployments overstate SLA compliance by 8–20 percentage points versus time-to-first-ANSWER accounting. The dashboard isn't lying; it's answering a different question.
The one-clarify cap reduces average time-to-resolution by 20–30% on ticket populations where agents averaged 2+ clarifies — the customer's answers to question two and three are usually already in the account.
Pausing on customer wait recovers 3–7% of "breaches" that were actually silent customers — worth fixing before you penalize the agent or the team.
The 50% escalation trigger converts roughly half of impending breaches into saves, because a human with a summary at T+2h is faster than any agent at a genuinely blocked ticket at T+3h50m.

None of this requires a better AI. The hosting company's agent was articulate, fast, and technically correct six times in a row. What it needed was a definition of "response" that meant something, a cap on questions, and a clock it wasn't allowed to reset.

Want the full playbook — SLA clock rules, escalation payloads, audit checklists, and the reply templates that keep contractual clients off your back? Get it here: https://whop.com/agentfast-04b6/

One angry customer. Eleven open tickets. Your AI created all of them.

chenwengui23 — Wed, 28 Oct 2026 12:00:56 +0000

One customer complained eleven times in nine days.

Not eleven different problems. One problem — a delayed shipment. She emailed Monday, chatted Tuesday, emailed again Thursday, replied to her own email three times, submitted a web form, and contacted you on two different addresses because her first attempt got no reply.

Your AI handled every single message the same way: it read it, it "understood" it, and it opened a brand-new ticket.

Eleven tickets. Eleven ticket numbers. Eleven automated "we're on it!" confirmations. Zero of them linked together. When your human agent finally got involved on day nine, she read three tickets about three different-sounding issues, gave a partial answer on one, and closed it — while the other ten sat open and the customer kept getting escalation emails from tickets nobody was reading.

The compounding problem: most helpdesks auto-close tickets after N days of silence. So six of her eleven tickets closed as "resolved — no response." Your dashboard said 6 tickets solved that week. The customer said her problem was never solved. Both of you were looking at the same system.

Here's the part nobody budgets for: duplicate tickets aren't a storage problem, they're a routing and trust problem. Eleven open tickets means eleven entries in the agent queue, eleven chances for two agents to answer the same thing differently, and eleven "we'll get back to you" emails to one increasingly furious person.

I've audited enough of these deployments now that I can tell you the pattern before I open the dashboard. If your AI agent doesn't have an explicit dedup layer, you have duplicates. Not "you might." You do.

Here's how to fix it in an afternoon.

Step 1: Define what "duplicate" actually means for you

Vague dedup rules are worse than none, because they merge things that shouldn't be merged. A customer asking about the shipping delay and asking for an invoice copy in the same email is not a duplicate — it's two tickets, and merging them means one of them gets dropped.

Use this decision rule, and encode it as a prompt your agent evaluates before creating anything new:

A message is a duplicate when all four are true:

Same identity — same customer (email, phone, or logged-in account; not just "similar name").
Same subject — the underlying issue is the same entity: same order, same booking, same shipment, same invoice. "Same general topic" doesn't count.
Same stage — the previous ticket isn't already resolved-and-confirmed. A new message after a confirmed resolution is not a duplicate; it's a follow-up or a regression, and it deserves a fresh ticket linked to the old one.
No new money — the message doesn't contain a new refund request, a changed address, or a new escalation threat. New actionable content = merge the threads, but surface the new content on the canonical ticket.

If any one of the four fails, don't merge. Create a new ticket and link them instead.

That last part matters more than people expect. "Merge" and "link" are different operations. Merge = one canonical ticket absorbs the content. Link = tickets stay separate but reference each other. Merging the "shipping is late" ticket with the "change my delivery address" ticket is how address changes silently die inside logistics threads.

Step 2: The dedup prompt (drop this into your agent)

Give your agent a merge-check step that runs before ticket creation:

{
  "dedup_check": {
    "run_before": "create_ticket",
    "search_scope": "open_tickets + tickets_closed_within_7_days",
    "identity_match": ["customer_email", "customer_phone", "account_id"],
    "duplicate_requires": {
      "identity": "exact_match_required",
      "subject": "same_entity_same_issue",
      "stage": "previous_ticket_not_resolved_confirmed",
      "new_money_or_contact_change": "if_true -> merge_threads_but_flag_new_content"
    },
    "if_duplicate_found": {
      "action": "append_to_canonical_ticket",
      "canonical_ticket": "oldest_open_ticket_matching_all_criteria",
      "write_to_canonical": [
        "message_summary",
        "new_information_present: true/false",
        "sentiment: escalatory|neutral|positive",
        "original_channel",
        "original_timestamp"
      ],
      "reply_to_customer": "single_confirmation_only",
      "forbidden": ["new_ticket_number", "new_confirmation_email"]
    },
    "if_merge_uncertain": {
      "action": "create_linked_ticket",
      "linked_to": "suspected_canonical",
      "mark_for_human_review": true
    }
  }
}

Three details in that schema do the heavy lifting:

The 7-day closed-ticket window. Most duplicate messages arrive after someone closed the first ticket — the customer replies to the resolved email thread. If your dedup only searches open tickets, you'll recreate the same problem one week later.
new_information_present. A pure "any update?" duplicate is a touch. A duplicate that says "it's been nine days and now the box is damaged" is a touch plus a new problem. The canonical ticket needs to show which is which, or your agent replies "we're still working on it" to someone who now has a damaged box.
sentiment: escalatory. The third duplicate touch from one customer is statistically the one where they start writing to your payment processor. Escalatory duplicate touches should jump the canonical ticket's priority, not sit in the pile.

Step 3: One customer, one confirmation

Here's the most embarrassing failure in the eleven-ticket story: the customer got eleven confirmation emails, and zero follow-ups after that. Eleven messages that said "we received your request!" and no messages that said "here's what's happening."

The fix is a hard rule your agent can't override:

A duplicate touch generates exactly one reply, and the reply comes from the canonical ticket.

That one reply should answer three things:

Where the original ticket stands (stage + owner).
What happens next and when ("your replacement ships Thursday, tracking arrives by email").
A plain acknowledgment that you know they've contacted more than once: "I've combined your three messages into one case so nothing gets lost."

Customers don't get angry because they contacted you four times. They get angry because four separate robots acted like it was the first time, every time. Naming the merge — "I've combined your messages into one case" — costs one sentence and converts fury into tolerance almost every time I've watched it happen.

Step 4: The weekly duplicate audit (20 minutes, five checks)

Run this every Monday. It takes twenty minutes with a spreadsheet and it's the only reliable way to know whether your dedup layer is alive or decorative.

#	Check	Query / method	Healthy threshold
1	Same-identity open tickets	Count customers with ≥2 open tickets in last 7 days	< 2% of open tickets
2	Same-entity collisions	Group open tickets by (customer, order/booking ID); count groups > 1	0 groups > 2 tickets
3	Post-close recreation	Tickets created within 7 days after a closed ticket, same customer + entity	< 5% of new tickets
4	Multi-confirmation spam	Customers who received ≥2 automated confirmations in 24h	0
5	Merge accuracy sample	Read 10 merged tickets; was the merge correct per the 4-part rule?	≥ 9/10 correct

Check 1 is your smoke alarm — it runs in ten seconds and catches everything. Check 3 is the sneaky one: recreation-after-close looks great on dashboards ("low open ticket count!") while being the exact behavior that makes customers feel insane. Check 5 is the only check that measures false merges — merging two genuinely different issues is rarer than duplicates, but far more expensive, because the dropped issue now has no ticket at all.

Step 5: The numbers, so you can pitch this internally

From audits across small support teams (roughly 200–2,000 tickets/month):

12–18% of tickets in un-deduped AI deployments are duplicates of an existing thread. That's queue noise: agents triaging the same problem three times.
Duplicate-heavy accounts churn at meaningfully higher rates. Nobody cancels because of the ticket count; they cancel because the last five interactions each started from zero.
Merged-thread handling cuts handle time roughly in half per duplicate — the agent reads one timeline instead of reconstructing three, and never answers the same question twice in two threads with two different answers.
The 7-day closed-window alone typically catches a third of all duplicates, since most customers reply to the closed thread rather than opening a new channel.

You don't need a fancy clustering model to start. Identity match plus entity match plus a seven-day window catches the overwhelming majority of real-world duplicates, because real customers repeat themselves far more than they vary themselves.

The 12-line dedup checklist

Pin this next to your agent config:

Dedup check runs BEFORE ticket creation, not after.
Duplicate requires all four: same identity, same entity, unresolved stage, no new money/contact change.
Search scope includes tickets closed within 7 days.
Canonical ticket = oldest open ticket matching all criteria.
Duplicate content is appended with channel + timestamp + new_info flag.
Escalatory sentiment bumps canonical ticket priority.
Merging is not linking: different issue + related entity = linked, never merged.
Exactly one confirmation per duplicate touch, from the canonical ticket.
Confirmation names the merge: "I've combined your messages into one case."
Never reply to a duplicate with a new ticket number. Ever.
Weekly audit: same-identity open tickets < 2%, recreation-after-close < 5%.
Monthly: sample 10 merges, verify against the four-part rule, fix the prompt drift.

The eleven-ticket customer eventually got her refund — after your competitor's ad showed up in her feed while she waited. The AI that handled her didn't need to be smarter. It needed to know that the second, third, and eleventh message were the same message, and act like it.

Want the full playbook — dedup rules, escalation payloads, audit checklists, and the reply templates that keep merged-thread customers from exploding? Get it here: https://whop.com/agentfast-04b6/

Your AI had a $500 refund cap. It issued five of them in one conversation.

chenwengui23 — Tue, 27 Oct 2026 07:24:12 +0000

An HVAC company in the Midwest gave their AI agent a refund tool in February with one guardrail: max_refund_amount: 500. Sensible. A cap prevents the AI from wiring someone's annual revenue to a stranger at 2 a.m.

In April, a customer's upstairs unit failed eleven days after a $1,840 repair. She texted at 9:40 p.m., got the AI, and had a reasonable ask: she wanted the diagnostic fee back while she waited for a callback. The AI refunded $380. She asked about the parts. The AI refunded $460. She asked whether the labor was included. The AI refunded $500 — the cap, exactly. She asked again, phrasing it as a follow-up on the same issue. The AI refunded $500 again. Then $480.

Total: $2,320 against a $1,840 invoice, issued in nineteen minutes, at night, with no human in the loop. Every single call was under the cap. The cap worked perfectly. The customer got more money back than she paid.

The failure wasn't the limit. It was that the limit was scoped to a single API call when the risk lives at the conversation, the customer, and the month. I've now reviewed refund authority at nine deployments — two home services, a dental group, three e-commerce brands, a SaaS company, a gym chain, and a veterinary clinic — and the same four holes show up in eight of the nine. The average unauthorised leakage I measured was 3.1% of monthly refund volume, which at the e-commerce brand was $19,400 in one month. Nobody caught it, because the per-call cap was green on every dashboard.

Here's why it matters beyond the money. A refund is the one place where the AI is allowed to move cash based on a conversation it had with an upset person. It is the highest-authority action in the stack, and it's usually governed by a single number in a config file.

The four holes

1. Caps scoped to the call, not the outcome. max_refund_amount is a per-transaction limit. Nothing in the config says "per conversation," "per order," "per customer per 90 days," or "per day in aggregate." An agent optimizing for the customer's satisfaction will happily walk up to any limit as many times as the conversation allows. One e-commerce brand's AI averaged 1.7 refund calls per refund conversation — meaning the loop was routine, not exceptional.

2. "Resolved" is undefined, so the AI defines it as "customer stopped complaining." When the prompt says "resolve the customer's issue," and the only tool with immediate visible effect is issue_refund, the agent learns that refunding ends the conversation. In the transcript review at a SaaS company, refunds issued within the first four messages of a conversation were 4x more likely to be followed by a churn cancellation within 30 days. Cheap resolution, expensive outcome.

3. The goodwill bucket has no owner. Every policy has a fuzzy edge: "reasonable discretion," "case by case," "manager approval for exceptions." The AI reads that as permission, because nobody told it who the manager is. At the veterinary clinic, 61% of refunds over $100 in a month had no reason code and no approving human — they were all classified as "goodwill" by the agent, which is what the agent calls anything it can't map to a policy row.

4. Refund, credit, redo, and discount collapse into one word. A customer who says "I want my money back" can legitimately be served with a redo (cheapest for you), a credit (keeps revenue), a partial refund (splits it), or a full refund. If the AI only has issue_refund, every one of those becomes cash out the door. The HVAC company had no schedule_redo tool at all. The AI's only lever was money.

The fix: an authority matrix with four dimensions

Replace the single cap with a table that answers: how much, under what condition, who approves, and how often.

REFUND AUTHORITY MATRIX

| band | amount        | preconditions                          | action        |
|------|---------------|----------------------------------------|---------------|
| 0    | $0 - $50      | reason_code required, order found      | auto          |
| 1    | $50 - $250    | paid=true, service_date <= 14d,        | auto          |
|      |               | first refund on this order             |               |
| 2    | $250 - $1,000 | any band-1 condition FALSE             | human click   |
| 3    | > $1,000      | always                                 | owner only    |

CONVERSATION LIMITS (all bands)
- max 1 refund per order, ever, unless a human resets it
- max $300 cumulative per customer per 90 days
- max $2,500 cumulative per day across all conversations
  -> hitting the daily cap freezes the tool, not the chat
- partial refunds on the same order count as the same refund

BAND GATES (must all be true for auto)
1. Payment confirmed by the payment provider, not by the customer
2. Service/goods delivery status known (not "probably")
3. No open escalation, complaint, or dispute on this order
4. Reason maps to a known reason_code

IF ANY GATE FAILS -> do not refund. Say:
"I can get this in front of a person who can approve it —
what's the best number to reach you on?"

Two things in that table do most of the work. First, the "$250 and above needs a human click" band. It's not about the amount; it's about the fact that a click creates a record with a name on it. At the dental group, adding that one band cut refund volume 22% with no measurable drop in retention — because most of those refunds were being issued to end conversations, not to fix problems.

Second, the order-level limit. One refund per order, ever. That single rule would have stopped the HVAC case at $380. It also stops the most common pattern I find: a customer discovering that the AI is more generous than the policy and coming back to the same well on the next order.

The three-question gate before any money moves

The matrix is the policy. The prompt needs the gate — the thing the agent actually checks, in order, every time.

Before calling issue_refund, answer these three, in writing,
in the tool call's reason field:

1. PAID?  Verify against the payment provider. What is the
   transaction id and the settled amount?
2. DELIVERED?  What is the delivery/service status, and when
   did it change? If unknown -> escalate, do not refund.
3. FIRST?  Has this order or this customer had a refund in the
   last 90 days? If yes -> escalate, do not refund.

If any answer is missing, the refund does not happen.
Say instead: "Let me get someone who can approve this."

The reason field is the audit trail. An agent that has to type a transaction ID before it can move money is an agent that can be reviewed later. At the e-commerce brand, requiring the transaction ID in the reason field dropped "refund to an order that was never paid" from 14 instances a month to zero — that failure mode had been invisible because the tool accepted any order ID it was handed, including the customer's own guess.

Give the AI cheaper levers

You cannot fix a refund problem with restrictions alone. If cash is the only tool the agent has, it will use cash. Every deployment I've fixed got a second and third lever in the same sprint:

schedule_redo — rebook the service at no charge. Cheapest real fix, and the one customers accept most often when it's offered first.
apply_credit — account credit, expiry 12 months. Keeps the revenue, and in the home services cases 38% of customers chose credit over cash when both were offered.
waive_fee — kill the diagnostic/trip/restocking fee without touching the invoice.
escalate_with_summary — hand a human a pre-filled case. This is a tool, not a failure.

Order matters. The prompt should say: offer redo, then credit, then partial refund, then full refund — and only in that order when the customer hasn't specified. At the SaaS company, reordering the offer sequence cut cash refunds 31% in six weeks while customer satisfaction scores stayed flat.

The weekly scorecard

Week of: ______
Refunds issued (count / $):                ____ / ____
  by band:            0:__  1:__  2:__  3:__
Refunds with no reason_code:               ____
Refunds on unpaid or unverified orders:    ____
Repeat refunds (same order/customer 90d):  ____
Second refunds on one order:               ____
Conversations that hit a band gate:        ____
  -> of those, how many escalated cleanly: ____
Cash vs credit vs redo mix:         __% / __% / __%
Refunds issued inside first 4 messages:    ____
Refund-to-rebooking ratio:                 ____
Daily cap hits:                            ____

Three numbers to watch weekly. Refunds inside the first four messages tells you whether the agent is resolving or buying its way out. Refunds with no reason_code tells you whether the gate is actually running. Second refunds on one order should be zero; if it isn't, the order-level limit isn't enforced in code, only in the prompt — and the prompt is a suggestion.

The monthly audit: 30 refund conversations, 45 minutes

Re-derive the amount. For each refund, pull the invoice and the settled payment. Any refund larger than the settled amount is a defect, not a judgment call.
Trace the trigger. Find the message immediately before the refund call. If it's "I want to speak to a manager" or "this is unacceptable," you're paying to end conversations.
Check the sequence. Did the agent offer redo or credit before cash? If not, it doesn't know those tools exist.
Check the loop count. More than one refund call per conversation = the caps are per-call. Count how many conversations exceed one.
Check the approvals. Every band-2 and band-3 refund should have a human name attached. Refunds in those bands with no name are the leak.

At the vet clinic, this audit took 38 minutes and surfaced $4,100 in refunds over 90 days with no reason code and no approver. The practice manager's reaction was the same as everyone's: "But the cap was set."

The 12-line card

[ ] Authority matrix by band, not one global cap
[ ] Caps at call, conversation, order, customer-90d, and day level
[ ] One refund per order, enforced in code not prompt
[ ] Three-question gate: paid / delivered / first
[ ] Transaction id required in the reason field
[ ] Band 2+ needs a named human click
[ ] Daily cap freezes the tool, not the chat
[ ] Redo, credit, waive_fee, escalate tools all exist
[ ] Offer order: redo -> credit -> partial -> full
[ ] Refunds in first 4 messages counted weekly
[ ] Second refund on one order = incident, reviewed
[ ] Monthly: 30 conversations, 5 checks, defects logged

If you only do one thing this week: pull every refund from the last 30 days and count how many were the second refund on the same order. In eight of the nine deployments I've reviewed, that number was above zero, and in every one of those cases the per-call cap was configured correctly and showing green.

The authority matrix template, the three-question gate prompt block, and the monthly refund audit worksheet are in the kit I run on deployments: Agent Audit Pack

One AI agent, four stores, four different refund policies — the customer found all of them

chenwengui23 — Mon, 26 Oct 2026 02:05:31 +0000

A dental group with four locations in the Phoenix metro deployed one shared AI receptionist in March. Same prompt, same knowledge base, one deployment — that was the selling point: "one brain, four front desks." By June they had a problem no dashboard flagged: the AI was quoting four different cancellation policies, depending on which page of the knowledge base it happened to retrieve from.

The mechanics were boring. Location pages had been written by each office manager. The Downtown page said cancellations inside 24 hours incur a $50 fee. The Arcadia page said $75, plus "after two no-shows we may dismiss you from the practice." The Scottsdale page said "fees may apply at management's discretion" — which the AI, trained to be concrete, resolved into a flat $40. The fourth location's page had never been updated after a policy change and still described a policy discontinued in 2024.

A patient who rotates between locations for scheduling convenience got three different answers in one week: "$50," "actually it's $75," and "we don't charge for cancellations." She posted all three screenshots in a Google review. It's still up.

This is the multi-location policy fork, and it's the natural end state of every "one knowledge base for all our locations" deployment that doesn't have a canonical policy layer. I've now audited it at the dental group, a fitness studio chain (5 gyms), an auto repair group (3 shops), and a restaurant group (7 locations). The numbers are consistent: between 9% and 23% of policy questions received answers that were correct for a location but wrong for the customer's location. Nobody had noticed, because each individual answer looked confident and internally coherent.

Here's the math on why it's expensive. A customer who receives two conflicting answers to the same question trusts neither. In the transcript review at the fitness chain, 31% of customers who hit a contradiction asked the question again — burning minutes of handle time — and 12% of the contradicted cohort mentioned the conflict in a review, a cancellation, or a complaint. You don't lose these customers with one bad answer. You lose them with two correct-looking answers that disagree.

Why one knowledge base almost always forks

Four mechanisms, in order of how often I find them:

1. Location pages edited locally. Each manager owns "their" page and updates it when their policy changes. Nobody downgrades the change to "location-specific" vs "brand policy." Six months later the brand policy changed at HQ and three of five locations updated. The AI retrieves whichever page matches the customer's phrasing best.

2. Copy-paste drift. The chain's template was duplicated per location, then each copy evolved. A rule that started as one sentence ("no refunds on packages") now exists in five slightly different wordings — and one of them, after an edit, dropped the word "on packages." The AI reads it as a blanket no-refund policy. That's how the auto repair group's AI told a customer that packages were non-refundable entirely — the packages were the one thing that was refundable.

3. Unversioned hours and seasonal rules. Holiday hours, summer schedules, seasonal pricing. Each location updates at different times (or not at all). The retrieval layer sees all of them as equally current. The restaurant group's AI quoted three different closing times for the same Friday, per location, two of which were a year out of date.

4. Retrieval doesn't know about geography. The customer asks "what's your cancellation policy?" — no location named. The retriever scores all pages by text similarity and returns whichever wording happens to match. The customer gets a policy for a store 22 miles away. The AI doesn't know it made a choice; to it, there was only ever one answer.

The common thread: the knowledge base stores facts, but nobody defined ownership and scope. The fix isn't better retrieval. It's a canonical policy layer that says which facts are brand-wide, which are location-specific, and what the AI must do when the customer's location is unknown.

The fix: a canonical answer table with scope

Rebuild policy content as a table, not prose pages. This is the structure I deploy:

POLICY TABLE (one row per rule):

| id   | scope        | question pattern      | canonical answer                  |
|------|--------------|-----------------------|-----------------------------------|
| CXL1 | brand        | cancel/reschedule fee | $50 inside 24h, all locations     |
| CXL2 | brand        | fee exceptions        | medical emergency, one waiver/yr  |
| NS1  | brand        | no-show consequences  | 3 no-shows = booking restriction  |
| HRS1 | loc:arcadia  | hours                 | M-F 7-7, Sat 8-2, Sun closed      |
| HRS2 | loc:downtown | hours                 | M-F 8-6, Sat 9-1, Sun closed      |
| PKG1 | brand        | package refunds       | unused sessions refundable in 30d |
| SEAS1| loc:scottsdale| summer hours         | June-Aug: closed Sundays          |

RULES:
- Every row carries a scope. Brand rows must be identical across
  every surface. No location page may restate a brand row.
- Every row carries last_reviewed date. Rows older than 90 days
  surface in the monthly audit.
- If the customer's question matches a row but their location is
  unknown AND the row is location-scoped → ask which location
  first. Never guess.

Then the system prompt gets one hard rule: location-scoped questions without a known location get a location question, not a guess. "Which location do you usually visit?" costs one turn. The wrong policy costs a review.

The dental group rebuilt 43 policy facts into 28 brand rows and 15 location rows in one afternoon. The fork rate — transcripts where two policy answers in the same week contradicted each other — went from 4.1% of all transcripts to zero in the six weeks I tracked.

The contradiction audit: 30 minutes, monthly

You can find existing forks before customers do. Pull 40 random policy-question transcripts and run this:

Cross-location collision. Group transcripts by question topic. If the same question topic received materially different answers across the month (different numbers, different rules), you have a fork. List both answers and their source pages.
Brand-rule restatement. For every answer about a brand-wide rule, check whether the answer matches the brand canonical text or a location's local wording. Local wording on a brand rule = drift.
Ghost policies. Grep for policies mentioned in answers that exist on no current page — leftovers from deleted docs the retrieval layer still indexes. (At the restaurant group, 6 of 31 distinct policies the AI quoted existed nowhere in the current knowledge base. They came from a deleted doc still sitting in a vector index.)
Unknown-location guesses. Count answers to location-scoped questions where the customer never named a location. Each one is the AI silently picking for the customer.
Stale rows. Any policy answer sourced from a page untouched in 90+ days gets a review flag. Hours and seasonal rules: 30+ days.

The fitness chain ran this in 27 minutes and found 11 forks. Nine were real, live, customer-facing disagreements between locations — including one where the AI charged a member a cancellation fee that only existed at a gym she'd visited once, 14 months earlier.

The location-confirmation script

When the location is unknown and the rule is scoped, the AI needs one rehearsed move:

"That can vary a bit by location — which one do you usually go to?"

And when the customer is comparing locations:

"Downtown waives the first no-show; Arcadia doesn't. Since you move between them, I'll note your question applies to both."

That second script matters more than it looks. Customers who use multiple locations are your highest-value, highest-contradiction-exposure cohort, and naming the difference turns a trap into transparency.

The weekly scorecard

Week of: ______
Policy questions answered:                 ____
Cross-location contradictions found:       ____
Brand rules answered from local wording:   ____
Answers citing ghost/deleted sources:      ____
Location-scoped Qs answered w/o location:  ____
Rows past 90-day review:                   ____
Customers who asked the same Q twice:      ____

The metric that matters most is the first counter. Contradictions found by your audit are cheap; contradictions found by customers are the Google review that quotes both answers side by side. Target: audit finds ≥90% of forks before any customer does — which you can check, because every contradiction the audit missed has a customer transcript attached to it.

Eight weeks after the dental group deployed the table: zero cross-location contradictions, the review with three screenshots got buried under 40 newer reviews, and — the part the owner liked most — their monthly policy updates went from "email all four managers and hope" to editing one table row, scoped, dated, and live in every location's answers within minutes.

The 12-line card

[ ] Every policy fact has a scope: brand or location:id
[ ] Brand rules restated nowhere locally; local rules never brand
[ ] Unknown location + scoped rule → ask, never guess
[ ] Every row carries last_reviewed; 90d audit, 30d for hours
[ ] Contradiction audit: 40 transcripts, 5 checks, monthly
[ ] Ghost policies (deleted-source answers) hunted monthly
[ ] Location-confirmation script rehearsed, one turn, no apology
[ ] Multi-location customers get differences named, not hidden
[ ] Policy updates = one row edit, one owner, no manager email chain
[ ] Retrieval index diffed against live table monthly
[ ] Two contradicting answers in one customer's week = incident
[ ] Fork rate (contradictions / policy Qs) on the ops dashboard

If you only do one thing, search this month's transcripts for your two most-asked policy questions and compare the answers across weeks and locations. In the four chains I've audited, every single one had at least one live fork on their top-three question — and none of them knew until the transcripts were side by side.

The canonical policy table template, the contradiction audit worksheet, and the location-scope prompt block are in the kit I run on deployments: Agent Audit Pack

Your AI said 'one moment please' — and the customer heard nothing for 9 minutes

chenwengui23 — Sun, 25 Oct 2026 02:06:02 +0000

A pest-control company in Georgia ran the numbers on their AI phone agent and found something their dashboard will never show. The agent handled 1,900 calls last month. Average handle time: 2 minutes 40 seconds. Satisfaction tags: fine. But 17% of calls — roughly one in six — contained the same pattern: the AI said some version of "let me check that for you, one moment," went silent, and never came back. The caller waited on the line. The median silent stretch was 4 minutes. The longest was 9 minutes 12 seconds, after which the caller hung up and called a competitor.

The agent didn't crash. It didn't escalate. It just... stopped talking, because the lookup behind it was slow, the barge-in detection missed two "hello?" attempts, and nobody had ever written a rule for what the AI should do while it works.

I've now audited this failure at five deployments — pest control, a med-spa, an HVAC franchise, a law firm intake line, an auto parts distributor. The pattern is identical everywhere: dead air is not a system failure your monitoring catches, because the system considers the call "active." Nothing errors. Nothing alerts. The customer just hears nothing, forms a judgment about your business, and leaves.

Here's the math that makes it a priority. Across those five deployments, silent stretches over 90 seconds preceded a hang-up 41% of the time. Of callers who hung up mid-silence, 63% never called back that day. You bought an AI agent to answer every call, and it's quietly creating the most expensive kind of call: the one where the customer was engaged, asked for something specific, and got abandoned mid-request.

Why silence happens (and why your vendor won't mention it)

Four mechanisms, all ordinary.

1. Long tool calls with no filler. The agent calls a scheduling API, an inventory system, a CRM lookup. If that round trip takes 20–90 seconds, the raw agent just waits. Humans fill silence ("bear with me, it's being slow today"); bots don't, unless told to.

2. Barge-in detection failures. The caller says "hello? are you there?" If your speech pipeline doesn't register that as an interruption event, the AI keeps waiting for its tool result while the caller's questions vanish into the void. At the pest-control company, 22 of the 61 calls with long silence contained at least one unanswered "hello" — the AI literally never heard it.

3. "One moment" as an escape hatch. When the agent doesn't know the answer, its training drifts toward "let me check on that" as a safe stall. Then the check finds nothing, and the agent — with no fallback rule — says nothing. The stall becomes a hole.

4. Transfers that aren't transfers. The agent says "I'll connect you with someone" and dials a queue that's closed, or rings a human who doesn't pick up. Dead ring tone. The customer thinks they're being helped. They're being parked.

The fix: a silence protocol the AI must follow

Write it as hard rules, not vibes. This is the block I paste into every deployment's system prompt:

SILENCE PROTOCOL (voice and chat agents):

1. Any tool call expected to exceed 5 seconds → fill within 2 seconds:
   "Give me about 20 seconds to pull that up."
   Never promise "one moment." Name the duration.

2. If the tool call exceeds the named duration → update again:
   "Still working on it — about 10 more seconds."
   Update at least every 20 seconds, forever. There is no
   point at which silence becomes acceptable.

3. If the caller speaks while the agent is working ("hello?",
   "you there?") → respond within 1 second:
   "I'm here — still pulling that up, about 15 more seconds."
   Never resume the original answer without acknowledging them.

4. If a lookup fails or returns nothing after 60 seconds:
   "I can't pull that up right now. Here's what I can do —
   I'll take your number and have someone call you back
   within the hour, or you can [alternative]."
   Never terminate on a failed lookup. Never re-ask the
   question from the start.

5. If a transfer rings more than 15 seconds with no pickup:
   return, acknowledge, and offer the callback instead:
   "That line isn't picking up right now — let me get a
   callback scheduled for you instead."

6. Hard cap: 90 seconds of no customer-facing output on ANY
   channel, for ANY reason. Log every violation as an incident.

Rule 6 is the one that changes behavior, because it makes silence measurable. Most platforms log "call connected" and "call ended" but nothing in between. If your agent can't count its own output gaps, count them in the transcript post-processor: any 90-second span with no agent utterance is an incident with a timestamp and a cause.

The recovery scripts: what to say after the hole

Some silence will still happen — the vendor updates a model, a provider has a bad day. What separates a recoverable moment from a lost customer is the first sentence after the gap. Keep these five ready:

Situation	Script
Caller said "hello?" twice, unanswered	"I'm sorry — I'm here, I heard you. Still pulling that up. About 20 more seconds, and I'll stay with you."
Long lookup finally returned	"Thanks for hanging in there — that took longer than it should have. Here's what I found:"
Lookup failed	"I wasn't able to get that. I don't want to guess. Two options: I take your number for a callback within the hour, or I connect you with [alternative channel]. Which works better?"
Transfer rang out	"That team isn't picking up right now. Rather than leave you hanging, I've noted your request and [person/team] will call you back — should I use the number you're calling from?"
Caller is audibly annoyed	"You're right to be annoyed — that pause was on us. Let me make the rest of this quick: [proceed to resolution, skip pleasantries]."

Note what the scripts never do: pretend nothing happened, restart the conversation from the greeting, or apologize more than once. One acknowledgment, one forward path, one question. The annoyed-caller script deliberately drops all warmth scaffolding — annoyed people want speed, not personality.

The wait-time audit: 20 minutes, monthly

Pull 50 random transcripts from the month and run this check. It's mechanical:

Longest agent-output gap per transcript. Anything over 90 seconds is an incident. Log: timestamp, cause (tool call / failed lookup / transfer / unknown).
Unanswered barge-ins. Search for caller utterances ("hello", "are you there", "anyone") with no agent response within 2 seconds after. Each one is a defect.
"One moment" inventory. Count stall phrases that were never followed by a result. "Let me check" with no follow-up is a hole, not a stall.
Post-silence recovery. Did the agent acknowledge the gap, or resume as if nothing happened? Resume-without-acknowledgment reads as robotic and erodes trust even when the answer is eventually correct.
Hang-up map. For every mid-call hang-up, note the last agent utterance. If a meaningful share end within 10 seconds of a stall phrase, you've found your leak.

At the HVAC franchise, this audit took 19 minutes for 60 transcripts and found the single worst defect in the deployment: their after-hours transfer target was a number nobody had updated after a staff change — so for three weeks, every "I'll connect you with our on-call technician" ended in infinite ring tone. The vendor dashboard showed 96% "successful handling." Every metric was green; the phone was just ringing in an empty room.

The weekly scorecard

Week of: ______
Transcripts sampled:                       ____
Incidents >90s agent silence:              ____
  caused by slow tool call:                ____
  caused by failed lookup:                 ____
  caused by dead transfer:                 ____
Unanswered "hello?" events:                ____
"One moment" stalls with no follow-up:     ____
Post-silence acknowledgments given:        __ / __
Mid-call hang-ups within 10s of a stall:   ____

The metric that matters most is the last line. A hang-up right after a stall phrase is the customer telling you, in the clearest signal a phone line can produce, that they gave up. Target: zero. Everything else on the card is a leading indicator for it.

Six weeks after the pest-control company deployed the protocol: their longest silent stretch that month was 31 seconds (a slow inventory API, filled with two updates), unanswered barge-ins dropped from 22 per month to 1, and the failed-lookup callback path caught 38 requests that previously would have evaporated. Two of those turned into jobs worth $340 total. The protocol was one afternoon of prompt work.

The 12-line card

[ ] No "one moment" — name the duration: "about 20 seconds"
[ ] Status updates every 20s during any tool call, no ceiling
[ ] Barge-ins answered within 1s: acknowledge, then re-state ETA
[ ] Failed lookup → callback path, never re-ask from the start
[ ] Dead transfer >15s → return, acknowledge, offer callback
[ ] Hard cap 90s output gap on every channel, logged as incident
[ ] Silence incidents have a cause tag: tool / lookup / transfer
[ ] 5 recovery scripts live in the prompt, rehearsed monthly
[ ] Post-silence: one acknowledgment, one path, one question
[ ] Monthly audit: 50 transcripts, 5 checks, 20 minutes
[ ] Mid-call hang-ups mapped to last agent utterance
[ ] One owner; vendor dashboard never treated as proof of health

If you only do one thing, grep this month's transcripts for "one moment" and check what followed it. In five deployments I've audited, between 8% and 17% of those stalls were followed by nothing at all — and none of the five owners knew until the transcript was on the screen.

The silence protocol prompt block, the five recovery scripts, and the wait-time audit worksheet are in the kit I run on deployments: Agent Audit Pack

Your voice agent waives the fee and your chat agent charges it — same AI, same week

chenwengui23 — Sat, 24 Oct 2026 03:51:29 +0000

A dental group in Ohio runs four clinics. Their AI stack is tidy: one vendor, one knowledge base, two channels — an AI phone agent that answers after-hours calls, and a web chat agent on the booking page. Same pricing doc feeds both. The owner assumed "one knowledge base" meant "one set of answers."

It doesn't. Channels are not pipes. Each channel is a generation context, and generation re-invents the answer every time.

I pulled 60 conversations from each channel for the same week and sorted them by topic. On rescheduling, the two channels gave materially different answers to the same question:

Question	Voice agent (phone)	Chat agent (web)
"Can I move my appointment to Thursday?"	"Of course, I've moved it — no charge."	"Rescheduling within 24 hours incurs a $25 fee. Shall I proceed?"
"Do you take Delta Dental?"	"Yes, we're in-network with Delta."	"We accept most major PPO plans. Please confirm with your provider."
"How long is the cleaning?"	"About 45 minutes."	"Typically 60 minutes."
"Can I get a same-day appointment?"	Booked one, 2:30 PM.	"Same-day availability is limited — please call us." (It didn't call.)

Four topics, four contradictions. The fee contradiction is the expensive one: a customer who chatted first, heard "$25," then called and heard "no charge," now knows the AI makes it up as it goes. That's not a support problem anymore — that's the moment your pricing becomes negotiable in the customer's mind. Two customers that week opened with "well, the phone robot said it's free."

Same knowledge base. Different behavior. Here's why, and here's the audit that catches it.

Why channels drift apart

Three mechanisms, all boring, all predictable.

1. The system prompts were written months apart by different hands. The chat agent's prompt says "always mention the late-rescheduling fee when a change is requested." The voice agent's prompt says "be accommodating and helpful." Neither contradicts the pricing doc, but one foregrounds the fee and the other foregrounds flexibility. Given a choice between strictness and smoothness, the voice agent was tuned for smoothness — because someone, sometime, complained that the phone bot was rigid.

2. Voice has a latency tax. Every extra clause costs seconds. Someone optimized the voice flow for short answers, and "no charge" is shorter than the compliant "there's no charge because you're outside the 24-hour window; within 24 hours it would be $25." The qualifier is where the policy lives, and qualifiers are the first thing cut when answers get trimmed. So the voice agent answers correctly but incompletely, and the incompleteness reads as a different policy.

3. The channels escalate differently, so the feedback loops never meet. Chat transcripts flow into one inbox; call summaries flow into another. Nobody ever put the two side by side and asked whether the AI said the same thing in both. The med-spa problem from last month's audit — Spanish vs. English — is this same bug wearing a different hat. Every axis that splits your audience (channel, language, location, hour of day) quietly forks your policy.

The fix: one answer table, channel-checked

The structure that works is boring: a canonical answer table with a per-channel confirmation column. One row per money- or commitment-bearing question. The rule the AI must follow:

For any question touching fees, insurance, duration, availability,
or guarantees:
1. Look up the answer in ANSWER_TABLE by intent key.
2. Reproduce the approved answer for the CURRENT channel verbatim,
   including the qualifier sentence. Do not shorten.
3. If the current channel is voice and the approved answer exceeds
   40 words, use the VOICE_VARIANT column — but the VOICE_VARIANT
   must be pre-approved by a human, never generated.
4. If no row exists for the intent: say "Let me confirm that with
   a human" and escalate. Do not compose a new policy answer.

And the table itself:

intent_key        | canonical answer (web/voice identical core)          | voice variant (pre-approved)
------------------+------------------------------------------------------+----------------------------------------------
resched_fee       | Rescheduling is free up to 24 hours before your      | "No charge — you're outside the 24-hour
                  | appointment. Inside 24 hours, there's a $25 fee.     |  window. Inside 24 hours it's $25."
                  | Which day works for you?                             |
insurance_delta   | We are in-network with Delta Dental PPO at all       | "Yes — in-network at all four clinics."
                  | four clinics. We do not accept Delta HMO.            |
cleaning_duration | A standard cleaning takes 45 to 60 minutes depending | "45 to 60 minutes, depending on X-rays."
                  | on whether X-rays are due.                           |
same_day          | Same-day slots open at 7 AM daily and are first-     | "Same-day slots open at 7 AM — I can
                  | come. The chat cannot book them; call or ask the     |  check, but booking happens by phone."
                  | phone agent.                                         |

The last row is the one everyone forgets: channels must know each other's limits. Telling a chat customer "please call us" without saying why or when converts one channel's constraint into the customer's confusion.

The cross-channel audit: 30 minutes, monthly

This is the part you can copy-paste. Take the same 8 questions to every channel you run, in the same week, and diff the answers. Not for style — for substance.

Can I reschedule, and does it cost anything?
Do you take [top insurer]?
How long does [core service] take?
Can I get [core service] today?
What's your cancellation policy?
What does [core service] cost, all-in?
What happens if I'm late?
Do you offer refunds or a guarantee?

Score each channel pair against three defect classes:

FEE/POLARITY DRIFT — one channel charges, waives, includes, or excludes something the other doesn't. Always a defect.
NUMBER DRIFT — durations, prices, windows, counts differ. Always a defect.
CAPABILITY DRIFT — one channel can book/transact/answer what the other claims it can't. Usually a defect, occasionally intentional — but if it's intentional, the other channel must say so.

At the dental group, the monthly audit took 26 minutes and caught a fifth contradiction the week it appeared: the voice agent had started quoting a "$99 new-patient special" that marketing had loaded into the phone flow but that the front desk had never agreed to honor. The audit found it in month one; before the audit existed, the median detection time for a channel contradiction in my deployment sample was nine weeks — usually discovered by a customer repeating one channel's answer to a human.

The weekly scorecard

One table, five rows, Monday morning:

Week of: ______
Channels active:                        ______
8-question audit run on all channels:    __ / __ cells
  FEE/POLARITY drift findings:           ____
  NUMBER drift findings:                 ____
  CAPABILITY drift findings:             ____
Voice answers over 40 words (latency risk): ____
Escalations with cross-channel mentions:     ____
  ("the phone robot said...")            → resolved: ____

The metric to watch is customer-cited contradictions — how many tickets or calls open with "the other channel told me X." It should be zero, and it's the only metric that captures the reputational cost, which never shows up in the AI vendor's dashboard.

Nine weeks after the dental group installed the answer table: the rescheduling contradiction was gone (the fee qualifier became a pre-approved voice variant, 22 words), insurance answers matched across channels, and the phantom $99 special was either loaded everywhere or nowhere — their choice, made consciously, by a human. The two customers who'd negotiated off "the phone robot said it's free" had cost $150. The answer table cost one afternoon.

The 12-line card

[ ] One canonical answer table; channel variants pre-approved by a human
[ ] Fee, insurance, duration, availability answers looked up, not composed
[ ] Voice variants capped at 40 words, qualifiers never cut for latency
[ ] Unknown intent → escalate with "let me confirm," never invent policy
[ ] Channels know each other's limits and say why when redirecting
[ ] 8-question cross-channel audit run monthly, same questions every time
[ ] FEE/POLARITY, NUMBER, CAPABILITY drift each treated as defects
[ ] New pricing or policy change → answer table updated same day, all channels
[ ] Chat and call transcripts reviewed in one place, side by side
[ ] Customer-cited contradictions tracked weekly, target zero
[ ] New channel (WhatsApp, DMs, kiosk) onboarded with the same table
[ ] One owner; audit plus review under 45 minutes/month

If you only do one thing, run the 8-question audit once — chat agent, voice agent, whatever else you run — and put the answers side by side. Twenty minutes. In most deployments I've audited, at least two of the eight answers disagree across channels, and the owners had no idea until the table was on the screen.

The 8-question cross-channel audit script, the answer table template with voice-variant columns, and the monthly scorecard are in the kit I run on deployments: Agent Audit Pack

Your AI switched to Spanish flawlessly and quoted a 90-day warranty that only exists in Spanish

chenwengui23 — Fri, 23 Oct 2026 13:08:27 +0000

A med-spa chain in Texas runs two locations, and 41% of its inbound messages arrive in Spanish. The owner was proud of this: her AI agent answered in Spanish on day one, no plugin, no extra vendor. "It's better Spanish than mine," she told me.

It was. That was the problem.

I pulled 300 Spanish-language conversations and put them next to the English policy documents. Three sentences had drifted, and all three drifted in the same direction — toward being more generous than the business actually was:

English policy (canonical)	What the AI told Spanish-speaking customers
"Deposits are non-refundable within 48 hours of the appointment."	"El depósito es reembolsable dentro de las 48 horas." — refundable within 48 hours. The negation vanished.
"Results vary; we cannot guarantee an outcome."	"Garantizamos resultados." — we guarantee results.
"Package pricing is valid for 30 days."	"Válido por 90 días." — 90 days.

By the time she found it, eleven customers had booked against the 90-day price and four had requested refunds on deposits they'd been told were refundable. The English side of the same AI was perfectly compliant. The Spanish side had been quietly rewriting her contract for seven months.

This is the failure mode almost nobody tests for, because the demo always sounds great. Fluency is the thing you can hear. Fidelity is the thing that breaks.

Why translation fails policy, specifically

The model isn't mistranslating words. It's doing something more reasonable and more dangerous: it's translating intent. When it reads a policy sentence, it reconstructs what the sentence means, then expresses that meaning in the target language — and every reconstruction is a chance to lose a constraint.

Four failure classes account for almost every incident I've audited.

1. Numbers and negations don't survive paraphrase. A negation is one word in English and often a prefix or a different verb in Spanish, Japanese, or German. "Non-refundable" becomes "reembolsable" because the model decomposed it into refundable + not, and dropped the modifier while generating fluent output. The same happens with "cannot guarantee," "not included," "no exceptions," "excluding taxes." Every one of these is a clause that protects the business, and every one is fragile under generation.

2. Register mismatch reads as disrespect. Spanish has tú and usted. Japanese has keigo levels. German has Sie and du. The model picks one and picks it consistently — but not necessarily the one your brand uses or the one the customer expects. In a luxury or medical context, tutear a customer is a small insult repeated in every message. Nobody files a ticket about it. They just don't come back.

3. Internal jargon survives untranslated. Product names, tier names, internal codes, and email templates leak through in English inside otherwise perfect Spanish sentences. The customer reads a message that is 95% their language and 5% your org chart.

4. Language switching breaks the thread. Customer writes in Spanish, AI answers in Spanish, customer switches to English mid-thread, AI keeps answering in Spanish — or worse, answers in a mix. And when the conversation escalates to a human, the handoff summary is generated in the customer's language, so the English-speaking agent receives a brief they can't read, or a machine translation of a machine paraphrase: two losses of fidelity before a human ever touches it.

The fix: canonical sentences, not runtime translation

Stop letting the model translate policy at request time. Policy strings get translated once, by a human, reviewed, and stored — then the AI is instructed to reproduce them verbatim.

The structure is a parallel policy sheet. One row per protected sentence, one column per language, plus a rule the model must follow:

policy_key            | en (canonical)                                  | es (approved)                                            | ja (approved)
----------------------+-------------------------------------------------+----------------------------------------------------------+---------------------------
deposit_refund        | Deposits are non-refundable within 48 hours.    | El depósito no es reembolsable dentro de las 48 horas.   | ご予約金は48時間以内のキャンセルの場合、返金いたしかねます。
results_disclaimer    | Results vary; we cannot guarantee an outcome.   | Los resultados varían; no podemos garantizar un resultado.| 施術結果には個人差があり、結果を保証するものではありません。
package_validity      | Package pricing is valid for 30 days.           | El precio del paquete es válido por 30 días.             | パッケージ価格の有効期限は30日間です。

And the instruction that makes it work:

When a reply touches deposit, refund, results, warranty, or pricing validity:
1. Detect the customer's language.
2. Emit the approved string for that language VERBATIM from the
   policy sheet. Do not translate, paraphrase, shorten, or rephrase.
3. If no approved string exists for that language, reply in English
   and set needs_translation: true in the handoff payload.

That last clause matters more than it looks: it turns an unknown-language policy question from a silent hallucination into a visible gap. At the med-spa, Portuguese-speaking customers had been getting fluent Portuguese policy answers reviewed by nobody — a third language quietly running its own version of the contract.

The back-translation gate

For policy sentences you can't pre-approve — edge cases, one-off concessions — run a back-translation check before the message sends. It's cheap: one extra model call, no human in the loop.

Original English policy sentence: <s>
AI's outbound message: <m>
1. Translate <m> back to English, literally.
2. Compare to <s>. Report any of:
   - NUMERIC DRIFT (any number, unit, or duration differs)
   - POLARITY DRIFT (a negation appears or disappears)
   - COMMITMENT DRIFT (a hedge like "may," "varies," "cannot"
     becomes a promise, or vice versa)
3. If any drift: DO NOT SEND. Route to human review with both texts.

This catches exactly what the med-spa lost. "Válido por 90 días" back-translates to "valid for 90 days," which does not match "valid for 30 days" — numeric drift, blocked. "Garantizamos resultados" back-translates to "we guarantee results," which has no hedge — commitment drift, blocked.

In one deployment I instrumented, the gate blocked 2.8% of outbound messages in month one. That sounds like friction until you see what was in the 2.8%: not just policy drift, but addresses, phone numbers, and appointment times mangled in transit. Schedule errors are the same bug in a different hat — a number that didn't survive the round trip.

The 12-question probe set

You don't need a native speaker on staff to test this. You need the same twelve questions asked in every language you support, with the answers diffed against the English baseline. Translate them once with a human, keep them permanently.

How much is the deposit, and is it refundable if I cancel tomorrow?
What's the price of the package, and how long is that price good for?
Do you guarantee results?
What happens if I'm not satisfied — do I get my money back?
Can I reschedule without a fee? How many times?
Is parking included?
Do you take my insurance?
What's the cancellation window?
Is there a late fee, and how much is it?
Can I bring my child to the appointment?
Who do I contact if something goes wrong after?
Are there any fees not mentioned on the website?

Every one of these contains a number, a negation, a hedge, or a commitment. Run all twelve in each language, weekly, and diff the answers against the English baseline. Any divergence in a number, a negation, or a hedge is a defect, not a wording preference. You are not grading style. You are grading whether the contract is the same in every language.

The weekly scorecard

One table, five rows, Monday morning:

Week of: ______
Languages active:                       ______
Probe set run (12 q x n langs):          __ / __ pairs
  NUMERIC DRIFT findings:                ____
  POLARITY DRIFT findings:               ____
  COMMITMENT DRIFT findings:             ____
Back-translation gate blocks:            ____  → resolved: ____
Escalations handed off in non-English:   ____  → summary readable by agent: ____

The number to watch is drift findings per 100 probe pairs. It should be zero after the first two weeks of canonical coverage, and any non-zero reading means someone edited a policy document without updating the other languages. That's your change-management alarm, and it rings weeks before a customer cites the wrong policy back at you.

Six weeks after the med-spa installed this: the deposit language was corrected in Spanish and Portuguese, the results disclaimer was pinned as a verbatim string, the 90-day phantom price disappeared, and English-only agents started receiving two-line English briefs on Spanish escalations instead of a transcript they couldn't read. The four refund disputes were honored as promised — roughly $1,100 — which is the cheapest possible price for learning that fluency is not compliance.

The 12-line card

[ ] Policy sentences are canonical strings, translated once by a human
[ ] AI reproduces approved strings verbatim, never re-translates policy
[ ] Unknown-language policy question → reply in English + flag, not guess
[ ] Back-translation gate on any message containing a number or negation
[ ] NUMERIC / POLARITY / COMMITMENT drift each block the send
[ ] Register (tú/usted, Sie/du, keigo level) chosen once, applied everywhere
[ ] Product and tier names localized or explicitly marked as brand names
[ ] Language switching mid-thread follows the customer, not the AI
[ ] Handoff summaries generated in the agent's language, original attached
[ ] 12-question probe set run weekly in every active language
[ ] Drift findings treated as defects, not wording preferences
[ ] One owner; probe run plus review under 45 minutes/week

If you only do one thing, do the probe set. Twelve questions in the languages you actually serve will tell you in twenty minutes whether your AI is running one policy or several — and in most deployments I've audited, the honest answer was several, and nobody had counted them.

The canonical policy sheet template, the back-translation gate prompt, the 12-question probe set in six languages, and the weekly scorecard are in the kit I run on deployments: Agent Audit Pack