DEV Community

chenwengui23
chenwengui23

Posted on

Your AI said the chat disappears in 30 days. It was still in five systems on day 400.

#ai

A regional HVAC and plumbing group — 26 branches, 18,000 conversations a month, a 240-person service org, an AI answering the front door of every branch — spent a quarter telling customers something that was not true about its own infrastructure. Not a pricing error. Not a policy misread. A statement about where its data lives, generated by a model that had never seen the data architecture.

The trigger was a $1,400 dispute over a compressor the customer said was quoted as covered. Her claim rested on a chat she had exported and emailed to herself. The company's written response to her demand letter stated that chat transcripts are purged after 30 days, and produced nothing. Opposing counsel then attached her export. The transcript was 137 days old and included a sentence from the AI that read: "This conversation is only stored for 30 days, so please save anything you need."

The case settled for $2,900 plus fees. The money was the cheap part. The expensive part was that the company had made a verifiable factual claim about its own systems, in writing, to a customer who was already in a dispute — and the claim was wrong in the one direction that looks like concealment.

What the logs actually showed

Ninety days, one retention question at a time:

  • 611 conversations contained a question about whether the chat was kept, how long, or who could see it
  • The AI answered with a specific retention number in 588 of them (96%) — and produced five different numbers: 7 days (34), 30 days (417), 90 days (89), "one year" (31), "as long as needed for your account" (17)
  • No number came from a system. All five came from prose in the system prompt, written by three different people over fourteen months
  • 74 erasure requests were processed in the quarter. 29 (39%) were closed as "deleted" while the transcript still existed in at least two stores
  • Agents burned 41 hours in the quarter manually hunting copies of chats for customers, auditors, and one lawyer
  • The transcripts themselves lived in five places: the helpdesk (no purge job at all), CRM activity notes (indefinite), the call/co-browse recording vault (13 months), the LLM provider's abuse-monitoring log (30 days, deletable only by opening a contract ticket — nobody ever had), and the analytics warehouse (25-month rolling partition)

Read that list again. Five stores, four different real retention periods, zero of them 30 days. The AI picked the number that sounded most reassuring and repeated it 417 times.

A retention promise is not a policy statement

This is the distinction that fixes the problem, and almost nobody makes it.

"We refund within 5 business days" is a policy statement. It describes what you choose to do. It can be wrong and you fix it by changing the policy.

"We delete this chat after 30 days" is a fact statement about your infrastructure. It is either true or false right now, and the model has no way to know which. It cannot be improved by better prompt wording, because the wording was never the issue — the model was answering a question whose answer lives in a configuration table it has never read.

So there are four different questions hiding behind "do you keep this?" and they need four different answers:

Question class Example Who can answer
Retention fact "How long do you save this chat?" Only the canonical retention table
Recording disclosure "Is this call being recorded?" Recording config of the specific channel, at call start
Copy request (DSAR) "Send me everything you have on me" Human process with a statutory clock
Erasure request "Delete everything you have" Human process, verified per store, with legal-hold check

The AI should be allowed to answer exactly one of those four — the first — and only by reading a row. The other three are process handoffs. The 29 bad erasure closures happened because the AI was allowed to confirm deletion, which is the single most dangerous sentence in support.

Step 1: One table, one row per place the data actually goes

Not a policy document. A table with a row per physical store, an owner, and a verification date.

{
  "retention_facts": [
    {
      "store": "helpdesk.tickets",
      "contains": ["chat_transcript", "attachments"],
      "retention_days": null,
      "retention_basis": "no_purge_job",
      "delete_mechanism": "manual_row_delete",
      "verified_at": "2026-11-02",
      "answer_allowed": false
    },
    {
      "store": "llm_provider.abuse_log",
      "contains": ["prompt", "completion"],
      "retention_days": 30,
      "retention_basis": "vendor_default",
      "delete_mechanism": "contract_ticket_only",
      "verified_at": null,
      "answer_allowed": false
    },
    {
      "store": "recording_vault",
      "contains": ["audio", "co_browse_video"],
      "retention_days": 395,
      "retention_basis": "scheduled_purge",
      "delete_mechanism": "purge_job_vault_07",
      "verified_at": "2026-11-10",
      "answer_allowed": true
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Two fields do the work.

answer_allowed is false for any store that has no verified delete path. In the case above, four of five rows would be false, which means the AI's correct answer to "how long do you keep this?" is not a number — it's "I can't give you an exact retention period from this chat. I can have our privacy team send you the specifics in writing." That sentence is unglamorous and it is the only honest one available.

verified_at is the date someone last proved the purge actually ran. A retention period nobody has verified is a guess with a number attached. Set it to null and the row is disqualified from being quoted, which is exactly what you want.

Step 2: The gate

{
  "retention_gate": {
    "retention_fact": {
      "say": "Everything I can quote is in our retention notice — I'll send it to you now rather than guess.",
      "action": "attach_retention_notice",
      "forbidden": ["any day count not read from a verified row", "approximations like 'about a month'"]
    },
    "is_this_recorded_at_call_start": {
      "say": "This call is recorded for quality and training. You can ask me to stop the recording at any point.",
      "action": "read_channel_recording_config",
      "forbidden": ["recording without disclosure", "disclosing a config that doesn't match the channel"]
    },
    "specific_message_history": {
      "say": "Your conversation history is attached to your account so the next person doesn't make you repeat it. I can't tell you the exact expiry from here.",
      "action": "handoff_to_privacy",
      "forbidden": ["inventing a per-message retention rule"]
    },
    "copy_request_dsar": {
      "say": "I'm logging this as a formal data request. Our privacy team will confirm receipt within 72 hours and send everything we hold.",
      "action": "create_dsar_ticket",
      "forbidden": ["pasting a partial transcript inline as if complete", "any promise of 'everything'"]
    },
    "erasure_request": {
      "say": "I've logged your deletion request. A person will confirm what can be deleted, and what we're required to keep, in writing.",
      "action": "create_erasure_ticket",
      "forbidden": ["the words 'deleted', 'permanently', 'erased'", "closing the ticket in chat"]
    },
    "legal_hold_active": {
      "say": "This account is under a retention hold, so I can't change what's stored. Our privacy team will explain what that means for your request.",
      "action": "route_legal_hold",
      "forbidden": ["accepting the erasure", "explaining the hold's reason"]
    },
    "vendor_copy_question": {
      "say": "Our AI provider processes these messages on our instructions; retention there is governed by our contract with them.",
      "action": "attach_subprocessor_list",
      "forbidden": ["claiming zero vendor copies", "naming a subprocessor not on the current list"]
    },
    "sensitive_or_minor_data": {
      "say": "This involves information with special handling rules. I'm routing it to a person now.",
      "action": "handoff_immediate",
      "forbidden": ["any retention answer at all"]
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Note what the erasure branch forbids: the words themselves. The AI is not permitted to say "deleted" in any form until a human has verified per-store execution. That one constraint would have prevented all 29 bad closures, because the closing sentence was the entire failure.

Step 3: A 200 from a delete endpoint is not a delete

The verification loop is where retention programs die. Deleting a row in the helpdesk does not touch the CRM note, the warehouse partition, or the vendor log. So the purge job has to prove itself:

def purge_and_prove(store, cutoff):
    before = store.count(where={"created_at": {"lt": cutoff}})
    store.delete(where={"created_at": {"lt": cutoff}})
    after = store.count(where={"created_at": {"lt": cutoff}})
    # a no-op delete that still returns 200 is the default failure mode
    if after >= before:
        alert(f"{store.name}: purge no-op, {before} rows still present")
        return False
    if after > 0:
        alert(f"{store.name}: partial purge, {after} rows remain past retention")
    return after == 0
Enter fullscreen mode Exit fullscreen mode

Run it on every store, write the result to the retention table, and stamp verified_at. Any store without a fresh stamp loses answer_allowed. That is the whole governance model: the AI can only quote what has been proven this month.

The forbidden list

Instead of... Say...
"We delete chats after 30 days" "I'll send you our retention notice"
"Your data has been deleted" "Your deletion request is logged; a person will confirm in writing"
"We don't share data with third parties" "Here's our current subprocessor list"
"This call isn't recorded" Read the channel config; disclose if it is
"About a month" / "roughly 90 days" No approximations. A number or nothing.
"Everything we have on you is in this chat" "Our privacy team will send everything we hold"

The Monday audit

-- 1. Retention answers with no source row
SELECT COUNT(*) FROM retention_answers
WHERE source_row_id IS NULL;
-- target: 0

-- 2. Stores past retention that still hold rows (purge drift)
SELECT store, rows_past_cutoff FROM retention_purge_runs
WHERE run_at > NOW() - INTERVAL 7 DAY AND rows_past_cutoff > 0;
-- target: 0 rows

-- 3. Erasure tickets closed without per-store verification
SELECT COUNT(*) FROM erasure_requests
WHERE status = 'closed' AND verified_stores < required_stores;
-- target: 0

-- 4. Stores with answers enabled but no verification this month
SELECT store FROM retention_facts
WHERE answer_allowed = 1 AND verified_at < NOW() - INTERVAL 31 DAY;
-- target: 0

-- 5. DSAR acknowledgement latency
SELECT AVG(TIMESTAMPDIFF(HOUR, received_at, acknowledged_at)) FROM dsar_requests;
-- target: <= 72 hours, zero nulls
Enter fullscreen mode Exit fullscreen mode

Query 2 is the one that predicts incidents. Every week it returns rows is a week your AI is making a claim your infrastructure is about to contradict.

The 12-line checklist

  1. Retention facts live in a table with one row per physical store — not in the system prompt
  2. Every row has an owner, a delete mechanism, and a verified_at date
  3. The AI answers a retention question only from a row with answer_allowed = true
  4. No row gets answer_allowed = true until a purge has been run and counted
  5. The AI never says "deleted", "permanently", or "erased" — humans confirm in writing
  6. Erasure tickets close only when every required store is verified, not when one API returns 200
  7. Legal holds are checked before an erasure is accepted, and their reason is never explained by AI
  8. Recording disclosure reads the channel config — no generic "this call may be recorded" when it isn't
  9. Vendor and subprocessor copies are named from a current list, never denied
  10. DSARs are acknowledged within 72 hours by a human process with a clock
  11. The weekly audit runs all five queries; any row in query 2 blocks retention answers for that store
  12. The retention notice is attached, not paraphrased — one document, one version, one URL

The five numbers the AI invented were all plausible. That's the trap: a model asked about data retention will produce something that sounds like a compliance answer, because it has read thousands of them. It has never read your S3 lifecycle rules. Until retention is a looked-up value with a proof date attached, every reassurance your AI gives about deleting data is a guess you will eventually have to defend in writing.


If you're deploying AI agents on customer-facing work, I write up a new failure mode like this one every day — with the schema, the gate, and the audit queries — at https://whop.com/agentfast-04b6/.

Top comments (0)