TL;DR
- Tokenizing at intake does nothing for the return path. Evidence agents query the CRM and raw identity comes back as a tool result, straight into model context.
- The detector cannot be a language model. The thing you are protecting against is the input.
- A regex proposes, a checksum confirms. Adding an identifier type is twenty lines.
- Tokenize into a vault, do not redact. Redaction burns the bridge you need to send the answer.
- Erasure tells you where your data actually went, and GDPR makes that a query you have to be able to run.
- The most important thing we shipped was the ability to attach the email an operator actually sent.
I thought there was one place personal data could leak out of my AI support system.
There are three. The model wasn't the leak.
In Part 2 I listed the tests we had added, including this assertion:
no PII leaks into hypothesis text
I watched it go green in CI and filed the problem as handled. It wasn't. That assertion watches the model's output, which is one door out of three, and the least dangerous of them.
Part 3 ended with productionization mostly done and a promise: that this post would cover the quiet failures, the behaviors we did not predict, and the feedback loops that changed the second iteration. That eval is the perfect example of all three, because nothing about it ever failed. It passed, every time, while watching the wrong door.
Two months later, here is the honest version. The system changed in kind, not in size, going from two runtime containers to twenty-seven services, and almost none of that came from features we planned.
What I had by Part 3 was a copilot: something that sits next to a support agent and hands them evidence. Re-reading that post, I claimed more than I had proved. I had made one ingest work for one stage of the process. That is not the same as running customer service.
A copilot can ignore where the customer's personal data ends up, because a human reads everything anyway. It does not need a work queue, because one person is looking at one card. It never needs to know what was actually sent, because it never claimed to have handled anything.
A customer-service system cannot ignore any of them. And each one arrived the same way the eval did: nothing threw, no alert fired, every check was green.
Lesson 12: the return path is a door
A ticket arrives. Evidence agents run in parallel: one asks the CRM who this customer is, one asks telemetry what fired, one checks provisioning. Each result becomes model context.
Now trace the personal data:
customer email ──► [strip] ──► "{a3f1...} reports a crash" ──┐
├──► model context
CRM lookup ──────────────────► {"name": "Jan Bakker", ──┘
"email": "jan@acme.nl",
"phone": "010-1234567"}
Both arrive in the same context window. Only one of them went through the stripper.
Intake tokenization never touched that second path. You cleaned the front door while the side door handed the same data straight back in.
So there are three boundaries, not one:
┌────────────────────┐
customer mail ──1──► │ │
│ model context │ ──3──► reply to customer
internal systems ──2──►│ │
└────────────────────┘
1 ingress inbound content best effort, cannot refuse
2 return path tool results you asked for deterministic, fail-closed
3 emission what the model writes detects, cannot prevent
Three doors, and they do not have equal guarantees. I had instrumented door 3 and partly covered door 1. The return path I had not considered at all. It is also the hardest of the three: the data comes from a system you trust, in a shape you do not control, in response to a query you deliberately made.
It has the least prior art too. There is plenty of guidance on scrubbing user input and filtering model output. For the return path I could not find a specification that treats it as a boundary at all.
Lesson 13: the detector cannot be a language model
If a model has to see raw personal data before deciding whether it is personal data, you have already lost. Detection has to finish before the model sees the text at all.
That rules out the obvious approach and leaves a deterministic one, split two ways. Microsoft Presidio supplies the recognizer registry, the context scoring and the replacement step. spaCy supplies the per-locale NER that tags the fuzzy things, people and organizations and locations, one model per supported language.
A regex proposes, a checksum confirms
spaCy is good at "Maria from accounting" and useless on a nine-digit national ID, because a structured identifier does not look like an entity. It looks like a number.
That is also where the thing is extensible. Adding an identifier type is a pattern plus a validator, and the validator does the real work. The Dutch BSN carries a checksum, so a loose match at a deliberately low score gets confirmed or discarded by arithmetic:
class NlBsnRecognizer(PatternRecognizer):
def __init__(self):
super().__init__(
supported_entity="NL_BSN",
patterns=[Pattern("nl_bsn", r"\b\d{8,9}\b", 0.3)],
supported_language="nl",
)
def validate_result(self, pattern_text):
return nl_bsn.is_valid(pattern_text) # python-stdnum, 11-proof
The regex on its own would tokenize every 8 to 9 digit number in the message: order numbers, timestamps, version codes. The checksum is what makes it precise. Every further identifier, a tax number or an IBAN or a postcode, is the same twenty lines.
Recall beats precision, except in stack traces
NER over dense business text in a non-English language is noisy. An ordinary street address survived un-redacted until we added a locale-specific recognizer for it. The default phone recognizer's region list did not include ours. Over-redaction costs a little context. A surviving secret costs everything.
One exception proves the bias cannot be blind: a .NET namespace in a stack trace reads exactly like a person's name. Inside a hard-marked technical region, person and location spans get dropped.
The bug only real text teaches you
Spans from two different mechanisms overlap partially:
"contact Jan Bakker at jan.bakker@acme.nl"
└─ PERSON ─┘
└────── EMAIL ──────┘
overlap ─┘
keep PERSON, drop EMAIL -> "contact {token} at bakker@acme.nl" leaked
union the two spans -> "contact {token}" covered
Keep one and drop the other, and the tail of the dropped span stays in the message as cleartext. Merge overlapping spans into their union and tokenize the whole substring, and every detected character is covered.
Lesson 14: tokenize, do not redact
Blacking values out is the obvious move. It is also wrong.
The point of the system is to send an answer to a real person at a real address. Destroy the data on the way in and you cannot send anything back out.
So values get replaced with placeholders, and the originals are stored encrypted in a scoped vault, resolved back at the human review step. The model stays a text-to-text function that never holds a key and never invokes the gate. The gate is deterministic middleware in the data path, so it cannot be prompt-injected away or skipped.
Two details worth stealing.
The token must reveal nothing.
{PERSON_1} mailed {PERSON_2} about {COMPANY_1} two people, one company, all typed
{a3f1c8de-...} mailed {7b02e914-...} about {c5d9...} nothing
A counter tells the reader how many distinct people are in the message. A category tells them what each value was. That is the metadata you were trying not to hand over, so the token is a random UUID and nothing else.
The matcher must be a closed allowlist, not "anything in braces":
_OPEN = re.compile(r"\{[^}]+\}") # matches {SOMETHING_1} too
_CLOSED = re.compile(r"\{[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-"
r"[0-9a-f]{4}-[0-9a-f]{12}\}")
With the open pattern, an attacker writes {SOMETHING_1} into a support email, your masking step treats it as already handled, and skips the span. You have given them a way to switch your detector off one region at a time.
Lesson 15: erasure tells you where the data really went
Under GDPR, a customer can ask you to delete their personal data and you have to actually do it. A copilot dodges this, because it reads from systems that already own the record. The moment you ingest and store customer email yourself, you control that copy, and "delete everything about this person" becomes a query you have to be able to run.
That forces the question you have been avoiding: where did all of this end up?
Further than I thought. One erasure has to reach interactions, cases, the audit log, operator notes, attachment blobs, the knowledge store, and the vault at multiple scopes. Miss one and you have not complied. You have only reported that you did.
The gap that taught me the most was a table I had stopped thinking about. The mailbox poller keeps a ledger of every message it has seen, for deduplication. It stores the raw body. It was designed as a dedup ledger, so it never got a lifecycle path: no retention, no expiry. Raw customer mail, kept forever, in a table nobody thought of as storage.
The fix is more interesting than the bug, because the obvious repair is wrong:
-- wrong: the dedup key goes with it, so the next delta sync
-- happily re-ingests the mail you just erased
DELETE FROM inbox_entries WHERE lower(sender) = :email;
-- right: destroy the content, keep the key
UPDATE inbox_entries
SET body = :redacted, sender = :redacted, subject = :redacted
WHERE lower(sender) = :email
AND body <> :redacted; -- idempotent, safe to re-run
You cannot delete the rows. An erasure that breaks idempotency re-creates the data it erased. Note the lower() as well: the stored sender is raw text from the mail API, and a case mismatch would silently under-redact while still reporting success.
Every table where data just accumulates is a retention decision you made by not making it.
Lesson 16: I had made it work for one stage, not for the process
Part 3 claimed the chat bot was only an ingest, and that any channel could feed the same endpoint. That part held. A mailbox poller now does exactly that, with no orchestrator changes.
But I was describing a smaller achievement than I thought. Real customer service is not a stage. It is intake, triage, ownership, drafting, review, sending, and knowing afterwards what happened.
It is not one channel either. The unit is the case, not the message, because the same customer problem can arrive by email, continue in a live chat, and get followed up by phone:
case
├── thread EMAIL "printing fails after update"
├── thread CHAT follow-up two days later
└── thread VOICE callback, resolved
Voice and chat are not built yet. Modelling them now is what stops the next channel from being a rewrite.
The chat channel only ever covered the middle of that process anyway. A reviewer working a stream of cases needs a worklist: filter, search, claim, reassign, approve. An adaptive card is a fine surface for one result and a terrible surface for a queue, so the operator experience left chat and became its own application.
None of that is AI work. All of it was required before the AI work could be used by more than one person, which is the real difference between a demo and a service.
Lesson 17: "deployed" is not a binary
Deployment turned out to have the same problem as that eval in Part 2. Every check was green and watching the wrong thing.
The most instructive failure of the two months: a pipeline that went green while shipping a stale image. Build succeeded. Deploy succeeded. Health checks passed. The running revision was the previous code.
Every signal said yes.
That is why deployment stopped being a step and became a protocol:
build ─► push ─► deploy green ─► verify replicas ─► flip traffic ─► watch
│ fail
└─► roll back, blue keeps serving
Blue-green instead of in-place update, replica verification before the flip, automatic rollback when the cutover fails. Add to that a production outage caused by a missing dependency in one image, present everywhere it was tested and absent where it ran.
The deploy pipeline is itself a system that can lie to you.
Lesson 18: the loop needs a keystone, and it is unglamorous
The biggest single piece of engineering in this period was a knowledge and reliability subsystem. The most important thing we shipped was much smaller: the ability to attach the email an operator actually sent back to the case it answered.
That sounds like admin. It is the keystone.
The system drafts an answer. A human edits it and sends it, often from their own mail client. At that moment the system knows what it proposed and has no idea what was said.
No recorded link between a case and the reply a human actually sent means no ground truth. And without ground truth, every piece of learning machinery upstream of it is theater.
It brings a second signal along nearly for free. Compare the draft to what was actually sent:
phi = SequenceMatcher(None, draft, sent).ratio()
if phi >= 0.98: credit = 1.0 # sent as drafted
elif phi >= 0.70: credit = 0.3 # edited on the way out
else: credit = 0.0 # rewritten: the draft earned nothing
A human rewriting your output is not a success case, whatever the model's confidence said about it. A system that scores it as one will learn its way into confident nonsense.
What I will not claim
Intake is not fail-closed, and cannot be. The three doors have different guarantees. Inbound content cannot be refused, so detection there is best-effort. The return path is deterministic and fail-closed. Emission can be inspected but not re-derived, so a check there detects a failure rather than preventing one. Underneath all of it: a control that fails open is not a control.
No detection recall number. Measuring it needs a labeled corpus of real customer mail, which is the thing I am trying not to accumulate. Structured identifiers are checksum-confirmed. For names in free text I have an over-redaction bias and no honest percentage. Anyone quoting you a PII recall figure for free text should be asked how they measured it.
No accuracy figure. There is no customer confirmation and no reopen signal joined to outcomes yet. Time-to-send is not time-to-resolution.
One thing is measured precisely, because it is a property of the runtime rather than a judgment about the world. Over 48 hours, a fully evidenced triage ran about 21 model calls and 88,000 tokens, roughly $0.40.
I am not going to turn that into a savings figure. What it replaces is the context assembly that opened Part 1: a quarter of an hour of tedious hunting across five systems, worth considerably more than forty cents of an engineer's time, and work nobody enjoys. The ratio is obviously lopsided. Making it precise would mean picking a salary, an hours-per-year and a minutes-per-ticket, multiplying them, and presenting the result as though it had been measured. A human still approves every answer, so none of this removes a person. It removes the fetching.
The contrast is the point. The easy thing to measure I can quote precisely. The things that matter most I cannot. A dashboard full of the former will quietly convince you that you have handled the latter.
What the traces are for
The reliability subsystem is built and deliberately switched off. It grades whether an answer type actually worked, and an autonomous evaluator can demote or quarantine a pattern but is structurally incapable of promoting one. The action vocabulary it gets contains no promotion. Moving closer to sending without a human requires a human.
It stays dark until real outcomes exist to feed it, which is what Lesson 18 was about. The loop is not closed. The piece that lets it start is in place.
Four posts later
Every part of this series turned out to be the same move at a different altitude.
Part 1 drew the reasoning boundaries: parallel evidence, structured outputs, a human approves anything that leaves the building. Part 2 drew the runtime boundaries, where the lesson was that runtime semantics matter more than diagrams. Part 3 drew the platform boundaries: async handoff, presentation limits, permissions that look granted and are not. This one drew the boundaries around the data, the process and the people. Those only show up once the thing stops being a demo and a customer is on the other end of it.
Find something implicit, make it explicit, make it checkable. Every time. None of the failures were clever. They were assumptions nobody wrote down, which is exactly why nothing threw when they turned out to be wrong.
Which brings me to the title. This series is called "I built a multi-agent AI support copilot," and I would not call it that today. A copilot is judged on whether its suggestion was useful. A customer-service system is judged on whether the problem went away, whether the customer's data was handled properly on the way through, and whether the person who pressed send could see what they were signing. Those are not harder versions of the same question. They are different questions, and answering them is the whole distance between the two names.
If you take one thing from four posts, take this: the check that passes while watching the wrong door is more dangerous than the check that fails. A failing check gets fixed on Tuesday. A passing one buys you months of unearned confidence. The eval in Part 2 wasn't wrong. It was answering a smaller question than I thought, and it took two months of production to notice.
The loop is not closed. But the system now records enough about its own behavior to eventually tell me whether it works, which is further than it could claim when this series started. When it closes, that will be worth writing about.
Thanks for reading.
Top comments (0)