How to keep PII out of your alert pipeline

Originally published at theculprit.ai/blog/keep-pii-out-of-alert-pipeline.

---

Every alert that crosses your wire has a PII problem you have not yet acknowledged.

This is not a sermon. It is a thing you can verify in about ten minutes. Open the last hour of whatever your team uses for paging — Slack, email, a webhook fan-out, a chat-ops channel — and grep for @. You will find customer email addresses in stack traces. You will find them in user-supplied form data echoed back into the error message. You will find them in the body of "user X did Y and it failed" notifications that someone wrote in 2022 and nobody has touched since. Now grep for 192., 10., 172.16. — there are your customer IPs, surfaced into a chat tool whose retention you do not control. Now grep for Bearer — those are the API tokens your authentication middleware accidentally included in the panic dump.

The reason this happens is not negligence. The reason it happens is that the alert pipeline is the one place in your stack where the rules of "what data is allowed to leave the boundary" were never written down. The application layer has an ORM that knows which fields are PII. The data warehouse has a column-level access policy. The customer-facing API has a serializer with explicit field allowlists. The alert pipeline has a console.error(err) and the assumption that whoever reads the alert is sufficiently trustworthy. That assumption stops being true the moment the alert routes to a third-party LLM, a vendor support portal, or a Slack workspace whose member list has drifted.

This piece is about what to actually do about that.

01 — Why the obvious fixes don't survive contact with reality

There are three obvious fixes. All three fail in interesting ways. Walking through why is most of the work.

Fix one: strip PII at the application layer before the alert is emitted. The pitch is clean — every logger.error() call site gets wrapped in a sanitizer that knows the domain types and redacts accordingly. In practice this fails along two axes. The first is enforcement: you cannot reliably grep your codebase for "every place a user-supplied string ends up in an error path." Stack traces capture local variables in some runtimes. Third-party libraries throw errors whose messages include the offending input verbatim. A request-validation middleware throws Invalid email: [e-mail] and there is no place in your application code where you "decided" to log that. The second is drift: even if you ship a sanitizer today, the next engineer adds a new field, a new exception type, a new integration that emits its own errors, and the sanitizer's allowlist quietly stops keeping up. The fix degrades to a security checklist item that nobody owns.

Fix two: drop alerts that contain PII. This sounds principled until you remember why you have an alert pipeline. The whole point is that something is broken, and the alert is your evidence. If your detector flags an alert as containing PII and your response is to discard it, you have built a system that hides the bugs that involve customer data — which is approximately every interesting bug. The detector also has false positives. Drop on false positive and you have built a system that drops random alerts at random rates. This fix tends to get rolled out enthusiastically, generate one P0 about a missed page, and get rolled back within a week.

Fix three: redact in the LLM prompt. This one is recent. The shape is "we use an AI tool to triage incidents; we'll add a redaction layer in front of the prompt." It fails because by the time the data reaches that redaction layer, it has already been written to your alert store, your queue, your log aggregator, your chat tool, and probably your email. The LLM prompt is not the boundary. The ingest path is the boundary. Redacting at the prompt is solving a symptom three hops downstream of the cause.

The pattern in all three is the same: each fix tries to add a filter at one specific point in the pipeline, while the real problem is that the pipeline has many exit points and the data is in plaintext at all of them.

02 — The shape of a fix that does work

The architecture worth building has four properties. None of them are novel. Most of them appear in compliance-driven sectors — healthcare, payments — where the obvious fixes have been tried and discarded for thirty years. They are unfamiliar to the observability stack only because observability has historically been treated as an internal tool, not as a data plane that crosses trust boundaries.

The properties:

1. The boundary is at ingest, not at presentation. The first thing that happens to an inbound alert is encryption-and-vault. Plaintext does not survive the receiving handler.
2. PII is replaced with reversible placeholders before any downstream consumer sees the payload. Correlation, LLM analysis, notification, log aggregation — all of these operate on a sanitized event whose fields read like <TOKEN_a1b2c3> rather than [e-mail].
3. The placeholder ↔ value map is per-tenant and encrypted with a tenant-scoped key. A leak of one tenant's vault cannot unlock another tenant's tokens. A leak of the application database without the per-tenant key cannot unlock anything.
4. Reveal is a single audited route. When an authorized user needs to see the original value — debugging a customer-reported incident, responding to a subpoena — they do so through one endpoint that checks tenant scope on every call and writes to an append-only audit log.

The composite property: there is no path through the system that produces plaintext PII as a byproduct. Producing plaintext requires a deliberate, authenticated, audited request. That is the bar.

Below is a sketch of the data flow. It is deliberately minimal — every edge here is load-bearing, and every node corresponds to a thing you have to either build or buy.

Two observations about this sketch. First, every arrow that leaves the tokenizer carries <TOKEN_…> placeholders, including the one to the LLM. Second, the REVEAL ROUTE is the only edge in the entire diagram where plaintext crosses a process boundary, and that crossing is audited.

03 — The four hard parts

The architecture above is easy to describe and tedious to build. Each of the four properties from §02 has a "how do we actually" attached to it that takes a week or two to get right.

03.1 — Encrypt before vault, with no plaintext window

The naive ingest handler looks like this:

export async function POST(req: Request) {
  const body = await req.json();
  await db.from('raw_alerts').insert({ payload: body });
  await queue.send({ alert_id: body.id });
  return new Response('ok');
}

This is wrong in a way that is not visible from the code. The raw_alerts table holds plaintext. Every backup of that table holds plaintext. Every read replica holds plaintext. Anyone with read access to the application database — your DBA, your ops engineer with break-glass credentials, the support engineer who joined last month and got read-only access by default — has plaintext access to every alert your customers have ever sent. The encryption needs to be inside the same statement as the insert, with the key sourced from a place that the application database does not itself hold.

The fixed shape uses Postgres' pgcrypto extension and a server-held symmetric key:

export async function POST(req: Request) {
  const rawBody = await req.text();
  const { error } = await db.rpc('vault_alert', {
    p_payload_text: rawBody,
    p_signature: req.headers.get('x-aiops-signature'),
  });
  if (error) return new Response('rejected', { status: 400 });
  await queue.send({ /* opaque pointer, no body */ });
  return new Response('ok');
}

The vault_alert RPC, defined once in a migration, performs the HMAC verification and the pgp_sym_encrypt(p_payload_text, current_setting('app.vault_key')) insert atomically. The application code never holds the key. The database never holds plaintext. The queue carries an opaque identifier — if the queue gets backed up, leaks, or is replayed, no PII is exposed.

The instinct to skip this step and "just trust the database" is strong. Resist it. The threat model is not "an attacker has root on Postgres." The threat model is "a backup ends up in S3 with the wrong ACL." Encryption-at-rest provided by the platform does not protect you from that. Per-row encryption with an application-held key does.

03.2 — A per-tenant token dictionary that does not become a privacy hole itself

The tokenizer's job is to walk a payload, find every PII match, and emit a sanitized version where each match is replaced with a placeholder. The natural shape is a dictionary table:

create table token_dictionary (
  tenant_id    uuid not null,
  placeholder  text not null,    -- e.g. "TOKEN_a1b2c3"
  encrypted_value bytea not null,
  pii_type     text not null,    -- "email" | "ipv4" | ...
  created_at   timestamptz not null default now(),
  primary key (tenant_id, placeholder)
);

There are three traps here. The first is using a global placeholder space — if TOKEN_a1b2c3 means [e-mail] for tenant A and 192.168.1.1 for tenant B, you have inadvertently built a side channel where tenant B's reveal endpoint can confirm whether a given placeholder was issued in tenant A. Always scope the lookup by (tenant_id, placeholder).

The second is encrypting the value with a single global key. The dictionary is, by construction, a high-value target — it is the table that, if leaked, undoes all the work above. The encryption key for encrypted_value should be derived per-tenant, ideally from a master key combined with tenant_id. A leak of the dictionary alone yields ciphertext you cannot decrypt without the master key. A leak of the master key alone yields nothing without the dictionary. You have to lose both.

The third is determinism. If you regenerate placeholders on every ingest, the same email address shows up under five different tokens across five alerts and your correlation engine can no longer tell that "the same customer keeps tripping the same bug." The fix: hash (tenant_id, normalized_value) and use the hash as the placeholder identifier. Same value within a tenant → same placeholder, every time. Different value or different tenant → different placeholder.

03.3 — Match gracefully: false positives are noise, false negatives are leaks

The detection layer is a regex set. Six patterns will catch most of what a typical alert payload contains:

export const PII_PATTERNS = [
  { type: 'email',        regex: /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g },
  { type: 'ipv4',         regex: /\b(?:(?:25[0-5]|2[0-4]\d|[01]?\d\d?)\.){3}(?:25[0-5]|2[0-4]\d|[01]?\d\d?)\b/g },
  { type: 'ipv6',         regex: /\b(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}\b/g },
  { type: 'phone',        regex: /(?:\+?1[-.\s]?)?(?:\(?\d{3}\)?[-.\s]?)?\d{3}[-.\s]?\d{4}\b/g },
  { type: 'ssn',          regex: /\b\d{3}-\d{2}-\d{4}\b/g },
  { type: 'high_entropy', regex: /\b[A-Za-z0-9+/=_-]{40,}\b/g },
];

Two notes on this set. The phone pattern is intentionally loose — it will match some things that look phone-shaped but are not (long order numbers, tracking IDs). That is the right tradeoff. A false positive becomes an opaque token in the alert; the engineer can still triage the alert and, if needed, reveal the original value. A false negative becomes a customer phone number sitting in your Slack history forever. The asymmetry is real. Tune toward false positives.

The high-entropy bucket exists because the most consequential leaks are not patterns we know about — they are bearer tokens, session IDs, and API keys whose format depends on whoever issued them. A 40+ character base64-ish blob is not always a credential, but it is almost never something you want surfaced to a third party. The threshold of 40 is tuned to skip UUID-without-dashes (32 chars) and short build SHAs while still catching JWTs and most provider tokens. Lower the threshold and you'll start catching legitimate identifiers; raise it and you'll start missing short access keys. There is no universally correct value. Pick one, instrument the false-positive rate against your traffic, and iterate.

What this set does not catch: free-text personal names ("John Smith"), structured-but-non-regex identifiers (most account numbers, license plates), and natural-language disclosures ("the customer mentioned their address is …"). For these you need either an ML-based classifier in front of the regex set or a structural rule that says "fields named customer_name, address, notes always tokenize regardless of contents." The structural rule is cheaper, more predictable, and much easier to audit.

03.4 — Reveal as a single audited route

Engineers will, eventually, need to see the original value. That is fine. The mistake is to scatter "decrypt this token" calls throughout the application. The right shape is a single route:

POST /api/incidents/:id/reveal
  body: { placeholders: ["TOKEN_a1b2c3", "TOKEN_b2c3d4"] }

The route checks: (a) the requesting user is authenticated; (b) the user's tenant matches the incident's tenant; (c) the user has the appropriate role for reveal (operator, admin); (d) the incident is one the user is permitted to view at all. Then, and only then, it pulls the encrypted values from the per-tenant dictionary, decrypts them server-side, returns plaintext over HTTPS, and writes a row to the audit log: who, when, which placeholders, which incident.

The audit log is not optional. It is the thing that turns "I trust my own employees" from an aspiration into something you can demonstrate to an auditor. It is also what lets you answer the question "did anyone access this customer's data between 2026-04-12 and 2026-04-19?" without combing through application logs. Build it as part of the reveal route from day one; retrofitting it later is annoying.

One subtle point: the reveal route should not return the entire decrypted payload by default. The caller has to name the placeholders they want. This sounds like friction, but it is what makes the audit log useful — instead of "user accessed incident 1234" (which tells you nothing about what was exposed), you get "user accessed TOKEN_a1b2c3 (email) and TOKEN_b2c3d4 (ipv4) on incident 1234" (which tells you exactly what data crossed the boundary). The friction is the point.

04 — What you give up

This is honest-tradeoffs time. Edge tokenization is not free.

Latency. The ingest handler now does an HMAC verify, an encrypted insert, and a queue publish before returning 200. On a warm Worker with the database in the same region, this is roughly 30–80ms more than a no-op write. For most alert pipelines, this is invisible — the sender does not care whether the 200 takes 5ms or 80ms. For pipelines where the sender is a tight retry loop with a low timeout, you may need to instrument and tune.

Cost. You are paying for: a worker invocation per alert, a queue message per alert, an encrypted column write per alert, a tokenizer invocation per alert, a sanitized-event write per alert. None of these are individually expensive. At a million alerts a day, the storage and compute add up to dollars, not hundreds of dollars. The expensive thing in this architecture is the LLM call for root-cause analysis, and that is bounded separately.

Operational complexity. You now have a key-rotation procedure. You have a backup-and-restore procedure that has to handle the dictionary as a separate concern. You have a "what do we do if the per-tenant key is lost" runbook (answer: you cannot recover the data, which is the point). You have an audit log that needs its own retention policy. None of these are crushing — they are all things compliance-driven sectors have been doing for decades — but they are real engineering work.

05 — What you get back

The thing you get back is not "compliance." Compliance is a side effect. The thing you get back is that the question "what would happen if our chat tool's history were leaked" stops requiring a long answer. You can route alerts to a third-party LLM without first negotiating a BAA, because there is no PHI in the prompt. You can grant a new vendor — error tracking, on-call, an analytics dashboard — read access to your alert stream without staging a security review for each one, because the alert stream does not contain customer data. You can ship faster.

The compliance posture follows. SOC 2 Type II's common criteria around confidentiality become almost mechanically satisfiable: encryption at rest (vault), in transit (HTTPS only), and a documented incident-response procedure with a 72-hour notification window. HIPAA's technical safeguards — access control, audit controls, integrity, person/entity authentication, transmission security — map cleanly onto the routes and tables described above. None of this means you are certified. It means that when you decide to pursue certification, the architecture work is already done and the auditor's questions become procedural rather than structural.

The other thing you get back is unchecked-code-review time for the rest of your team. Every PR no longer has to be scrutinized for "did this introduce a path that logs customer data." That path doesn't exist. There is one place in the codebase where plaintext appears, and it is a route handler with seventeen lines of authorization logic in front of it.

06 — Where to start, if you want to start

If you're convinced and looking at your own pipeline, the order matters. Do not start with the tokenizer; start with the vault.

1. Pick the first byte of plaintext PII to remove from a downstream consumer. A reasonable first target is your incident-management tool, since it is usually the most-shared and the least-controlled. The goal of week one is "no plaintext customer email addresses in any incident-management notification we send."
2. Build the encryption-at-ingest path. Move every alert-emitting service to write to the encrypted vault first; let the existing pipeline keep running off the encrypted blob, decrypted in the consumer. This is the dangerous step — get this wrong and you lose alerts. Run it in shadow mode for a week.
3. Build the tokenizer. Start with the four highest-confidence patterns (email, IPv4, IPv6, SSN). Run it against historical traffic; measure your false-positive and false-negative rates against a hand-labeled sample of 200 alerts. Iterate the regex set until you can defend the numbers.
4. Cut over the downstream consumers to the sanitized events. Keep the vault as a fallback for "we lost an alert" debugging.
5. Build the reveal route and the audit log. Migrate any "I'll just SSH in and SELECT it" workflows to use the route. Turn on RLS and the lint rules that prevent the back door from being reopened.

You can be at step 5 in six to eight weeks of focused work. The thing that gates the timeline is not the engineering — it is figuring out which integrations in your existing pipeline are actually emitting PII, and what the structural rules need to be to stop them. That part is investigative.

If you'd rather not build the pipeline yourself, that is exactly what Culprit is. The architecture above is a description of what we ship. The piece is here because we want the architecture to be the default approach to alerting, not a thing you have to discover by getting bitten.