AI assistance disclosure: This article was drafted with the help of Claude. All technical content, design decisions, code references, and screenshots reflect production systems I designed and operate at airCloset; the prose was revised by me prior to publication.
Hi, I'm Ryan, CTO at airCloset.
In Part 1, I walked through the four monitoring axes (application / infrastructure / CI / LLM) and the deliberately different shape each one ends up in. That's the write-side of the observability stack, more or less wrapped up.
But shaping the write side isn't the end of the story. The moment production data flows through the stack, you have to block the path PII can take to slip in — and that's true with or without AI. It's the kind of classic observability problem where, if you cut corners, you walk straight into a leak incident.
Historically, the set of people who could read logs mostly overlapped with the set who could read the DB. For engineers with DB access, logs weren't an additional path to personal data — which put log-side defenses in a position where hardening them didn't meaningfully move the overall defense line for most organizations.
AI breaks that premise. Non-engineers pulling logs over MCP don't have DB access. Logs became, for the first time, a path where someone without DB access can reach personal data. On top of that, log content now flows into AI's input, which introduces new exposure surfaces: transmission to the model, and re-surfacing in the model's output. Log PII protection has shifted from "hygiene worth doing" to "required as a trust-boundary redesign." That's the premise this post starts from.
And on top of that, if the observability stack isn't queryable by AI, the whole "AI-consumable observability" goal from Part 1 falls apart.
Part 2 is about how I reconciled these two — protecting PII while keeping searchability for AI — and how that combination ends up driving Self-Healing from CI failure to PR proposal.
The Observability Stack Is a Natural Path for PII
App emits a log → it lands in Loki → AI queries it through MCP. Stand up this naive flow and you get:
- Customer email addresses and phone numbers in error logs
- Order response payloads riding inside traces
- DB query logs that emit full table rows
Plain-text PII pooling in the observability stack means AI can search it directly. This isn't really an AI problem, it's an observability problem: the stack itself becomes a PII conduit. At the same time, if you scrub PII completely, you lose "I want to investigate Customer A's support ticket" as a query, which is a normal support workflow.
cortex (the internal AI platform) had to reconcile both. The key principle was: don't make "block the PII path" and "search by PII" mutually exclusive.
Note: "cortex" here refers to airCloset's internal AI platform codename. Unrelated to Snowflake Cortex, Palo Alto Networks Cortex, etc.
Multi-Layer PII Design — Six Layers
cortex's PII handling is six layers, each with a different role:
| Layer | Purpose | Mechanism |
|---|---|---|
| Write: BQ Policy Tag | Column-level access control |
pii_high / pii_medium / pii_low three-tier taxonomy. Without fine-grained reader on the column, SELECT errors out with Access Denied (pure CLS (Column-Level Security) — no dynamic masking) |
| Write: ETL DLP | Strip plain-text PII from derived tables | Cloud DLP redacts during transforms (customer support data, etc.). Placeholders like [EMAIL_ADDRESS] / [PHONE_NUMBER] preserve the structure |
| Write: log hashing | Plain text never reaches Loki | App-side hash via hashEmail (HMAC-SHA256 → 12-char prefix; key lives outside the observability stack) before log emit |
| Search: same function on both sides | Look up a specific customer's logs without ever touching plain text | Query-side runs the same hashEmail before sending to Loki |
| Output: MCP masking | Mask when AI consumes | Column-name detection masks the local part (e.g. r***@air-closet.com), keeping @domain so first-response triage can still tell which domain the account belonged to |
| Identity separation | Internal staff email is handled in a separate track from customer PII | HMAC-signed by Edge Router as auth attribution; not part of the masking pipeline |
The fourth row — search with the same function on both sides — is where the security / usability tradeoff gets really tight.
I'll use email as the running example, but the six layers guard more than email. PII spans names (including phonetic readings), phone numbers, addresses, postal codes, dates of birth, card and bank details, external-service IDs, and more. The anonymization technique varies by the nature of the field — same-function hashing to preserve correlation (email, phone), partial masking (names, addresses), full redaction (card numbers, tokens) — and that call is made per field. What stays constant is the structure: which of the six layers guards it, and how. That's the reusable part of the design.
And this anonymization isn't confined to observability logs (Loki) either. An MCP tool that queries a service DB, for instance, pulls customer names, addresses, and phone numbers into its result set, so the same PII anonymization rules run before anything is handed back to the AI. The consistent rule is "anonymize PII on every data path that reaches the AI," applied across data-source types, not just one.
Hash on Both the Write and Search Sides
Naively "remove PII from logs" and you can no longer answer "let me look up Customer A's logs." But if you hash at write time and store that hash in the log, the search side can run the same hash function over the input and find the matching record. Plain-text email never touches either end.
Concretely:
Write side:
// Application code
logger.info("Subscription updated", {
user: hashEmail(user.email), // → '7a3f9c2e0b1d' (HMAC-SHA256 12-char prefix)
plan: "monthly",
});
// → Only the hashEmail result ends up in Loki
Search side (when you want to pull a specific customer's logs):
Here's the awkward part. "Pull up Customer A's logs" — the naive way to build it hands the raw email to the AI, which then passes it to an MCP tool to search. But that means handing plain-text PII to the AI (the model, and the vendor behind it). Guard the inside of Loki with hashes all you want; it leaks at the search input, one step earlier.
So in cortex the search tool takes a non-PII ID, resolves it to an email inside the MCP server, hashes it there, and returns only the hash. The email exists only inside the MCP server and never reaches the model:
// MCP tool resolve_email_hash (runs server-side)
// Input is an ID (non-PII). The email is never returned to the caller = the AI.
const email = await resolveEmailById(userId); // resolved from the DB, server-side
const hash = hashEmail(email, secret); // same function, same key as the write side
// → the AI gets back only the hash, never the email
The AI takes that hash and searches Loki via Grafana MCP as {service_name="subscription"} |~ "${hash}". Both the write side and the search side run the same hashEmail with the same key, so logs from the same customer collapse to the same hash. Meanwhile:
- Plain-text email never enters Loki
- The query string Loki sees doesn't contain plain-text email either (only the hashed value reaches it)
- And the AI (the model) never receives plain-text email either. All it touches is a non-PII ID and hashes that already live in Loki. The plain-text email never leaves the trust boundary of the MCP server.
- Enumeration resistance comes from keeping the HMAC key outside the stack. Email is a low-entropy, enumerable input space, so a bare one-way hash (plain SHA-256, etc.) is breakable. The hash function is public, so once logs leak, an attacker just hashes a list of likely emails on their own machine and matches against the leaked values, no key required. HMAC folds a secret key into the hash computation itself, so an attacker who doesn't have the key can't even turn a candidate email into "the same shape as the leaked hash." They never get onto the brute-force field. Keep the key only at the write side and the search tool, never in Loki itself, and you get "a log leak alone doesn't expose the plaintext unless the key leaks too", one more condition an attacker has to satisfy
- Truncating to a 12-char prefix (48 bits) means collisions are possible in theory, but negligible at customer-base scale. By the birthday problem, the 50% collision point sits around 20M records (≈ 2^24.5), and below that the expected collision count stays tiny. More to the point, a collision wouldn't leak plaintext anyway: this hash is a correlation key for identifying a customer's logs, not a security boundary, so the worst case is "another customer's logs occasionally land on the same hash", a degradation of correlation accuracy, not a disclosure
This reuses the property "same input → same hash" of hash functions in the form "the same function on both sides makes search work." The security / debug usability tradeoff compresses cleanly.
And of course, this is all just the app log layer. The BQ side is protected by Policy Tag-based column-level access control as its own layer (rows 1–2 of the table above). The whole thing is multi-layered.
What makes the "take an ID, resolve and hash inside" shape work is that plain-text email never crosses the trust boundary of the MCP server. The easy implementation (hand the AI a raw email, let the tool search) leaks the plaintext to the model at the search input, no matter how well you guard the inside of Loki. You could argue "the vendor's terms say it won't leave," but that's a dependency on terms, and it's weak under audit. Take an ID and hash inside, and you keep plaintext away from the model structurally, not contractually. When I said up top that PII protection has become "a trust-boundary redesign," this is the kind of design call I meant.
An aside: when I was working this out, I asked an AI for help, and it suggested building an admin screen where a human manually turns emails into hashes. That's one way to keep PII away from the model, sure, but it doesn't fit autonomous operation — a human has to step in before any investigation can start. cortex is built to run all the way through to "fixed before anyone notices" self-healing, so a solution that inserts a human isn't on the table. "Take an ID, hash inside the MCP server" came out of that constraint. What counts as an acceptable solution was, in the end, a design judgment on my side.
Integration Surface — "Humans = Web, AI = MCP" on the Same Backend
Three backends (Prometheus / BigQuery / Loki) now carry the observable data, and PII is handled. The next question is who queries them, and how. The common trap is to build "human dashboard aggregations" and "AI data feeds" separately. The moment you do:
- Two implementations chasing the same question
- Numbers drift between them
- It becomes unclear which is canonical
- Aggregations for AI and for humans update on different schedules
cortex's choice: share one observability backend; only the consumer-facing interface differs.
Human side: AI Operations Portal
There's an internal portal (codenamed PI Lab) that aggregates dashboards by monitoring target:
- Claude Code usage (the cc-usage screen from Part 1)
- MCP tool usage (by server / tool / user / team)
- Infrastructure cost (Gemini / GCP / AWS / GitHub on one screen)
- Alert state, deploy history, etc.
Here's what the MCP usage dashboard actually looks like:
Over the past 30 days, service-product-graph had 37,946 calls (with 7,106 errors), gws had 19,350, db-graph had 17,297 — and that's just the top. Which MCP is used how much, where the failures are showing up — all visible at a daily glance. (The "high error rate" some servers seem to have is partly typed errors counted in — expected rejections like "permission denied" — so the interpretation needs care.) The "annotation graph MCP, ~50,000 calls / 73 users" figure from the previous series came from this same view.
These pages on the React side pull from BQ / Prometheus / Loki through an internal API. The aggregation logic lives at the API layer.
AI side: MCP
When AI agents need the same data, they go through purpose-specific MCPs:
- Grafana MCP — LogQL / PromQL queries against Loki / Mimir / Prometheus / Tempo. Natural-language questions like "What time window had the most errors on Service X last week?" are the agent's job to translate into LogQL / PromQL before they go over MCP
-
BQ MCP (via cortex-product-graph) — SQL queries against
claude_usage.claude_usage/cortex.mcp_tool_calls
The design pivot: the human dashboard and the AI MCP share the same backend. No separate "AI aggregation table" and "human aggregation table." Build the observability backend once, then provide a consumer-specific interface layer (web dashboard / MCP) on top.
In DDD terms, MCP and the web dashboard are both just presentation layers — different I/O channels into the same domain (the observability backend). Treating MCP as "something special" leads to duplicate implementations; treating it as one presentation layer form keeps the design clean.
That's exactly why "the observability stack is visible to AI" actually holds. Build the backend, but without an AI-facing presentation layer (= MCP), AI can't query it. MCP is the piece that makes "hand it to AI" actually work.
The Real Driver of Self-Healing
The layer that keeps the observability stack from being "just a screen to look at" is Self-Healing. I covered the full picture in AI Harness Series Part 4, so I'll skip the details here, but from the observability side, the start and end of the chain are clear:
The flow:
- Detect — Production alert / CI failure fires a Loki LogQL alert
- Deliver — POST to event-relay (the internal webhook hub)
- Launch — auto-review bot starts up (= an agent backed by Claude Code)
- Gather context — The bot pulls full logs via Grafana MCP, traces related PR / commit / code via Product Graph MCP
- Propose — File a fix PR
- Verify — If CI passes, the bot auto-merges; if not, another bot reviews
So the starting point of Self-Healing is whether the observability stack can hand "what broke" to AI in the right shape. If errors aren't recognized / stacktraces aren't preserved / related code (PR / commit / graph) isn't reachable — any of those missing and the chain stops cold. (The specific failure modes are in the next section.) Put another way:
The quality of observability is the ceiling for AI autonomous operation.
That's the central claim of Part 2. Reframe the observability stack as "input that drives AI," not "monitoring infrastructure," and the priorities of your design decisions shift accordingly.
What's Still Open — Defining "What Counts as an Error" and the Stacktrace Design
The biggest remaining issue, honest version.
You can polish the observability stack to a mirror finish, but if the design of what counts as an error and whether the stacktrace survives falls apart, all of it is wasted. I touched on this earlier in AI Harness Series Part 2 in the context of cortex's internal knowledge graph, and it shows up on the observability side too.
Concretely, here are the failure modes:
-
try ~ catchswallows the error without logging → nothing reaches the observability stack - catch does log, but at
console.log-equivalent info level → not recognized as an error - Error gets emitted, but only
error.messageis written; stacktrace is dropped → AI can't trace back to the original code - An async error goes unhandled and the process falls over
These are all problems at the code that creates the observability entry point, not at the observability stack itself. No matter how polished the stack is, if the faucet at the entry point is broken, nothing flows out.
What's in place today is three layers, none of them complete:
-
lint (static) — The
no-silent-catchrule blocks empty catches and.catch(() => null)-style swallows. But once there's any function call inside the catch, lint is satisfied — so patterns like "demote tologger.info(err.message)" or "log onlyerror.messageand drop the stacktrace" slip through statically -
Guideline document — Rules like "use
serializeError(error)to store stacktrace as a structured field" and "droppingstackvialogger.error(err.message)is a Major violation" are written down in the internal guidelines. But static checking can't enforce these; they rely on human / AI review - AI auto-review — The PR auto-review bot does look at test coverage including "are error cases being tested," but it has no observability-specific checklist, so it can't systematically catch stacktrace design quality
In other words: "There's a guideline, lint catches some, AI review catches some, but it's not airtight" is the honest description. The real gap is that at the moment new code is being written, there isn't a harness that proactively suggests / completes "this should be treated as an error, this should keep its stacktrace." Auto-review picks things up at PR time, but a proactive harness for the observability entry-point design itself isn't built yet.
"Observability stack: done. Observability target design: still on humans." That's the honest picture. Closing that gap with a harness is the next step.
Closing — Static Edition + Dynamic Edition Are Lined Up; Merging Them Is the Next Series
The code-graph series was about reshaping a static analysis graph so AI could query it — handing the structure of code as fact. This two-part series was about handing what's happening in production right now, also as fact.
| Shape | What's Handed Over | |
|---|---|---|
| Static edition (code-graph + db-graph + annotation graph) | 3-graph parallel + SAME_ENTITY | Code and meaning |
| Dynamic edition (Part 1 + this post) | Prometheus / BQ / Loki + MCP | Production behavior and cost |
The honest part: these two still sit side by side, not joined. For cortex's stated principle of "don't let AI infer — hand it facts" to truly reach completion, the next step is to pour dynamic data into the static graph and merge them. This is the exact same gap I flagged as the "absence of dynamic analysis" open issue at the end of code-graph Part 2: putting "how often is this edge actually used in production?" on the static graph's nodes. That's when "hand it as fact" reaches its final form.
Layer Self-Healing on top of static + dynamic and you get "AI autonomously operates," which works today. But merging the two editions into one graph is still ahead — that's the next series.
And one more time, observability target design (what counts as an error, whether stacktrace survives) is what really sets the ceiling. Harness-ifying that is the next homework item.
Thanks for reading this far.
Top comments (16)
The resolve-inside-the-MCP-server design is the right structural move, and the reasoning for rejecting the admin-screen alternative is the most honest part of the post, most writeups don't admit they considered the human-in-the-loop version and picked autonomy over it on purpose.
The birthday-bound math checks out for the customer base you have today, but it's computed against today's size, not the trust boundary's actual lifetime. 20M records at 50% collision sounds comfortably far off, until the company grows for five years and nobody revisits the constant. A correlation key that degrades gracefully at current scale can still degrade silently at future scale, because nothing in the design re-checks the assumption, it was true when written and nobody's watching whether it's still true.
The other thing I didn't see addressed: what happens on HMAC key rotation. If the key ever needs to rotate, either every historical log becomes uncorrelatable with new ones (a quiet loss of the "look up Customer A's history" capability you built the whole six-layer design to preserve), or the old key has to be retained somewhere to re-hash on read, which reopens exactly the key-outside-the-stack boundary the enumeration-resistance argument depends on. Worth stating which of those two you'd actually do, because right now the design reads as if the key never rotates, and "never rotates" is a strong assumption for a secret whose whole job is staying secret.
Both land, and the second one (rotation) is a real gap in the post, the design as written does read as if the key is immortal, and you're right that "never rotates" is a bad assumption for a secret.
The thing that resolves both of your points is one I didn't make explicit: logs aren't kept forever. They rotate out on a retention window, and once you lean on that, the clean design is to put the hash and the key on the same clock as the logs they serve.
Concretely: tie a key's lifetime to the retention window of the logs hashed under it. When you rotate, you keep the old key only as long as logs hashed with it are still alive, and re-hash on read against whichever key covers that log's era. The moment those logs age out, the old key is deleted with them. That dodges both horns of your dilemma, correlation survives within a log's lifetime because the covering key is still there, and the key never accumulates or lives forever outside the stack because it dies when its logs do.
It also softens your first point as a side effect. The birthday-bound math is against a bounded population, not an ever-growing one, because retention caps how many records coexist. The collision assumption stops being a constant nobody revisits and becomes a function of the retention window, which is a number someone is already looking at.
Full honesty: this is the design, not the current state. It's early enough that the key doesn't rotate yet and retention-coupling isn't wired in, so today it genuinely is the immortal-key version you called out. You named the exact thing that has to get built before rotation is ever needed. Good catch, both of them.
Tying the key to the retention window is the clean fix for the immortal-key problem, and I think it exposes a tradeoff the post's original goal has to answer next. If each era gets its own key, the same email hashes to a different value in era N and era N+1. That's fine for a single log lookup, but it breaks the exact capability the six-layer design was built to preserve: look up Customer A's whole history. A search spanning two eras can't recognize the same person across the key boundary without re-hashing every candidate against every live era key and matching independently, which turns a hash lookup back into a fan-out.
So the honest question is whether cross-era correlation is a deliberate loss (search only ever answers within-this-retention-window, which may be fine, most investigations don't need five years back) or whether there's a second, longer-lived index key that exists precisely to survive rotation, in which case that key inherits the immortal-key problem you just solved for the log-hash key, just moved one layer over. Worth stating which one this is, because right now the design reads as having quietly traded correlate-forever for correlate-within-a-window without saying so out loud.
It's the window, and I'd argue the window is the right boundary rather than a quiet downgrade, because it falls out of the same retention clock instead of being a separate decision.
Concretely, I wouldn't build a second long-lived index key, that just moves the immortal-key problem one layer over like you said. I'd set the key lifetime equal to the retention window, which caps the number of live keys at two: any log still alive was hashed under either the current key or the previous one, because anything older than one rotation has already aged out of Loki. So a lookup hashes the input under at most two keys and ORs the results. The fan-out you're describing is real but fixed at 2x, not an unbounded re-hash. Hide that behind the MCP tool for agents and the API layer for humans, and the caller still asks "Customer A's history" once.
So it's not correlate-forever and it's not correlate-within-one-key. It's correlate-as-far-back-as-the-logs-still-exist, which is the honest ceiling anyway: you can't correlate history that's already been rotated out of Loki, key or no key. The retention window was always the real limit on how far back a lookup can see. Tying the key lifetime to it just makes the crypto boundary agree with the boundary that already existed, and keeps the live-key count at two by construction. You're right it should be stated out loud, though, "correlation is bounded by retention, by design" is the line the post is missing.
Fixed at 2x and derived from the same retention clock instead of a separate decision is the version that actually closes it, correlate-as-far-back-as-the-logs-still-exist is the honest ceiling and tying the crypto to it removes an assumption instead of adding one. Worth stating the one edge case before this ships: at the exact rotation boundary, is there a moment where a log written a second before rotation and read a second after needs a key that's already been retired, or does retention lag rotation by enough margin that this never actually happens in practice? If the rotation and the retention-based deletion aren't atomic with each other, you could briefly need a third key or have zero keys covering a thin sliver of logs right at the seam. Probably a non-issue if rotation cadence is much shorter than retention window, but worth the one-sentence guarantee in the design doc so nobody has to rediscover it during an incident. Good exchange, this is the kind of thread that's worth linking back to when the immortal-key question comes up again.
Right on the seam, and here's the one-sentence guarantee: the only logs that can fall into a zero-key gap are the ones already at the retention edge, i.e. the ones being aged out anyway. A log written a second before rotation and read a second after is, by definition, a log that's crossing out of the retention window at that same moment, so "can't find a key for it" and "it's past retention" collapse into the same case. As long as deletion never runs ahead of rotation (retention lags rotation, never leads it), the sliver only ever contains logs that were already leaving. That makes the guarantee "no live log is ever without a covering key," which is the line for the design doc, exactly as you said, so nobody rediscovers it mid-incident.
This whole thread was a genuine pleasure. You pushed on the two things I'd have most regretted leaving implicit, retention-coupling and the rotation seam, and both are sharper in the post now because of it. Linking back here when the immortal-key question resurfaces is exactly right. Thanks, Mike.
The dual-sided HMAC is the right shape for keeping plaintext out, but I'd name what it moves rather than removes. A deterministic hash is a stable pseudonym: same email, same value, which is what makes search work and also what lets anyone with query access reconstruct one person's whole history under that token. Plaintext egress is closed; linkability is not.
And the search side is an online oracle: it hashes arbitrary input and looks for a match, so anyone who can query can confirm whether an email they already suspect is in the logs, one guess at a time. HMAC stops offline brute force of the stored hash, but the search endpoint answers the membership question for free. Neither breaks your design, they just belong in the threat model beside it: the boundary is real for plaintext, and does not cover correlation or confirmation. For a known-domain field like email, that gap is where re-identification actually lands.
Both correct, and both worth naming explicitly in the threat model. But they land on the other side of the boundary this design draws, so let me make the boundary itself explicit, because I left it too implicit in the post.
The searcher here is assumed to already have PII access. Support and engineering can already see this person's data in the DB; that's their job. What this design removes is not their ability to correlate or confirm, they already have it, it's the need to route plaintext PII through the model to do a log investigation. The boundary is "does the AI ever see the plaintext," not "can an authorized human correlate." So the linkability you describe (one stable pseudonym reconstructs a history) and the online oracle (confirm-by-query) are both real, but they're capabilities the searcher already holds by virtue of DB access. The hash doesn't grant them; it just lets the same authorized person do the log side without handing the email to the model.
Where your framing sharpens mine: for someone who has query access but not DB access, those two gaps would be a genuine escalation, and that's exactly the case to guard. We keep the search side auditable (every resolve + query is logged) precisely so that "authorized human doing their job" and "someone fishing the oracle" are distinguishable after the fact. That's a detection control, not a prevention one, and you're right that for a known-domain field like email, confirmation is the sharp edge. Naming it beside the plaintext boundary is the honest way to draw the diagram, and I'd rather have both lines on it than pretend the boundary covers more than it does.
Agreed on the boundary, and the audit trail is the right call for the human-without-DB case. The one place it thins is the direction your post is driving toward. Detection works because a person fishing the oracle looks anomalous, but an agent doing self-healing queries at volume by design, so a confirmation probe hides inside its own normal traffic. The baseline is the anomaly you would otherwise flag. For that persona I would pair the audit log with a prevention control on the search side, scoping a lookup to a case context or capping distinct identifiers per session, so an authorized query is bounded and not just logged. Good exchange.
You're right that detection thins exactly where the series is heading, and that's the sharp version of it. Once the querying persona is an agent doing self-healing at volume, "anomalous" stops being a usable signal because volume is the baseline.
Where I'd push on the framing: an agent firing a confirmation probe isn't really the agent's malice, it's almost always an agent that's been prompt-injected into doing it. And once injection is on the table, the oracle is a small downstream symptom, the same compromised path can exfiltrate through any tool it can reach, not just the hash lookup. So I'd put the prevention control at the injection layer (tool-permission scoping, input provenance, output review) rather than teaching the PII search side to count distinct identifiers. Capping the oracle while injection is unhandled feels like bolting one window shut.
That said, your per-session bound is the right move as defense-in-depth, and whether it's worth it is a product call. For a domain where a single re-identification is catastrophic (health, finance), a second wall on the search side that holds even after injection succeeds is worth the cost. For an internal platform where the searcher and the agent are both inside the trust boundary, I'd spend that budget on hardening the injection surface first. Good exchange, genuinely, this is the kind of threat-model back-and-forth that's hard to get.
You're right that injection is the root and the oracle is one symptom, harden the injection surface first, I'm with you there. The one thing I'd keep beside that is why the search wall earns its place: tool-scoping, provenance, and output review are all probabilistic, their false-negative rate is unknown and the attacker gets to lower it. The per-session bound is the one control whose guarantee doesn't move when injection gets better. So I'd frame the product call less by domain and more by that, put the deterministic wall wherever a probabilistic guarantee failing is unacceptable, which is often wider than just health and finance. Good exchange, genuinely.
That reframe is the version I'll keep: put the deterministic wall wherever a probabilistic guarantee failing is unacceptable. Cleaner than drawing it by domain, and it generalizes past security. Thanks for pushing on every layer of this, genuinely one of the best exchanges I've had here. Enjoy the sand in Niterói.
Smart take on PII: hash at write and search time so AI can find what it needs without exposing data. Curious about collision handling and performance overhead in practice. Also loved the CI-to-PR self-healing angle—chef's kiss.
Thanks, really glad the self-healing angle landed.
On both counts the honest answer is "bounded, so neither really bites." Collisions are birthday-bound against a population capped by the log retention window, not an ever-growing set, so the space stays comfortably clear at our scale. If that ever stops being true, the collision constant just becomes a retention knob someone's already watching. Performance is a non-issue in the same spirit. It's one hash on write and one on search, which is nothing next to the query and the model cost it sits beside. The expensive part of an investigation was never the hash.
the entry point design gap is where this lands for me. the stack is solid but the faucet is still owned by humans and lint, and that's the actual ceiling for autonomous operation.
pattern i've seen work: type the log call site directly. if logger.error requires an Error object and not a string, dropped stacktraces stop being a guideline violation and become a type error the compiler catches before CI. the lint rule enforces the type, not the convention. is that the direction you're thinking for the harness, or does the problem look different from where you're sitting?
Yeah, that's the direction, and I already run a weaker version of it. Lint doesn't warn here, it blocks. A catch that swallows without logging fails the build, and so does a log call that drops the stacktrace. Typing the call site so logger.error takes an Error and not a string is the stronger form of the same move, and I agree it's better. It moves the same block from CI lint to the compiler, which is strictly earlier and harder to skip past.
But that closes the half that was always mechanizable. "The stacktrace got dropped" is a correctness property, and correctness properties belong in the type system. The ceiling I meant sits one level up, and types don't reach it. It's deciding whether a condition is an error at all.
Concrete case from this week. A scheduled importer walks a list of articles and one returns 404 because it isn't published yet. Is that an error? I decided no, it's an expected skip, log a warning and exit 0. A rate-limit exhaustion on the same loop is an error, exit 1. Nothing in the type system tells you which is which. logger.error taking an Error forces me to log a real Error once I've decided it's an error, but the decision that a 404 here means "not ready" rather than "broken" is a claim about what the absence means in the business, and that claim is the faucet.
So the framing I'd land on is that you push every mechanizable property down to the compiler precisely so the only thing left at the top is the irreducible classification call. Your typed-call-site move is how you clear the mechanizable layer. What stays human isn't "did we handle the error right," it's "is this even an error." That one doesn't compile.