DEV Community: Ryosuke Tsuji

When Shipping Gets Cheap, Deciding Gets Expensive: Guardrails for AI-Era Cloud Costs

Ryosuke Tsuji — Wed, 05 Aug 2026 23:40:03 +0000

AI assistance disclosure: This article was drafted with the help of Claude. All technical content, design decisions, code references, and screenshots reflect production systems I designed and operate at airCloset; the prose was revised by me prior to publication.

Hi, I'm Ryan, CTO at airCloset.

We spent a week going through the cloud bill for our internal AI platform. The results, up front:

Cloud Run instance-based billing down 84% (44% on the single service we tested first)
Container image storage down 65% after revisiting how many versions we keep
The thing that mattered most wasn't any single cut. It was the gate at merge time and the daily visibility

But the savings aren't really what I want to write about. When implementation gets cheap, the cost of deciding whether to build something stays exactly where it was, and suddenly it's the expensive part.

Slow implementation used to be a decision gate

This isn't a post about AI, exactly. But AI is what makes this particular thing grow.

Building anything used to cost real time. You'd estimate it, review the design, carve out the engineering time. That process is tedious, and it also had a side effect nobody designed for: it forced you to ask whether the thing was worth building at all. The slowness itself was the gate.

Make implementation fast and that gate comes off. You can ship an idea the same day you have it. That's genuinely good, and it makes "just build it and find out" the right call far more often than it used to be.

What also comes off, though, is any reason to hold back. Each individual thing is small enough that nobody objects. And what's left behind isn't engineering hours, it's the running cost of whatever you shipped. That part doesn't show up on the day you build it. It accumulates quietly afterward.

The starting point: we couldn't attribute spend to an app

What kicked this off was a handful of occasions where usage-based spend grew more than we expected after a particular change. BigQuery scan volume, and Vertex AI / Gemini calls.

The frustrating part was that we couldn't attribute the spend back to a specific app after the fact. BigQuery does let you label query jobs, and job history is there. But several of our apps run under the same service account, so knowing which principal ran a query didn't tell us which app was responsible. We could push labels through every call path, but until that's fully rolled out you're in the same position.

If you can't attribute after the fact, you stop it before it grows. That's where the idea of putting a gate at the entrance came from.

Visibility comes before any of the cuts

Here's the conclusion first: the highest-leverage part of this whole week was the gate at merge time and the daily reporting, not any individual optimization. Cuts are one-time. Left alone, the same things pile up again.

The gate: changes above a threshold go to a human

We added cost as a review dimension for our AI reviewer. If a PR pushes usage-based spend past a threshold, it gets handed to a human. Three thresholds:

Category	Threshold
Ongoing daily cost increase	¥2,000/day (~$13)
One-off execution cost	¥10,000 (~$65)
Monthly storage growth	¥1,000/month (~$6)

The design choice I care about here: the AI doesn't do the arithmetic. It reads the change and measures quantities like scan volume, call counts, and token counts. Converting to currency and comparing against the threshold happens in a deterministic script. Hand the money math to a model and you get occasional arithmetic slips, the wrong unit price, and answers that drift between runs. A gate that gives different answers on different days isn't a gate.

Unit prices come from our actual billing export (spend ÷ usage), not from the published rate card. Hardcoding list prices means carrying three separate sources of drift: USD conversion, exchange rate, and committed-use discounts.

Storage gets its own threshold because cumulative cost behaves differently from execution cost. Stop a job and its daily cost goes to zero. Data you've written keeps billing until someone deletes it. So for storage we ask a different question: is there an expiration policy, and what does this add per month going forward? That's a forward projection, computed from call volume times payload size, not a measurement of what already happened.

Daily reporting: post to Slack twice a day

We post per-app cost to Slack twice a day. A few decisions in there worth mentioning.

Post even on days when nothing crossed a threshold. "Nothing to see today" is information, and more importantly, the reader needs to be able to tell the difference between a quiet day and a broken reporter. A monitor that only speaks up when something's wrong can't tell you it's dead.

Compare against the same weekday last week, not yesterday. Batch volume varies by day of week, so a day-over-day comparison flags every Monday as a spike, and people stop reading it within a week.

Treat "no data last week" as an increase. This is the dangerous case. A batch that just started running is exactly the thing that surprises you. Percent-change metrics tend to drop these on the floor because the math is undefined when the baseline is zero, so we handle that case explicitly.

The backstop: put a ceiling on usage-based services

Gates and reporting still leak. There are paths that don't go through a PR at all, like manual runs, external triggers, and usage patterns nobody anticipated.

So for anything that bills by usage, set a ceiling that matches how you actually operate. BigQuery, Vertex AI, and Gemini all let you cap this with quotas.

The important part is to derive the ceiling from measured usage, not from what the platform allows.

Take BigQuery. On-demand pricing has a default cap of 200 TiB per day, per project (that's been the default since September 2025; before that it was unlimited). Per-TiB pricing varies by region, but at a few dollars per TiB, that ceiling works out to somewhere north of a thousand dollars a day. Run that for a full month and you're at tens of thousands of dollars.

Which means the default setting is: spend that much, every month, without anyone being told. That isn't a ceiling.

We cap BigQuery scan volume at roughly 1.5x our normal usage. "Ten times normal, to be safe" has the same problem as the default. A ceiling only means something if it sits where a runaway actually stops.

One thing to be clear about: we can do this because this is internal infrastructure. Capping at 1.5x means accepting that things stop when they hit the cap. For an internal platform, you rerun it tomorrow. Do the same thing on a customer-facing path and you've built yourself an outage. Only put this kind of ceiling on things that are allowed to stop. For anything touching end users, set the ceiling much higher and handle it with alerting and autoscaling instead.

Given that constraint, the side effects are fine. A legitimate job occasionally hits the cap, and when it does, that's a prompt to go find out why something needed 1.5x normal volume. Which is its own kind of detection. Alerts tell you afterward. A quota stops it while it's happening. With usage-based billing that difference matters.

If you try to run any of this as a habit rather than a system, it evaporates the first busy week. It only works once it's structural. Gate at the entrance, daily reporting for the continuous view, quota as the hard stop. Three layers, and only together do they make the invisible visible.

Case 1: when the default is the expensive one, speed builds debt

Now the things we actually found. Cloud Run first.

Cloud Run has two billing models: instance-based, where CPU is always allocated, and request-based, where it's allocated only while handling a request. 97% of our Cloud Run spend was instance-based.

The cause was structural. Looking at where we define Cloud Run services in our repo, most of them didn't specify cpuIdle at all. There's no shared factory, so every new service silently landed on the expensive side unless someone thought to set it.

Worth knowing: deploy from the console or gcloud and the default is request-based, the cheap one. Define the same service declaratively in IaC with explicit resource limits and cpuIdle ends up false, which puts you on instance-based. So click it together and you get the cheap default; write it as code and you get the expensive one. The more disciplined your infrastructure practice, the easier this is to miss.

Which is the whole point from earlier. When the default sits on the expensive side, going faster builds debt automatically. The per-service difference is small enough that nobody notices.

The name reads backwards

There's a trap in the naming. The setting is called cpuIdle.

Value	Behavior	Billing
`true`	CPU allocated only during requests	Request-based (cheaper)
`false` (IaC default)	CPU always allocated	Instance-based (pricier)

Read true as "keeps running while idle" and you have it exactly backwards. And the failure mode is nasty: no exception, no error, just the work that happens after the response quietly not happening. Flip a service that does fire-and-forget background work to true and the work disappears without a trace in the logs.

So we reduced the decision to one question: does this service keep working after it returns a response?

The exclusion list is narrower than it looks

When we picked which services to switch, we started with a generous exclusion list of anything that "probably needs CPU all the time." Most of it turned out not to. Two things about Cloud Run's behavior are not obvious.

First, CPU is allocated during container startup regardless of this setting. A service that loads a large dataset from BigQuery at boot, synchronously, before the web server starts, looks like an obvious "needs constant CPU" case. But that work happens during startup, so switching it is fine.

Second, long timeouts are not a reason to exclude something. A batch that takes 3,600 seconds of synchronous work is handling a request that entire time, so it has CPU. "It's a heavy job, so it needs constant CPU" doesn't hold.

What actually needs excluding is services that keep working after the response goes out, the fire-and-forget ones. That's the whole list. Getting there took several rewrites of the exclusion set.

Settings like this always over-exclude when you decide by intuition. Exclude based on a condition you can state, not on a service feeling like it needs it. Whether you can compress the rule into one line is a decent proxy for whether you've understood it.

Results, and stopping the regression

The first service we switched came down 44% on its own. Rolling it out brought instance-based billing down 84% overall. Everything is switched except the handful of services doing fire-and-forget work.

Then we added a CI guard that fails on service definitions with no cpuIdle. The part I like: specifying cpuIdle: false explicitly is also a violation. If a service genuinely needs constant CPU, it goes in an allowlist with a written reason. The point is to make someone type out why.

And with guards like this, we now write a violation on purpose and confirm the build actually fails. A guard that's been written isn't necessarily a guard that runs, and there's nothing to tell you when it isn't. All you're left with is the belief that you're covered, which is worse than knowing you aren't.

Case 2: the data can't tell you how recovery actually works

Container image storage was the other large line item. The fix was trivial: we went from keeping 10 image versions to keeping 2. Storage dropped 65%.

The interesting part is how you get to that number. From the data alone, all you can say is "maybe we need 10." How far back you might roll to isn't something the data records.

Here's the actual reasoning. If the image is gone, you restore the code from git and deploy it again. So keeping images isn't an archive that lets you return to any point in history. It's a way to get back to the previous version quickly in an emergency. Framed that way, two is enough.

What's doing the work there isn't data, it's knowing how recovery actually happens in your organization. Whether restoring from git is a safe assumption. How long a deploy takes. Whether anyone has ever needed to go further back than one version in an emergency. None of that is in your logs or your metrics.

Ask an AI to look into it and you'll usually get "keep more versions, to be safe," because that's what the data supports on its own. What a human brings is knowing what the retention is for.

Case 3: we were running full CI for every review round

The CI change that mattered most was about when things run, not what.

A reviewer flags something, you push a fix, they look again. Every one of those pushes ran the full set of jobs: build, lint, test, and knip.

The structure is the same with human reviewers. It's just that AI review makes the rounds more frequent and much faster, so waste that was always there becomes obvious. Measured, we average 3.4 rounds per PR (counting a round as one flag-and-fix cycle), with 6.8 minutes of heavy jobs per round. Which means running the same test suite three or four times before review even converged. We were paying for a full test run against code that still had open comments on it.

So we moved the heavy jobs behind review approval. Every push runs a lightweight check set (1.2 minutes), and when the reviewer approves, a bot triggers the heavy CI.

Light checks per round, heavy jobs once at the end. 3.4 × 6.8 min becomes 3.4 × 1.2 min + 6.8 min, roughly 53% less.

Skipping and not-running are not the same thing

There's an implementation trap here. Our first attempt used if: conditions to skip jobs inside the same workflow, and that was the wrong design.

State	Branch protection treats it as
Job skipped via `if:`	Passing
Workflow never runs (no check exists)	Not reported, blocks the merge

Skip a required check and you've effectively disabled branch protection. The green check appears without the tests having run, and the PR is mergeable. That fails open.

Never start the workflow and no check is created, so the unreported required check blocks the merge. That fails closed.

That asymmetry is the whole design. The heavy jobs now live in a separate workflow triggered only by workflow_dispatch. If the dispatch fails, the checks stay unreported and the merge stays blocked.

The cost optimization and the safety net came out of the same decision. "Spend less" and "stop when something's wrong" tend to share a shape.

Also: deciding not to do something, with arithmetic

We also skip the heavy jobs on PRs that only touch documentation. The design question there was whether to split the docs-only detection into its own job.

A dedicated job is structurally cleaner, but every PR then pays for one more runner start, about 25 seconds. Docs-only PRs are around 4% of recent merges, and they save roughly 300 seconds of heavy jobs each.

Run the expected value: 96% of PRs paying 25 seconds outweighs 4% of PRs saving 300. So the detection happens inside an existing job and we skipped the dedicated one.

"Make the structure cleaner" always sounds right. In a cost context, you can decide against it with arithmetic. Go by which version feels more elegant and you'll miss reversals like this one.

Case 4: it looks free at first, then the data grows

One more: BigQuery MERGE.

When you want to insert data without creating duplicates, MERGE is the obvious choice. Match on a key, update if it's there, insert if it isn't. One statement, and it's idempotent.

The catch is that MERGE reads the target every time, no matter how many rows you're inserting. Even for a single row, it has to scan the target side to confirm that row isn't already there.

That's awkward from a cost perspective. While the table is small, it looks like nothing. Nothing is wrong on the day you write it. But as the data accumulates, the scan per run grows with it. Your execution frequency hasn't changed, and the cost climbs anyway.

And because nobody edited any code, this never shows up in PR review. A gate at merge time asks "how much does this change add," so anything that grows without being changed is out of scope by construction. This is the purest version of what I described at the top: invisible on the day you build it, accumulating quietly afterward.

Where the duplicate check should live

So we moved the duplicate check to Firestore, and BigQuery only gets the insert.

Look the key up in Firestore, and only insert the rows that come back as new. BigQuery becomes append-only, and the scan for reconciliation goes away entirely.

Same shape as Case 2. Not "how do we optimize the MERGE" but does BigQuery need to be the thing doing the duplicate check at all. BigQuery is excellent at scanning large volumes and aggregating. Asking it to confirm whether one key exists is using it against the grain. Firestore is the opposite: point lookups are its job, aggregation isn't. Push each to the side it's good at and both end up doing something natural.

Why you need something that notices

The point here isn't "don't use MERGE." It has its place, and we haven't replaced all of ours.

The point is that cost grows in two different ways: expensive from day one, and expensive after it grows into it. The first kind, review catches. The second kind, review never sees.

Which is what the daily reporting is for. Comparing against the same weekday last week, "this app's cost keeps climbing and nobody has touched it" eventually shows up on its own. That's the reason a gate alone isn't enough.

What the freed-up time is actually for

When implementation gets cheap, the gate that slow implementation used to provide comes off with it
What's left is the running cost of everything you shipped, which is invisible on the day you ship it
And it grows in two ways: expensive from day one, and expensive once the data grows into it. Review only ever catches the first kind
So the time you gained goes into asking whether it should exist
Habits erode, so make it structural: a gate at merge, daily reporting, quotas on usage-based services, guards in CI
And accept that some calls can't be automated: how recovery works, how far back you'd roll, whether this thing needs to exist at all
Push what can be systematized into systems, and spend human attention on what can't. That's what reallocating the time means

Shipping faster is unambiguously good. But spend all of the gains on shipping more and the weight of what you shipped catches up with you later. We put a week into this cleanup. The next move isn't to schedule another one, it's to build more of the machinery that keeps things from piling up in the first place.

GitHub Actions Getting Expensive? We Cut CI Costs to a Quarter With a One-Line Change

Ryosuke Tsuji — Wed, 29 Jul 2026 23:52:17 +0000

AI assistance disclosure: This article was drafted with the help of Claude. All technical content, design decisions, code references, and screenshots reflect production systems I designed and operate at airCloset; the prose was revised by me prior to publication.

Hi, I'm Ryan, CTO at airCloset.

We've migrated our GitHub Actions runners twice: GitHub-hosted to Blacksmith, then Blacksmith to Namespace. The results, up front:

Per-run CI cost is roughly a quarter of what we paid on GitHub-hosted
The slow tail (p90) is 37% shorter
Runs that silently never finish went from 32 to zero
Each migration was a one-line change

This post is the measured record: how we measured, what we found, and the one migration that failed. If you're deciding whether to move off GitHub-hosted runners, the numbers here should help.

With AI in the loop, CI swells quietly

First, why this matters now. Once AI agents become the main driver of development, CI cost grows on two axes at the same time.

The first axis is run count. Agents open PRs and push fixes faster than humans do, so the number of pushes climbs. In our environment, a single test workflow runs 3,000 to 3,500 times a month. Reviews are done by AI too, and every fix push triggers CI again, so this number only grows as development accelerates.

The second axis is task count. When development gets faster, you start wanting CI to do all the things that never used to be worth the cost. Another lint layer, stricter coverage gates, docs consistency checks. Each run gets longer.

It's run count multiplied by task count. Leave it unattended and the bill grows quietly, but steadily.

First things first: if you fit in the free tier, stay put

Let me state the opposite conclusion before anything else: if you fit inside the free tier, there is no reason to migrate. Public repositories get standard runners for free, and private ones come with 2,000 minutes a month on the Free plan and 3,000 on Team.

In fact, across airCloset, some repositories are still on GitHub-hosted runners today. Anything that fits inside the 3,000 free minutes stays where it is. There is no reason to move something that runs for free onto something you pay for.

This post is for teams that blew past the free tier a long time ago and are watching the bill climb every month.

The migration path

Here's the road we took.

Period	Runner	Trigger
Until late March	GitHub-hosted (2-core)	--
Late March to June	Blacksmith (4 vCPU)	Speed and cost
Late June to now	Namespace (4 vCPU/8GB)	Speed, cost, and the hang problem below

Along the way we also evaluated self-hosting on AWS spot instances, and decided against it because every job would pay the instance startup overhead. CI is a world where boot latency compounds, so always-pooled runners win.

Each migration is literally a one-line change to runs-on:

 jobs:
   test:
-    runs-on: ubuntu-latest
+    runs-on: nscloud-ubuntu-24.04-amd64-4x8

The rest is a one-time GitHub App connection in the provider's console, and the whole edit takes a few minutes. Standard features like actions/cache keep working as-is. With migration cost this close to zero, the decision comes down to measurements.

Migration 1: GitHub-hosted to Blacksmith

The motivation was simple: speed and cost. We compared successful runs of the same test workflow in the two weeks on either side of the migration, a window where the workflow content was identical, using run history from the GitHub API. Note that run counts were far lower back then, which is why the sample sizes look modest.

	Median	p90	n
GitHub-hosted (2-core)	340s	598s	237
Blacksmith (4 vCPU)	139s (-59%)	157s (-74%)	130

To be clear about what this is: it is not "Blacksmith is 2.4x faster at the same size." We doubled the machine, from 2-core to 4 vCPU.

The real point is the unit price. GitHub-hosted 2-core costs $0.006/min, and the 4-core larger runner costs $0.012/min. Blacksmith charges $0.004/min for 2 vCPU and $0.008/min for 4 vCPU. In other words, for 1.33x what GitHub charges for 2 cores, you rent twice the machine. Run time shrank to 41%, so per-run cost worked out to roughly 45% less (1.33x price times 0.41x time), while runs got 2.4x faster. Faster and cheaper. "The same money buys twice the machine" is the actual nature of this kind of migration.

The snag with Blacksmith: flaky install hangs

Blacksmith earned its keep at that price, but one problem in daily operation was impossible to ignore: dependency installation (the npm install step) would occasionally hang in silence, and CI would never finish.

We never pinned down a reproduction; it was flaky. GitHub Actions jobs default to a 6-hour timeout, so left alone, a hung run keeps billing and occupying a runner for 6 hours. For a while, people noticed stuck runs, cancelled them by hand, and re-ran them, and that happened a lot. We then tightened the timeout to 30 minutes, and even after that, 32 runs died at the timeout without anyone noticing, over about two and a half months. Manual cancels aren't in that count, so 32 is a floor. In our first three weeks on Namespace: zero.

The raw number may look small. But when this lands in the middle of an autonomous agent loop, a silent stall becomes a stall of the whole loop. A human notices "huh, CI is stuck," cancels, and re-runs. An agent loop needs detection and retry machinery built separately, and that quietly piled up operational cost.

One note in fairness: this is what we observed during our usage window, March through June of this year. It may well be improved by now, and I can't guarantee it reproduces elsewhere.

Migration 2: Blacksmith to Namespace, including the failure

The motivation was again speed and cost, plus the hang problem above. But this migration failed once, and we rolled it back. The failure seems worth sharing as-is, so here is what happened.

We didn't estimate our concurrency

Namespace's plans cap concurrent capacity, counted in vCPUs. The Developer plan (the cheapest, with a free trial) allows 32 vCPUs on Linux. With 4 vCPU/8GB machines, that's 8 machines at a time.

Because AI agents develop in parallel, our CI concurrency is high, and peaks blow well past 8 machines. From day one of the migration, runs piled up in the queue and CI clogged. We rolled back to Blacksmith after two days, upgraded to the Business plan (160 vCPUs, or 40 machines), re-migrated, and it has been stable since.

The lesson is simple: measure your repository's peak concurrency before you switch. Pull run history from the GitHub API and the peak number of simultaneously running jobs falls out mechanically. Pick a plan on price and speed alone and you'll take the same detour we did.

Measured: the median shrank, and the tail shrank more

Amusingly, the rollback handed us a clean controlled experiment. The two rollback days and the five days right after re-migration ran exactly the same workflow content, so the runner is the only variable.

	Median	p90	n
Blacksmith (4 vCPU, rollback window)	378s	839s	335
Namespace (4 vCPU/8GB)	339s (-10%)	526s (-37%)	311

The median improved by about 10%, but the bigger story is the tail. p90 dropped from 839s to 526s, 37% shorter, and the hangs went to zero. "Occasionally slow, occasionally stuck" simply disappeared, and for agent-loop operation that is worth more than a median improvement, because the duration of a loop iteration is set by its slowest link.

The cost math

Published unit prices (Linux x64, as of July 2026):

Runner	Size	Price/min
GitHub-hosted standard	2-core	$0.006
GitHub larger runner	4-core	$0.012
Blacksmith	2 vCPU	$0.004
Blacksmith	4 vCPU	$0.008
Namespace	4 vCPU/8GB	$0.004 (prepaid)

A 4 vCPU Namespace machine costs less per minute than GitHub's 2-core. That's the single biggest lever. Normalize per run with the measured times, and GitHub-hosted to Blacksmith cut about 45%, Blacksmith to Namespace roughly halved it again. In total, about a quarter of the GitHub-hosted era.

Three caveats.

Namespace has a plan fee: $100/month on Team, $250/month on Business, each with the same amount of usage included. And if you fit in the free tier, GitHub Actions costs zero. No unit price beats free. That's why the earlier "stay put" advice comes first.
$0.004/min is the prepaid rate (the usage included in your plan). Overage runs at $0.006/min, which is the same as GitHub's 2-core. The assumption is that you pick a plan your usage fits inside.
Prices change. These are published rates at the time of writing, so check the pricing pages (GitHub / Blacksmith / Namespace) when you decide.

Which tier are you in?

Three stages, by scale.

You fit in the free tier: stay on GitHub-hosted. Do nothing
You're past the free tier and the bill is starting to sting: Blacksmith or Namespace, either way a one-line runs-on change buys twice the machine for the same money. Blacksmith has 3,000 free minutes a month and Namespace has a 30-day trial, so trying costs nothing
AI agents are your main developers and CI concurrency is high: choose on tail latency and stability, and estimate peak concurrency before you migrate

Wrap-up

With AI-driven development, CI cost grows as run count times task count
GitHub-hosted to Blacksmith: 1.33x the unit price for twice the machine, 59% faster runs, about 45% cheaper per run
Blacksmith to Namespace: median -10%, p90 -37%, hangs from 32 to zero, at half the unit price
Namespace caps concurrency by plan. Estimate your peak concurrency before switching
If you fit in the free tier, don't migrate. If you're past it, start with one line of runs-on

If your GitHub Actions bill keeps growing, Namespace is a choice I can recommend with measurements behind it. I hope the numbers help you make your own call.

AI-Native Redesign: The Principles Don't Change — Only the Machinery Does

Ryosuke Tsuji — Tue, 28 Jul 2026 00:13:26 +0000

AI assistance disclosure: This article was drafted with the help of Claude. All technical content, design decisions, code references, and screenshots reflect production systems I designed and operate at airCloset; the prose was revised by me prior to publication.

Hi, I'm Ryan, CTO at airCloset (a fashion-rental subscription service based in Japan).

"Everything changes with AI" is the prevailing mood. My experience building and then running an internal AI platform (cortex) points the other way. The principles don't change at all. Only the machinery does. This post is about what I've come to treat as principle, what I've concluded should be broken, and the thinking behind that split.

Disclaimer: "cortex" in this article is the internal codename for the AI platform built in-house at airCloset. It is unrelated to existing commercial services like Snowflake Cortex or Palo Alto Networks Cortex.

I've written about the individual pieces before: code-graph, product-graph, db-graph, biz-graph, AI-Observability, the auto-review harness, and Self-Healing. This post isn't about any of them. It's about the design principle sitting behind all of them, one abstraction level up, more essay than build log.

The principle, in one sentence: how do we make accurate information accessible? It's an old question. Libraries, legal case books, encyclopedias, search engines — every era has had its own answer using whatever tools that era gave it. Even the technology revolutions people call "paradigm shifts" mostly just changed the means. The underlying question didn't move.

Now AI has arrived, and my read (probably not a controversial one) is that its shift is at least on the scale of the internet, possibly larger. As with every previous paradigm shift, the means of answering "how do we make accurate information accessible?" will get redesigned from the ground up. That's what this post is about: AI-Native Redesign — a view where you rebuild the whole design with AI treated as a given, instead of bolting an "AI tool" or a RAG layer onto a design that was optimized for humans doing everything.

The Underlying Principle

Splitting the Principle into Three Nodes

The frame I use: split the principle into three nodes — creation / maintenance / consumption. Someone (or something) creates it. Someone maintains it. Someone consumes it.

A library: creation = the catalog and classification system, maintenance = adding new titles and updating the shelves, consumption = someone borrowing a book. Legal case books and lawyers: creation = courts writing their rulings, maintenance = new cases getting added to the corpus, consumption = a lawyer looking up a matching case for a client. Engineering docs at a company: creation = writing the spec or README, maintenance = updating it when the code changes, consumption = another engineer reading it while implementing. In every case, each of the three nodes has someone owning it, and the whole thing only works when all three keep turning.

As long as the three of them are held together by a chain of trust and incentive, the system holds itself up. If one of them drops out, the whole thing slides into a negative spiral and quality decays. I'll come back to the specific mechanism later in this chapter.

"Information" here changes shape depending on the domain — knowledge in books, legal precedent, internal documentation at a company, runtime behavior of a service, customer trends, the reasoning behind a decision. What they all share is this: as long as it lives only in one person's head, the moment scale kicks in the whole system stops working. It only starts to work at scale once the information is stored in a form you can look up, and reachable when someone needs it.

What Has Changed, What Hasn't

The means have changed a lot. Clay tablets, oral tradition, manuscripts, dictionaries and indices, the printing press, legal codes and lawyers, library classification, then in the last few decades: wikis and paper docs, APM and structured logs, searchable knowledge bases, distributed tracing, web-scale search engines, BI dashboards, RAG. Each era tried to optimize the three-node balance with whatever tools it had.

Wikis fit the 90s because "humans write, humans update, humans search" was the only shape available. APM appeared in the 2000s because storage got cheap enough to hold the telemetry that machines generate. Each generation had its own subject and its own tooling to answer with.

But the underlying question hasn't moved. Every era in every domain was solving the same problem — "make accurate information accessible" — with that era's tools. And in some sense, each generation was consciously designing the three-node balance. But once humans are in the loop, they cut corners on writing, forget to update, get things wrong, eventually stop maintaining, and the whole thing breaks. Some systems have held up (library systems, legal frameworks, established academic disciplines), but most have fallen into the negative-spiral side because the human limit was the binding constraint.

What makes AI different is that it dramatically widens the range where creation and maintenance can run without much human labor. Automating creation and maintenance is itself old news — deterministic systems have covered a huge amount of it, so widely that we don't even notice anymore. It's not just engineering infrastructure (APM, CI/CD, log collection, schema validation). Media services run on the same shape: articles get created, updated, and deleted through a system, then distribution pushes them to the app, the website, and the print edition. Bank transactions, e-commerce catalogs, routing data in a map app, the timeline in a social network. Deterministic automation is everywhere, quiet enough that it's easy to forget it's there.

But deterministic automation has a ceiling. There's a class of work it can't touch: qualitative judgment, articulating design context, updating documentation to follow a code change, adding annotations — anything that needs interpretation and context. AI is the first thing that reaches into that zone. And it can even sit on the consumption side (in the APM example, this means AI running the "look at the dashboards, close the improvement loop" work that humans used to do). Once the human requirement drops structurally, many domains can start running a positive spiral for the first time. But how to wire this expanded capability into your own organization's loops is still a human design call. That's what this post — AI-Native Redesign — is about.

Where this post lands
AI isn't a replacement for deterministic automation. It's a new capability that brings the zones deterministic tools never reached into the automation envelope — but redesigning your organization's loops around that expanded capability is what turns it into a change on the ground.

A Closer Look at Each Node

Creation. Putting information into a form that can be looked up later. Writing the structure of your code down as docs. Instrumenting the runtime to emit logs and traces. Leaving the reasoning behind a decision as a design doc. Turning a customer trend into a KPI definition. Anything that leaves something behind in a form someone (or something) can come back to.

Maintenance. Keeping what you already wrote from drifting away from its source as that source changes. Code changes, the doc should follow. Service topology changes, the metric definitions should follow. Customer trends shift, KPIs should follow. Decisions get overridden, the record should follow. If creation is a one-shot act, maintenance is the standing chore that never ends.

Consumption. Actually reaching for what's stored and using it to decide or act. Humans reading. Machines querying. Alerts firing. AI agents pulling context. All of it counts.

These aren't sequential phases (creation → maintenance → consumption). They're a single system held together by mutual trust and incentive — you can't evaluate any one of them in isolation.

The Negative Spiral — Why It Collapses in Most Real Places

Concretely, the interdependence looks like this:

If it doesn't get consumed, no one is motivated to create it. Nobody keeps writing docs no one reads. No engineer polishes a dashboard no one looks at.
If it doesn't get maintained, it becomes unfit to consume. A doc from six months ago gets a "probably stale" tag in someone's head and is skipped. A dashboard whose metric definition drifted since the last product change quietly seeds wrong decisions.
If it doesn't get created, there's nothing to maintain in the first place. Information that was never stored in a queryable form can't be tracked as it changes.

As long as all three sides believe "my part is worth doing," the loop keeps turning. But if any one of them gets too expensive, the other two lose their incentive too. "Nobody's going to read it, and it'll rot anyway," "I updated it but no one cared," "when I search it's stale or wrong" — three separate excuses that reinforce each other, and the negative spiral picks up speed.

How this collapses in most places
If any one of the three nodes gets too expensive, the other two lose their reason to invest. The reason documentation cultures don't stick, monitoring stacks go stale, and knowledge bases quietly hollow out isn't a tool problem — it's structural. The balance between the three nodes has to hold, or the whole thing decays in silence.

That's why patching a single node never fixes it. Many people have seen the "our Confluence has 100k pages and no one reads or updates them" version of this. Not a Confluence problem — what happens when at least two of the three nodes (creation and maintenance) stay expensive, and the fix that gets applied is a search feature on the consumption side. The loop never closes.

The Rest of This Post

Everything from here on works off the same principle: make information accessible, and run it as a three-node loop.

Chapter 2 lines up concrete cases from around us (internal docs, monitoring, business data, schema management, security) and shows how each one has been carrying its own version of the three-node problem.
Chapter 3 argues why AI is the pivot, from the angle of "the first thing that brings zones deterministic automation couldn't reach into range of automation."
Chapter 4 walks through the AI-native implementations in cortex (code-graph / db-graph / biz-graph / product-graph / Observability) side by side.
Chapter 5 goes into the difference between "adding an AI tool" and AI-Native Redesign, and why keeping pre-AI design in place while adding AI is a losing pattern.
Chapter 6 sketches what widens between organizations that have multi-layered self-sustaining loops and organizations that don't — the evolution-speed gap I think grows non-linearly over time.

Each chapter reads independently, but read in order, they form a single chain: unchanging principle → common applications → what's special about AI → concrete implementations → the redesign argument → what's next.

Where Deterministic Automation Couldn't Reach

I said earlier that deterministic automation has already reached almost everywhere. And I said that deterministic automation still has zones it can't touch. This chapter breaks those remaining zones down.

If you look at each domain and split it into the three nodes, two categories show up.

Type 1: Domains Where One Node Is Still Human-Only

Domains where deterministic automation covers some nodes but not others. Laying them out side by side:

Domain	Creation	Maintenance	Consumption
Internal docs	Manual (specs / design docs / runbooks)	Manual (keeping them current)	Manual (reading them)
Observability	Automatic (APM / logs / metrics)	Automatic (metric def changes still manual)	Manual (dashboard triage, improvement cycles)
Business data (BI reports)	Automatic (SQL / ETL)	Automatic (schema changes still manual)	Manual (analysis, interpretation)
Schema management (basic CRUD)	Automatic (ORM generation)	Manual (migration decisions)	Automatic (validation)
Security posture	Automatic (logs, anomaly detection)	Automatic (rule updates still manual)	Manual (threat assessment, alert triage)

Every remaining human dependency is in the parts that need qualitative judgment or contextual interpretation. In most of these, the bottleneck settles on the consumption side — "the information is there, but no one uses it" is the classic symptom. Humans interpreting a room full of dashboards and log streams at once hits a cognitive ceiling, and that ceiling has been the structural bottleneck. Internal docs is a special case where all three nodes are human, which is why it's the most obvious pain point.

Type 2: Domains Where Building the Artifact Wasn't Worth the Return

Separate from Type 1, there's a deeper category: the zones we never tried in the first place. The individual data and code are there, but the cost of building the "system that structures the relationships between them" never matched the payoff. This covers two subtypes — relationships that already exist but aren't visible to humans (like connections across code, or across tables) and relationships that were never even defined (like the causal link between a marketing initiative and a KPI). I'll come back to Type 2a and Type 2b in a moment.

What's interesting is that the underlying work — static analysis, SQL aggregation, graph-building pipelines — is often perfectly writable in deterministic logic. The reason nothing existed here wasn't "we couldn't build it." It was "the ROI didn't clear the bar."

Break down why no one built it and you get problems on both sides — a double whammy:

Cost side. Even if deterministic logic can build it, standing up the system to do so (the static analyzer, the extraction pipeline, the graph substrate) is real engineering effort.
Payoff side. Even if you build it, humans can't consume it usefully. A human traversing a graph node-by-node isn't a real workflow, and no semantic search layer existed to sit on top.
When cost is high and payoff is thin, no one signs up.
So the thing that would have been useful just... didn't exist.

Type 2a: Making existing relationships visible (an analysis problem).

Relationships that are already there in the code or the data, but that humans can't see or follow. For example:

Boundary connections across a codebase (which API is called from other repos, and where)
Semantic relationships between tables (which set of tables represents the same business entity)
Log correlation across distributed services (the causal chain of logs across services)

These are all "the information already exists, but no human can analyze it" failure modes.

Type 2b: Designing new relationships (a design problem).

Relationships that don't exist anywhere yet — you have to design a conceptual model first, then extract against it. For example:

The causal link between an initiative and a KPI (which campaign moved which number)
The pairing of a decision to its outcome (how the call in a design doc played out after implementation)
The structure connecting customer segments to behavior patterns

These are a deeper failure mode: "the information doesn't exist at all — you have to design it before you can extract it." The classic example is relationships that individuals hold in spreadsheets or in their heads, which don't scale up to an organization.

In my earlier biz-graph post (Making Initiative Impact Analysis Explorable with Graph RAG + MCP), I put the difference this way:

db-graph made existing relationships discoverable. biz-graph designed relationships that didn't exist yet and produced them. The first is an analysis problem, the second is a design problem.

Type 1 vs. Type 2

Even though they're both "zones deterministic automation couldn't reach," they behave differently:

Type 1: "this piece is human, so it's slow or stuck" — a bottleneck in an existing workflow.
Type 2: "this didn't exist to begin with" — opening a domain that was never charted.

Type 2 has the larger ceiling. Removing a bottleneck makes an existing workflow faster. Charting new territory makes decisions and insights possible that weren't possible before.

The Pattern Underneath Both

What Type 1 and Type 2 share: the parts that resisted rule-based encoding.

Type 1's leftover work: "is this dashboard anomaly a real incident or a known false positive?" "when and how do we run this migration?" "what should this doc say to its reader?"
Type 2's leftover work: "which pieces of code represent the same boundary?" "which initiative moved which KPI?" "how do you lay out a conceptual schema like Week × MetricDomain?"

The common thread is qualitative or contextual judgment, or semantic connection. Those can't be written as rules, which is why they've been left standing.

That framing sets up the next chapter — why AI is the pivot. AI:

covers the remaining qualitative-judgment nodes from Type 1, and
shifts both sides of the Type 2 double whammy (it lowers the cost of building deterministic extraction systems, and it becomes the missing consumer that can pick up the resulting artifact semantically).

Not only "the zones deterministic automation couldn't reach," but also "the zones deterministic automation could reach but wasn't worth building" — both come into range of automation for the first time. That's what makes AI new.

Why AI Is the Pivot

The previous chapter split "zones deterministic automation couldn't reach" into two types: the leftover qualitative-judgment nodes in Type 1, and the cost-vs.-payoff bind in Type 2. This chapter goes into why AI is the first thing that can address both.

What's Genuinely New About AI

Every automation technology before AI stayed inside the region where you can write deterministic logic. Enumerate rules, put in branches, match patterns. It's a very powerful stack, and as I said earlier, it's threaded through nearly everything in modern life.

But it hit a class of work it couldn't touch — the same class the previous chapter arrived at from a different angle: qualitative judgment, contextual interpretation, semantic connection. A judgment that a human "kind of understands" explodes into an unmanageable condition tree the moment you try to write it as code. What Type 1's leftover human-only nodes and Type 2's un-built structuring systems had in common was that neither survived rule-based encoding.

What AI brings for the first time is making that whole class machine-processable. An LLM handles judgments that don't survive being turned into code by treating them as statistical patterns. This isn't an extension of the deterministic stack. It's a new capability, orthogonal to it.

Three Directions of Change

This new capability shifts systems in three distinct directions. They map onto the Type 1 / Type 2 split from the previous chapter: direction 1 addresses Type 1, and directions 2 and 3 address Type 2.

Direction 1: automate the remaining qualitative-judgment nodes in Type 1.

In Type 1 domains, the human-only node — dashboard interpretation, threat assessment, documentation updates, KPI analysis — is now something AI can carry.

Direction 2: cut the cost of building the structuring system.

The Type 2 side "deterministic logic could build this, but the effort was too much." AI assists at each step of building the system (spec design, code generation, testing, debugging), which drops the barrier to standing up these systems. Extraction pipelines that used to take months now regularly land in days. As a concrete data point, the initial version of cortex's biz-graph (the MCP server that handles initiative × KPI causality) went from implementation to Pulumi deployment in one day.

Direction 3: consume structured data semantically.

This one dissolves Type 2's "even if we build it, no one can use it." A graph a human couldn't traverse by hand, AI walks with a mix of graph traversal and semantic search. A question like "which of last month's marketing initiatives contributed most to new-user acquisition?" gets answered with both structures working together.

Deterministic-First

Design principle: deterministic-first
If a thing can be written deterministically, write it deterministically. Keep the surface where AI does inference as narrow as necessary — this is what "containing hallucinations" actually means in practice.

Let AI touch parts that deterministic logic could have handled, and you widen the hallucination surface for nothing. The "containing hallucinations" phrase I've used across earlier posts is really this call — where to draw the line between deterministic and AI.

Leaning deterministic also gets you side benefits:

Idempotency. Same input, same output, every time. Critical for testing, auditing, and reproducibility.
Cost, by orders of magnitude. Inference calls are token-metered. Deterministic execution is typically 10× or more cheaper.

Where AI goes is a decision, not a default. "AI can do it" isn't the criterion. Deterministic where deterministic works. AI only where deterministic doesn't. That division of labor sits at the core of AI-Native Redesign.

Some concrete pairings:

Static analysis, SQL, pipelines → deterministic (Type 2 creation side).
The structured artifact those produce → AI consumes it semantically (Type 2 consumption side).
Wherever qualitative judgment is what's actually needed → AI takes it (Type 1).

Why AI Requires Whole-System Redesign

With earlier automation tools (APM, CI/CD, monitoring, BI), the standard move was to drop them on top of existing operations. They stood alone and didn't disturb the existing workflow, so adding them was enough.

AI is different. "Add an AI tool to the existing system" won't solve what I described above. The reason is that AI's value doesn't come from any single tool feature — it comes from shifting the balance across all three nodes.

Replace just the human node in Type 1 with AI, and if the downstream consumption workflow doesn't line up, no one uses the AI's output.
Stand up new Type 2 structure, and if consumption is still shaped around a human doing the judging, the artifact never actually gets used.
If you don't redesign the boundary between "AI handles this" and "deterministic handles this," ownership gets ambiguous fast.

This is why "add an AI tool" isn't enough. The whole system has to be redesigned with AI as a given. That's the AI-Native Redesign thesis, and I go into it in more detail later on.

Examples from cortex

This chapter walks through the concrete systems running in cortex and shows how the principles I've laid out play out in each of them. There's a dedicated post for each system that I'll link inline. Here I'm keeping the focus on the same three questions: where's the deterministic layer, where's AI, and how was the division decided.

code-graph

code-graph surfaces the code connections across 46 repositories (API boundaries, DB boundaries, event boundaries) into a single knowledge graph. Type 2a — making existing relationships visible.

Creation. Deterministic (tree-sitter static analysis). Function, class, and import call relationships get extracted as written.
Maintenance. Static analysis re-runs on code changes. Maintenance stays deterministic.
Consumption. AI over MCP, combining semantic search and graph traversal.
Hallucination containment. Boundary nodes (API endpoint, DB table, event topic) are materialized explicitly during static analysis, so AI's inference range is scoped to "stop at the boundary."

db-graph

db-graph covers relationships between database tables and the business context around them (Type 2a). Both the ORM-level JOIN relationships and the business-entity-level semantic relationships get graphed.

Creation (structure). Deterministic (static analysis of the ORM, schema extraction) + human review as the guarantee.
Creation (business context). AI generation + human review as the guarantee.
Maintenance. ORM and schema changes get detected deterministically; drift in the business context is caught in human review.
Consumption. AI answers natural-language questions like "which tables are involved in the customer purchase cycle" by combining graph traversal and semantic search.
Hallucination containment. Structure stays deterministic. Business context is AI-generated but gated by human review as the final check. The reason db-graph goes through human review while cortex-product-graph runs on AI review comes down to blast radius: wrong DB structure or wrong business context feeds directly into organizational decision-making, so the risk of being wrong is much larger than for a code error. The call isn't just "if AI is accurate enough, let AI handle it" — it's paying the cost of human review whenever the downside of being wrong is large.

biz-graph

biz-graph covers the causal relationship between initiatives and KPIs (Type 2b — designing new relationships). Unlike db-graph, there's no "JOIN target" sitting between initiatives and KPIs to begin with. The relationship has to be designed by a human first.

Creation. AI (extracting structure from initiative slide decks) + deterministic (KPI data extraction, embedding-based similarity edges) + human schema design (someone defines conceptual anchors like Week and MetricDomain).
Maintenance. New initiative decks and KPI updates keep flowing in to keep the graph current — slide parsing on the AI side, the value pipeline deterministic.
Consumption. An AI agent handling "what's the causal relationship between last month's social campaigns and this week's new-user counts?" traverses Initiative → Week → co-occurring KPIs on the graph.
Hallucination containment. The human-designed schema (conceptual anchors like Week and MetricDomain) is the deterministic frame around AI. AI can only reason inside that frame. KPI extraction, similarity edges, graph construction — all deterministic. AI is confined to the consumption side and to the judgment calls during slide extraction.

cortex-product-graph

cortex-product-graph is cortex's main knowledge graph, unifying cortex's own code, DB schema, docs, and Pulumi IaC. AI is used heavily in cortex development itself, and this system is a good example of how the AI-Native Redesign principles land in a working setup.

Creation (structure). Deterministic (ts-morph extracts @graph-* JSDoc from code and Pulumi IaC, then merges with the documentation and with db-graph).
Creation (code + annotation). Developer + AI assist (Claude Code / Codex generate code and the annotations at the same time).
Maintenance. cortex's AI review runs per-PR and checks code logic, doc consistency, and annotation drift together, filing REQUEST_CHANGES when they don't line up.
Consumption. AI over MCP, combining semantic and structural search.
Hallucination containment. cortex's AI review looks at code + annotation + docs together on every PR. Even PRs that get merged without a human reviewer go through AI as a review layer. Because the code and its @graph-* annotations sit next to each other in JSDoc (code and intent as an SSoT inside the same file), AI spots the gap between the code change and its intent immediately, which is why AI review stays accurate. This is also a bet on AI-Readability — writing code in a form that's not only readable by humans but also structurally parseable by AI agents.

Observability + Self-Healing

AI-Observability handles the four monitoring axes (Application / Infrastructure / CI / LLM), and the loop from there to Self-Healing is where AI-Native Redesign is at its clearest. Type 1 — removing the consumption-side bottleneck.

Creation + maintenance. Deterministic (OpenTelemetry, metrics, logs, traces, deterministic alerts).
Consumption (detection). Deterministic alert thresholds fire incidents.
Consumption (judgment and response). AI cross-references log and trace context (pulled via Grafana MCP in practice) with the relevant source code / tables / docs (found by traversing cortex-product-graph), produces a root-cause hypothesis, and then a fix PR.
Maintenance loop. The generated fix PR is quality-checked by the AI review flow that runs on top of cortex-product-graph before it merges.
Hallucination containment. AI never touches production directly. AI's output is always a PR — something reviewable. cortex-product-graph + AI review + auto-merge chain acts as the final gate.

This is the fully-formed loop of AI-Native Redesign: monitoring → detection → AI inference → PR generation → AI review → merge. The whole loop turns, and cortex ends up in the "fixed before we notice" state (the title of the Self-Healing post).

The Pattern Underneath All Five

Building these five systems, some design calls kept recurring by choice.

Creation side stays deterministic by default. Anything that reads "the data as written" — static analysis, ORM, OTel, SQL, extraction pipelines — leans deterministic.
AI is confined to consumption and to the meaning layer on top of structured artifacts. Graph traversal, semantic search, annotation generation — the judgment work that doesn't survive being written as rules — is where AI sits.
There's always a hallucination-containment mechanism. Boundary nodes, AI review, human review — the specific mechanism differs, but every system has some form of lid on AI output. AI's free-writing zone is kept narrow, and even inside that zone, its output goes through a review layer.
AI review quality depends on the context foundation. AI can do comprehensive PR review in cortex because cortex-product-graph exists as the structured context foundation. Without it, AI would only see local information from the PR diff, and couldn't judge consistency with the rest of the codebase or the docs. Before "where and how do we use AI," the question that comes first is: "what context can we give AI to reason over?"
The review choice has a risk-profile component. Even when AI review would be accurate enough, if the downside of being wrong is large, human review can still be the right call. As I mentioned with db-graph, it's a comparison between the cost of being wrong and the cost of human labor.
Human design judgment sits above the whole thing. Schema design (biz-graph), guideline definitions (auto-review), monitoring target selection (Observability) — these are human calls.

The pattern to notice: the same AI-Native Redesign principles land in different shapes depending on the subject and the goal.

AI-Native Redesign vs. "Adding an AI Tool"

So far the argument has been: AI is the first thing that reaches into the zones deterministic automation couldn't, and making that actually work needs whole-system design — a context foundation like cortex-product-graph, or a closed loop like Observability + Self-Healing.

This chapter goes into why the "just add AI to the existing system" approach — the approach that skips the whole-system redesign — falls short.

What "Adding an AI Tool" Usually Looks Like

Over the past year I've put a range of AI tools into the organization myself. Here are the shapes I've tried at least once:

AI summary on dashboards. AI reads the dashboard and gives you the takeaway.
AI-generated docs. Docs get produced from code.
AI PR review. AI reads the PR and comments.
AI search on the internal knowledge base. Natural-language queries against internal knowledge.
AI-assisted coding. Claude Code, Cursor, etc.

Each one was useful in isolation. But honestly, the results topped out around 1.x — worth the deployment cost, but nowhere near a paradigm shift.

That was the point where I had to step back and ask what it would take for the organization to use AI better. What came out of that were the three failure modes below, and the AI-Native Redesign direction that follows from them.

Failure Mode 1: The Three-Node Balance Stays Optimized for Humans

Existing systems were shaped, within the capabilities of their era, around "humans handle every node." That assumption is baked into things you don't usually see as design choices:

Dashboards: density and count set for what a human can consume.
Documentation: structure and granularity set for what a human can read.
Code: conventions and granularity set for what humans write and review.

Add AI on top and AI has to operate inside "the balance optimized for humans." AI's actual strengths — watching a room full of dashboards at once, traversing docs structurally, verifying code exhaustively — get suppressed because the surrounding two nodes stay shaped for humans.

Failure Mode 2: AI Is Asked to Judge Without a Context Foundation

As the earlier examples showed, AI review works comprehensively only when there's a context foundation like cortex-product-graph. Ask AI to judge without one, and it only sees local information, and its actual value doesn't come out.

PR review AI: seeing only the PR diff, all it can do is comment on coding style.
Dashboard AI summary: summarizes the numbers on that one dashboard; the relationship to the rest of the system is invisible.
Doc AI search: keyword match or semantic search, each a local result.

The feeling "AI is shallow" is usually not about the AI. It's about the missing context foundation the AI was supposed to reason over.

This is adjacent to what's now being called context engineering. I've written about the retrieval-side design in an earlier agentic Graph RAG post, but in the three-node frame, retrieval quality is only the consumption node. The failure here is upstream — the creation side never produced a form that could be pulled as context.

Failure Mode 3: The Creation Side Doesn't Shift into a Form AI Can Consume

Of the three directions I laid out earlier, Direction 1 (automate the leftover qualitative-judgment nodes from Type 1) can be reached by adding AI to an existing system. But Direction 2 (cut the cost of building structuring systems) and Direction 3 (consume structured data semantically) both require the creation side to change into a form AI can consume.

JSDoc @graph-* annotations on code express structure and business intent as an SSoT (Single Source of Truth) → AI can understand structure and intent together.
Logs emitted as structured events, correlated with traces and metrics from other services → AI can follow causal chains across a distributed system.
Docs restructured from standalone files into data tied to code, design, and domain → AI can pull them as context.

These are creation-side design changes. Adding a feature to the existing system doesn't produce them. "Add AI on the consumption side" alone caps AI's ceiling at whatever the input-side constraints are — a rough format, implicit context, purely local data.

What AI-Native Redesign Actually Is

Flip the three failure modes and you get what AI-Native Redesign is:

Rebalance all three nodes around "AI + human" as the assumption. Which node gets AI and which stays human is redesigned from scratch. Not "add AI to a human-balanced system" — draw a new balance where AI-carried nodes are first-class parts of it.
Build the context foundation first. The structured context AI can reason over (something like cortex-product-graph) gets built first, and AI review and self-repair go on top of it. The opposite order — "put AI in, then notice context is missing" — is what fails.
Change the creation side. Reshaping the creation side into a form AI can consume — annotations, structured events, fine-grained docs — is part of the redesign. Consumption-side additions alone aren't enough.

The five cortex implementations from earlier all did these three. code-graph shaped structure through static analysis into a form AI could reach (creation-side change). cortex-product-graph became the context foundation for judgment itself. Observability + Self-Healing redesigned all three nodes with AI in the mix, from monitoring through to auto-repair.

This is the underlying reason the evolution-speed gap I sketch in the next chapter widens over time. The gap between "add an AI tool" and "AI-Native Redesign" is bigger than a linear-vs.-exponential ROI gap, from what I've seen.

Life After AI-Native Redesign

What Building cortex Has Actually Felt Like

Some things I couldn't see back when we were just deploying individual AI tools have come into view while building and running cortex.

Fix PRs now ride straight into AI review and auto-merge with no human in the path — 115 of them via Self-Healing alone in the last 30 days (details in the Self-Healing post).
Drift between docs, annotations, and code gets repaired automatically in places that used to sit uncorrected.
The cost of chasing "how was this actually built again?" through past decisions has dropped.
Individual writing speed hasn't changed much. What did change: the quality bar for what actually ships, and the fact that people other than me can now contribute. Monthly merged PRs going from the 10–23 range through March to 518 in April 2026 came from the workflow switch (main push → PR + AI review + auto-merge), not from writing more. The number is really the shape of "the ceiling of 'reviewed manually by me' came off, so this scale and this quality bar can now be sustained by more people than just me" (data in the harness intro post).

This isn't "add an AI tool, get 1.x." It's what happens when multiple self-sustaining loops start turning across layers. Qualitatively a different kind of change from a one-off productivity bump.

What Multi-Layered Self-Sustaining Loops Actually Look Like

Each of the cortex systems has its own self-sustaining loop.

code-graph. Every code change updates the graph, AI reviews using the updated graph.
cortex-product-graph. Every PR keeps annotations and code aligned, and AI review accuracy tightens with each pass.
Observability + Self-Healing. Monitoring detects an incident, AI produces a fix PR, AI review checks it, and it merges.
biz-graph. Initiative-to-KPI relationships get extracted continuously and stay in a form usable for decision-making.

These loops run independently, but they're connected through cortex-product-graph as a shared context foundation. The output of one loop becomes the input to another — that shape of connection.

Once this kind of multi-layer loop starts running inside the organization, it changes how time gets spent at a fundamental level. Not "AI helps" — closer to "a substantial share of daily work completes inside the loops."

Where This Structure Could Rot

If the three-node symmetry is the axis, then AI-Native systems' own negative spiral is a question that has to be asked too. The circular dependency I presented as a virtuous cycle (AI review maintains cortex-product-graph, cortex-product-graph supports AI review) turns into a self-amplifying error loop the moment errors get into the foundation. AI reasoning confidently and consistently wrong on top of a contaminated context foundation is a real, symmetric failure mode of this design, not a hypothetical.

The defenses split three ways. The reason db-graph puts human review at the final gate is entry-side containment — narrowing the flow of contamination into the foundation wherever the downside of being wrong is large. Materializing boundary nodes explicitly through static analysis is inference-range containment — narrowing where AI is allowed to reason. And keeping cortex-product-graph in a form that can always be regenerated from deterministic extraction + annotations + docs is recoverability — a way back once contamination is detected. All three are held together by "the foundation is never something AI alone writes into." The signal that this failure mode is starting to show is drift in AI review's own accuracy metrics (REQUEST_CHANGES rate, false positive / negative rate) that no one can explain — which is the symmetric-side extension of what I meant in the AI-Observability post when I argued LLMs should be the fourth monitoring axis.

How I Read the Evolution-Speed Gap

The rest is my read. I don't know how much of this generalizes to other organizations.

One objection I want to head off: "this sounds like something you need a dedicated platform team for." airCloset doesn't have one. I built cortex solo, alongside my CTO duties, and now people from the business side, not just engineering, are shipping on top of that harness. The fact that this was reachable at all is itself Direction 2 (AI cutting the cost of building structuring systems) in action.

Between organizations running AI as individual tools and organizations that have assembled multiple self-sustaining loops across layers, my sense is the evolution-speed gap widens over time.

In the first, AI is a useful tool, but decision-and-implementation speed itself doesn't change much.
In the second, the full cycle — decision → implementation → detection → repair — gets an order of magnitude faster.

Compound that gap over time and how much you can get done in the same period starts to diverge — gradually or sharply. "Exponentially" would be overstating it, but at least the way I see the gap widening, "linear" doesn't describe it either.

What This Post Was Trying to Say

Take the timeless question — "how do we make accurate information accessible?" — and redesign for it with AI as a given capability. That's the idea at the center of AI-Native Redesign.

AI isn't a replacement for deterministic automation. It's a new capability that brings zones deterministic automation couldn't reach into the automation envelope for the first time.
Adding AI in isolation doesn't work. The positive spiral only kicks in when all three nodes are rebuilt together.
Doing that requires a context foundation AI can reason over (something like cortex-product-graph), built ahead of the AI layer.
The whole thing is a stack of self-sustaining loops, and the more of them turn together, the more the organization's evolution speed changes.

One last thing. I scoped this post to information access, but the creation / maintenance / consumption loop shows up far more widely than that. Cultural transmission, the survival of a business, the growth of an academic field, a living language itself — all of them turn on the same structure, where creation dries up without consumption and quality decays without maintenance. Dead languages, lost traditions, failed companies, the internal wiki no one reads — push far enough and they hollow out through the same mechanism. Which raises a question I can't answer here: how far past information systems does "AI structurally lowers the cost of creation and maintenance" actually reach? I'll leave that one open.

The cortex build is one instance of trying this out. Different organizations and different subjects will land somewhere else, but the underlying question — "how do we make information accessible?" — should be the same. If this post is useful as material for laying an AI-Native answer over other contexts, that's what I was hoping for.

Observability Design for the AI Era — Reconciling PII Protection With AI Searchability, and Driving Self-Healing

Ryosuke Tsuji — Mon, 13 Jul 2026 23:50:57 +0000

AI assistance disclosure: This article was drafted with the help of Claude. All technical content, design decisions, code references, and screenshots reflect production systems I designed and operate at airCloset; the prose was revised by me prior to publication.

Hi, I'm Ryan, CTO at airCloset.

In Part 1, I walked through the four monitoring axes (application / infrastructure / CI / LLM) and the deliberately different shape each one ends up in. That's the write-side of the observability stack, more or less wrapped up.

But shaping the write side isn't the end of the story. The moment production data flows through the stack, you have to block the path PII can take to slip in — and that's true with or without AI. It's the kind of classic observability problem where, if you cut corners, you walk straight into a leak incident.

Historically, the set of people who could read logs mostly overlapped with the set who could read the DB. For engineers with DB access, logs weren't an additional path to personal data — which put log-side defenses in a position where hardening them didn't meaningfully move the overall defense line for most organizations.

AI breaks that premise. Non-engineers pulling logs over MCP don't have DB access. Logs became, for the first time, a path where someone without DB access can reach personal data. On top of that, log content now flows into AI's input, which introduces new exposure surfaces: transmission to the model, and re-surfacing in the model's output. Log PII protection has shifted from "hygiene worth doing" to "required as a trust-boundary redesign." That's the premise this post starts from.

And on top of that, if the observability stack isn't queryable by AI, the whole "AI-consumable observability" goal from Part 1 falls apart.

Part 2 is about how I reconciled these two — protecting PII while keeping searchability for AI — and how that combination ends up driving Self-Healing from CI failure to PR proposal.

The Observability Stack Is a Natural Path for PII

App emits a log → it lands in Loki → AI queries it through MCP. Stand up this naive flow and you get:

Customer email addresses and phone numbers in error logs
Order response payloads riding inside traces
DB query logs that emit full table rows

Plain-text PII pooling in the observability stack means AI can search it directly. This isn't really an AI problem, it's an observability problem: the stack itself becomes a PII conduit. At the same time, if you scrub PII completely, you lose "I want to investigate Customer A's support ticket" as a query, which is a normal support workflow.

cortex (the internal AI platform) had to reconcile both. The key principle was: don't make "block the PII path" and "search by PII" mutually exclusive.

Note: "cortex" here refers to airCloset's internal AI platform codename. Unrelated to Snowflake Cortex, Palo Alto Networks Cortex, etc.

Multi-Layer PII Design — Six Layers

cortex's PII handling is six layers, each with a different role:

Layer	Purpose	Mechanism
Write: BQ Policy Tag	Column-level access control	`pii_high` / `pii_medium` / `pii_low` three-tier taxonomy. Without fine-grained reader on the column, SELECT errors out with `Access Denied` (pure CLS (Column-Level Security) — no dynamic masking)
Write: ETL DLP	Strip plain-text PII from derived tables	Cloud DLP redacts during transforms (customer support data, etc.). Placeholders like `[EMAIL_ADDRESS]` / `[PHONE_NUMBER]` preserve the structure
Write: log hashing	Plain text never reaches Loki	App-side hash via `hashEmail` (HMAC-SHA256 → 12-char prefix; key lives outside the observability stack) before log emit
Search: same function on both sides	Look up a specific customer's logs without ever touching plain text	Query-side runs the same `hashEmail` before sending to Loki
Output: MCP masking	Mask when AI consumes	Column-name detection masks the local part (e.g. `r***@air-closet.com`), keeping `@domain` so first-response triage can still tell which domain the account belonged to
Identity separation	Internal staff email is handled in a separate track from customer PII	HMAC-signed by Edge Router as auth attribution; not part of the masking pipeline

The fourth row — search with the same function on both sides — is where the security / usability tradeoff gets really tight.

I'll use email as the running example, but the six layers guard more than email. PII spans names (including phonetic readings), phone numbers, addresses, postal codes, dates of birth, card and bank details, external-service IDs, and more. The anonymization technique varies by the nature of the field — same-function hashing to preserve correlation (email, phone), partial masking (names, addresses), full redaction (card numbers, tokens) — and that call is made per field. What stays constant is the structure: which of the six layers guards it, and how. That's the reusable part of the design.

And this anonymization isn't confined to observability logs (Loki) either. An MCP tool that queries a service DB, for instance, pulls customer names, addresses, and phone numbers into its result set, so the same PII anonymization rules run before anything is handed back to the AI. The consistent rule is "anonymize PII on every data path that reaches the AI," applied across data-source types, not just one.

Hash on Both the Write and Search Sides

Naively "remove PII from logs" and you can no longer answer "let me look up Customer A's logs." But if you hash at write time and store that hash in the log, the search side can run the same hash function over the input and find the matching record. Plain-text email never touches either end.

Concretely:

Write side:

// Application code
logger.info("Subscription updated", {
  user: hashEmail(user.email), // → '7a3f9c2e0b1d' (HMAC-SHA256 12-char prefix)
  plan: "monthly",
});
// → Only the hashEmail result ends up in Loki

Search side (when you want to pull a specific customer's logs):

Here's the awkward part. "Pull up Customer A's logs" — the naive way to build it hands the raw email to the AI, which then passes it to an MCP tool to search. But that means handing plain-text PII to the AI (the model, and the vendor behind it). Guard the inside of Loki with hashes all you want; it leaks at the search input, one step earlier.

So in cortex the search tool takes a non-PII ID, resolves it to an email inside the MCP server, hashes it there, and returns only the hash. The email exists only inside the MCP server and never reaches the model:

// MCP tool resolve_email_hash (runs server-side)
// Input is an ID (non-PII). The email is never returned to the caller = the AI.
const email = await resolveEmailById(userId); // resolved from the DB, server-side
const hash = hashEmail(email, secret);        // same function, same key as the write side
// → the AI gets back only the hash, never the email

The AI takes that hash and searches Loki via Grafana MCP as {service_name="subscription"} |~ "${hash}". Both the write side and the search side run the same hashEmail with the same key, so logs from the same customer collapse to the same hash. Meanwhile:

Plain-text email never enters Loki
The query string Loki sees doesn't contain plain-text email either (only the hashed value reaches it)
And the AI (the model) never receives plain-text email either. All it touches is a non-PII ID and hashes that already live in Loki. The plain-text email never leaves the trust boundary of the MCP server.
Enumeration resistance comes from keeping the HMAC key outside the stack. Email is a low-entropy, enumerable input space, so a bare one-way hash (plain SHA-256, etc.) is breakable. The hash function is public, so once logs leak, an attacker just hashes a list of likely emails on their own machine and matches against the leaked values, no key required. HMAC folds a secret key into the hash computation itself, so an attacker who doesn't have the key can't even turn a candidate email into "the same shape as the leaked hash." They never get onto the brute-force field. Keep the key only at the write side and the search tool, never in Loki itself, and you get "a log leak alone doesn't expose the plaintext unless the key leaks too", one more condition an attacker has to satisfy
Truncating to a 12-char prefix (48 bits) means collisions are possible in theory, but negligible at customer-base scale. By the birthday problem, the 50% collision point sits around 20M records (≈ 2^24.5), and below that the expected collision count stays tiny. More to the point, a collision wouldn't leak plaintext anyway: this hash is a correlation key for identifying a customer's logs, not a security boundary, so the worst case is "another customer's logs occasionally land on the same hash", a degradation of correlation accuracy, not a disclosure

This reuses the property "same input → same hash" of hash functions in the form "the same function on both sides makes search work." The security / debug usability tradeoff compresses cleanly.

And of course, this is all just the app log layer. The BQ side is protected by Policy Tag-based column-level access control as its own layer (rows 1–2 of the table above). The whole thing is multi-layered.

What makes the "take an ID, resolve and hash inside" shape work is that plain-text email never crosses the trust boundary of the MCP server. The easy implementation (hand the AI a raw email, let the tool search) leaks the plaintext to the model at the search input, no matter how well you guard the inside of Loki. You could argue "the vendor's terms say it won't leave," but that's a dependency on terms, and it's weak under audit. Take an ID and hash inside, and you keep plaintext away from the model structurally, not contractually. When I said up top that PII protection has become "a trust-boundary redesign," this is the kind of design call I meant.

An aside: when I was working this out, I asked an AI for help, and it suggested building an admin screen where a human manually turns emails into hashes. That's one way to keep PII away from the model, sure, but it doesn't fit autonomous operation — a human has to step in before any investigation can start. cortex is built to run all the way through to "fixed before anyone notices" self-healing, so a solution that inserts a human isn't on the table. "Take an ID, hash inside the MCP server" came out of that constraint. What counts as an acceptable solution was, in the end, a design judgment on my side.

Integration Surface — "Humans = Web, AI = MCP" on the Same Backend

Three backends (Prometheus / BigQuery / Loki) now carry the observable data, and PII is handled. The next question is who queries them, and how. The common trap is to build "human dashboard aggregations" and "AI data feeds" separately. The moment you do:

Two implementations chasing the same question
Numbers drift between them
It becomes unclear which is canonical
Aggregations for AI and for humans update on different schedules

cortex's choice: share one observability backend; only the consumer-facing interface differs.

Human side: AI Operations Portal

There's an internal portal (codenamed PI Lab) that aggregates dashboards by monitoring target:

Claude Code usage (the cc-usage screen from Part 1)
MCP tool usage (by server / tool / user / team)
Infrastructure cost (Gemini / GCP / AWS / GitHub on one screen)
Alert state, deploy history, etc.

Here's what the MCP usage dashboard actually looks like:

Over the past 30 days, service-product-graph had 37,946 calls (with 7,106 errors), gws had 19,350, db-graph had 17,297 — and that's just the top. Which MCP is used how much, where the failures are showing up — all visible at a daily glance. (The "high error rate" some servers seem to have is partly typed errors counted in — expected rejections like "permission denied" — so the interpretation needs care.) The "annotation graph MCP, ~50,000 calls / 73 users" figure from the previous series came from this same view.

These pages on the React side pull from BQ / Prometheus / Loki through an internal API. The aggregation logic lives at the API layer.

AI side: MCP

When AI agents need the same data, they go through purpose-specific MCPs:

Grafana MCP — LogQL / PromQL queries against Loki / Mimir / Prometheus / Tempo. Natural-language questions like "What time window had the most errors on Service X last week?" are the agent's job to translate into LogQL / PromQL before they go over MCP
BQ MCP (via cortex-product-graph) — SQL queries against claude_usage.claude_usage / cortex.mcp_tool_calls

The design pivot: the human dashboard and the AI MCP share the same backend. No separate "AI aggregation table" and "human aggregation table." Build the observability backend once, then provide a consumer-specific interface layer (web dashboard / MCP) on top.

In DDD terms, MCP and the web dashboard are both just presentation layers — different I/O channels into the same domain (the observability backend). Treating MCP as "something special" leads to duplicate implementations; treating it as one presentation layer form keeps the design clean.

That's exactly why "the observability stack is visible to AI" actually holds. Build the backend, but without an AI-facing presentation layer (= MCP), AI can't query it. MCP is the piece that makes "hand it to AI" actually work.

The Real Driver of Self-Healing

The layer that keeps the observability stack from being "just a screen to look at" is Self-Healing. I covered the full picture in AI Harness Series Part 4, so I'll skip the details here, but from the observability side, the start and end of the chain are clear:

The flow:

Detect — Production alert / CI failure fires a Loki LogQL alert
Deliver — POST to event-relay (the internal webhook hub)
Launch — auto-review bot starts up (= an agent backed by Claude Code)
Gather context — The bot pulls full logs via Grafana MCP, traces related PR / commit / code via Product Graph MCP
Propose — File a fix PR
Verify — If CI passes, the bot auto-merges; if not, another bot reviews

So the starting point of Self-Healing is whether the observability stack can hand "what broke" to AI in the right shape. If errors aren't recognized / stacktraces aren't preserved / related code (PR / commit / graph) isn't reachable — any of those missing and the chain stops cold. (The specific failure modes are in the next section.) Put another way:

The quality of observability is the ceiling for AI autonomous operation.

That's the central claim of Part 2. Reframe the observability stack as "input that drives AI," not "monitoring infrastructure," and the priorities of your design decisions shift accordingly.

What's Still Open — Defining "What Counts as an Error" and the Stacktrace Design

The biggest remaining issue, honest version.

You can polish the observability stack to a mirror finish, but if the design of what counts as an error and whether the stacktrace survives falls apart, all of it is wasted. I touched on this earlier in AI Harness Series Part 2 in the context of cortex's internal knowledge graph, and it shows up on the observability side too.

Concretely, here are the failure modes:

try ~ catch swallows the error without logging → nothing reaches the observability stack
catch does log, but at console.log-equivalent info level → not recognized as an error
Error gets emitted, but only error.message is written; stacktrace is dropped → AI can't trace back to the original code
An async error goes unhandled and the process falls over

These are all problems at the code that creates the observability entry point, not at the observability stack itself. No matter how polished the stack is, if the faucet at the entry point is broken, nothing flows out.

What's in place today is three layers, none of them complete:

lint (static) — The no-silent-catch rule blocks empty catches and .catch(() => null)-style swallows. But once there's any function call inside the catch, lint is satisfied — so patterns like "demote to logger.info(err.message)" or "log only error.message and drop the stacktrace" slip through statically
Guideline document — Rules like "use serializeError(error) to store stacktrace as a structured field" and "dropping stack via logger.error(err.message) is a Major violation" are written down in the internal guidelines. But static checking can't enforce these; they rely on human / AI review
AI auto-review — The PR auto-review bot does look at test coverage including "are error cases being tested," but it has no observability-specific checklist, so it can't systematically catch stacktrace design quality

In other words: "There's a guideline, lint catches some, AI review catches some, but it's not airtight" is the honest description. The real gap is that at the moment new code is being written, there isn't a harness that proactively suggests / completes "this should be treated as an error, this should keep its stacktrace." Auto-review picks things up at PR time, but a proactive harness for the observability entry-point design itself isn't built yet.

"Observability stack: done. Observability target design: still on humans." That's the honest picture. Closing that gap with a harness is the next step.

Closing — Static Edition + Dynamic Edition Are Lined Up; Merging Them Is the Next Series

The code-graph series was about reshaping a static analysis graph so AI could query it — handing the structure of code as fact. This two-part series was about handing what's happening in production right now, also as fact.

	Shape	What's Handed Over
Static edition (code-graph + db-graph + annotation graph)	3-graph parallel + SAME_ENTITY	Code and meaning
Dynamic edition (Part 1 + this post)	Prometheus / BQ / Loki + MCP	Production behavior and cost

The honest part: these two still sit side by side, not joined. For cortex's stated principle of "don't let AI infer — hand it facts" to truly reach completion, the next step is to pour dynamic data into the static graph and merge them. This is the exact same gap I flagged as the "absence of dynamic analysis" open issue at the end of code-graph Part 2: putting "how often is this edge actually used in production?" on the static graph's nodes. That's when "hand it as fact" reaches its final form.

Layer Self-Healing on top of static + dynamic and you get "AI autonomously operates," which works today. But merging the two editions into one graph is still ahead — that's the next series.

And one more time, observability target design (what counts as an error, whether stacktrace survives) is what really sets the ceiling. Harness-ifying that is the next homework item.

Thanks for reading this far.

Observability Design for the AI Era — Application / Infrastructure / CI / LLM, Each in Its Own Shape

Ryosuke Tsuji — Mon, 06 Jul 2026 23:44:23 +0000

AI assistance disclosure: This article was drafted with the help of Claude. All technical content, design decisions, code references, and screenshots reflect production systems I designed and operate at airCloset; the prose was revised by me prior to publication.

Hi, I'm Ryan, CTO at airCloset.

In the previous series, code-graph deep dive (Part 2), I wrote about making a 46-repo codebase semantically searchable for AI. The final issue I left open in that piece was the absence of dynamic analysis:

What lives on the graph is the fact that "this edge exists statically." How often that edge actually gets used in production isn't recorded.

A graph that gives you static facts is one thing. Telling AI what's actually happening in production right now is a separate problem. So the same shaping discipline I applied to the static graph needs to apply to the observability stack too.

This post is the first half of that story. I split it into two: Part 1 (this post) covers how I shape four different monitoring surfaces (application / infrastructure / CI / LLM). Part 2 covers PII handling, the integration surface, and Self-Healing.

What Does "Observable to AI" Even Mean?

The biggest lesson from the code-graph series was: the data has to be shaped before AI can consume it. Throwing 46 repositories of source at a model blows past the context window and invites hallucination. So we shaped it — static analysis into a graph, boundary nodes given meaning, SAME_ENTITY joins between graphs — and only then handed it over.

The observability stack has the exact same problem. Throw raw production logs at AI and you get:

Sheer log volume that drowns the context window
No way for the model to tell errors from noise
Metrics, logs, and traces that don't link to each other
Questions like "what are we spending right now" that raw logs don't answer at all

In other words, logs have to be reshaped before AI can use them. Same problem, different domain.

The catch is that the right shape depends on what you want AI to answer. At cortex (the internal AI platform), I split the monitoring surface into four axes and let each one settle into its own form:

Note: "cortex" here refers to airCloset's internal AI platform codename. Unrelated to Snowflake Cortex, Palo Alto Networks Cortex, etc.

Monitoring target	What you want AI to answer	Shape
Application	"What's happening in production right now?" (exploration)	log + trace
Infrastructure	"Do we have enough resources? Anything down?" (time series)	metric
CI	"What broke? Since when?" (alert + history)	log + alert
LLM	"How much are we spending? Who's using how much?" (real-time + structured aggregation)	metric + structured records

"Just push everything through OTel and dump it all in Loki" is an option. But the moment you do, you're asking one backend to answer wildly different kinds of questions — real-time "what's spending right now" alongside "monthly cost broken down by team via SQL" — and one of them is going to suffer. Splitting by purpose is the choice I made.

Let me walk through each of the four axes. Application and infrastructure are the foundation, so I'll keep those brief. CI and LLM are where the AI-era design judgments actually surface, so I'll dig into those.

Application — OTel + Loki + Tempo, the Standard Stack

The foundation is unremarkable. Every cortex application is instrumented with OpenTelemetry, with traces going to Tempo, logs to Loki, and metrics to Mimir — the standard Grafana Cloud setup.

There's no special trick here. What matters is the discipline: every app emits logs and traces in the same shape. That uniformity is what lets AI later run something like {service_name="<service>"} |~ "error" through MCP and investigate across services.

I covered the actual instrumentation in AI Harness Series Part 4 (Self-Healing), so I'll leave the details there. The point worth repeating is: a standard OTel stack, properly laid down, is the precondition for everything AI-driven that comes later.

Infrastructure — Cloud Run / BigQuery / Pub/Sub Metrics, All Into Mimir

cortex runs on GCP and stitches together Cloud Run, Cloud Run Jobs, BigQuery, Pub/Sub, Cloud Tasks, and the usual suspects. Each GCP resource's metrics (CPU, memory, execution count, latency, queue dwell time, etc.) flow through Cloud Monitoring into Mimir.

Nothing special here either — just standard GCP metrics, all gathered into one Mimir instance. But that "one place" property pays off later: AI can answer "which service used the most CPU last week?" or "is there a worker with a clogged queue?" naturally, because everything is queryable from a single store. MCP picks it up from there.

That's it for the foundation. Standard observability stacks are well-documented elsewhere; go read Grafana's and OpenTelemetry's docs if you want the details.

The interesting AI-era design judgments are in the next two axes — CI and LLM.

CI — Ship Logs to Loki via Post-Hoc Pull, Not Webhook Push

cortex runs CI on GitHub Actions, and I ship every CI log into Grafana Loki.

"Why? GitHub Actions has a perfectly good UI for that" is a reasonable question. The reasons are concrete:

Having AI hit the GitHub Actions API on every investigation is slow and auth-heavy. Ingesting into Loki once means AI can query it ad-hoc
One Loki instance holds CI logs and application logs together, so you can cross-query them
LogQL alerts turn CI failure into a structured signal
AI can ask "any tests that have been broken since last week?" in natural language

But the shipping mechanism is unusual. The choice cortex made:

Don't push logs from inside the CI run. After the run finishes, pull them from the GitHub API.

Concretely:

When the Test job ends, a workflow_run event fires
A separate workflow dedicated to log shipping triggers
That workflow pulls logs from the GitHub API (/repos/.../actions/jobs/.../logs)
Ships them to Grafana Cloud as structured JSON (job / status / ref / pr / commit / output, etc.) via OTLP /v1/logs

Filter on {service_name="ci", ref="main", status="failure"} and you get just the main-branch CI failures, cleanly.

Why pull instead of push:

CI execution and observability decouple. If shipping fails, the test run is unaffected. You can also retry / replay shipping independently
No path for PR code to touch the API key. The shipping workflow runs in the default-branch context and uses base-repo secrets, not whatever a fork PR brought. The test workflow itself never touches the Grafana API key — that's a structural guarantee, not a "we trust it won't leak"
Shipping failure becomes observable. If shipping lives inside CI, a shipping bug means the observability stack goes silent — and you don't notice. Split them, and the shipping workflow's success / failure is itself something you can alert on

The moment a main-branch failure shows up, a LogQL alert fires and Slack gets pinged. That's the trigger for Self-Healing, which I cover in Part 2.

LLM — Gemini and Claude Code, Two Different Shapes

The last axis is LLM observability. cortex uses both Gemini API and Claude Code (Anthropic's official CLI) heavily, and since both cost money, I want visibility into how they're used (though the billing models differ — Gemini is pay-per-use, Claude Code is a subscription, and that difference matters later). The reason I shape them differently isn't really about "what kind of question" — it's about where you can instrument — the instrumentation locus:

Gemini — I own the calling code, so I can wrap every call with a common helper and emit metrics inline. Prometheus is the natural fit.
Claude Code — It's an external CLI; I can't wrap its calls from the inside. Usage shows up as records after the fact. A structured store (BigQuery) is the natural fit.

The "real-time vs SQL aggregation" framing of the question is a consequence of where you can instrument, not the cause. With that clarified, here's how each one plays out.

Gemini — Prometheus, Cost Visible in Real Time via Client-Side Estimation

cortex uses Gemini everywhere: db-graph table description generation, code-graph field type inference, general context generation. What I want to see is what's expensive right now, with no lag. If a runaway prompt or batch job kicks off, I don't want to wait until tomorrow's billing report.

So every Gemini call goes through a common wrapper (traceGeminiCall) that emits four metrics per call:

gemini.tokens.total — cumulative tokens (labels: model / service / type=prompt|completion)
gemini.requests.total — request count (labels: model / service / status)
gemini.request.duration — latency histogram
gemini.cost.usd — estimated cost (labels: model / service)

The design choice that splits opinions is: who computes the cost? Two options:

A. Pull from Google Cloud Billing API after the fact — accurate, but billing lags by hours to a day, and there's no per-task cost granularity
B. Compute client-side from token counts × a price table — instant, with per-task granularity attached by you, but the price table needs upkeep

I picked B. The price table lives in a constant called GEMINI_PRICING and gets manually bumped whenever Google moves prices. Just gemini-3-flash / gemini-3-pro with input/output unit prices each. Nothing fancy.

The real reason for B is real-time visibility:

Billing lags by hours to a day. A runaway prompt or batch bleeds cost all night before tomorrow's billing surfaces it. Computing client-side, tokens times price right after the call, lets you see "what's expensive right now" at the service level (code-graph / gcs-transformer / db-dictionary and so on, app/pipeline-grained) within minutes. That's a speed billing can never match.
Price table maintenance is light (Google doesn't change prices often), so the upkeep cost is trivial.
Cloud Billing API authentication, fetching, normalization, fan-out is its own pipeline of weight you'd have to maintain.

Then I emit gemini_cost_usd_USD_total as a cumulative Prometheus counter (the doubled usd_USD comes from OTel meter name gemini.cost.usd combined with the unit USD during Prometheus exporter conversion) and PromQL can answer "how much did we spend in the last hour" directly: sum(increase(gemini_cost_usd_USD_total[1h])). Alert fires at $1/hour, info severity, into Slack. In practice this is less an aggregation surface I query after the fact and more a tripwire: the threshold-crossing Slack alert is how a runaway gets caught.

One line worth drawing here: the gemini.cost.usd counter carries exactly two labels, model and service, and service is coarse (a bounded set of app/pipeline names). Try to push call-site-level identity onto the label, "what did that one prompt cost," and the label combinations blow up across many repos and inference types until the time-series DB can't absorb them. So the Prometheus side stays a tripwire: coarse service granularity, immediate alerting, nothing finer. The per-prompt attribution question, "which prompt burned the most this week," isn't a time-series question at all, it's a SQL one. That wants the token records in BigQuery with as much call-site context as you care to attach, which is the same reason Claude Code goes to BQ below. "I can instrument this call" and "this should live as a time series" are separate claims, and the fine-grained aggregation is where Gemini and Claude Code converge back onto the same backend.

Prometheus is what you want when the question is "right now."

Claude Code — Send to BigQuery, Built for SQL Aggregation

Every developer at the company uses Claude Code. But the economics differ from Gemini: it's a subscription, so token usage doesn't translate straight into a dollar figure. What I'm after here is less the cost itself and more the usage picture — who's using how much, how many tokens per repo, how well the cache is landing — so I can turn it into better usage.

The question that split opinion: "Should Claude Code usage go to Loki too?"

The answer: No, into BigQuery.

Why? Because Claude Code usage is, fundamentally, a structured ledger:

email — the user
repository — which repo it was used in
timestamp — when
input_tokens / output_tokens
cache_creation_input_tokens / cache_read_input_tokens — prompt-cache effectiveness included

And the questions you want to ask look like:

"Last week, what's the cumulative spend for Team A members?"
"How much did edits on Repo X cost over the past month?"
"What's the prompt-cache hit ratio difference between teams?"

All of these are SQL aggregation questions. LogQL aggregation and joins on Loki are painful. BigQuery, with a DAY partition and email as the primary key, just writes naturally.

So the Claude Code → BigQuery pipeline runs in four stages:

Emit — A bundled analyzer in Claude Code POSTs UsageInput (token info only, no email) to an internal endpoint
Auth proxy — A Cloudflare Edge Router worker validates CORTEX_API_KEY and stamps the user's email onto the request as X-Cortex-User-Email
Ingest — A Cloud Run API dedupes and publishes to Pub/Sub
Persist — A Cloud Run worker pulls from Pub/Sub, validates the schema, and streaming-inserts to BigQuery

Two structural points worth calling out:

Identity authority lives at the Edge Router. User identity is resolved exactly once, there. The emit side (Claude Code) never holds the email. This shuts down whole classes of client-side id-spoofing and social-engineering paths structurally
Pub/Sub gives async decoupling. Ingest and worker are separate, so backpressure on the worker doesn't affect ingest response times. On failure, Pub/Sub DLQ retries up to five times

What sits in BigQuery is visible day-by-day through the internal portal I'll cover in Part 2. Here's what it actually looks like:

The numbers are interesting enough to mention briefly: in the last 30 days, 78.0B tokens / 384K messages / 47 users / 79 repositories. The one to focus on is Cache Read Input at 75.1B (96% of total) — prompt-cache is dramatically effective. On a subscription this doesn't show up as a dollar figure, but cache read tokens carry roughly 1/10 the effective input rate, so if you were paying per-token API pricing for the same usage, this works out to roughly 7× more efficient at the blended input level versus the cache-less counterfactual. Being able to see usage efficiency as a concrete number like this is the point of the visualization; "aggregation-shaped backend matched to the question" is the design choice that makes this kind of metric fall out of SQL naturally and show up daily. Doing the same thing in LogQL would be a battle.

As a side note: MCP tool-call logs end up in BigQuery too (cortex.mcp_tool_calls), but via a simpler path — each MCP server just writes records directly, no OTel in the loop. The "annotation graph MCP used ~50,000 times by ~73 people" figure from the previous series came from this exact table.

The core point of this layer is: don't dogmatically force everything through OTel — match the tool to the qualitative nature of the aggregation.

To Be Continued

That's the four axes (application / infrastructure / CI / LLM) and the design judgments behind each. The write-side of the observability stack is wrapped up.

But shaping the write side isn't the whole story. The moment production data flows through the stack, PII becomes a constraint you have to design around. And the data has to actually be consumable by AI through MCP, with a thoughtful integration surface for both humans (web dashboards) and AI (MCP). Connect all of that, and the real driver of Self-Healing comes into focus from the observability side. That's the Part 2 story.

Thanks for reading. Part 2, "Observability Design for the AI Era — Reconciling PII Protection With AI Searchability, and Driving Self-Healing," is out now. Read on.

Making the Context Across 46 Repositories Semantically Searchable for AI

Ryosuke Tsuji — Mon, 29 Jun 2026 23:50:05 +0000

AI assistance disclosure: This article was drafted with the help of Claude. All technical content, design decisions, code references, and screenshots reflect production systems I designed and operate at airCloset; the prose was revised by me prior to publication.

Hi, I'm Ryan, CTO at airCloset.

In Part 1, I wrote about unifying 46 repositories of production code into a single knowledge graph via static analysis. The graph itself got built, but I closed the post with four open issues: no semantic search, node explosion, having to open the file to actually know what a function does, and the cost of writing a new parser every time a new boundary pattern showed up.

This Part 2 is about how I solved the first one — the entry-point problem (no semantic search). The other three are left exactly as Part 1 described them — I'll come back to them at the end, together with the new issues that surfaced once the entry-point problem was out of the way.

The reason to start with the entry-point problem is simple: if the graph exists but the only way to reach it is grep, the model ends up inferring anyway. The whole point — "give the model verified facts, not inference" — falls apart. So the entry-point problem had to be solved before the others.

The Hint Was in db-graph

Months earlier, I'd already solved the same structural problem in a different domain — the db-graph project.

Internally, we had a large number of DB tables spread across many services, and no single person had the full picture. Different people knew different pieces well, but the whole map didn't fit in anyone's head. So I built db-graph: extract schemas statically from ORM definitions, generate per-table descriptions with Gemini, embed them as 768-dimensional vectors in the graph, and make the whole thing semantically searchable in natural language.

At the time of that article it covered 991 tables. Today it spans 21 schemas / 1,133 tables / 10,815 columns, and finding data in natural language without knowing table names is just how people work now.

The pattern that proved out there:

Static-analysis graph + AI-generated context = natural-language semantic search works.

Bringing the Same Pattern to code-graph

If it worked for db-graph, it should work for code-graph. The moment that thought landed, I noticed something:

code-graph already contains "DB table nodes" as boundary nodes — they're one of the boundary node types I covered in Part 1.

So if I just join code-graph and db-graph, code-graph automatically inherits db-graph's semantic context. Without writing a single annotation, the existing assets alone make the graph meaningfully richer.

That's where the idea of "joining graphs" first came up — not treating each graph as its own island, but designing the joins between them.

But API / Event / Page Still Need Meaning — and Annotating Every Function Is Off the Table

Joining db-graph took care of DB context. But the remaining boundaries (API / Event) and the graph's entry-point type (Page) still need meaning attached. Static analysis alone can't pull intent out of those, so context has to come from somewhere else.

The choice was clear: write the intent directly into the code via annotations (the same approach used by cortex's internal knowledge graph, which I covered in AI Harness Series, Part 2).

The catch: you can't annotate all the functions across 46 repos. There must be tens of thousands of them. Asking established teams running an existing production codebase to retroactively annotate everything is just not realistic.

But here's the second realization:

What matters is just the boundary nodes. So if I only annotate around the boundaries, that's enough.

When an AI agent asks "what breaks if I change this code" or "what other repos call this API," what it needs isn't a per-function logic explanation. It needs boundary intent — what is this screen for, what does this API return, what milestone in the business does this Event mark.

= Minimum annotations, maximum meaning. That became the heart of the design.

Designing the annotation graph

Putting it together (internally we call this annotation graph service-product-graph, or SPG):

Three graphs sit as peers, joined by SAME_ENTITY edges. There's no hierarchy — you can start from any graph and reach the others.

code-graph (structure) — functions / classes / boundary nodes from static analysis (46 repos)
db-graph (DB context) — 1,133 tables, semantically described
annotation graph (intent) — @graph-* tags written only around boundaries

The entry point for AI agents is a single MCP server that traverses all three graphs. AI agents never hit db-graph directly — the annotation graph's MCP server proxies db-graph calls on their behalf.

The annotation graph has 7 node types: Page / Section / Dialog / Field / Action / Api / Task. The early version was screen-focused and called screen-graph, but once it grew to cover backend Api / Task, it was renamed to service-product-graph.

An Annotation Example

Here's what an annotation looks like (fictional, but close in shape to the real ones):

/**
 * @graph-page /home
 * @graph-business Main screen. Members can see what they're currently renting, buy items, and initiate returns.
 * @graph-label Home Screen
 * @graph-has-section banners, wearing-items, wearing-return, delivery-status
 * @graph-has-dialog buying-modal, return-modal
 * @graph-navigates-to /return-procedure, /checkout, /my-karte
 * @graph-calls GET /api/v1/wearing
 * @graph-reads admin_delivery_orders, admin_rental_items
 * @graph-flow styling-loop
 * @graph-status monthly-member
 */

Two things matter here:

@graph-business carries the intent text (in our actual codebase it's written in Japanese). This is exactly what gets vectorized — it's the substance of semantic search.
@graph-flow / @graph-status carry where this sits in the member lifecycle (free signup → monthly subscription → styling loop → cancellation, etc.) and which member segment it's for. They add a second dimension of meaning: "this screen shows up inside the styling loop for monthly members."

There's also @graph-case (the conditional pattern tag that test cases derive from), but that's for another time.

Running Annotations Without Interfering With the Day-to-Day Dev Workflow

This is where it gets practical.

Once I committed to building annotation graph, here were the constraints:

Engineers run normal product dev with human code review
AI review isn't wired up on every repo yet — cortex's fully automated review (covered in AI Harness Series, Part 6) only works inside the cortex monorepo
Asking humans to review annotations on top of their normal review load is a non-starter
Even a split like "humans review the code, the AI reviews the annotations" inside the same PR mixes two review streams together and just confuses everyone

In other words: don't mix humans and AI inside the same PR.

The solution was to physically separate annotations onto their own branch.

Leave main untouched; engineers' normal flow stays exactly as it was
Stand up a separate annotation branch that's the AI's exclusive territory
When main changes, a webhook fires
The annotation branch handles generating the diff annotations and reviewing them — the AI does both, end-to-end
From the engineer's side, they only touch main and don't even need to know annotations exist

This is the "every line of code passes through an AI gate" ideal from AI Harness Series, Part 6, adapted to the constraints of an existing organization. cortex (the internal AI platform) is a monorepo I assemble from scratch, so "every commit passes the AI gate" actually holds there. For the 46-repo production system, that precondition doesn't hold. So instead of giving up on the ideal, I split it: engineers' workflow on one branch, AI's annotation workflow on another, both running in parallel.

Protecting Cross-Graph Consistency With an SLO

Just running the annotation pipeline doesn't guarantee the quality of the joins between the three graphs (code-graph / db-graph / annotation graph). So there's a set of SLOs that automatically check the consistency across the entire graph.

The main rules:

API chain connectivity — at least 95% of HANDLES_API handlers must have downstream function calls (= no handlers that receive an API and then do nothing)
DB access completeness — at least 80% of DB read/write edges must be joined to db-graph column nodes (= code-graph's DB boundaries are connected to db-graph's meaning)
Event field resolution — at least 70% of Event edges must carry field-level information
No ambiguous edges — name-resolution-ambiguous edges must be 0 (severity: error)

These are really just a naive question — "shouldn't the boundaries connect to each other?" — turned into an SLO. If anything drops below threshold, an alert fires, and the trustworthiness of the whole graph gets defended every day.

The daily boundary-analysis cron from Part 1 (5% connection-rate drop = alert) was code-graph-only. This is a cross-graph SLO — it guards the joins between graphs themselves. Add a parser to one repo, write a new annotation, change a schema — whatever happens, by the next morning a quality drop in any join becomes visible.

Joining the Static Graph and the Annotation Graph via SAME_ENTITY Bridges

I've been writing "join" casually, but the actual joining wasn't that straightforward.

Static-analysis API / Page / Task nodes and annotation graph API / Page / Task nodes are created as separate nodes. They mean the same thing, but their names / paths / identifiers don't match by themselves — there's nothing automatic about lining them up.

To connect them, we generate a separate edge type called SAME_ENTITY. There are three bridges:

API bridge — API path normalization with a 4-stage fallback
1. Per-repo prefix conversion (e.g., normalize console-side /console/api/ to /api/)
2. Version stripping (/v1.x/ → /)
3. Parameter normalization (unify /:id, /{id} to /:dynamic)
4. Exact match → tolerate trailing ? → strip trailing :dynamic? → finally fall back to a dynamic-dispatch boundary :dynamic, loosening progressively
Page bridge — 6 strategies applied in priority order (URL direct match, component path match, itemId match, PascalCase normalization match, parent-directory linking, strip dynamic segments and match parent URL)
Task bridge — 8 per-repo patterns

There was also one operational footgun. The first implementation used INSERT NOT EXISTS to avoid duplicates. But BigQuery's streaming-buffer visibility lag let duplicates slip in — in one repo the edges doubled from 106 to 214 overnight. We fixed it by rewriting to MERGE INTO to make the operation idempotent.

The Result: Entering the Graph from "the subscription-fee calculation"

With all of this in place, the entry-point problem from the end of Part 1 was finally solved:

"the subscription-fee calculation for members seems off"

Throw this natural-language query at annotation graph and vector search returns the related nodes (Page / Api / Function / DB table) as facts. From there, SAME_ENTITY takes you over to code-graph functions, including callers and callees in other repos. From the DB boundaries in code-graph, you can cross into db-graph and pull the relevant columns.

The entry point can be anywhere — "what calls this table?" starts from db-graph, "what's the blast radius of this function?" starts from code-graph, both walk the same connected network. From a single natural-language query, or from a specific node, you can now traverse all three graphs and get every relevant piece of code plus every relevant DB schema.

The Part 1 lament — "the graph is there but the entry point is missing" — could finally be put to bed.

Real Usage Numbers

From 2026-04-16 (first production deployment) to the time of writing — about 2.5 months — the annotation graph's MCP server has handled ~50,000 calls from ~73 users. The breakdown:

Engineers (PI Division + QA + relevant engineering teams) — ~47,000 calls / 51 users
Non-engineers (stylists / customer support / mall operations / executives / administration) — ~2,800 calls / 21 users

The interesting line is the second one. "Search the codebase in natural language" is usually an engineer's tool — but once the entry-point problem was solved, people outside engineering started using it too, asking things like "how does this feature actually work?" or "what's in this DB?" in their own words.

This is adjacent to the "non-engineers writing specs with AI" trend I covered in AI Harness Series, Part 5 — a graph that can be queried by meaning starts to matter org-wide. Call volume is overwhelmingly dominated by engineers, of course. The interesting thing is the range of job roles starting to pick it up. That's the real impact of solving the entry-point problem.

MCP as the Single Front Door

The MCP server is the cross-graph entry point. It exposes six tools — service search / service detail / API detail / data-flow tracing / impact-radius tracing / business-rule full-text search — and that's the only entry point AI agents ever touch.

One design choice worth calling out: AI agents never talk to db-graph directly. The annotation graph's MCP proxies db-graph calls. From the agent's side, the mental model stays simple: "ask one MCP and get everything back."

That makes the full chain — "Screen → API → Code → DB → Column" — traversable in a single MCP tool call.

April–May Timeline of Trial and Error

Same approach as Part 1 (pulling commits from Jan–Mar). For Part 2, the key commits are from April–May.

April: Expansion and the First Bridges

2026-04-14 ─ refactor(graph): rename screen-graph to service-product-graph — declaration that the scope expands from screen-only to whole-service
2026-04-15 ─ feat(graph): add Api and Task node types to service-product-graph parser — Api / Task node types added
2026-04-15 ─ feat(mcp): add cross-graph tools to service-product-graph MCP — cross-graph tools land (the single front door across all three graphs)
2026-04-15 ─ feat(graph): add SAME_ENTITY bridge edges between service-product-graph and code-graph — first bridges
2026-04-18 ─ feat(graph): resolve Redis keys to code-graph boundary nodes — boundary resolution through Redis
2026-04-19 ─ feat(service-product-graph): add EventBridge EMITS_TO support + SAME_ENTITY bridge
2026-04-20 ─ feat(code-graph, service-product-graph): improve SAME_ENTITY boundary bridge coverage — 4-stage fallback locked in
2026-04-21 ─ feat(auto-review): SPG annotation auto-maintenance pipeline — AI auto-maintenance pipeline (= what Part 1 hinted at with "humans alone can't, but AI can")
2026-04-22 ─ feat(service-product-graph): add Task SAME_ENTITY bridge to code-graph — all three bridges in place

May: Stabilizing and Expanding

2026-05-01 ─ Annotation generation moves from local execution to a Cloud Run Job; operation stabilizes
2026-05-05 ─ feat(spg): add mall repos to SPG indexing — mall repos indexed
2026-05-06 ─ feat(spg): add Go-aware parser — Go support
2026-05-06 to 08 ─ Page bridge strategies expanded to six, connection rate hits 100%

What This Timeline Says

April 15 was the day "expansion + cross-graph tools + bridges" landed in close succession. Over the next week, "Redis / EventBridge / Task bridges / annotation auto-maintenance" stacked up week over week.

In particular, the annotation auto-maintenance pipeline on April 21 is where the "humans alone can't do this, but AI can" promise from Part 1 got cashed in. From that point on, annotation shifted from "humans grind through writing them" to "design the whole operation assuming AI writes them."

What Still Isn't Solved

Solving the entry-point problem didn't make everything clean. A few issues remain.

1. Maintaining Annotation Coverage

The frontend side is annotated heavily. Backend / Go / batch are still thin. Some nodes will always be missing annotations — that's structural, and you can't drive it to zero. It's an ongoing operational issue.

2. Bridge Mis-Joins Aren't Fully Eliminated Structurally

The Page bridge in particular has cases where multiple annotation Pages map to the same boundary — that's structural and unavoidable. Adding more strategies got coverage to 100%, but guaranteeing "every join is correct" 100% is hard.

3. No Dynamic Analysis

The graph only carries the fact that "this edge exists statically." How often that edge actually gets used in production isn't recorded. Piping production execution counts back into the static graph and surfacing dead-code edges as a separate signal — that's still untouched.

4. Onboarding Cost When a New Repo Joins Production

Every time a new repo enters production, the bridge normalization rules and per-repo patterns need adjusting. This is the annotation-graph-side version of Part 1's fourth issue (the cost of adding a new parser for every new boundary pattern).

Closing: Not "Thrown Away," but "Evolved"

In Part 1's closing note, I touched on the fact that the cortex side (the internal AI platform) bailed out of the code-graph approach early and bet on an annotation-based knowledge graph instead. The bail-out was fast enough that calling it "thrown away" wouldn't be wrong — but looking back across this whole series, the more accurate word is "evolved."

What it evolved into, in the end, is three graphs joined as peers:

code-graph (structure)
db-graph (DB context)
annotation graph (boundary intent)

Joined by SAME_ENTITY, served to the agent through MCP. The thing static analysis alone couldn't deliver — querying by meaning — became workable by reusing the db-graph success pattern and adding minimal annotations only at the boundaries.

And one more framing: paired with the AI Harness Series, Parts 1–6, this series sits as:

AI Harness series — how to live with AI when you're assembling the system from scratch yourself
code-graph-deep-dive series (Part 1 + Part 2) — how to live with AI inside an existing organization's running production system

= the same philosophy (design without trusting AI), implemented under two different sets of constraints.

Thanks for reading this far.

Got the Top 7 Badge — honestly thrilled 🙌

Ryosuke Tsuji — Wed, 24 Jun 2026 00:21:42 +0000

Cyberpunk cat RPGs and robot personalities

Jess Lee for The DEV Team

Jun 23

Building One Knowledge Graph Across 46 Repositories With Static Analysis

Ryosuke Tsuji — Mon, 22 Jun 2026 23:54:01 +0000

AI assistance disclosure: This article was drafted with the help of Claude. All technical content, design decisions, code references, and screenshots reflect production systems I designed and operate at airCloset; the prose was revised by me prior to publication.

Hi, I'm Ryan, CTO at airCloset.

This post is about unifying a production codebase spanning 46 repositories across multiple services into one knowledge graph, using static analysis.

Internally we call it code-graph, and I built it between January and March of this year.

Three things I want to write down:

Why "just letting AI read the code" isn't enough, and why I had to chase down the connections that cross repository boundaries
How I extracted boundaries across 46 repos and a zoo of frameworks (jQuery / AngularJS / Express / NestJS / TypeORM / Redux Axios ...)
What 3 months of trial and error solved, and what it didn't

This is Part 1, covering the construction of code-graph itself, the painful parts, and the issues that remained. Part 2 is about service-product-graph (SPG) — a layer I built on top of code-graph to compensate for what static analysis couldn't do alone.

What Was This For?

A long-running production codebase usually looks something like this:

Multiple services and multiple teams touching it
Each era's framework still alive and mixed in
Dependencies via API, DB, and Event are tangled — not clean 1:1 front-to-back relationships:
- The same API gets called from multiple repositories (= n:1 callers)
- The same DB table is written to and read from across multiple repositories (= n:n)
- For Events, just looking at the emit side doesn't tell you how completely the subscribe side is covered — it's practically untraceable

The starting point was wanting to ask AI: "show me the blast radius," "tell me what breaks if I change this," — across this entire codebase.

The naive answer is: "just hand all 46 repositories worth of code to AI and let it analyze."

But that doesn't work, for two reasons:

Context window: 46 repositories × years of accumulated code is just not a size you can hand to an AI in one shot
Hallucination: even if you could, "read everything and extract the relationships" is an inference task. It misses things, it makes mistakes. That's not usable for impact analysis on a production system

So the first idea I landed on was: build a knowledge graph externally, via static analysis. That's the starting point of code-graph.

Scale: 46 Repositories

The target splits into two graphs:

air-closet graph (37 repos): a graph that spans multiple services like airCloset, Men's, WMS, and more
mall graph (9 repos): airCloset Mall and related

So 46 repositories in total.

The thing to notice is that this isn't "one service with 37 repos." It's a collection of multiple services that adds up to that scale. Making the dependencies that cross service boundaries visible as cross-repo edges is exactly what the boundary nodes discussion below is about.

Why Boundary Nodes Matter

This is the heart of the article.

When you want AI to understand code, getting it to "read what's in front of it, plus what's next to it" is honestly not hard. grep, open the file, hand it to the model — that works fine.

For a small codebase, that's enough. But at scale, you hit the context window and hallucination problems mentioned above. I suspect most readers can relate.

One way to improve this is to statically analyze the codebase, convert it into a knowledge graph, and serve it to AI through MCP. That's the approach.

The first step was static analysis with tree-sitter (an OSS library that parses source code into syntax trees — it supports a lot of languages and is what VS Code and similar editors use for syntax highlighting; I genuinely recommend it if you want to build something in this space). It's a great tool, but on its own it doesn't solve everything.

What it doesn't solve is tracing relationships that cross boundaries — APIs, databases, and so on. tree-sitter can extract the relationships between variables, functions, and other in-language constructs. But it can't extract those boundaries.

The thing that humans and AI alike get stuck on, in practice, is exactly that — code connections that cross boundaries:

The same API is being called from another repo you weren't looking at
- The frontend in repo A and the nightly batch in repo C might both hit /api/v1/users/me
- Looking at just one of the repos, AI has no way of knowing
The same DB table is being read or written by some batch process you don't know about
- When you're modifying service-side code, some batch in a different location might be reading and writing the same table
- Misjudge the blast radius and you get data inconsistency
The subscribers for this event might not be fully accounted for
- With distributed pub/sub, looking only at the emit side doesn't let you cover the subscribe side
- Something runs somewhere you don't know about

In short: getting AI to understand the code on the other side of a boundary, without hallucinating. That's the goal.

If you have boundary nodes, AI can answer "this API is also called from repo X" as a fact. Instead of asking AI to infer, you hand it a fact that's already been resolved.

Yes, there is inference during the extraction phase — TypeScript Compiler and Gemini both contribute. But the results are persisted as confirmed values in the graph, and a daily boundary-analysis cron (covered below) lets us notice drift the next morning. By the time AI consumes the graph, only verified facts flow to it.

AI has a tendency to answer "with whatever it can see" rather than saying "I don't know." That's where silent hallucinations creep in — wrong answers that neither AI nor the human catches. Boundary nodes are what physically prevents that. They give AI a verified place to stand.

Construction: tree-sitter Base, With TypeScript Compiler and Gemini Where Needed

Normal code structure (function calls, class inheritance, imports) is relatively straightforward to extract with tree-sitter. Walk the AST, turn functions / methods / classes / fields into nodes, connect references with edges. Just grind through it.

The catch is that while tree-sitter is great at building syntax trees, it's weak on type information and scope resolution. To accurately follow a field access chain like user.preferences.theme, you need to resolve what type the variable user is and where it's defined. tree-sitter alone can't reach that.

So for field-access resolution we use TypeScript Compiler API and Gemini in combination. tree-sitter extracts the structure → TypeScript Compiler resolves variables and types → for the dynamic cases that even that can't reach, Gemini infers. Three stages with distinct responsibilities, which is how we push field-access accuracy up.

We define 21 edge types:

CALLS (function call) / EXTENDS (inheritance) / IMPLEMENTS (interface implementation), etc. — the basic structure tree-sitter can give us
CALLS_API (caller) / HANDLES_API (handler) — API boundary
EMITS_TO (emitter) / SUBSCRIBES_TO (subscriber) — Event boundary
WRITES_TO / READS_FROM — DB boundary
and more

The real battle starts when you try to extract the boundary edges (CALLS_API / HANDLES_API / EMITS_TO / SUBSCRIBES_TO / WRITES_TO / READS_FROM).

Extracting and Joining Boundary Nodes: 3 Months of Trial and Error (Jan–Mar)

Unlike normal code, boundaries (API endpoints, DB tables, Event topics) are written in wildly different ways depending on the framework, language, technical area, library, repository, and the person who wrote it.

Take "define an API endpoint": is it Express? NestJS with a @Get() decorator? A Fastify route? Each one produces a completely different AST shape. And the same repo can contain multiple patterns simultaneously.

And it's not just extraction that's hard. Joining the extracted boundaries on the graph is its own headache. For the same API path or DB table name, you get:

Casing variation: camelCase / snake_case / PascalCase
Trailing-slash variation (/users/me vs. users/me)
The boundary name itself is a variable (${baseUrl}/users/me)

…all mixed together. Normalizing all of that and correctly joining caller to handler, emitter to subscriber, writer to reader was genuinely the painful part.

And this had to happen across all 46 repositories × the framework zoo.

Looking back at the actual git history from that period, you see new parsers and detectors being added almost every week, noise filters going in, and concept renames landing. Here are the main commits from January through March, in order (the commit prefix starts as graph-rag — the stack was originally named after the "knowledge graph + RAG for LLM consumption" framing — and is renamed to code-graph on February 15; a few late-February commits still carry a short-lived graph prefix from that transition):

January: Starting Out, and Realizing tree-sitter Alone Isn't Enough

2026-01-15 ─ feat(graph-rag): add TypeScript parser with tree-sitter — the starting commit
2026-01-15 ─ feat(graph-rag): add graph builder with BigQuery storage — graph data is written to BigQuery
2026-01-19 ─ feat(graph-rag): add TypeScript Compiler-based variable resolution for field extraction — realized that tree-sitter alone couldn't resolve variable types, brought in the TypeScript Compiler API alongside it

February: Framework Diversity, Fighting Noise

2026-02-02 ─ feat(graph-rag): add frontend parser for jQuery/Vanilla JS codebase — jQuery / Vanilla JS frontend code
2026-02-03 ─ feat(graph-rag): add AngularJS Page detection for frontend BFS — AngularJS page detection (older framework, still very much running)
2026-02-15 ─ refactor(code-graph): consolidate 18 MCP tools into 5 with deep subgraph traversal — the toolset had ballooned to 18, consolidated to 5 (also the moment the stack was unified under the name code-graph)
2026-02-18 ─ fix(code-graph): reduce graph noise by filtering Type nodes, external lib CALLS, and Storybook files — noise reduction: filter out Type nodes, external library CALLS, Storybook files
2026-02-19 ─ fix(code-graph): extract path aliases from tsconfig paths in addition to make-symlink + fix(code-graph): resolve @alias path imports for CommonJS symlink patterns — the path-alias pain: tsconfig paths, make-symlink, and on top of that the CommonJS symlink pattern — three different mechanisms to support
2026-02-19 ─ feat(code-graph): add stop_at=boundary option to trace_connections — option to stop traversal at boundary nodes (explicit traversal scoping / node-explosion mitigation)
2026-02-21 ─ feat(graph): add typeORM JOIN detection, NestJS decorator parsing, Fetcher API detection — TypeORM JOINs / NestJS decorators / Fetcher API support
2026-02-21 ─ fix(graph): pass fullFileCode to Redux Axios variable resolver for scope-based extraction — Redux Axios variable resolver fix

March: Concept Cleanup and Precision

2026-03-08 ─ refactor(code-graph): rename __external__ to __boundary__ — concept cleanup: standardize on "boundary node" rather than "external resource"
2026-03-16 ─ refactor: remove db-dictionary from code-graph stack — split the DB schema dictionary (the layer that lets you look up table / column definitions) off into its own graph to evolve independently
2026-03-24 ─ fix(code-graph): infer table names from dynamic variable names — table-name inference from dynamic variable names
2026-03-24 ─ feat(code-graph): add orphan boundary node cleanup script — cleanup script for orphan boundary nodes

What This Timeline Tells You

Every single week there's a new framework or pattern being handled. The work of "extracting boundary nodes" is, fundamentally, adding parsers for each new way people write the boundary.

Just listing the frameworks / mechanisms that showed up:

tree-sitter (TypeScript / JavaScript / Go / Dart (Flutter))
TypeScript Compiler (variable resolution)
jQuery / Vanilla JS
AngularJS
Express / Koa / Fastify
NestJS (decorator parsing)
TypeORM (DB JOIN detection)
Fetcher API
Redux Axios (variable resolver)
3 different path-alias schemes (tsconfig paths / make-symlink / CommonJS symlink)

This isn't a "TypeScript / JavaScript / Go / Dart static analysis" story you can wrap up in one sentence. The air-closet codebase is a collection of long-running production systems where every era's framework still coexists. We had to pick up, from the AST, the era-specific meaning of "here's an API endpoint," "here's a DB call," "here's an Event subscription."

Why I Was So Particular About Accuracy

90% is completely unusable.

Take "list every piece of code that calls this API." If you recall only 90% of the callers, then 10% of the relevant code is invisible to AI. When you're using code-graph for blast-radius investigation, that invisible 10% is what causes the incident. That's single-hop recall.

And it gets worse the further you walk. For multi-hop graph traversal, every hop multiplies in: at 0.9 per hop you get 0.81 at 2 hops, 0.729 at 3, ~0.59 at 5, ~0.35 at 10 — after just a handful of hops you're at less than half. Push it to 0.99 and you get 0.98 at 2 hops, 0.95 at 5, ~0.90 at 10. Whether the system is usable in practice is decided by that single-digit difference between 90% and 99% — and it bites you on both axes: single-hop recall when you're enumerating, multi-hop confidence when you're traversing.

So every time a new boundary pattern showed up, we'd add a new custom parser, aiming to keep the boundary connection rate above 99%. We can't measure extraction recall directly — there's no ground-truth "every boundary that should exist" denominator — so the indicator we actually measure daily is "what fraction of callers / handlers are correctly connected on the graph" = the connection rate. The next section is about how that's monitored.

Boundary Analysis Is Running Today

The code-graph we built is still running daily.

Concretely, a boundary-analysis cron runs at JST 7:00 every morning. What it does:

API boundaries: match CALLS_API (caller) with HANDLES_API (handler), and aggregate cross-repo connection rates
Event boundaries: match EMITS_TO (emit) with SUBSCRIBES_TO (subscribe)
DB boundaries: aggregate cases where WRITES_TO and READS_FROM from different repositories touch the same table (= implicit cross-repo DB dependency)

The day-over-day numbers get compared, and if the connection rate drops by more than 5%, we get a Grafana alert.

This whole thing only makes sense because we have boundary nodes to compare against. We're monitoring the quality of the extracted boundaries themselves on a daily cadence. The kind of drift the connection rate catches by the next morning: "a parser fell behind a new pattern and a class of boundaries went invisible," "the repository layout changed and path aliases stopped resolving." There are failure modes the connection rate alone can't see — a caller-side parser regression that drops callers entirely will leave the surviving handlers still looking "connected" to whatever callers remain, and the missing ones slip out silently. That's a separate axis we cover with day-over-day absolute node counts per repo / pattern.

What Still Doesn't Work

Even after all that, a handful of issues remain that I can't solve at the root.

1. No Semantic Search (an Entry-Point Problem)

The search MCP tool only does LIKE-based substring matching.

If you're in the middle of development and want to follow connections starting from a function you're already looking at, that's fine — you can pull it up by function name or filename directly.

The problem shows up when you're investigating a production bug or a customer support ticket. You have no idea what filenames or function names are involved at the start. When the input is "the subscription-fee calculation for members seems off," and you want to walk to the related code from there, no natural-language query into the graph means you can't find the entry point in the first place.

The intent was: "instead of grepping the whole codebase, navigate relevance via graph RAG." What we ended up with is a structure where you have to grep at the entry point and infer your way in.

2. Node Explosion

If you naively turn the AST into a graph, every builtin function, anonymous function, and internal utility becomes a node. The map call you don't care about, the internal helper you don't care about — they're all nodes.

Trigger a traversal starting from one node, and within a few hops you're dragging in helpers, types, and primitives until the node count explodes. There's no axis built into the graph structure for "filter by relevance."

We work around it with explicit controls like stopping traversal at boundary nodes, but that's a workaround, not a root fix.

3. To Know What a Function Actually Does, You Still Have to Read the File

The graph tells you "something is here," "this calls out to another repo." But what the function actually does still requires opening the file.

That makes the graph slow on its own. The codebase-investigation tool we built later uses the graph to narrow down candidate files and then hands those to Git Server MCP to actually read — but the underlying graph-only resolution limit doesn't go away.

4. Operational Cost of Adding Parsers for Every New Boundary Pattern

Every time a new framework or library enters the codebase, we have to learn "how do they write boundaries in this thing" and add a new parser.

The parser directory already has 10+ custom detectors / extractors. There's no sign of the maintenance and extension cost going down — every time a new tech stack enters the codebase, the same work repeats.

Side Note: A Different Call Elsewhere — cortex

Note: "cortex" in this section is the internal codename for an AI platform I've been building in-house at airCloset. Unrelated to existing commercial products like Snowflake Cortex or Palo Alto Networks Cortex.

Setting code-graph aside for a moment: I also have a separate project — cortex — where I'm building an in-house AI platform from scratch (currently a single monorepo with 100+ apps).

On that project I did initially try the same approach as code-graph, but bailed out early and went with an annotation-based knowledge graph instead:

It's a monorepo I'm assembling myself, so I can realistically annotate everything at once
Use JSDoc tags to write intent directly into the code, and build the graph from that
Vectorize that intent and store it on the node, so semantic search works

The decision to "write intent into the code and graph it" — and the trial and error that led to it — I covered in detail in a separate series. If interested: AI Harness Series, Part 2 (The Knowledge Graph at the Heart of cortex)

Annotation-Based Won't Work for Production Systems

And no, you can't take the same approach for the production-side codebase that code-graph deals with:

Annotating all 46 repos at once isn't realistic
Long-running production systems, touched by multiple teams, with mixed frameworks
The precondition "put annotations into the code" doesn't hold

So the choice was: keep code-graph (static analysis) as the base, and evolve by layering on additional graph layers to compensate.

How we're trying to solve the issues above, I cover in Part 2.

To Be Continued

That's it for Part 1. Part 2 covers how we try to get past the issues above.

The real story is less "thrown away" and more "evolved."

Thanks for reading this far.

AI Isn't Something to Trust — It's Something to Design

Ryosuke Tsuji — Tue, 16 Jun 2026 00:02:03 +0000

AI assistance disclosure: This article was drafted with the help of Claude. All technical content, design decisions, code references, and screenshots reflect production systems I designed and operate at airCloset; the prose was revised by me prior to publication.

Hi, I'm Ryan, CTO at airCloset.

Disclaimer: "cortex" in this article is the internal codename for an AI platform built in-house at airCloset. It is unrelated to existing commercial services like Snowflake Cortex or Palo Alto Networks Cortex.

Across the five posts of this series I've worked through how cortex's harness is put together, one piece at a time: the overall picture, the knowledge graph, Auto Review, Self-Healing + Recurrence Prevention, and non-engineer PRs. Having walked through all of them, I want to step one level down for the wrap-up. Why am I building this thing in the first place? That's what this post is about.

The five posts might look independent, but the root is one thing, and the series doesn't close cleanly without that one thing being put into words. Together with the philosophy, I want to look back at the failures that don't show up when you only write about what worked — what I threw away, where I tripped — as a reference point for anyone trying something similar.

Series Index

#	Theme	Key scene	Article
1	Series intro: cortex's harness	PRs auto-merge / incidents self-heal before you notice	ai-harness-intro
2	Product Graph (cpg)	Code, docs, DB, infra unified into one graph	cortex-product-graph
3	AI PR review	webhook → AI review → auto-fix → squash merge	cortex-auto-review
4	Self-Healing + observability + auto-added guardrails	Alert → AI investigates → fix PR + new lint/type gate → auto redeploy	cortex-self-healing
5	Democratizing the maintenance phase	Domain experts open PRs to production; the harness owns the quality gate	cortex-non-engineer-prs
6	Series wrap-up	The underlying philosophy plus a retrospective on the failures and lessons	This post ← you are here

Origin — What I Was Thinking About in 2025

When I started building cortex, there was one question I wanted to answer:

How do I get AI to understand the system accurately?

If AI could understand the system accurately, then PR review, bug investigation, and fixes could all be delegated, and even non-engineers could open up their own development. Conversely, as long as I was stuck on "understand it accurately," everything downstream was sitting on unstable ground. So I spent a lot of time on the prerequisite layer before any of the individual mechanisms.

The two obvious approaches both hit walls.

Wall 1: The Context Window Limit

The first reflex is "just give it all the information it might need." Stuff the codebase, docs, DB schema, infra definitions all into the prompt, and AI gets the whole picture.

That fails on size. Codebase + docs + schemas + infra at our company doesn't come close to fitting into any realistic context window.

"Surely context windows will keep growing, and this'll work eventually?" — the more I thought about it, the less of a future I saw in that direction.

Even with a model whose context window is very large like Gemini, behavior gets unstable when you push it close to the limit. Middle information gets dropped, irrelevant tokens skew the conclusion sideways. This isn't a model-selection problem; it's a structural attention problem. The more unrelated tokens you mix in, the more the attention ratio toward relevant tokens drops mechanically. This is the documented "lost in the middle" phenomenon (information placed at the start and end of long inputs gets used; information placed in the middle is effectively ignored), and stuff the context window full and you routinely end up in a state where the information you thought you handed over isn't actually visible to the model.

"Lost in the middle" itself may get mitigated as long-context models improve, so I treat it as empirical supporting evidence rather than the core argument. The real wall is the recursive one beneath it: even if "size" is solved, you immediately need a higher-level context to judge which tokens are necessary and which aren't. That problem is recursive and can't be resolved by context window size, in principle. Information has to be structured, or AI doesn't make correct judgments. That's true of humans too — but humans are a notch better off, because LLMs don't notice they don't know, and they answer with confidence anyway. Silently wrong is worse than visibly stuck.

The keep-growing-context-windows path didn't have a real resolution in sight.

Wall 2: Don't Lean on Learning Either

The other obvious move is to make AI itself learn. Fine-tune per organization, teach it our codebase, our docs, our business. I considered it. Currently not doing it.

Two reasons. One: getting learning into actual production was still research-phase (in 2025 then; still in 2026 as I write this) and the road to real deployment is still long. The other is thornier: even if you could learn it, "forgetting" is extremely hard.

A business system has to reflect "the current truth." When the design changes, the DB schema changes, the business rules change, you want to actively erase old knowledge. But "delete just this piece of what's baked into the LLM weights" is unsolved at the research level — there's even a field name for it, machine unlearning, which tells you how hard it is. And on top of that, teaching the model new things also destroys unrelated existing knowledge (called destructive interference / catastrophic forgetting). Lean on learning and both hit at once: the cost of keeping things consistent explodes.

Rather than treating "doesn't learn" as a downside, I came around to: because it doesn't learn, swapping out the external knowledge is enough to reflect the current state, and the consistency story is much simpler. That was the call at the time.

The Way Out — GraphRAG + MCP

With no future in the context-window direction or the learning direction, I came across the GraphRAG concept.

GraphRAG itself is widely discussed elsewhere; for me, what it meant was the framing: "supply only the context that's needed, at the moment it's needed." Combined with MCP (Anthropic's protocol for connecting LLMs to external tools), AI can go fetch what it needs on its own.

What was decisive was that this structure lets AI traverse the graph agentically. Rather than "read everything and find related parts by inference," AI gets to the node it needs and pulls the fact out. Which leads to:

Instead of making AI infer, supply facts as context.

That one sentence became the core of cortex's entire design philosophy.

The first thing I built was a static-analysis-based code-graph, which I then threw away after trial and error, and arrived at the annotation-based product-graph (cpg) — details in the trial-and-error section.

I Don't Trust AI to Begin With

The origin section in one line:

I don't trust AI to fill in the blanks for me.

"Don't trust" here is not the same as "have no faith in." This isn't doubting Claude / GPT / Gemini's generation quality. What I mean is:

It doesn't know context it wasn't handed.
It doesn't, on its own and without being told, produce the ideal state.

The first one is a truth no amount of model progress will change. Architecturally, LLMs can't know things that weren't in the training data and aren't in this session's context. "Surely smarter models will pick up on it" — I don't think that future is coming. Smarter is a real direction; smart alone doesn't compensate for not knowing.

The second one is about responsibility, and humans owning it. AI can't decide on its own what "ideal" means. When it tries, it lands on a generic best-practice answer slightly off from the actual situation. Ideal depends on the business, the organization, the moment in time — none of which is visible to AI unless a human verbalizes it and hands it over.

So that conviction is not underestimating AI's capability; it's a design decision to not let AI auto-complete the prerequisites.

Mastering AI is not about giving it freedom — it's about confining its output to a predictable range.

The mechanism for confining it is the harness this series has been describing.

So I Build Harnesses to Hold AI to Determinism

Reading each post through the lens of "don't make AI infer; lean on determinism" surfaces that the five of them are all the same conviction showing up in different layers.

Part 2 — Knowledge Graph: Instead of making AI search the codebase, this mechanism tilts toward making the codebase legible. With @graph-* annotations, code / docs / DB / infra are unified into one graph, so AI doesn't have to grep + infer to find related parts. This is the direct implementation of "supply facts as context" from the origin section. → cortex-product-graph

Part 3 — Auto Review Dimensions: Nine review dimensions (responsibility / severity / type SSoT / etc.) are fixed in advance. When AI does the review, what to check isn't something it gets to infer. "Looking at the PR as a whole" gives AI too much room for inference, so dimensions are split and each is judged as its own question. Dimensions = locked by the harness, evaluation = AI's job. → cortex-auto-review

Part 4 — Self-Healing + Recurrence Prevention: Alert → investigation → fix PR → redeploy. The flow itself is fixed. AI doesn't get to think through "how should we respond to incidents" each time. And Recurrence Prevention — adding lint / CI gates so the same trap can't be stepped on twice — is mechanical refusal at the gate, not trust-AI-not-to-do-it-again. Or put differently: I don't expect AI never to repeat a mistake. → cortex-self-healing

Part 5 — Non-Engineer PRs: If the harness weren't holding quality, business-side folks opening PRs directly to production wouldn't survive a single day. Conversely, with the three mechanisms above stacked up (context locked, dimensions locked, traps locked out mechanically), the person closest to the requirements can ship the change directly. The translation layer and the engineering priority queue disappeared as a downstream consequence of the determinism push. → cortex-non-engineer-prs

So what's covered across the five posts is "don't make AI infer; lean on determinism" implemented at different layers. The root is one conviction.

What "Don't Make AI Infer, Lean on Determinism" Actually Means

Let me sharpen this phrase that's come up a few times.

"Lean on determinism" does not mean "give AI zero room to infer." Code generation, judging review findings, hypothesizing root causes from error logs — these are domains where AI not inferring is the same as no work getting done.

Where I want to lean on determinism is in domains where variance isn't allowed. Specifically:

Which part of the codebase to look at — don't have AI guess by analogy; pull it deterministically from the knowledge graph
Which review dimensions to apply — don't let AI pick "the important-looking dimensions"; lock the dimension list in advance
How to respond to incidents — don't make AI think through the workflow each time; fix the alert → fix PR path
Not stepping on the same trap twice — don't ask AI to "try to be careful"; let lint / CI mechanically refuse it

What implements this line — where inference is allowed vs. where it isn't — is the harness. To borrow the metaphor from Part 5, the harness lays down rails you can't fall off. On top of the rails, AI runs free (inference works as inference); but it can't fall off the rails sideways.

Put differently, this is equivalent to the framing in Part 2 (cortex-product-graph): "where hallucination gets confined." Saying "no inference allowed" isn't quite right — the harness isn't a thing that makes hallucination go to zero. It's a thing that confines hallucination to places where hallucination is OK (i.e., the inference-allowed zone). The structure and facts about the codebase are pulled deterministically, so the retrieval process itself has no opening for hallucination; hallucinations on the judgment side get filtered downstream by tests / lint / dimension-by-dimension reviews. The places where hallucination is allowed and the places where it isn't are physically split by the harness. That's the continuation of the Part 2 framing.

Step back one more level and what the harness is really doing is shifting when inference happens. The annotations and descriptions on the graph were also written by AI originally — there is inference baked into them. But that inference is write-time — happens once, reviewed, then frozen — not read-time (happens every query, unverified at the point of use). The graph is frozen, reviewed inference, which is exactly why the read side can treat it as fact. "Leaning on determinism" can be rephrased as not letting unverified inference run on every query.

This is also the underlying basis for Part 1 (Series Intro)'s claim "models commoditize; harnesses differentiate." Model-side quality is converging across Claude / GPT / Gemini, but the harness is codebase-specific and business-specific, so this is where org-level differentiation actually comes from.

Worth flagging: the position of this boundary moves with model capability. As agentic search and reasoning get stronger, today's "must be deterministic" zone might be tomorrow's "inference is good enough" zone — and in fact cortex itself depends on AI's ability to traverse the graph agentically. But the boundary itself never disappears. How information is structured, where the line gets drawn between fact and inference — whether you hold that line explicitly as a design decision is what differentiates organizations, across every model generation.

A note: this framing isn't confined to cortex's harness. The same stance shapes db-graph MCP, the natural-language interface over internal DB schemas, and Sandbox MCP, which lets non-engineers safely publish AI-built apps. It's the through-line in any platform we build that's based on AI doing meaningful work.

One level more abstract: the individual features aren't where the value is. The value sits in the conviction itself. cortex / db-graph / Sandbox MCP are all that one conviction translated into our own use cases.

The way I think about "design":

Design is translating an abstract principle into a concrete implementation that fits your own use cases.

It's not drawing class diagrams, and it's not laying out architecture diagrams — it's the translation work of "how does this principle take shape under our business / codebase / constraints?" That's where each organization's distinctiveness lives, and that's the value that can't be copied.

Said the other way: another organization copying cortex's surface doesn't reproduce the substance. What gets asked of every org is how it translates this principle into its own use cases.

Trial and Error That Got Me Here

Everything I've written about above is the form that ended up working. Getting to that form involved a lot of throwing away. Two representative examples worth keeping on record, plus one shorter one.

I Spent Two Months on Static-Analysis code-graph, Then Threw It Out

The first thing I built was static-analysis-based code-graph: extracting AST data — imports, call graphs, type dependencies — and putting that into a graph DB. At a glance, the obvious implementation of "make AI understand the codebase."

Why two months? code-graph wasn't just cortex; it spanned our consumer-facing services and internal-system repositories too — over 40 repos in total (cortex being one of them). The mechanically-extractable AST data (imports / call graphs / type dependencies) was usable as-is via tree-sitter, but each repo had its own API endpoints / DB schema / event definitions / Pub/Sub topology, and extracting those boundary nodes (where an app meets the outside) goes beyond mechanical AST analysis and had to be implemented per-repo-type — that's where the time went.

So I spent two months out of the first three on this, and got something that worked end-to-end.

And then I threw it away.

Why: static analysis is great at capturing structure, but it can't traverse on intent or business context. Concretely, three things broke:

No semantic entry point for search — if I want to query the codebase with "show me the function calculating member subscription billing," I can't get there unless I already know the function name or file. A graph built only from static analysis has no semantic-tag entry pointing to "what is this code for?"
The graph contains only code — internal helpers / utilities / types / arguments all become nodes, so traversal from any function blows up within a few hops, dragging in helpers and primitives. There's no axis to filter on semantic relatedness
What I actually wanted was code + DB schema + docs + infra on one graph — given a function, I want to pull, in one query, the DB tables it touches, the docs where the design lives, and the linked business requirement. A code-only graph just can't do that

→ I switched to the annotation-based approach (@graph-* JSDoc tags write the business intent into the code, and that gets unified with DB schema / docs / infra into one graph). Searchable semantically, and when you traverse, only related stuff comes back. That's the current product-graph (cpg). Don't drag sunk cost forward and you'll get to the final form — discarding two months of investment instead of trying to recoup it was the foundation for everything that came after.

Setting Coverage 90% as a Solo Target Broke the Implementations

Test coverage is still gated at 90%+ (as covered in Part 3). That part hasn't changed. But there was a period when Coverage was treated as a standalone target, and during that period the implementation visibly got worse.

Specifically:

Heavy default-value use that hides branches: function(input = {}) style writes the missing-input branch out of the test path. Coverage goes up, protection against unexpected input is gone
Catch-and-swallow over throw: try / catch returning null. Don't throw → no need to test "doesn't throw," and Coverage is satisfied. Invalid state silently propagates
Early returns that flatten too much: dump complex conditions through an "early return" escape. Tests pass; what should have been validation just isn't there anymore

Result: Coverage 90%, quality lower than before. When you look at Coverage alone, the shortest path to "satisfy it" is a weaker implementation that passes the tests.

Two lessons:

Set a number as a target, and the number becomes the goal. Coverage is a "minimum floor" — not "a goal to hit"
Don't evaluate any metric alone. Coverage has to be evaluated alongside responsibility separation / exception design / boundary value coverage / etc.

Then, as the follow-up: I added linting that mechanically closes off the routes that let you weaken implementations to satisfy Coverage. Two specific examples:

no-silent-catch: AST-level ban on empty catch and silent-handler patterns like .catch(() => null). Catch bodies have to have a function call (logger included) / re-throw / new / await — otherwise it's an error. Catches the "weaken throws to satisfy Coverage but lose observability in production" pattern structurally. The violation message routes you to @cortex/otel/logger for structured logging, so the chain through Cloud Run OTel → Loki / Grafana stays intact
vitest-strong-matchers: bans weak matchers like toBeTruthy / toBeDefined / toContain / toBe(true|false) / expect.any / expect.objectContaining. Catches "any assertion that passes" patterns at the AST level, and points you instead toward toStrictEqual / toMatchInlineSnapshot that pin down the full output. This is one notch above Coverage — a test quality concern — but it lines up because the same reflection applies: don't let a number become the goal

On top of that, cortex's testing guideline opens with "Coverage is not the goal, just a supporting indicator," and threshold-lowering / istanbul ignore workarounds get bounced as Critical in Auto Review. So even when Coverage is satisfied, "this is intentionally deleting a branch" / "this is swallowing the exception" comes back as a Major finding.

From the lesson "a single metric warps implementation," we descended through guideline that states the principle → lint that mechanically rejects → Auto Review that evaluates as a dimension before Coverage 90% finally functioned as the "minimum floor" it should have been all along. This too sits in the lineage of the Recurrence Prevention mechanism from Part 4 (so the same trap can't be stepped on twice).

Parallel Sub-Agent → Sequential Evaluation

Third: an internal-structure call about Auto Review. Distribute the 9 dimensions to parallel sub-agents and evaluate concurrently — the plausible-looking design ("parallel = faster, parallel should also hold quality") I tried first and ended up throwing out.

What actually happened: time, cost, and accuracy all got worse.

Time got worse: each sub-agent has its own startup, its own context load, its own result aggregation overhead. "9-way parallel = 9x faster" didn't hold; there were even cases where sequential evaluation in one session ended up faster
Cost got worse: each sub-agent loads PR diff + guidelines + related code independently — common context loads ran 9 times. Token consumption measured at just under 4x — not the naive 9x (the context other than diff is shared across many dimensions, which is what kept it from blowing up to a full 9x)
Accuracy didn't hold: parallel sub-agents don't see each other's verdicts, so the same problem comes back as "APPROVE" from one and "REQUEST_CHANGES" from another. Duplicate findings show up too. Without a "what kind of PR is this as a whole?" pass to anchor on, dimensional findings drift toward local optima and the overall picture gets worse

Switching to sequential evaluation: same session goes through 9 dimensions in sequence, so context loads once, and each dimension's call has the previous dimension's verdict in front of it. All three — time, cost, accuracy — improve simultaneously.

Of course, sequential evaluation introduces order dependence between dimensions — earlier verdicts can shape later ones. That's a real trade-off, and I accepted it knowingly. Inter-dimension consistency at the cost of some order sensitivity is more useful as a 9-dimension review than fully independent dimensions that contradict each other.

The takeaway: the distributed-systems intuition that "parallel = faster, parallel = quality holds" breaks its own assumptions in an AI harness. Unlike parallelizing across CPU cores on your machine, with AI the context isn't shared memory; it's per-process state. Sequential evaluation in one session ends up better on speed, token efficiency, and inter-dimension consistency at the same time — a structural property that's easy to miss at design time.

What This Section Is Really Saying

The form I've described across the series is the result of a lot of trial and error. Not starting with the right answer and laying it out from there. The decisions of throwing things away with sunk costs included, the trap of letting a metric I chose turn into the goal, the distributed setup that looked natural but worked against me — those are the things I walked through before landing at the current shape.

Not easy. I don't pretend it was. But if you do walk through it, real results follow — that's the honest read on it now.

Closing the Series

What I most wanted to communicate across these six posts comes down to one thing:

AI coding is not about "how to use AI" — it's about designing the environment AI runs in.

Or, put another way:

AI isn't something to trust. It's something to design.

Assuming a large codebase: prompt engineering / model selection / tool selection — each matters individually, but polishing them alone doesn't get you to auto-merging PRs, auto-healing incidents, or non-engineer development. Getting there requires building a codebase / business flow / observability / repair cycle where AI doesn't need to infer. That's not an individual AI skill — that's an environment-design problem (conversely, for a small project of a few dozen files, today's AI models work fine standalone. Harnesses become essential when scale exceeds what one person can hold in their head.).

And the conviction at the root of environment design is, repeating myself, "I don't trust AI to fill in the blanks for me" — looking the reality in the face that context that wasn't handed over isn't known, and the ideal state doesn't happen without being told. Once you accept that premise, what to build clarifies naturally.

Looking back, four decisions ended up being the ones that mattered:

Locked the conviction first: putting words to the root ("AI isn't something to trust") gave priority order to every mechanism. If I'd started from technique, I don't think I'd have made it to the current form
Invested with throwing-out as the default: like I did with code-graph at the two-month mark, I went into things with "throwing this out is OK." Drag sunk cost forward and you can't move forward
Refused standalone numerical targets: the moment a metric like Coverage 90% becomes the goal on its own, implementations warp. Designed the system so it gets evaluated alongside other dimensions
Designed for "no inference," not around AI's capability: I prioritized building structure where AI doesn't have to infer, instead of relying on what AI can do. That's what made the system stable end-to-end, I think

If even one of these is useful to someone starting on something similar, that would be great.

Afterword — Where Engineering Careers Are Heading

Slipping off the wrap-up topic — this is something I've been turning over recently, written here in a "loosely held thought" tone, so feel free to skim.

As harnesses mature, I think engineering work splits along two directions.

One direction is value creation from problem identification and business design. In the world of Part 5 — where non-engineer PRs work — "writing code" stops being scarce, and the actual scarce thing becomes the ability to define what to build. The person closest to the requirements (a PMO, a business manager, a domain-deep engineer) ends up driving Claude Code through to the merged PR themselves. This direction looks less like "engineer" and more like a business designer who moves between domain and implementation.

The other direction is building the foundation that lets all of that happen safely and quickly. Non-engineers can open PRs to the production repo only because the harness underneath holds quality — knowledge graph, Auto Review, Self-Healing, Recurrence Prevention, lint, CI, tests, observability stack, all interlocked. Designing / maintaining / evolving that gets harder, not easier. As the house-builder side, rail-layer side, this demands deep infra understanding / security instinct / observability design / a feel for AI's architectural quirks.

I'm building cortex, so I'm spending more time on the latter; building "a foundation where the business can run its own changes" is genuinely fun for me. That said, I'm not the type who fully commits to one side — I move between listening to business questions and assembling the foundation, and the satisfaction from each is its own kind. This isn't a "which is more important?" question — the harness exists precisely so the former is possible, and the former being alive is what gives the latter meaning. They're mutually dependent.

Maybe the era of polishing just "coding ability" is shifting slightly. Where to put your value — or whether to move between both directions — becomes a question more engineers will need to choose into intentionally.

Six posts in, thanks for sticking with me to the end.

#	Theme	Key scene	Article
1	Series intro: cortex's harness	PRs auto-merge / incidents self-heal before you notice	ai-harness-intro
2	Product Graph (cpg)	Code, docs, DB, infra unified into one graph	cortex-product-graph
3	AI PR review	webhook → AI review → auto-fix → squash merge	cortex-auto-review
4	Self-Healing + observability + auto-added guardrails	Alert → AI investigates → fix PR + new lint/type gate → auto redeploy	cortex-self-healing
5	Democratizing the maintenance phase	Domain experts open PRs to production; the harness owns the quality gate	cortex-non-engineer-prs
6	Series Final	The underlying philosophy plus a retrospective on the failures and lessons	This post

Part 5 ("The Author Doesn't Have to Be an Engineer") has been generating sharp comments. Worth a read for the thread alone.

Ryosuke Tsuji — Thu, 11 Jun 2026 01:06:48 +0000

Self-healing guardrails for business-side PRs

Ryosuke Tsuji

Jun 8

The Author Doesn't Have to Be an Engineer: How the Harness Holds Quality

#ai #devops #engineering #github

16 min read

The Author Doesn't Have to Be an Engineer: How the Harness Holds Quality

Ryosuke Tsuji — Mon, 08 Jun 2026 23:32:30 +0000

AI assistance disclosure: This article was drafted with the help of Claude. All technical content, design decisions, code references, and screenshots reflect production systems I designed and operate at airCloset; the prose was revised by me prior to publication.

Hi, I'm Ryan, CTO at airCloset.

Disclaimer: "cortex" in this article is the internal codename for an AI platform built in-house at airCloset. It is unrelated to existing commercial services like Snowflake Cortex or Palo Alto Networks Cortex.

In Part 1 (Series Intro), I wrote about how cortex's harness has matured to the point where non-engineers (business-side managers, PMOs, and the like) can open PRs to the production repository. The harness here is the runtime foundation for AI in production -- the combination of the knowledge graph, Auto Review, Self-Healing, and Recurrence Prevention covered across Parts 1 through 4.

Part 5 is what comes next: that harness has now reached the layer of who actually writes the code.

"Surely an engineer is still checking afterward, right?" -- I expect a lot of readers will land here with that question. So this post leads with one concrete example before anything else.

Part 5 covers:

What kinds of PRs are actually shipping -- two recent ones in detail
What works and what doesn't -- the boundary between adding on top of an existing stack and standing up new infrastructure
Why this holds for non-engineers -- how the four mechanisms from Parts 1-4 carry it
What's next -- into toC services -- the direction of travel for consumer-facing scale

The deeper toC implementation story will live in a separate post; here you'll get the framing and the direction.

Series

#	Theme	Key scene	Article
1	Series intro: cortex harness	PRs merging unattended / incidents fixed before anyone notices	ai-harness-intro
2	Product Graph (cpg)	Code / docs / DB / infra unified into one graph	cortex-product-graph
3	Auto PR review	webhook -> AI review -> auto-fix -> squash merge	cortex-auto-review
4	Self-Healing + observability + auto-added guardrails	Alert -> AI investigates -> fix PR + new lint/type gate -> auto redeploy + same pattern auto-rejected from then on	cortex-self-healing
5	Democratizing the maintenance phase	Domain experts open PRs to production; the harness owns the quality gate	This article ← you are here
6	Series Final	The underlying philosophy plus a retrospective on the failures and lessons	cortex-philosophy

Start with one scene

A +1,742 line / 41 file PR lands on the internal dashboard web app. Title: "PL dashboard ver.2". The change opens up project visibility to managers and team leads across multiple business units, scoping what each person sees to their own division or team. It adds an SSoT in the shared types package, new routes on the API server with SQL involving INNER JOIN and LEFT JOIN, new pages and view-state on the web app, and a personal-settings surface -- the whole stack of things you'd expect for a real feature.

The point is, this isn't a typo fix or a string swap. Entities, repositories, API routes, screens, filters, personal settings -- every layer you'd normally touch for a feature got touched. A few days of work for an experienced engineer, in scale terms.

The review-fix cycle ran like this:

PR open (+1,742 / 41 files)
auto-review pass 1: Major finding (a permission-scope fall-through -- data from other divisions leaking into the view that shouldn't be there) plus a handful of Minor items
author bot push: closes the scope fall-through, addresses the Minor items
auto-review pass 2: Nit items remaining, plus a lint catch (no-empty-function)
author bot push: lint clean
auto-review pass 3: still some COMMENTED nits, not yet APPROVE
author bot push (iteration 2): hardens loading skeleton, reverts an unnecessary JSDoc tweak
auto-review pass 4: APPROVED → CI green + APPROVE both met → auto-merge → production

From PR open to merge: four review-fix rounds, three author-bot pushes, zero human reviewers in the loop. The reviews come from the auto-review bot, the fixes come from an author bot (an automated review-response agent that the PR author has running on their machine), the final APPROVE is submitted by the AI, and an auto-merge script picks it up the instant CI is green. Production lands with 56/56 shared type checks (SSoT), 2,284/2,284 API tests, 1,113/1,113 web specs, and 0 lint errors. (cortex splits the lint job between oxlint for general checks and a custom eslint plugin for the @graph-* rules.)

The second review pass is worth noting. "Scope fall-through" is a somewhat technical finding -- a hole in the permission filter meant data from divisions other than your own could leak into the view. This is an internal dashboard, so it's not an external-leak incident, but "only see what's relevant to you" is the whole point of a dashboard like this -- losing it doesn't just risk an information slip, it drowns the user in noise that they shouldn't be filtering through in the first place. That's the kind of issue that's easy to merge by mistake and painful to notice in production. The fact that auto-review caught it on pass one and bounced it back for the author side to fix is what makes this whole flow viable for non-engineers. Without that loop, a PR of this size from a non-engineer would be a bad bet.

And: the author of this PR is not an engineer. A business-side teammate handed a feature description to Claude Code, leaned on the knowledge graph (covered in Part 2) to pull in the relevant existing code, and the +1,742 line PR is what came back. The four review-fix rounds above are what happened next.

That setup lines up directly with the central claim of this post: the person who knows the business requirements best, instead of organizing them and handing them to an engineer, runs them through Claude Code to production themselves.

Quick clarification on "write." When I say "write" in this article, I don't mean typing line by line in an editor. I mean handing the business requirements to Claude Code, judging the resulting diffs and AI review comments with domain knowledge, and seeing it through to a production merge -- the whole arc. Most of the actual diff is written by Claude Code; review feedback is handled by the author bot. What the human does is three things: put what they want into words, make the judgment calls along the way ("does this fit, is this off"), and sign off when it's ready to merge. None of that is implementation work in the technical sense. That's what "write" means here.

There's still a learning curve, of course -- the prompts you give Claude Code, where to point it for context. But none of that is learning to program. What you need is the ability to articulate what you want clearly, not syntax or framework knowledge.

The harness covers quality, so even at +1,742 lines / 41 files, this works.

Out of scope: this post does not cover the path where non-engineers freely ship apps to a sandbox environment instead of opening PRs against the production repo. That's a different mechanism, covered in an earlier post: Bridging "I Want to Build" and "I Want to Publish Safely" for Non-Engineers with a Custom Sandbox MCP. This post is specifically about opening PRs against the production repo -- the front door that's traditionally been engineer-only.

When you need a change, you can make it yourself

The point of the previous section is this:

When you need a change, you make it yourself, without flagging an engineer.

When that holds, work like:

"I want a new metric on the dashboard"
"The aggregation filter doesn't match how the business actually operates"
"I want a small business-support feature embedded in the production app"

stops queueing behind whatever an engineer is in the middle of. The fix lands when the need lands.

Think about the old flow. Someone on the business side notices a small thing that needs to change. They write the requirements up. They open a ticket or a Slack thread for an engineer. The engineer is in the middle of something else, so it queues. When they finally get to it, the interpretation drifts from what the business actually meant, there's a back-and-forth, a review pass, and only then does it ship. Even a small change takes days to a week in wall-clock time.

That's the cost of a translation layer between business understanding and code, and it gets worse the busier the engineer is. The business's improvement cycle ends up paced by engineering's backlog.

When the person who knows the requirements writes the change themselves, that translation layer and that queue both disappear.

Here are two recent examples of that working.

Two non-engineer PRs that recently shipped

PR	Kind	Size	What changed
PR 1	Deep bug fix	+348 -177 / 7 files	The dashboard's actuals number was unfairly exceeding the target. Root cause: the "which teams to aggregate" definition was asymmetric between target side and actuals side. Fix lifts the shared "teams to include" list into its own file and points both sides at it. Tests added too.
PR 2	Feature build on top of existing stack	+1,742 -227 / 41 files	The PL dashboard v2 from the opening scene. Entities, repositories, API, UI -- all touched, but the web app itself (the stack) was already standing; this rides on top.

Different shapes, but both are non-engineer PRs that made it all the way to merge.

PR 1 -- a deep root-cause fix

This started from "the number looks wrong" on the business side, and the PR went all the way down to a data-integrity issue. The surface symptom: "the actuals number on the dashboard exceeds the monthly target, with the achievement reading 101% even though the team knows that's not real." The lazy fix would be a fudge factor or a clamp on the display. That's not what happened.

The author dug into the aggregation queries and pinned the real cause: the actuals side and the target side were reading from different tables, and the definition of "which teams count" wasn't symmetric between them. Teams that don't carry a target value (designers, PMOs, and so on) didn't show up on the target side but were getting counted on the actuals side, so the numerator was inflated against the denominator.

The fix is structural, not cosmetic. A single file defines "the teams in scope for this aggregation" as a shared list, and both sides reference it. No future drift between target-side and actuals-side definitions -- it's locked in by the shared constant.

The handling of "what data falls out of an aggregation" and "are the target and actuals sides really symmetric" is the kind of thing engineers miss too. A non-engineer working through it down to the structural level and fixing it there is what stands out about this PR.

PR 2 -- a big feature build on top of an existing stack

This is the PR the opening scene walked through. +1,742 / 41 files spanning entity, repository, API, and UI -- a scale of change that's well past what people usually mean when they say "modification."

What lets a non-engineer ship this size of change is that the web app itself (the stack) is already standing. Nobody's standing up a new app, no new Cloud Run service needed defining, no new dependency packages, no new directory structure. The change adds a route, a page, and a repository entry inside the existing structure that's already there. It rides on what's been built.

This is the "on top of an existing stack" range. That's where the boundary is, and the next section spells it out.

Note on terminology: "modification" in this article is broader than "small tweaks to existing logic." It includes adding new entities, new endpoints, and new pages on top of an existing stack. The line I'm drawing is between building on top of a stack vs. standing the stack up in the first place.

What works, what doesn't

The principle: standing up a stack is hard, building on top of one isn't

The cleanest dividing line for non-engineer development isn't "modification vs. new development." It's "on top of an existing stack" vs. "stand up a new stack."

Standing up a new stack (work that starts from infrastructure: a new web app from scratch, a new Cloud Run service defined from a Dockerfile, a brand-new BigQuery pipeline) → engineering work
Adding to an existing stack (a new page in an app that already exists, a new endpoint on an existing API, a new data source on an existing pipeline) → non-engineers can do this

All three of the example PRs above sit on the second side. The stack itself was already built (by me, for the most part), so they get to work inside it. "Stand up a new app from scratch" or "define infrastructure (Dockerfile / IaC) from zero" are still engineer territory.

Put another way: renovations and new rooms inside an existing house are open to anyone. Building the house itself is engineering. Get the structure wrong -- the load-bearing parts, the wiring, the plumbing -- and the cost of recovery is high. That's the part of stack design where there's still too much "if this is wrong, everything downstream breaks" risk to hand to AI.

What's left for engineers: laying the rails -- the stack and the harness itself

The flip side: the rail-laying work -- standing up a stack, and extending the harness itself -- is what non-engineers don't touch yet. Both require a different kind of knowledge:

Infrastructure: containers, IaC, the operational characteristics of cloud services. Cloud Run resource ceilings, cold starts, Pub/Sub at-least-once semantics, BigQuery partition / cluster design, how Pulumi stacks split. Get this wrong and a thing that compiles can still fall over in production
Authentication for external integrations: OAuth, webhooks, how you handle API keys and where they sit in Secret Manager. One small slip leaks credentials into the repo or lets a webhook fire something you didn't intend
Security fundamentals: what to never expose, where to sanitize, where the privilege boundary cuts. SQL injection, XSS, SSRF, broken authorization -- "it works" isn't enough here
Harness design and extension: adding a new Auto Review dimension, changing Self-Healing logic, writing a new lint rule (e.g. in eslint-plugin-graph), structuring guidelines. Decisions that require understanding how the whole flywheel hangs together -- the most meta layer

That last bullet -- harness extension -- has an important implication: for non-engineers to keep being able to ship to production, someone has to keep the harness evolving. Recurrence Prevention (Part 4) is the automatic loop that adds lint / CI guards / guidelines per trap. But the architecture of the harness itself -- the structure of dimensions, the calibration of judgment, the design of the Self-Healing flow, the shape of the knowledge graph -- those are a meta layer that still requires engineering judgment.

Concrete case: the current nine Auto Review dimensions ([Graph] / [Architecture] / [Security] / [Test] / [Doc] / [Impact] / [Observability] / [AI-Antipattern] / [Recurrence]) were designed by observing past incidents and fix patterns. When a tenth dimension becomes necessary -- say, a "breaking change check on dependency upgrades" axis -- decisions about responsibility splits with existing dimensions and where to set thresholds are made by looking at the whole structure. That's the kind of engineering work that stays.

The harness provides "rails you can't derail from." Laying those rails -- and laying the foundation those rails sit on -- is a different job, and it's still on engineering. Engineers lay the rails; anyone can run on them. That's the boundary today.

Why this works for non-engineers

This is a short recap, because everything that makes it work was already covered in Parts 1 through 4. Four mechanisms reinforcing each other -- that's what lets non-engineers operate safely on top of an existing stack.

① The knowledge graph pulls relevant code from "what you want to do"

cortex-product-graph from Part 2 -- the unified graph fusing code, docs, DB schema, and infrastructure into one knowledge base (implementation name: cpg) -- carries this layer.

Non-engineers don't need to know function names or repo structure. A natural-language question like "I want to add a metric column to the dashboard" goes to Claude Code, which hits the knowledge graph with a semantic search and gets back the relevant nodes -- the screen, the API, the DB, the docs -- in one or two hops. You can get started without knowing the technical vocabulary.

For PR 2: the author told Claude Code "I want PL dashboard v2 with division/team scoping for non-PI-Div PMOs and team leads," and the knowledge graph pulled the existing /projects route, project-repository.ts, FilterHeaders.tsx, and ProjectTable.tsx as the relevant nodes. The author never needed to know what file to edit. That's how the translation layer drops out.

② Auto Review enforces quality at the gate

The 9-dimension automated review from Part 3 is the next layer. [Graph] / [Architecture] / [Security] / [Test] / [Doc] / [Impact] / [Observability] / [AI-Antipattern] / [Recurrence] -- the AI returns REQUEST_CHANGES on what's missing and loops with the author bot until APPROVE -- the four-round example from the opening scene is exactly this in motion.

The point is this: the first PR doesn't have to be perfect. The author doesn't need to ship a completed, security-hole-free version on the first try. Push the initial PR and the rest gets sorted by the auto-review and the author bot bouncing off each other. The reason the author bot doesn't spin off into a loop of confused fixes is that the knowledge graph holds the full codebase context: changes are made with structural awareness of what they touch, so misreadings of the review feedback don't compound.

③ Self-Healing catches what slips through to production

Part 4 covered Self-Healing. If something does break in production, the AI starts from the alert, investigates root cause, opens a fix PR, and gets it auto-redeployed -- the entire loop runs without humans. Incidents triggered by a non-engineer's change recover on their own, hands-off. That's what makes the bar to opening a PR feel survivable.

This isn't "non-engineers are safe because nothing can go wrong." It's "even if something goes wrong, the harness has it covered." The system is designed to minimize damage, not eliminate failure. The three-layer construction (Observation → Repair → Strengthening) from Part 4 is what makes that net real.

④ Recurrence Prevention keeps the trap count from growing

The Recurrence Prevention loop from the back half of Part 4. Every trap that gets stepped on gets nailed down in the same PR, so the next attempt at the same pattern gets caught. The form depends: mechanizable traps become lint or CI guards; less-mechanizable ones become entries in the guideline docs (docs/gotchas, severity docs) that the AI reviewer reads. Either way, the catch happens before merge. Non-engineers contribute to this loop too -- when they hit a trap, the doc entry that prevents the next person from hitting it can come from them.

As this compounds, the rails get denser. Where there was once a loose "don't go that way" guideline, every incident adds another small rail saying "or this way, or this way, or this way," and the lane that's safe to walk gets clearer. The denser the rails, the safer non-engineers are in the lane.

→ The four pieces aren't independent components. Each one's output feeds the next one's input. This is the Guides + Sensors flywheel from Part 1 in action. I won't re-explain the details since they're in the prior posts, but non-engineers shipping to production is the result of all four wheels turning together. Take any one out and the level of upfront knowledge required to write to production jumps, and the whole thing collapses.

Next -- carrying the pattern to consumer-facing services

cortex is an internal AI platform, so the system as it stands can't be lifted into a toC production service as-is. The biggest issue is the difference in quality bar. For toC, "detect after user impact → Self-Healing fix" is too late. The requirement becomes: incidents don't happen, and when something is about to ship, there's review and testing on top of human sign-off.

That said, the shape of the harness -- a knowledge graph for context, 9-dimension AI review, an author bot responding to feedback -- carries over directly. The thing that changes is the final step: cortex's auto-merge becomes "AI does the prep, a human signs off." Not by giving up the AI's range, but by having the AI handle the heavy lifting (test writing, environment setup, test runs, the 9-dimension review) and leaving only the final APPROVE on a human. "If the human sign-off stays, engineer time doesn't really decrease, does it?" -- but historically engineers were spending the bulk of their time on the implementation, the test writing, the environment setup, the self-review, the back-and-forth on review. Sign-off itself is the smallest piece of that pie. With AI doing the prep work, what an engineer spends time on shifts from implementation labor to quality judgment.

A caveat on the knowledge graph: it only earns its keep at large codebase scale. If the codebase fits in one AI context window, a cross-repo graph is unnecessary. The reason cortex (100+ apps) and the toC side (40+ repos) need one is because the scale forces it.

The concrete plan is real (extending the knowledge graph across the toC side's 40+ repositories, designing the AI-prep flow, etc.), and the full version goes in a separate post.

Wrap-up

The person who knows the business requirements best, instead of writing them up for an engineer, runs them through to production directly. Quality is held by the harness, so what's required from the writer is domain knowledge and the ability to direct an AI well. Business asks stop queuing behind engineering, and the cycle speeds up
The four mechanisms from Parts 1-4 (knowledge graph / Auto Review / Self-Healing / Recurrence Prevention) form a reinforcing flywheel. The first PR doesn't have to be perfect, and what does break is repaired automatically. That's the design
The boundary: engineers lay the rails, anyone can run on them. Standing up the stack (infrastructure, authentication, security) and extending the harness itself (new lint rules, new review dimensions, Self-Healing flow design) stay on engineering
Carrying this to consumer-facing toC services, the knowledge graph (a 40+ repo cross-repo graph on the service side) covers the context layer, but the quality bar shifts, so auto-merge becomes "AI prep + human sign-off." Details in a separate post

In Part 6 I'll wrap the series with the philosophy at the foundation -- why this design, what got given up, what got kept. The series so far has been about "the parts that are working"; Part 6 puts the failures and the dead ends on the table too, including the gap between the philosophy and the actual implementation. A retrospective for myself, and -- I hope -- a reference for anyone heading down a similar road.

Fixed Before Anyone Notices, Stronger After Every Fix: Self-Healing + Recurrence Prevention

Ryosuke Tsuji — Mon, 01 Jun 2026 23:57:25 +0000

AI assistance disclosure: This article was drafted with the help of Claude. All technical content, design decisions, code references, and screenshots reflect production systems I designed and operate at airCloset; the prose was revised by me prior to publication.

Hi, I'm Ryan, CTO at airCloset.

Disclaimer: "cortex" in this article is the internal codename for an AI platform built in-house at airCloset. It is unrelated to existing commercial services like Snowflake Cortex or Palo Alto Networks Cortex.

In Part 3 I covered AI reviewing AI PRs -- the auto-review pipeline that defends quality at the PR stage.

This post is the other side: defending quality in production, via Self-Healing. A production alert fires, an AI investigates it, opens a fix PR, the PR goes through the same auto-review pipeline from Part 3, gets auto-merged and auto-redeployed. And the same fix PR is required to add a new Guide -- whether that's a lint rule, CI guard, type constraint, or guideline update -- so the same anti-pattern gets auto-rejected from then on. The guardrails grow every time.

"Incidents get fixed automatically" is catchy on its own, but on its own it's probably not enough in the long run. You have to close the recurrence class while you fix the incident -- self-healing plus self-strengthening -- before the quality gates start to compound over time.

Start with last month's numbers

115 Self-Healing PRs merged in the past 30 days.

Effectively all of them merged and deployed without human involvement.

Humans only step in when the AI judges "this is not something code can fix."

That's the current state of "incident response" at cortex.

Don't read "115 = 115 user-impacting incidents" though. Roughly:

About half (54) are Deploy Failed-style alerts -- CI / Pulumi deploy step caught a failure, the AI absorbed it before it shipped to production. Recently the [Recurrence] loop (covered later) has been piling up countermeasures here, so this bucket is trending down anecdotally
The remaining 61 are production-runtime alerts (Service Error Log Detected / Pipeline Failure / Generator Failure etc.) -- the service is running in production, but an error-log threshold or consecutive-failure threshold tripped. The AI absorbed them before they propagated to user impact

So it's less "incident response" than "production anomalies that monitoring caught, fixed 115 times by AI before anyone woke up." The number of incidents humans actually have to acknowledge is in the low single digits per month.

There's also a clear pattern of the same service firing repeatedly (one ETL-ish service alone accounts for 25 of the 61) -- which is exactly what the [Recurrence] loop covered later is supposed to eliminate by turning into lint or type gates. That's the back half of this post.

One more honest note: the recent month's number is slightly inflated. The codebase had a fair number of "silent catch" patterns -- catch blocks that swallow exceptions without logging anything. We added the no-silent-catch lint rule and swept the existing silent catches in batches, which exposed previously hidden production errors as alerts. So part of the spike is "monitoring caught up to reality." Once the [Recurrence] loop converts these into lint over time, the number should converge. "Things we couldn't see, we can see now" is a quality improvement -- what we're seeing is the catch-up phase.

One more thing worth saying: doing this by hand is utterly unsustainable. Running 115 manual cycles of "ack alert -> read logs -> context switch -> understand the code -> fix -> open PR -> review -> deploy" would bankrupt any team's engineering bandwidth. The system absorbs them without anyone noticing, and converts the fix into a new Guide (lint / CI guard / type constraint / guideline) at the same time -- that's the actual subject of this post.

The moment an alert fires, the AI starts an investigation, traces Loki / Product Graph / git blame to root cause, opens a fix PR, runs it through the auto-review from Part 3, APPROVE -> auto-merge -> auto-redeploy. One full loop.

Series

#	Theme	Key scene	Article
1	Series intro: cortex harness	PRs merging unattended / incidents fixed before anyone notices	ai-harness-intro
2	Product Graph (cpg)	Code / docs / DB / infra unified into one graph	cortex-product-graph
3	Auto PR review	webhook -> AI review -> auto-fix -> squash merge	cortex-auto-review
4	Self-Healing + observability + auto-added guardrails	Alert -> AI investigates -> fix PR + new lint/type gate -> auto redeploy + same pattern auto-rejected from then on	This article ← you are here
5	Democratizing the maintenance phase	Domain experts open PRs to production; the harness owns the quality gate	cortex-non-engineer-prs
6	Series Final	The underlying philosophy plus a retrospective on the failures and lessons	cortex-philosophy

Big picture -- the three layers: Observation, Repair, Strengthening

For Self-Healing to work, you need an Observation layer in front and a Strengthening layer (recurrence prevention) behind it. Self-Healing itself is the middle Repair layer. The "self-healing + self-strengthening" loop only spins up when all three are in place.

Prerequisites: The three layers only stand up on top of two prior pieces: cpg (the unified code / docs / DB / infra knowledge graph from Part 2) and the Observability stack covered in this post.

No Observability -> the observation layer is empty, nothing gets detected -> the repair layer never even fires

No cpg -> the AI cannot see "where else does this trap exist" -> the repair layer does symptom-level patching at best, and the strengthening layer's horizontal expansion stops working

Put differently: trying to copy this setup without those two will just multiply incidents. An AI that blindly looks at error logs and rewrites production code is just speeding up the rate at which gh pr create ships accidents. cpg and Observability are the minimum bar for being able to delegate auto-repair to AI.

Note also that cortex is a several-hundred-thousand-line codebase, and at that scale loading the whole codebase as AI context is impossible for the AI as well (let alone for a human). Tell the AI to trace impact with just grep and file reads, and it'll run out of context window before it finds anything. cpg is what lets it ask "which other code does this function's change ripple into" and get the answer in one hop. Small repos may not need this. Past a certain scale, cpg is not optional, it's required.

In Fowler's Guides / Sensors terms from Part 1, cpg and Observability are the substrate that supports both Guides (pre-execution controls like lint) and Sensors (post-execution gates like auto-review and Self-Healing). Observability feeds Sensors via firing alerts; cpg feeds the Guides side by supplying the auto-review with impact-scoping context. Neither belongs on one side only -- they're foundational to both, and Self-Healing and auto-review only function on top of this substrate. That's the structural claim this post is built around.

Repair -> Strengthening loop">

Layer	Role	Key components
Observation	Real-time detection of production anomalies	OTel SDK / Loki / Mimir / Tempo / Faro / Grafana / Pino logs with trace_id
Repair	AI receives the alert, investigates root cause, opens a fix PR, auto-review, auto-merge, auto-redeploy	Event Relay -> SSE -> `self-healing` mode script -> claude -p (worktree) -> gh pr create
Strengthening	The fix PR is required to add a new Guide (lint / CI guard / type constraint / guideline). The same anti-pattern can't reach production again	`@cortex/eslint-plugin-graph` (26 rules), `scripts/check-*.ts` (13 guards), `recurrence-prevention.md`, the `[Recurrence]` lens of auto-review

I'll walk through them in order.

Observation -- where do the alerts come from?

cortex's production observability is built on Grafana Cloud + OpenTelemetry:

OTel SDK (the shared @cortex/otel package) -- every service calls initOtel({ serviceName }) at its entry point. Trace / metric / log all go out via OTLP to Grafana Cloud
Loki (logs) -- Pino structured logs get trace_id automatically. trace and log are cross-referenced
Mimir (metrics) -- Cloud Run / pipeline / Gemini API token usage, etc.
Tempo (traces) -- distributed tracing
Faro (frontend) -- captures browser JS errors / performance / network failures
Grafana -- dashboards + Alert Rules + Notification Policy

We also have a strict definition of log levels, anchored on business impact:

Level	Definition	Examples
`warn`	Business-foreseeable, does not need immediate action (retryable / self-recovers).	Search query returned 0 results, optional field unset, short retry due to rate limit
`error`	Data recovery / re-run will definitely be needed afterward. Impact expected to be under 20%.	"User record that should exist isn't there," BigQuery insert failure, per-record enrichment failure
`fatal`	The feature as a whole fails for 20%+ of requests. Service-continuity broken, fatal config missing, full upstream outage.	OTel init failure, required secret missing at startup, full input data source outage for a pipeline

The key point is to not pick the level mechanically based on the exception class name like NotFoundError. Same "record not found" situation: "this record must exist and doesn't" is error / fatal; "user search returned 0 hits" is warn. The level is decided by business impact -- "does this require data recovery later," "is the whole feature down" -- not by the type. Without this discipline you simultaneously get monitoring fatigue and missed critical incidents. Self-Healing reacts mainly to error-threshold trips; fatal is the human-escalation side.

Alert Rules are managed declaratively in Pulumi, grouped by service into categories like BOT / Pipeline / Transformer / Generator / Gemini / CI / Deploy / Service Catch-All. When we add a new service, one line in infra code spins up the dashboards and alerts automatically.

This is "the infrastructure that lets the AI see the same things humans see." Self-Healing picks up alerts coming off this stack.

What Observability can't catch, Self-Healing can't fix either

Honest disclaimer: Self-Healing can only react to what the observation layer can detect as an anomaly. "Observability is everything" is literally true here.

What the current stack catches is roughly logic-level errors -- exceptions, error logs, deploy failures, external-API call failures, threshold-based metric anomalies.

What it doesn't catch:

UI errors -- the logic ran, no error logs, but the screen shows something different from intent / shows the wrong value. Faro catches client-side JS exceptions and network failures, but "the logic ran and the output is just wrong" never fires an alert
Silent data corruption -- aggregated values slowly drift, bad values get into a table. Unless it crosses a threshold or schema check, nothing detects it
Perceived UX degradation -- requests feel slow, the UX feels off. Only catchable once SLO / latency thresholds trip

So Self-Healing is "AI replacing the human in the loop for incidents the observation layer can catch." The coverage of the observation layer itself is the prerequisite. Holes in observation stay as blind spots that neither auto-review nor Self-Healing reaches.

This isn't really a limitation of Self-Healing -- it's the importance of growing the observation stack, which cortex keeps investing in continuously. (From Part 1, Observability is one of the "supporting foundations" beneath the flywheel.)

Repair -- the Self-Healing flow

MODE=self-healing runs the same webhook-server script as the auto-review setup from Part 3, but listening for Grafana firing alerts.

The textual flow looks like:

[Grafana Alert Rule firing]
   ↓ POST /webhook/grafana
[Event Relay (in-house)] -- persisted in Firestore
   ↓ SSE push (event: grafana-alert)
[self-healing mode script]
   ↓ throttle check (same fingerprint skipped for 4h)
   ↓ 👀 reaction in Slack to signal "I'm on it"
   ↓ git worktree add -b hotfix/auto-alert-{service}-{ts} origin/main
   ↓ run claude -p inside the worktree
     - search related code via Product Graph MCP
     - pull error logs from Loki via Grafana MCP
     - identify root cause and fix
     - update tests as needed
     - conventional commit
   ↓ git push + gh pr create
[fix PR]
   ↓ auto-review (the Part 3 pipeline)
   ↓ APPROVE -> auto-merge -> auto-redeploy
[recovered]
   ↓ ✅ in the Slack thread

What happens when the AI judges "this is not fixable in code"

Not every alert is fixable by code. The implementation has a rule: "if you judge it unfixable, exit without changing anything." In that case Slack gets a notification of the form "This alert cannot be addressed in code. Investigation: ..." -- including what the AI investigated.

Worth clarifying on the numbers side: the headline 115 is "Self-Healing runs that reached PR-created -> merged -> deployed." This "unfixable, exit clean" case is a separate bucket, happening several times a month (external transient outages, infra / config issues that aren't code, cases too complex for the AI to judge confidently). The "humans show up" bucket is this separate one -- it isn't a "some of the 115 failed" failure rate.

Here's what a real Slack message looks like (a case where a transformer service's GitHub PAT had expired):

ℹ️ This alert cannot be addressed in code. Investigation:

Investigation summary

Checked the error logs in Loki for the past hour. The single error source for the target service right now is expiration / revocation of the GitHub PAT (Personal Access Token).

What I found

The BQ snapshot pipeline is healthy: 7 consecutive [BQ] Snapshot rows inserted successes. The earlier autodetect type-drift problem was resolved in an earlier PR that put SAFE_CAST(... AS INT64) in place.

The single ongoing error (multiple occurrences in the past hour): GithubUnauthorizedError: GitHub API /commits/master unauthorized (401): PAT likely expired/revoked -- the /check-drift handler returns 401 every time Cloud Scheduler invokes it.

Root cause

The GitHub PAT held in Secret Manager for this service has expired or been revoked. The code already classifies the 401 with a dedicated GithubUnauthorizedError and is already sending the "PAT needs rotation" notification to Slack. The detection / notification design on the code side is working as intended.

Resolution: exit without changes

This is an external-credential operational issue (PAT expiry), and what needs to happen is a human rotates the PAT in Secret Manager. Not the kind of incident code can resolve. Therefore: no code change, no commit, no PR.

Operator actions:

Add a new version of the target secret in GCP Console / Secret Manager with a valid PAT (repo scope, read access to the target repository)

No Cloud Run revision redeploy needed (secretKeyRef version:latest is referenced)

At this level of detail, "what was investigated, why code can't fix it, what the human should do" all come out in one Slack message. Open the thread and the operator can act immediately. The productivity gap vs. "alerts just forwarded blindly" is significant.

Deduplication

A throttle ensures the same fingerprint (Grafana's unique alert identifier) is not re-processed for 4 hours. Without this, alerts that fire again before the fix PR has merged would spawn another worktree, another fix PR, and so on -- an easy infinite loop.

We also permanently skip any alertname containing credential. Credential incidents carry leakage risk if the AI touches them, so they're explicitly escalated to humans.

Self-Healing and Part 3 auto-review -- "the fixer AI" and "the reviewer AI" are independent

This is the most consequential design choice of the agent setup, so calling it out explicitly.

PRs opened by Self-Healing are not special PRs, just fix PRs. They go through the Part 3 auto-review pipeline under exactly the same conditions -- the 9 lenses (Graph / Architecture / Security / Test / Doc / Impact / Observability / AI-Antipattern / Recurrence) get checked in order. Critical / Major findings -> REQUEST_CHANGES; Nit-only / no findings + CI green -> APPROVE -> auto-merge.

The important bit: this is not a monolithic "AI fixing AI" loop. The fixer-side AI and the reviewer-side AI are fully independent:

Different process, different session: the self-healing-mode AI and the reviewer-mode AI are launched as separate claude -p processes. They do not share context
Different input sources: the fixer builds the problem from Grafana alert + Loki + cpg. The reviewer judges from the PR diff + cpg + review guidelines
Different objectives: the fixer is optimizing for "stop the incident." The reviewer is judging "does this violate the 9 lenses or the severity contract?" A deliberate separation of concerns where the two roles' incentives are intentionally misaligned

As a result, PRs the fixer dashed off get blocked by the reviewer (REQUEST_CHANGES -> back to the fixer). The AI does not approve its own output. "Just-make-it-work" fixes don't get through.

This is the often-debated review-independence problem in LLM-agent operation, solved here in the obvious way: split the work across separate agents.

A concrete example: meet subscription's 409 ALREADY_EXISTS

Take the alert from the Google Meet recording auto-fetch service I covered in the Meeting Intelligence post. On 2026-05-21, Self-Healing opened a fix PR titled fix(meet-xxx): auto-fix for Service Error Log Detected.

The trigger error from Loki:

Workspace Events API request failed: 409 Conflict
"Subscription associated with the resource already exists."

How the AI investigated:

Pinned the error in Loki -- ran {service_name="meet-xxx"} | json | level=~"ERROR|error|Error" via Grafana MCP, picked up the Failed to renew Meet subscription stack trace
Traced the call path in Product Graph -- identified renewSubscriptions -> createMeetSubscription
Cross-referenced past PRs -- the "opposite-direction inconsistency" (name in Firestore but missing from Google = 404) had already been self-healed in another PR with patchMeetSubscriptionTtl -> null fallback. The current direction (still on Google's side but missing from Firestore = 409) was the gap
Verdict: "the same pattern may exist elsewhere" -- a [Recurrence] decision matrix "horizontal expansion required" case

Instead of a quick patch, it implemented the same-direction self-healing symmetrically to the opposite-direction fallback that was already there:

Made createMeetSubscription idempotent
If POST returns 409, extract the existing Subscription name from the response and call patchMeetSubscriptionTtl
The caller writes the return value back into Firestore, so the next renewal converges to the normal PATCH path (self-healing)
Per the existing graph/no-silent-catch lint, JSON.parse failures are also logger.warn + serializeError for structured logging
Three tests added

This is what "Self-Healing pushing all the way to root cause and rolling the fix out horizontally" looks like in practice. "Close the recurrence class, don't just suppress the symptom" (the spirit of recurrence-prevention.md) executed autonomously by the AI.

Strengthening -- Guides (lint + guidelines) grow automatically

This is the layer that keeps Self-Healing from being just "auto-repair."

In Fowler's Guides / Sensors terms from Part 1, the Strengthening layer is the place where Guides grow -- i.e. the pre-execution controls that prevent AI from deviating in the first place. cortex's Guides come in two flavors:

Machine-read Guides: lint / type / CI guard / coverage thresholds / Prettier -- enforced at commit / CI time
Human-and-AI-read Guides: guidelines like recurrence-prevention.md, severity.md, ai-antipattern.md, etc. -- used as decision criteria by auto-review

The 9 lenses, severity contract, and no-downgrade rules from Part 3 are the latter; the auto-added lints in Part 4 are the former. Together they form the Guides surface. Lints are "formalized guidelines," guidelines are "lints that haven't been formalized yet."

The Sensors side -- Self-Healing and auto-review -- grow these Guides every time they run:

Self-Healing's root-cause investigation finds "the same pattern exists elsewhere" -> demands horizontal expansion + a new lint (= new Guide)
Auto-review's [Recurrence] lens blocks PRs that fix without adding lint
Both depend on cpg to see impact scope across the codebase

cpg is what lets the AI ask "where else does this trap exist." Self-Healing and auto-review (= the Sensors side) share cpg as a substrate, and each run thickens Guides by one notch.

What happens every time Self-Healing runs (the recurrence-prevention-first flow)

Every fix PR Self-Healing opens is checked for [Recurrence] by auto-review. The decision matrix:

Situation	Required action	Form
Same trap stepped on 2+ times	Lint required (custom ESLint rule / type constraint / CI guard)	Machine (new Guide)
Pattern may exist elsewhere	Horizontal expansion required (cpg traversal for similar nodes, fix all of them in this PR)	Investigation + fix
Cannot be machine-checked but worth formalizing	Add to an existing guideline	Guideline entry
One-off, no value in formalization	Nothing (bug fix only)	--

When the "stepped on 2+ times" situation applies, the fix PR can't merge without a new lint included. So every Self-Healing run produces:

Horizontal expansion via cpg -- not just the immediate fix target, every similar node enumerated
A new Guide added in the same PR -- ESLint custom rule / type constraint / CI guard / guideline entry, one of the four
All existing violations cleared in the same PR -- no warn-as-deferral, error on first introduction
Auto-review -> auto-merge -> auto-redeploy -- the regular Part 3 pipeline
Going forward, writing the same pattern gets mechanically rejected by CI / lint -- the recurrence class is structurally closed

"Add the guard while you fix the bug" runs as a self-sustaining loop driven by Self-Healing.

"We'll do it later" and "introduce as `warn`" are banned

A couple of important contract clauses from the guidelines:

"Plan to lint later," "lint when we refactor," "another PR will handle this" -- all banned. If it can be addressed in this PR, it must be
"Existing violations remain, so introduce as warn and promote to error later" -- not accepted. This is deferral in disguise. The responsibility for the warn->error promotion goes nowhere and the rule rots
If you add a lint rule, fix all existing violations in the same PR and ship at error

These extend the no-downgrade rules from Part 3 -- preempting the typical escape hatches.

The "step on it, mechanize it" lineage

Custom Guides currently piled up in cortex:

graph/no-silent-catch (ESLint) -- the source of the "inflated number" mentioned in the intro. Bans catch blocks that swallow exceptions
Stacktrace-preservation guideline (codified as a Major violation in observability.md, caught by auto-review) -- forbids logger.error(err.message) style logs that drop the stack and keep only the message string. Forces the err field to hold serializeError(error) so name / message / stack are preserved as structured fields. Observability is everything here, so logs that drop stack info are treated as inherently broken
cortex-quality/require-fetch-timeout (oxlint -- a Rust-implemented JS/TS lint that runs ESLint-compatible rule sets, dozens of times faster than ESLint due to the Rust impl. cortex uses oxlint for the standard ruleset and ESLint for custom rules that need AST-level work) -- mandates signal: AbortSignal.timeout(...) on external fetch calls. Born from a case where a no-timeout fetch hung indefinitely and triggered a Cloud Tasks redelivery storm
graph/no-bq-string-timestamp-param (ESLint) -- from a case where passing TIMESTAMP as a string to a BigQuery query parameter NULLed the value out through a serializer bug and silently failed every INSERT
graph/require-firestore-ignore-undefined (ESLint) -- forces ignoreUndefinedProperties: true on new Firestore(). From a case where a single NULL row caused a 100% failure rate in a sync batch
check-otel-env-injection (CI guard) -- the recurrence prevention for the Cloud Run OTel env injection case below
TypeScript type tightening (type level) -- tighter function signatures, branded types for ID disambiguation, exhaustive discriminated unions, etc. Patterns that can't be lint-caught but are catchable at the type level get closed from the type side

These aren't textbook-learnable rules -- they're "stepped on once, then mechanized." The number of traps the organization has stepped on translates directly into the number of Guides piled up (across ESLint / oxlint / CI guard / types).

How does the AI write a lint rule without breaking it?

Three structural things keep this sane:

Existing rules are the template: the custom-rule directory already holds 26 custom rules, each as .ts + .test.ts pairs. New rules follow the same shape, so the AI never has to write the AST-walking boilerplate from scratch
Tests first: violation / pass fixtures go into .test.ts first, implementation fills in TDD-style. Coverage threshold (90% statements + branches) is gated by the Part 3 auto-review, so a lint without tests cannot merge
lint / type / CI guard sit in the same "mechanize" bucket: the decision matrix in recurrence-prevention.md groups lint / type constraint / CI guard together as the "lint-required" row, and leaves the choice within that bucket (write it as a lint? express it at the type level? add a separate CI guard?) to the AI based on how much AST work is involved and whether runtime semantics matter. Traps that need AST inspection but actually hinge on runtime behavior usually end up as a type constraint (branded type / discriminated union / signature tightening) rather than a custom lint

So "AI writes a lint rule" is supported by existing rule corpus + the test harness + the mechanize-bucket selection criteria -- three together. The path where the AI hand-rolls raw ESLint API and bricks something is structurally closed.

A concrete example: Cloud Run OTel env injection -> promoted to CI guard

Multiple services hit this trap: when a Cloud Run Service / Job is defined in Pulumi, forgetting to inject OTEL_EXPORTER_OTLP_ENDPOINT and GRAFANA_CLOUD_API_KEY via secretKeyRef causes OTel init to be skipped in production, no trace/log reaches Grafana, and incidents become silently invisible.

The normal response would be "we'll be more careful next time." At cortex:

Incident surfaces -> Self-Healing opens a fix PR (adds the env injection to the affected service)
Auto-review's [Recurrence] decides "same trap stepped on -> lint required"
The same PR adds scripts/check-otel-env-injection.ts (CI guard) -- mechanically asserts OTel env injection across all Cloud Run resource definitions under infra/
All other existing services get their env injection added in the same PR
Merge -> deploy -> any future write of the same kind gets rejected by CI

That's what "the guardrails grow every time Self-Healing runs" looks like in practice. The trap is "stepped on -> mechanically checked from then on."

Where Guides stand right now (in numbers)

Snapshot of cortex's Guide inventory:

Category	Count	Notes
Custom ESLint rules (`@cortex/eslint-plugin-graph`)	26	`no-silent-catch` / `require-firestore-ignore-undefined` / `no-bq-string-timestamp-param` etc.
CI guards (`scripts/check-*.ts`)	13	`check-otel-env-injection` / `check-cloudscheduler-oidctoken-audience` etc.
Standard oxlint rules (set to `error`)	183	Base config ships everything at error
TypeScript strict gates (baseline)	9	`strict` / `noImplicitAny` / `strictNullChecks` / `noUncheckedIndexedAccess` etc.
TypeScript type tightening (per-recurrence)	grows over time	branded type / discriminated union / function-signature tightening etc. Patterns that can't be lint-caught but can be type-caught are closed from the type side
Test coverage thresholds	statements + branches 90%	Uniform across all packages
Prettier	1 config	Format auto-fix
Guidelines	the entire review-guidelines repo	Used as the decision basis by auto-review

The first two categories plus the type-tightening row -- Custom ESLint, CI guard, type tightening -- are the part that compounds over time through the [Recurrence] lens every time Self-Healing or auto-review runs. The guardrails grow with time. That's the substance of the Strengthening layer.

The whole loop, from the top

When you compose the three layers:

[production anomaly] -> Observation layer (OTel/Loki/Grafana) -> Alert firing
                                              ↓
                                       Event Relay -> SSE
                                              ↓
[Self-Healing mode script]
   - claude -p in worktree
   - root cause via cpg + Loki + git blame
   - commit fix
   - (if applicable) add new lint / type gate too
   - gh pr create
                                              ↓
[Auto-review (Part 3)] -- 9 lenses in order, especially [Recurrence] forces
                         recurrence-prevention action (lint / horizontal expansion / guideline entry)
                                              ↓
                          APPROVE + CI green
                                              ↓
[auto-merge -> Turborepo build -> Pulumi parallel deploy]
                                              ↓
[production recovered + same anti-pattern mechanically rejected from now on]

The loop completes without human intervention. Not just repair, but the quality gates that grow with every repair -- that's the "auto-recovery + auto-strengthening" substance at cortex.

That said, as the front of the article spelled out, the loop is only viable because cpg and Observability exist. cpg makes horizontal expansion possible; Observability turns production anomalies into structured data. With those two in place at the foundation, AI can stand on the side that does Repair and Strengthening. Self-Healing is not a standalone mechanism. It's a Sensor riding on top of cortex's Guides (cpg + Observability + lint + guidelines). That's the single most important framing in this post.

Self-Healing by the numbers

Breaking the headline down further.

Main firing categories

What kicked off Self-Healing in the past 30 days (with the mapping back to the front-of-post 2 buckets):

Category	Bucket
Service Error Log Detected (most frequent)	Production-runtime (61 side)
Pipeline Failure -- data pipeline failing a configured number of times in a row	Production-runtime (61 side)
Generator Failure -- AI generation jobs (embedding / annotation etc.) failing	Production-runtime (61 side)
Deploy Failed -- deploy step failures (Pulumi up / Cloud Run revision failed)	Deploy step (54 side)

Alert-firing to production-recovery time

Median 30 minutes to 1 hour. Roughly:

Alert firing -> AI investigation start: under 1 minute (Event Relay + SSE)
AI investigation + fix + PR open: 3-8 minutes
Auto-review (including the Part 3 10.8 review-fix iterations on average): 20-45 minutes
Auto-merge + deploy: 3-10 minutes

Many of these finish before anyone wakes up (alert fires early morning -> by the time people come in, there's just a ✅ in Slack).

What changed / Bridge to Part 5

We've now covered the cortex picture across Parts 1-4:

Part 1: the cortex big picture and harness-engineering framing
Part 2: Product Graph (cpg) -- the AI's "brain"
Part 3: auto-review -- defending quality at the PR stage
Part 4 (this post): Self-Healing + Observability + auto-added guardrails -- defending quality in production while growing the quality gates themselves

The engineering role has shifted, over the last half-year, from "write, review, fix, merge, deploy, incident-respond" -- all of that -- toward looking at the whole system from above and tuning it. human-on-the-loop, working at the Policy layer.

Part 5 covers the harness reaching the "who writes the code" layer. The center of it is domain experts (business-side managers, PMOs — non-engineers) opening PRs to production, with a concrete walk-through of a +1,742 line / 41 file feature PR that landed with zero human reviewers in the loop. What guarantees the quality is the harness stack built across this series — "whoever writes, the harness owns the quality gate" is the Part 5 framing.

The toC service expansion gets a brief mention at the end for direction, but the full implementation discussion lives in a separate post.

The actual series wrap-up is Part 6. The center of it is the underlying philosophy -- why I picked this design, what I gave up, what I kept. Alongside that, since the series so far has been mostly "what's working," I want to look back at the failures and dead ends behind that surface, and the gap between the philosophy and the implementation. A retrospective for myself, and -- hopefully -- a reference for anyone starting down a similar path.