DEV Community: Ryosuke Tsuji

AI Isn't Something to Trust — It's Something to Design (Series Final)

Ryosuke Tsuji — Tue, 16 Jun 2026 00:02:03 +0000

Hi, I'm Ryan, CTO at airCloset.

Disclaimer: "cortex" in this article is the internal codename for an AI platform built in-house at airCloset. It is unrelated to existing commercial services like Snowflake Cortex or Palo Alto Networks Cortex.

Across the five posts of this series I've worked through how cortex's harness is put together, one piece at a time: the overall picture, the knowledge graph, Auto Review, Self-Healing + Recurrence Prevention, and non-engineer PRs. Having walked through all of them, I want to step one level down for the wrap-up. Why am I building this thing in the first place? That's what this post is about.

The five posts might look independent, but the root is one thing, and the series doesn't close cleanly without that one thing being put into words. Together with the philosophy, I want to look back at the failures that don't show up when you only write about what worked — what I threw away, where I tripped — as a reference point for anyone trying something similar.

Series Index

#	Theme	Key scene	Article
1	Series intro: cortex's harness	PRs auto-merge / incidents self-heal before you notice	ai-harness-intro
2	Product Graph (cpg)	Code, docs, DB, infra unified into one graph	cortex-product-graph
3	AI PR review	webhook → AI review → auto-fix → squash merge	cortex-auto-review
4	Self-Healing + observability + auto-added guardrails	Alert → AI investigates → fix PR + new lint/type gate → auto redeploy	cortex-self-healing
5	Democratizing the maintenance phase	Domain experts open PRs to production; the harness owns the quality gate	cortex-non-engineer-prs
6	Series wrap-up	The underlying philosophy plus a retrospective on the failures and lessons	This post ← you are here

Origin — What I Was Thinking About in 2025

When I started building cortex, there was one question I wanted to answer:

How do I get AI to understand the system accurately?

If AI could understand the system accurately, then PR review, bug investigation, and fixes could all be delegated, and even non-engineers could open up their own development. Conversely, as long as I was stuck on "understand it accurately," everything downstream was sitting on unstable ground. So I spent a lot of time on the prerequisite layer before any of the individual mechanisms.

The two obvious approaches both hit walls.

Wall 1: The Context Window Limit

The first reflex is "just give it all the information it might need." Stuff the codebase, docs, DB schema, infra definitions all into the prompt, and AI gets the whole picture.

That fails on size. Codebase + docs + schemas + infra at our company doesn't come close to fitting into any realistic context window.

"Surely context windows will keep growing, and this'll work eventually?" — the more I thought about it, the less of a future I saw in that direction.

Even with a model whose context window is very large like Gemini, behavior gets unstable when you push it close to the limit. Middle information gets dropped, irrelevant tokens skew the conclusion sideways. This isn't a model-selection problem; it's a structural attention problem. The more unrelated tokens you mix in, the more the attention ratio toward relevant tokens drops mechanically. This is the documented "lost in the middle" phenomenon (information placed at the start and end of long inputs gets used; information placed in the middle is effectively ignored), and stuff the context window full and you routinely end up in a state where the information you thought you handed over isn't actually visible to the model.

"Lost in the middle" itself may get mitigated as long-context models improve, so I treat it as empirical supporting evidence rather than the core argument. The real wall is the recursive one beneath it: even if "size" is solved, you immediately need a higher-level context to judge which tokens are necessary and which aren't. That problem is recursive and can't be resolved by context window size, in principle. Information has to be structured, or AI doesn't make correct judgments. That's true of humans too — but humans are a notch better off, because LLMs don't notice they don't know, and they answer with confidence anyway. Silently wrong is worse than visibly stuck.

The keep-growing-context-windows path didn't have a real resolution in sight.

Wall 2: Don't Lean on Learning Either

The other obvious move is to make AI itself learn. Fine-tune per organization, teach it our codebase, our docs, our business. I considered it. Currently not doing it.

Two reasons. One: getting learning into actual production was still research-phase (in 2025 then; still in 2026 as I write this) and the road to real deployment is still long. The other is thornier: even if you could learn it, "forgetting" is extremely hard.

A business system has to reflect "the current truth." When the design changes, the DB schema changes, the business rules change, you want to actively erase old knowledge. But "delete just this piece of what's baked into the LLM weights" is unsolved at the research level — there's even a field name for it, machine unlearning, which tells you how hard it is. And on top of that, teaching the model new things also destroys unrelated existing knowledge (called destructive interference / catastrophic forgetting). Lean on learning and both hit at once: the cost of keeping things consistent explodes.

Rather than treating "doesn't learn" as a downside, I came around to: because it doesn't learn, swapping out the external knowledge is enough to reflect the current state, and the consistency story is much simpler. That was the call at the time.

The Way Out — GraphRAG + MCP

With no future in the context-window direction or the learning direction, I came across the GraphRAG concept.

GraphRAG itself is widely discussed elsewhere; for me, what it meant was the framing: "supply only the context that's needed, at the moment it's needed." Combined with MCP (Anthropic's protocol for connecting LLMs to external tools), AI can go fetch what it needs on its own.

What was decisive was that this structure lets AI traverse the graph agentically. Rather than "read everything and find related parts by inference," AI gets to the node it needs and pulls the fact out. Which leads to:

Instead of making AI infer, supply facts as context.

That one sentence became the core of cortex's entire design philosophy.

The first thing I built was a static-analysis-based code-graph, which I then threw away after trial and error, and arrived at the annotation-based product-graph (cpg) — details in the trial-and-error section.

I Don't Trust AI to Begin With

The origin section in one line:

I don't trust AI to fill in the blanks for me.

"Don't trust" here is not the same as "have no faith in." This isn't doubting Claude / GPT / Gemini's generation quality. What I mean is:

It doesn't know context it wasn't handed.
It doesn't, on its own and without being told, produce the ideal state.

The first one is a truth no amount of model progress will change. Architecturally, LLMs can't know things that weren't in the training data and aren't in this session's context. "Surely smarter models will pick up on it" — I don't think that future is coming. Smarter is a real direction; smart alone doesn't compensate for not knowing.

The second one is about responsibility, and humans owning it. AI can't decide on its own what "ideal" means. When it tries, it lands on a generic best-practice answer slightly off from the actual situation. Ideal depends on the business, the organization, the moment in time — none of which is visible to AI unless a human verbalizes it and hands it over.

So that conviction is not underestimating AI's capability; it's a design decision to not let AI auto-complete the prerequisites.

Mastering AI is not about giving it freedom — it's about confining its output to a predictable range.

The mechanism for confining it is the harness this series has been describing.

So I Build Harnesses to Hold AI to Determinism

Reading each post through the lens of "don't make AI infer; lean on determinism" surfaces that the five of them are all the same conviction showing up in different layers.

Part 2 — Knowledge Graph: Instead of making AI search the codebase, this mechanism tilts toward making the codebase legible. With @graph-* annotations, code / docs / DB / infra are unified into one graph, so AI doesn't have to grep + infer to find related parts. This is the direct implementation of "supply facts as context" from the origin section. → cortex-product-graph

Part 3 — Auto Review Dimensions: Nine review dimensions (responsibility / severity / type SSoT / etc.) are fixed in advance. When AI does the review, what to check isn't something it gets to infer. "Looking at the PR as a whole" gives AI too much room for inference, so dimensions are split and each is judged as its own question. Dimensions = locked by the harness, evaluation = AI's job. → cortex-auto-review

Part 4 — Self-Healing + Recurrence Prevention: Alert → investigation → fix PR → redeploy. The flow itself is fixed. AI doesn't get to think through "how should we respond to incidents" each time. And Recurrence Prevention — adding lint / CI gates so the same trap can't be stepped on twice — is mechanical refusal at the gate, not trust-AI-not-to-do-it-again. Or put differently: I don't expect AI never to repeat a mistake. → cortex-self-healing

Part 5 — Non-Engineer PRs: If the harness weren't holding quality, business-side folks opening PRs directly to production wouldn't survive a single day. Conversely, with the three mechanisms above stacked up (context locked, dimensions locked, traps locked out mechanically), the person closest to the requirements can ship the change directly. The translation layer and the engineering priority queue disappeared as a downstream consequence of the determinism push. → cortex-non-engineer-prs

So what's covered across the five posts is "don't make AI infer; lean on determinism" implemented at different layers. The root is one conviction.

What "Don't Make AI Infer, Lean on Determinism" Actually Means

Let me sharpen this phrase that's come up a few times.

"Lean on determinism" does not mean "give AI zero room to infer." Code generation, judging review findings, hypothesizing root causes from error logs — these are domains where AI not inferring is the same as no work getting done.

Where I want to lean on determinism is in domains where variance isn't allowed. Specifically:

Which part of the codebase to look at — don't have AI guess by analogy; pull it deterministically from the knowledge graph
Which review dimensions to apply — don't let AI pick "the important-looking dimensions"; lock the dimension list in advance
How to respond to incidents — don't make AI think through the workflow each time; fix the alert → fix PR path
Not stepping on the same trap twice — don't ask AI to "try to be careful"; let lint / CI mechanically refuse it

What implements this line — where inference is allowed vs. where it isn't — is the harness. To borrow the metaphor from Part 5, the harness lays down rails you can't fall off. On top of the rails, AI runs free (inference works as inference); but it can't fall off the rails sideways.

Put differently, this is equivalent to the framing in Part 2 (cortex-product-graph): "where hallucination gets confined." Saying "no inference allowed" isn't quite right — the harness isn't a thing that makes hallucination go to zero. It's a thing that confines hallucination to places where hallucination is OK (i.e., the inference-allowed zone). The structure and facts about the codebase are pulled deterministically, so the retrieval process itself has no opening for hallucination; hallucinations on the judgment side get filtered downstream by tests / lint / dimension-by-dimension reviews. The places where hallucination is allowed and the places where it isn't are physically split by the harness. That's the continuation of the Part 2 framing.

Step back one more level and what the harness is really doing is shifting when inference happens. The annotations and descriptions on the graph were also written by AI originally — there is inference baked into them. But that inference is write-time — happens once, reviewed, then frozen — not read-time (happens every query, unverified at the point of use). The graph is frozen, reviewed inference, which is exactly why the read side can treat it as fact. "Leaning on determinism" can be rephrased as not letting unverified inference run on every query.

This is also the underlying basis for Part 1 (Series Intro)'s claim "models commoditize; harnesses differentiate." Model-side quality is converging across Claude / GPT / Gemini, but the harness is codebase-specific and business-specific, so this is where org-level differentiation actually comes from.

Worth flagging: the position of this boundary moves with model capability. As agentic search and reasoning get stronger, today's "must be deterministic" zone might be tomorrow's "inference is good enough" zone — and in fact cortex itself depends on AI's ability to traverse the graph agentically. But the boundary itself never disappears. How information is structured, where the line gets drawn between fact and inference — whether you hold that line explicitly as a design decision is what differentiates organizations, across every model generation.

A note: this framing isn't confined to cortex's harness. The same stance shapes db-graph MCP, the natural-language interface over internal DB schemas, and Sandbox MCP, which lets non-engineers safely publish AI-built apps. It's the through-line in any platform we build that's based on AI doing meaningful work.

One level more abstract: the individual features aren't where the value is. The value sits in the conviction itself. cortex / db-graph / Sandbox MCP are all that one conviction translated into our own use cases.

The way I think about "design":

Design is translating an abstract principle into a concrete implementation that fits your own use cases.

It's not drawing class diagrams, and it's not laying out architecture diagrams — it's the translation work of "how does this principle take shape under our business / codebase / constraints?" That's where each organization's distinctiveness lives, and that's the value that can't be copied.

Said the other way: another organization copying cortex's surface doesn't reproduce the substance. What gets asked of every org is how it translates this principle into its own use cases.

Trial and Error That Got Me Here

Everything I've written about above is the form that ended up working. Getting to that form involved a lot of throwing away. Two representative examples worth keeping on record, plus one shorter one.

I Spent Two Months on Static-Analysis code-graph, Then Threw It Out

The first thing I built was static-analysis-based code-graph: extracting AST data — imports, call graphs, type dependencies — and putting that into a graph DB. At a glance, the obvious implementation of "make AI understand the codebase."

Why two months? code-graph wasn't just cortex; it spanned our consumer-facing services and internal-system repositories too — over 40 repos in total (cortex being one of them). The mechanically-extractable AST data (imports / call graphs / type dependencies) was usable as-is via tree-sitter, but each repo had its own API endpoints / DB schema / event definitions / Pub/Sub topology, and extracting those boundary nodes (where an app meets the outside) goes beyond mechanical AST analysis and had to be implemented per-repo-type — that's where the time went.

So I spent two months out of the first three on this, and got something that worked end-to-end.

And then I threw it away.

Why: static analysis is great at capturing structure, but it can't traverse on intent or business context. Concretely, three things broke:

No semantic entry point for search — if I want to query the codebase with "show me the function calculating member subscription billing," I can't get there unless I already know the function name or file. A graph built only from static analysis has no semantic-tag entry pointing to "what is this code for?"
The graph contains only code — internal helpers / utilities / types / arguments all become nodes, so traversal from any function blows up within a few hops, dragging in helpers and primitives. There's no axis to filter on semantic relatedness
What I actually wanted was code + DB schema + docs + infra on one graph — given a function, I want to pull, in one query, the DB tables it touches, the docs where the design lives, and the linked business requirement. A code-only graph just can't do that

→ I switched to the annotation-based approach (@graph-* JSDoc tags write the business intent into the code, and that gets unified with DB schema / docs / infra into one graph). Searchable semantically, and when you traverse, only related stuff comes back. That's the current product-graph (cpg). Don't drag sunk cost forward and you'll get to the final form — discarding two months of investment instead of trying to recoup it was the foundation for everything that came after.

Setting Coverage 90% as a Solo Target Broke the Implementations

Test coverage is still gated at 90%+ (as covered in Part 3). That part hasn't changed. But there was a period when Coverage was treated as a standalone target, and during that period the implementation visibly got worse.

Specifically:

Heavy default-value use that hides branches: function(input = {}) style writes the missing-input branch out of the test path. Coverage goes up, protection against unexpected input is gone
Catch-and-swallow over throw: try / catch returning null. Don't throw → no need to test "doesn't throw," and Coverage is satisfied. Invalid state silently propagates
Early returns that flatten too much: dump complex conditions through an "early return" escape. Tests pass; what should have been validation just isn't there anymore

Result: Coverage 90%, quality lower than before. When you look at Coverage alone, the shortest path to "satisfy it" is a weaker implementation that passes the tests.

Two lessons:

Set a number as a target, and the number becomes the goal. Coverage is a "minimum floor" — not "a goal to hit"
Don't evaluate any metric alone. Coverage has to be evaluated alongside responsibility separation / exception design / boundary value coverage / etc.

Then, as the follow-up: I added linting that mechanically closes off the routes that let you weaken implementations to satisfy Coverage. Two specific examples:

no-silent-catch: AST-level ban on empty catch and silent-handler patterns like .catch(() => null). Catch bodies have to have a function call (logger included) / re-throw / new / await — otherwise it's an error. Catches the "weaken throws to satisfy Coverage but lose observability in production" pattern structurally. The violation message routes you to @cortex/otel/logger for structured logging, so the chain through Cloud Run OTel → Loki / Grafana stays intact
vitest-strong-matchers: bans weak matchers like toBeTruthy / toBeDefined / toContain / toBe(true|false) / expect.any / expect.objectContaining. Catches "any assertion that passes" patterns at the AST level, and points you instead toward toStrictEqual / toMatchInlineSnapshot that pin down the full output. This is one notch above Coverage — a test quality concern — but it lines up because the same reflection applies: don't let a number become the goal

On top of that, cortex's testing guideline opens with "Coverage is not the goal, just a supporting indicator," and threshold-lowering / istanbul ignore workarounds get bounced as Critical in Auto Review. So even when Coverage is satisfied, "this is intentionally deleting a branch" / "this is swallowing the exception" comes back as a Major finding.

From the lesson "a single metric warps implementation," we descended through guideline that states the principle → lint that mechanically rejects → Auto Review that evaluates as a dimension before Coverage 90% finally functioned as the "minimum floor" it should have been all along. This too sits in the lineage of the Recurrence Prevention mechanism from Part 4 (so the same trap can't be stepped on twice).

Parallel Sub-Agent → Sequential Evaluation

Third: an internal-structure call about Auto Review. Distribute the 9 dimensions to parallel sub-agents and evaluate concurrently — the plausible-looking design ("parallel = faster, parallel should also hold quality") I tried first and ended up throwing out.

What actually happened: time, cost, and accuracy all got worse.

Time got worse: each sub-agent has its own startup, its own context load, its own result aggregation overhead. "9-way parallel = 9x faster" didn't hold; there were even cases where sequential evaluation in one session ended up faster
Cost got worse: each sub-agent loads PR diff + guidelines + related code independently — common context loads ran 9 times. Token consumption measured at just under 4x — not the naive 9x (the context other than diff is shared across many dimensions, which is what kept it from blowing up to a full 9x)
Accuracy didn't hold: parallel sub-agents don't see each other's verdicts, so the same problem comes back as "APPROVE" from one and "REQUEST_CHANGES" from another. Duplicate findings show up too. Without a "what kind of PR is this as a whole?" pass to anchor on, dimensional findings drift toward local optima and the overall picture gets worse

Switching to sequential evaluation: same session goes through 9 dimensions in sequence, so context loads once, and each dimension's call has the previous dimension's verdict in front of it. All three — time, cost, accuracy — improve simultaneously.

Of course, sequential evaluation introduces order dependence between dimensions — earlier verdicts can shape later ones. That's a real trade-off, and I accepted it knowingly. Inter-dimension consistency at the cost of some order sensitivity is more useful as a 9-dimension review than fully independent dimensions that contradict each other.

The takeaway: the distributed-systems intuition that "parallel = faster, parallel = quality holds" breaks its own assumptions in an AI harness. Unlike parallelizing across CPU cores on your machine, with AI the context isn't shared memory; it's per-process state. Sequential evaluation in one session ends up better on speed, token efficiency, and inter-dimension consistency at the same time — a structural property that's easy to miss at design time.

What This Section Is Really Saying

The form I've described across the series is the result of a lot of trial and error. Not starting with the right answer and laying it out from there. The decisions of throwing things away with sunk costs included, the trap of letting a metric I chose turn into the goal, the distributed setup that looked natural but worked against me — those are the things I walked through before landing at the current shape.

Not easy. I don't pretend it was. But if you do walk through it, real results follow — that's the honest read on it now.

Closing the Series

What I most wanted to communicate across these six posts comes down to one thing:

AI coding is not about "how to use AI" — it's about designing the environment AI runs in.

Or, put another way:

AI isn't something to trust. It's something to design.

Assuming a large codebase: prompt engineering / model selection / tool selection — each matters individually, but polishing them alone doesn't get you to auto-merging PRs, auto-healing incidents, or non-engineer development. Getting there requires building a codebase / business flow / observability / repair cycle where AI doesn't need to infer. That's not an individual AI skill — that's an environment-design problem (conversely, for a small project of a few dozen files, today's AI models work fine standalone. Harnesses become essential when scale exceeds what one person can hold in their head.).

And the conviction at the root of environment design is, repeating myself, "I don't trust AI to fill in the blanks for me" — looking the reality in the face that context that wasn't handed over isn't known, and the ideal state doesn't happen without being told. Once you accept that premise, what to build clarifies naturally.

Looking back, four decisions ended up being the ones that mattered:

Locked the conviction first: putting words to the root ("AI isn't something to trust") gave priority order to every mechanism. If I'd started from technique, I don't think I'd have made it to the current form
Invested with throwing-out as the default: like I did with code-graph at the two-month mark, I went into things with "throwing this out is OK." Drag sunk cost forward and you can't move forward
Refused standalone numerical targets: the moment a metric like Coverage 90% becomes the goal on its own, implementations warp. Designed the system so it gets evaluated alongside other dimensions
Designed for "no inference," not around AI's capability: I prioritized building structure where AI doesn't have to infer, instead of relying on what AI can do. That's what made the system stable end-to-end, I think

If even one of these is useful to someone starting on something similar, that would be great.

Afterword — Where Engineering Careers Are Heading

Slipping off the wrap-up topic — this is something I've been turning over recently, written here in a "loosely held thought" tone, so feel free to skim.

As harnesses mature, I think engineering work splits along two directions.

One direction is value creation from problem identification and business design. In the world of Part 5 — where non-engineer PRs work — "writing code" stops being scarce, and the actual scarce thing becomes the ability to define what to build. The person closest to the requirements (a PMO, a business manager, a domain-deep engineer) ends up driving Claude Code through to the merged PR themselves. This direction looks less like "engineer" and more like a business designer who moves between domain and implementation.

The other direction is building the foundation that lets all of that happen safely and quickly. Non-engineers can open PRs to the production repo only because the harness underneath holds quality — knowledge graph, Auto Review, Self-Healing, Recurrence Prevention, lint, CI, tests, observability stack, all interlocked. Designing / maintaining / evolving that gets harder, not easier. As the house-builder side, rail-layer side, this demands deep infra understanding / security instinct / observability design / a feel for AI's architectural quirks.

I'm building cortex, so I'm spending more time on the latter; building "a foundation where the business can run its own changes" is genuinely fun for me. That said, I'm not the type who fully commits to one side — I move between listening to business questions and assembling the foundation, and the satisfaction from each is its own kind. This isn't a "which is more important?" question — the harness exists precisely so the former is possible, and the former being alive is what gives the latter meaning. They're mutually dependent.

Maybe the era of polishing just "coding ability" is shifting slightly. Where to put your value — or whether to move between both directions — becomes a question more engineers will need to choose into intentionally.

Six posts in, thanks for sticking with me to the end.

#	Theme	Key scene	Article
1	Series intro: cortex's harness	PRs auto-merge / incidents self-heal before you notice	ai-harness-intro
2	Product Graph (cpg)	Code, docs, DB, infra unified into one graph	cortex-product-graph
3	AI PR review	webhook → AI review → auto-fix → squash merge	cortex-auto-review
4	Self-Healing + observability + auto-added guardrails	Alert → AI investigates → fix PR + new lint/type gate → auto redeploy	cortex-self-healing
5	Democratizing the maintenance phase	Domain experts open PRs to production; the harness owns the quality gate	cortex-non-engineer-prs
6	Series Final	The underlying philosophy plus a retrospective on the failures and lessons	This post

Part 5 ("The Author Doesn't Have to Be an Engineer") has been generating sharp comments. Worth a read for the thread alone.

Ryosuke Tsuji — Thu, 11 Jun 2026 01:06:48 +0000

Self-healing guardrails for business-side PRs

Ryosuke Tsuji

Jun 8

The Author Doesn't Have to Be an Engineer: How the Harness Holds Quality (Series Part 5)

#ai #devops #engineering #github

16 min read

The Author Doesn't Have to Be an Engineer: How the Harness Holds Quality (Series Part 5)

Ryosuke Tsuji — Mon, 08 Jun 2026 23:32:30 +0000

Hi, I'm Ryan, CTO at airCloset.

Disclaimer: "cortex" in this article is the internal codename for an AI platform built in-house at airCloset. It is unrelated to existing commercial services like Snowflake Cortex or Palo Alto Networks Cortex.

In Part 1 (Series Intro), I wrote about how cortex's harness has matured to the point where non-engineers (business-side managers, PMOs, and the like) can open PRs to the production repository. The harness here is the runtime foundation for AI in production -- the combination of the knowledge graph, Auto Review, Self-Healing, and Recurrence Prevention covered across Parts 1 through 4.

Part 5 is what comes next: that harness has now reached the layer of who actually writes the code.

"Surely an engineer is still checking afterward, right?" -- I expect a lot of readers will land here with that question. So this post leads with one concrete example before anything else.

Part 5 covers:

What kinds of PRs are actually shipping -- two recent ones in detail
What works and what doesn't -- the boundary between adding on top of an existing stack and standing up new infrastructure
Why this holds for non-engineers -- how the four mechanisms from Parts 1-4 carry it
What's next -- into toC services -- the direction of travel for consumer-facing scale

The deeper toC implementation story will live in a separate post; here you'll get the framing and the direction.

Series

#	Theme	Key scene	Article
1	Series intro: cortex harness	PRs merging unattended / incidents fixed before anyone notices	ai-harness-intro
2	Product Graph (cpg)	Code / docs / DB / infra unified into one graph	cortex-product-graph
3	Auto PR review	webhook -> AI review -> auto-fix -> squash merge	cortex-auto-review
4	Self-Healing + observability + auto-added guardrails	Alert -> AI investigates -> fix PR + new lint/type gate -> auto redeploy + same pattern auto-rejected from then on	cortex-self-healing
5	Democratizing the maintenance phase	Domain experts open PRs to production; the harness owns the quality gate	This article ← you are here
6	Series Final	The underlying philosophy plus a retrospective on the failures and lessons	cortex-philosophy

Start with one scene

A +1,742 line / 41 file PR lands on the internal dashboard web app. Title: "PL dashboard ver.2". The change opens up project visibility to managers and team leads across multiple business units, scoping what each person sees to their own division or team. It adds an SSoT in the shared types package, new routes on the API server with SQL involving INNER JOIN and LEFT JOIN, new pages and view-state on the web app, and a personal-settings surface -- the whole stack of things you'd expect for a real feature.

The point is, this isn't a typo fix or a string swap. Entities, repositories, API routes, screens, filters, personal settings -- every layer you'd normally touch for a feature got touched. A few days of work for an experienced engineer, in scale terms.

The review-fix cycle ran like this:

PR open (+1,742 / 41 files)
auto-review pass 1: Major finding (a permission-scope fall-through -- data from other divisions leaking into the view that shouldn't be there) plus a handful of Minor items
author bot push: closes the scope fall-through, addresses the Minor items
auto-review pass 2: Nit items remaining, plus a lint catch (no-empty-function)
author bot push: lint clean
auto-review pass 3: still some COMMENTED nits, not yet APPROVE
author bot push (iteration 2): hardens loading skeleton, reverts an unnecessary JSDoc tweak
auto-review pass 4: APPROVED → CI green + APPROVE both met → auto-merge → production

From PR open to merge: four review-fix rounds, three author-bot pushes, zero human reviewers in the loop. The reviews come from the auto-review bot, the fixes come from an author bot (an automated review-response agent that the PR author has running on their machine), the final APPROVE is submitted by the AI, and an auto-merge script picks it up the instant CI is green. Production lands with 56/56 shared type checks (SSoT), 2,284/2,284 API tests, 1,113/1,113 web specs, and 0 lint errors. (cortex splits the lint job between oxlint for general checks and a custom eslint plugin for the @graph-* rules.)

The second review pass is worth noting. "Scope fall-through" is a somewhat technical finding -- a hole in the permission filter meant data from divisions other than your own could leak into the view. This is an internal dashboard, so it's not an external-leak incident, but "only see what's relevant to you" is the whole point of a dashboard like this -- losing it doesn't just risk an information slip, it drowns the user in noise that they shouldn't be filtering through in the first place. That's the kind of issue that's easy to merge by mistake and painful to notice in production. The fact that auto-review caught it on pass one and bounced it back for the author side to fix is what makes this whole flow viable for non-engineers. Without that loop, a PR of this size from a non-engineer would be a bad bet.

And: the author of this PR is not an engineer. A business-side teammate handed a feature description to Claude Code, leaned on the knowledge graph (covered in Part 2) to pull in the relevant existing code, and the +1,742 line PR is what came back. The four review-fix rounds above are what happened next.

That setup lines up directly with the central claim of this post: the person who knows the business requirements best, instead of organizing them and handing them to an engineer, runs them through Claude Code to production themselves.

Quick clarification on "write." When I say "write" in this article, I don't mean typing line by line in an editor. I mean handing the business requirements to Claude Code, judging the resulting diffs and AI review comments with domain knowledge, and seeing it through to a production merge -- the whole arc. Most of the actual diff is written by Claude Code; review feedback is handled by the author bot. What the human does is three things: put what they want into words, make the judgment calls along the way ("does this fit, is this off"), and sign off when it's ready to merge. None of that is implementation work in the technical sense. That's what "write" means here.

There's still a learning curve, of course -- the prompts you give Claude Code, where to point it for context. But none of that is learning to program. What you need is the ability to articulate what you want clearly, not syntax or framework knowledge.

The harness covers quality, so even at +1,742 lines / 41 files, this works.

Out of scope: this post does not cover the path where non-engineers freely ship apps to a sandbox environment instead of opening PRs against the production repo. That's a different mechanism, covered in an earlier post: Bridging "I Want to Build" and "I Want to Publish Safely" for Non-Engineers with a Custom Sandbox MCP. This post is specifically about opening PRs against the production repo -- the front door that's traditionally been engineer-only.

When you need a change, you can make it yourself

The point of the previous section is this:

When you need a change, you make it yourself, without flagging an engineer.

When that holds, work like:

"I want a new metric on the dashboard"
"The aggregation filter doesn't match how the business actually operates"
"I want a small business-support feature embedded in the production app"

stops queueing behind whatever an engineer is in the middle of. The fix lands when the need lands.

Think about the old flow. Someone on the business side notices a small thing that needs to change. They write the requirements up. They open a ticket or a Slack thread for an engineer. The engineer is in the middle of something else, so it queues. When they finally get to it, the interpretation drifts from what the business actually meant, there's a back-and-forth, a review pass, and only then does it ship. Even a small change takes days to a week in wall-clock time.

That's the cost of a translation layer between business understanding and code, and it gets worse the busier the engineer is. The business's improvement cycle ends up paced by engineering's backlog.

When the person who knows the requirements writes the change themselves, that translation layer and that queue both disappear.

Here are two recent examples of that working.

Two non-engineer PRs that recently shipped

PR	Kind	Size	What changed
#1573	Deep bug fix	+348 -177 / 7 files	The dashboard's actuals number was unfairly exceeding the target. Root cause: the "which teams to aggregate" definition was asymmetric between target side and actuals side. Fix lifts the shared "teams to include" list into its own file and points both sides at it. Tests added too.
#1557	Feature build on top of existing stack	+1,742 -227 / 41 files	The PL dashboard v2 from the opening scene. Entities, repositories, API, UI -- all touched, but the web app itself (the stack) was already standing; this rides on top.

Different shapes, but both are non-engineer PRs that made it all the way to merge.

#1573 -- a deep root-cause fix

This started from "the number looks wrong" on the business side, and the PR went all the way down to a data-integrity issue. The surface symptom: "the actuals number on the dashboard exceeds the monthly target, with the achievement reading 101% even though the team knows that's not real." The lazy fix would be a fudge factor or a clamp on the display. That's not what happened.

The author dug into the aggregation queries and pinned the real cause: the actuals side and the target side were reading from different tables, and the definition of "which teams count" wasn't symmetric between them. Teams that don't carry a target value (designers, PMOs, and so on) didn't show up on the target side but were getting counted on the actuals side, so the numerator was inflated against the denominator.

The fix is structural, not cosmetic. A single file defines "the teams in scope for this aggregation" as a shared list, and both sides reference it. No future drift between target-side and actuals-side definitions -- it's locked in by the shared constant.

The handling of "what data falls out of an aggregation" and "are the target and actuals sides really symmetric" is the kind of thing engineers miss too. A non-engineer working through it down to the structural level and fixing it there is what stands out about this PR.

#1557 -- a big feature build on top of an existing stack

This is the PR the opening scene walked through. +1,742 / 41 files spanning entity, repository, API, and UI -- a scale of change that's well past what people usually mean when they say "modification."

What lets a non-engineer ship this size of change is that the web app itself (the stack) is already standing. Nobody's standing up a new app, no new Cloud Run service needed defining, no new dependency packages, no new directory structure. The change adds a route, a page, and a repository entry inside the existing structure that's already there. It rides on what's been built.

This is the "on top of an existing stack" range. That's where the boundary is, and the next section spells it out.

Note on terminology: "modification" in this article is broader than "small tweaks to existing logic." It includes adding new entities, new endpoints, and new pages on top of an existing stack. The line I'm drawing is between building on top of a stack vs. standing the stack up in the first place.

What works, what doesn't

The principle: standing up a stack is hard, building on top of one isn't

The cleanest dividing line for non-engineer development isn't "modification vs. new development." It's "on top of an existing stack" vs. "stand up a new stack."

Standing up a new stack (work that starts from infrastructure: a new web app from scratch, a new Cloud Run service defined from a Dockerfile, a brand-new BigQuery pipeline) → engineering work
Adding to an existing stack (a new page in an app that already exists, a new endpoint on an existing API, a new data source on an existing pipeline) → non-engineers can do this

All three of the example PRs above sit on the second side. The stack itself was already built (by me, for the most part), so they get to work inside it. "Stand up a new app from scratch" or "define infrastructure (Dockerfile / IaC) from zero" are still engineer territory.

Put another way: renovations and new rooms inside an existing house are open to anyone. Building the house itself is engineering. Get the structure wrong -- the load-bearing parts, the wiring, the plumbing -- and the cost of recovery is high. That's the part of stack design where there's still too much "if this is wrong, everything downstream breaks" risk to hand to AI.

What's left for engineers: laying the rails -- the stack and the harness itself

The flip side: the rail-laying work -- standing up a stack, and extending the harness itself -- is what non-engineers don't touch yet. Both require a different kind of knowledge:

Infrastructure: containers, IaC, the operational characteristics of cloud services. Cloud Run resource ceilings, cold starts, Pub/Sub at-least-once semantics, BigQuery partition / cluster design, how Pulumi stacks split. Get this wrong and a thing that compiles can still fall over in production
Authentication for external integrations: OAuth, webhooks, how you handle API keys and where they sit in Secret Manager. One small slip leaks credentials into the repo or lets a webhook fire something you didn't intend
Security fundamentals: what to never expose, where to sanitize, where the privilege boundary cuts. SQL injection, XSS, SSRF, broken authorization -- "it works" isn't enough here
Harness design and extension: adding a new Auto Review dimension, changing Self-Healing logic, writing a new lint rule (e.g. in eslint-plugin-graph), structuring guidelines. Decisions that require understanding how the whole flywheel hangs together -- the most meta layer

That last bullet -- harness extension -- has an important implication: for non-engineers to keep being able to ship to production, someone has to keep the harness evolving. Recurrence Prevention (Part 4) is the automatic loop that adds lint / CI guards / guidelines per trap. But the architecture of the harness itself -- the structure of dimensions, the calibration of judgment, the design of the Self-Healing flow, the shape of the knowledge graph -- those are a meta layer that still requires engineering judgment.

Concrete case: the current nine Auto Review dimensions ([Graph] / [Architecture] / [Security] / [Test] / [Doc] / [Impact] / [Observability] / [AI-Antipattern] / [Recurrence]) were designed by observing past incidents and fix patterns. When a tenth dimension becomes necessary -- say, a "breaking change check on dependency upgrades" axis -- decisions about responsibility splits with existing dimensions and where to set thresholds are made by looking at the whole structure. That's the kind of engineering work that stays.

The harness provides "rails you can't derail from." Laying those rails -- and laying the foundation those rails sit on -- is a different job, and it's still on engineering. Engineers lay the rails; anyone can run on them. That's the boundary today.

Why this works for non-engineers

This is a short recap, because everything that makes it work was already covered in Parts 1 through 4. Four mechanisms reinforcing each other -- that's what lets non-engineers operate safely on top of an existing stack.

① The knowledge graph pulls relevant code from "what you want to do"

cortex-product-graph from Part 2 -- the unified graph fusing code, docs, DB schema, and infrastructure into one knowledge base (implementation name: cpg) -- carries this layer.

Non-engineers don't need to know function names or repo structure. A natural-language question like "I want to add a metric column to the dashboard" goes to Claude Code, which hits the knowledge graph with a semantic search and gets back the relevant nodes -- the screen, the API, the DB, the docs -- in one or two hops. You can get started without knowing the technical vocabulary.

For #1557: the author told Claude Code "I want PL dashboard v2 with division/team scoping for non-PI-Div PMOs and team leads," and the knowledge graph pulled the existing /projects route, project-repository.ts, FilterHeaders.tsx, and ProjectTable.tsx as the relevant nodes. The author never needed to know what file to edit. That's how the translation layer drops out.

② Auto Review enforces quality at the gate

The 9-dimension automated review from Part 3 is the next layer. [Graph] / [Architecture] / [Security] / [Test] / [Doc] / [Impact] / [Observability] / [AI-Antipattern] / [Recurrence] -- the AI returns REQUEST_CHANGES on what's missing and loops with the author bot until APPROVE -- the four-round example from the opening scene is exactly this in motion.

The point is this: the first PR doesn't have to be perfect. The author doesn't need to ship a completed, security-hole-free version on the first try. Push the initial PR and the rest gets sorted by the auto-review and the author bot bouncing off each other. The reason the author bot doesn't spin off into a loop of confused fixes is that the knowledge graph holds the full codebase context: changes are made with structural awareness of what they touch, so misreadings of the review feedback don't compound.

③ Self-Healing catches what slips through to production

Part 4 covered Self-Healing. If something does break in production, the AI starts from the alert, investigates root cause, opens a fix PR, and gets it auto-redeployed -- the entire loop runs without humans. Incidents triggered by a non-engineer's change recover on their own, hands-off. That's what makes the bar to opening a PR feel survivable.

This isn't "non-engineers are safe because nothing can go wrong." It's "even if something goes wrong, the harness has it covered." The system is designed to minimize damage, not eliminate failure. The three-layer construction (Observation → Repair → Strengthening) from Part 4 is what makes that net real.

④ Recurrence Prevention keeps the trap count from growing

The Recurrence Prevention loop from the back half of Part 4. Every trap that gets stepped on gets nailed down in the same PR, so the next attempt at the same pattern gets caught. The form depends: mechanizable traps become lint or CI guards; less-mechanizable ones become entries in the guideline docs (docs/gotchas, severity docs) that the AI reviewer reads. Either way, the catch happens before merge. Non-engineers contribute to this loop too -- when they hit a trap, the doc entry that prevents the next person from hitting it can come from them.

As this compounds, the rails get denser. Where there was once a loose "don't go that way" guideline, every incident adds another small rail saying "or this way, or this way, or this way," and the lane that's safe to walk gets clearer. The denser the rails, the safer non-engineers are in the lane.

→ The four pieces aren't independent components. Each one's output feeds the next one's input. This is the Guides + Sensors flywheel from Part 1 in action. I won't re-explain the details since they're in the prior posts, but non-engineers shipping to production is the result of all four wheels turning together. Take any one out and the level of upfront knowledge required to write to production jumps, and the whole thing collapses.

Next -- carrying the pattern to consumer-facing services

cortex is an internal AI platform, so the system as it stands can't be lifted into a toC production service as-is. The biggest issue is the difference in quality bar. For toC, "detect after user impact → Self-Healing fix" is too late. The requirement becomes: incidents don't happen, and when something is about to ship, there's review and testing on top of human sign-off.

That said, the shape of the harness -- a knowledge graph for context, 9-dimension AI review, an author bot responding to feedback -- carries over directly. The thing that changes is the final step: cortex's auto-merge becomes "AI does the prep, a human signs off." Not by giving up the AI's range, but by having the AI handle the heavy lifting (test writing, environment setup, test runs, the 9-dimension review) and leaving only the final APPROVE on a human. "If the human sign-off stays, engineer time doesn't really decrease, does it?" -- but historically engineers were spending the bulk of their time on the implementation, the test writing, the environment setup, the self-review, the back-and-forth on review. Sign-off itself is the smallest piece of that pie. With AI doing the prep work, what an engineer spends time on shifts from implementation labor to quality judgment.

A caveat on the knowledge graph: it only earns its keep at large codebase scale. If the codebase fits in one AI context window, a cross-repo graph is unnecessary. The reason cortex (100+ apps) and the toC side (40+ repos) need one is because the scale forces it.

The concrete plan is real (extending the knowledge graph across the toC side's 40+ repositories, designing the AI-prep flow, etc.), and the full version goes in a separate post.

Wrap-up

The person who knows the business requirements best, instead of writing them up for an engineer, runs them through to production directly. Quality is held by the harness, so what's required from the writer is domain knowledge and the ability to direct an AI well. Business asks stop queuing behind engineering, and the cycle speeds up
The four mechanisms from Parts 1-4 (knowledge graph / Auto Review / Self-Healing / Recurrence Prevention) form a reinforcing flywheel. The first PR doesn't have to be perfect, and what does break is repaired automatically. That's the design
The boundary: engineers lay the rails, anyone can run on them. Standing up the stack (infrastructure, authentication, security) and extending the harness itself (new lint rules, new review dimensions, Self-Healing flow design) stay on engineering
Carrying this to consumer-facing toC services, the knowledge graph (a 40+ repo cross-repo graph on the service side) covers the context layer, but the quality bar shifts, so auto-merge becomes "AI prep + human sign-off." Details in a separate post

In Part 6 I'll wrap the series with the philosophy at the foundation -- why this design, what got given up, what got kept. The series so far has been about "the parts that are working"; Part 6 puts the failures and the dead ends on the table too, including the gap between the philosophy and the actual implementation. A retrospective for myself, and -- I hope -- a reference for anyone heading down a similar road.

Fixed Before Anyone Notices, Stronger After Every Fix: Self-Healing + Recurrence Prevention (Series Part 4)

Ryosuke Tsuji — Mon, 01 Jun 2026 23:57:25 +0000

Hi, I'm Ryan, CTO at airCloset.

Disclaimer: "cortex" in this article is the internal codename for an AI platform built in-house at airCloset. It is unrelated to existing commercial services like Snowflake Cortex or Palo Alto Networks Cortex.

In Part 3 I covered AI reviewing AI PRs -- the auto-review pipeline that defends quality at the PR stage.

This post is the other side: defending quality in production, via Self-Healing. A production alert fires, an AI investigates it, opens a fix PR, the PR goes through the same auto-review pipeline from Part 3, gets auto-merged and auto-redeployed. And the same fix PR is required to add a new Guide -- whether that's a lint rule, CI guard, type constraint, or guideline update -- so the same anti-pattern gets auto-rejected from then on. The guardrails grow every time.

"Incidents get fixed automatically" is catchy on its own, but on its own it's probably not enough in the long run. You have to close the recurrence class while you fix the incident -- self-healing plus self-strengthening -- before the quality gates start to compound over time.

Start with last month's numbers

115 Self-Healing PRs merged in the past 30 days.

Effectively all of them merged and deployed without human involvement.

Humans only step in when the AI judges "this is not something code can fix."

That's the current state of "incident response" at cortex.

Don't read "115 = 115 user-impacting incidents" though. Roughly:

About half (54) are Deploy Failed-style alerts -- CI / Pulumi deploy step caught a failure, the AI absorbed it before it shipped to production. Recently the [Recurrence] loop (covered later) has been piling up countermeasures here, so this bucket is trending down anecdotally
The remaining 61 are production-runtime alerts (Service Error Log Detected / Pipeline Failure / Generator Failure etc.) -- the service is running in production, but an error-log threshold or consecutive-failure threshold tripped. The AI absorbed them before they propagated to user impact

So it's less "incident response" than "production anomalies that monitoring caught, fixed 115 times by AI before anyone woke up." The number of incidents humans actually have to acknowledge is in the low single digits per month.

There's also a clear pattern of the same service firing repeatedly (e.g. gcs-transformer is 25 of the 61) -- which is exactly what the [Recurrence] loop covered later is supposed to eliminate by turning into lint or type gates. That's the back half of this post.

One more honest note: the recent month's number is slightly inflated. The codebase had a fair number of "silent catch" patterns -- catch blocks that swallow exceptions without logging anything. We added the no-silent-catch lint rule and swept the existing silent catches in batches, which exposed previously hidden production errors as alerts. So part of the spike is "monitoring caught up to reality." Once the [Recurrence] loop converts these into lint over time, the number should converge. "Things we couldn't see, we can see now" is a quality improvement -- what we're seeing is the catch-up phase.

One more thing worth saying: doing this by hand is utterly unsustainable. Running 115 manual cycles of "ack alert -> read logs -> context switch -> understand the code -> fix -> open PR -> review -> deploy" would bankrupt any team's engineering bandwidth. The system absorbs them without anyone noticing, and converts the fix into a new Guide (lint / CI guard / type constraint / guideline) at the same time -- that's the actual subject of this post.

The moment an alert fires, the AI starts an investigation, traces Loki / Product Graph / git blame to root cause, opens a fix PR, runs it through the auto-review from Part 3, APPROVE -> auto-merge -> auto-redeploy. One full loop.

Series

#	Theme	Key scene	Article
1	Series intro: cortex harness	PRs merging unattended / incidents fixed before anyone notices	ai-harness-intro
2	Product Graph (cpg)	Code / docs / DB / infra unified into one graph	cortex-product-graph
3	Auto PR review	webhook -> AI review -> auto-fix -> squash merge	cortex-auto-review
4	Self-Healing + observability + auto-added guardrails	Alert -> AI investigates -> fix PR + new lint/type gate -> auto redeploy + same pattern auto-rejected from then on	This article ← you are here
5	Democratizing the maintenance phase	Domain experts open PRs to production; the harness owns the quality gate	cortex-non-engineer-prs
6	Series Final	The underlying philosophy plus a retrospective on the failures and lessons	cortex-philosophy

Big picture -- the three layers: Observation, Repair, Strengthening

For Self-Healing to work, you need an Observation layer in front and a Strengthening layer (recurrence prevention) behind it. Self-Healing itself is the middle Repair layer. The "self-healing + self-strengthening" loop only spins up when all three are in place.

Prerequisites: The three layers only stand up on top of two prior pieces: cpg (the unified code / docs / DB / infra knowledge graph from Part 2) and the Observability stack covered in this post.

No Observability -> the observation layer is empty, nothing gets detected -> the repair layer never even fires

No cpg -> the AI cannot see "where else does this trap exist" -> the repair layer does symptom-level patching at best, and the strengthening layer's horizontal expansion stops working

Put differently: trying to copy this setup without those two will just multiply incidents. An AI that blindly looks at error logs and rewrites production code is just speeding up the rate at which gh pr create ships accidents. cpg and Observability are the minimum bar for being able to delegate auto-repair to AI.

Note also that cortex is a several-hundred-thousand-line codebase, and at that scale loading the whole codebase as AI context is impossible for the AI as well (let alone for a human). Tell the AI to trace impact with just grep and file reads, and it'll run out of context window before it finds anything. cpg is what lets it ask "which other code does this function's change ripple into" and get the answer in one hop. Small repos may not need this. Past a certain scale, cpg is not optional, it's required.

In Fowler's Guides / Sensors terms from Part 1, cpg and Observability are the substrate that supports both Guides (pre-execution controls like lint) and Sensors (post-execution gates like auto-review and Self-Healing). Observability feeds Sensors via firing alerts; cpg feeds the Guides side by supplying the auto-review with impact-scoping context. Neither belongs on one side only -- they're foundational to both, and Self-Healing and auto-review only function on top of this substrate. That's the structural claim this post is built around.

Repair -> Strengthening loop">

Layer	Role	Key components
Observation	Real-time detection of production anomalies	OTel SDK / Loki / Mimir / Tempo / Faro / Grafana / Pino logs with trace_id
Repair	AI receives the alert, investigates root cause, opens a fix PR, auto-review, auto-merge, auto-redeploy	Event Relay -> SSE -> `self-healing` mode script -> claude -p (worktree) -> gh pr create
Strengthening	The fix PR is required to add a new Guide (lint / CI guard / type constraint / guideline). The same anti-pattern can't reach production again	`@cortex/eslint-plugin-graph` (26 rules), `scripts/check-*.ts` (13 guards), `recurrence-prevention.md`, the `[Recurrence]` lens of auto-review

I'll walk through them in order.

Observation -- where do the alerts come from?

cortex's production observability is built on Grafana Cloud + OpenTelemetry:

OTel SDK (the shared @cortex/otel package) -- every service calls initOtel({ serviceName }) at its entry point. Trace / metric / log all go out via OTLP to Grafana Cloud
Loki (logs) -- Pino structured logs get trace_id automatically. trace and log are cross-referenced
Mimir (metrics) -- Cloud Run / pipeline / Gemini API token usage, etc.
Tempo (traces) -- distributed tracing
Faro (frontend) -- captures browser JS errors / performance / network failures
Grafana -- dashboards + Alert Rules + Notification Policy

We also have a strict definition of log levels, anchored on business impact:

Level	Definition	Examples
`warn`	Business-foreseeable, does not need immediate action (retryable / self-recovers).	Search query returned 0 results, optional field unset, short retry due to rate limit
`error`	Data recovery / re-run will definitely be needed afterward. Impact expected to be under 20%.	"User record that should exist isn't there," BigQuery insert failure, per-record enrichment failure
`fatal`	The feature as a whole fails for 20%+ of requests. Service-continuity broken, fatal config missing, full upstream outage.	OTel init failure, required secret missing at startup, full input data source outage for a pipeline

The key point is to not pick the level mechanically based on the exception class name like NotFoundError. Same "record not found" situation: "this record must exist and doesn't" is error / fatal; "user search returned 0 hits" is warn. The level is decided by business impact -- "does this require data recovery later," "is the whole feature down" -- not by the type. Without this discipline you simultaneously get monitoring fatigue and missed critical incidents. Self-Healing reacts mainly to error-threshold trips; fatal is the human-escalation side.

Alert Rules are managed declaratively in Pulumi, grouped by service into categories like BOT / Pipeline / Transformer / Generator / Gemini / CI / Deploy / Service Catch-All. When we add a new service, one line in infra code spins up the dashboards and alerts automatically.

This is "the infrastructure that lets the AI see the same things humans see." Self-Healing picks up alerts coming off this stack.

What Observability can't catch, Self-Healing can't fix either

Honest disclaimer: Self-Healing can only react to what the observation layer can detect as an anomaly. "Observability is everything" is literally true here.

What the current stack catches is roughly logic-level errors -- exceptions, error logs, deploy failures, external-API call failures, threshold-based metric anomalies.

What it doesn't catch:

UI errors -- the logic ran, no error logs, but the screen shows something different from intent / shows the wrong value. Faro catches client-side JS exceptions and network failures, but "the logic ran and the output is just wrong" never fires an alert
Silent data corruption -- aggregated values slowly drift, bad values get into a table. Unless it crosses a threshold or schema check, nothing detects it
Perceived UX degradation -- requests feel slow, the UX feels off. Only catchable once SLO / latency thresholds trip

So Self-Healing is "AI replacing the human in the loop for incidents the observation layer can catch." The coverage of the observation layer itself is the prerequisite. Holes in observation stay as blind spots that neither auto-review nor Self-Healing reaches.

This isn't really a limitation of Self-Healing -- it's the importance of growing the observation stack, which cortex keeps investing in continuously. (From Part 1, Observability is one of the "supporting foundations" beneath the flywheel.)

Repair -- the Self-Healing flow

MODE=self-healing runs the same webhook-server script as the auto-review setup from Part 3, but listening for Grafana firing alerts.

The textual flow looks like:

[Grafana Alert Rule firing]
   ↓ POST /webhook/grafana
[Event Relay (in-house)] -- persisted in Firestore
   ↓ SSE push (event: grafana-alert)
[self-healing mode script]
   ↓ throttle check (same fingerprint skipped for 4h)
   ↓ 👀 reaction in Slack to signal "I'm on it"
   ↓ git worktree add -b hotfix/auto-alert-{service}-{ts} origin/main
   ↓ run claude -p inside the worktree
     - search related code via Product Graph MCP
     - pull error logs from Loki via Grafana MCP
     - identify root cause and fix
     - update tests as needed
     - conventional commit
   ↓ git push + gh pr create
[fix PR]
   ↓ auto-review (the Part 3 pipeline)
   ↓ APPROVE -> auto-merge -> auto-redeploy
[recovered]
   ↓ ✅ in the Slack thread

What happens when the AI judges "this is not fixable in code"

Not every alert is fixable by code. The implementation has a rule: "if you judge it unfixable, exit without changing anything." In that case Slack gets a notification of the form "This alert cannot be addressed in code. Investigation: ..." -- including what the AI investigated.

Worth clarifying on the numbers side: the headline 115 is "Self-Healing runs that reached PR-created -> merged -> deployed." This "unfixable, exit clean" case is a separate bucket, happening several times a month (external transient outages, infra / config issues that aren't code, cases too complex for the AI to judge confidently). The "humans show up" bucket is this separate one -- it isn't a "some of the 115 failed" failure rate.

Here's what a real Slack message looks like (a styling-pattern-watch-transformer case where the GitHub PAT had expired):

ℹ️ This alert cannot be addressed in code. Investigation:

Investigation summary

Checked the error logs in Loki for the past hour. The single error source for styling-pattern-watch-transformer right now is expiration / revocation of the GitHub PAT (Personal Access Token).

What I found

The BQ snapshot pipeline is healthy: 7 consecutive [BQ] Snapshot rows inserted successes. The earlier autodetect type-drift problem was resolved in an earlier PR that put SAFE_CAST(... AS INT64) in place.

The single ongoing error (multiple occurrences in the past hour): GithubUnauthorizedError: GitHub API /commits/master unauthorized (401): PAT likely expired/revoked -- the /check-drift handler returns 401 every time Cloud Scheduler invokes it.

Root cause

The GitHub PAT held in Secret Manager for this service has expired or been revoked. The code already classifies the 401 with a dedicated GithubUnauthorizedError and is already sending the "PAT needs rotation" notification to Slack. The detection / notification design on the code side is working as intended.

Resolution: exit without changes

This is an external-credential operational issue (PAT expiry), and what needs to happen is a human rotates the PAT in Secret Manager. Not the kind of incident code can resolve. Therefore: no code change, no commit, no PR.

Operator actions:

Add a new version of the target secret in GCP Console / Secret Manager with a valid PAT (repo scope, read access to the target repository)

No Cloud Run revision redeploy needed (secretKeyRef version:latest is referenced)

At this level of detail, "what was investigated, why code can't fix it, what the human should do" all come out in one Slack message. Open the thread and the operator can act immediately. The productivity gap vs. "alerts just forwarded blindly" is significant.

Deduplication

A throttle ensures the same fingerprint (Grafana's unique alert identifier) is not re-processed for 4 hours. Without this, alerts that fire again before the fix PR has merged would spawn another worktree, another fix PR, and so on -- an easy infinite loop.

We also permanently skip any alertname containing credential. Credential incidents carry leakage risk if the AI touches them, so they're explicitly escalated to humans.

Self-Healing and Part 3 auto-review -- "the fixer AI" and "the reviewer AI" are independent

This is the most consequential design choice of the agent setup, so calling it out explicitly.

PRs opened by Self-Healing are not special PRs, just fix PRs. They go through the Part 3 auto-review pipeline under exactly the same conditions -- the 9 lenses (Graph / Architecture / Security / Test / Doc / Impact / Observability / AI-Antipattern / Recurrence) get checked in order. Critical / Major findings -> REQUEST_CHANGES; Nit-only / no findings + CI green -> APPROVE -> auto-merge.

The important bit: this is not a monolithic "AI fixing AI" loop. The fixer-side AI and the reviewer-side AI are fully independent:

Different process, different session: the self-healing-mode AI and the reviewer-mode AI are launched as separate claude -p processes. They do not share context
Different input sources: the fixer builds the problem from Grafana alert + Loki + cpg. The reviewer judges from the PR diff + cpg + review guidelines
Different objectives: the fixer is optimizing for "stop the incident." The reviewer is judging "does this violate the 9 lenses or the severity contract?" A deliberate separation of concerns where the two roles' incentives are intentionally misaligned

As a result, PRs the fixer dashed off get blocked by the reviewer (REQUEST_CHANGES -> back to the fixer). The AI does not approve its own output. "Just-make-it-work" fixes don't get through.

This is the often-debated review-independence problem in LLM-agent operation, solved here in the obvious way: split the work across separate agents.

A concrete example: meet subscription's 409 ALREADY_EXISTS

Take the alert from the Google Meet recording auto-fetch service I covered in the Meeting Intelligence post. On 2026-05-21, Self-Healing opened a fix PR titled fix(meet-subscription-renewal): auto-fix for Service Error Log Detected.

The trigger error from Loki:

Workspace Events API request failed: 409 Conflict
"Subscription associated with the resource already exists."

How the AI investigated:

Pinned the error in Loki -- ran {service_name="meet-subscription-renewal"} | json | level=~"ERROR|error|Error" via Grafana MCP, picked up the Failed to renew Meet subscription stack trace
Traced the call path in Product Graph -- identified renewSubscriptions -> createMeetSubscription
Cross-referenced past PRs -- the "opposite-direction inconsistency" (name in Firestore but missing from Google = 404) had already been self-healed in another PR with patchMeetSubscriptionTtl -> null fallback. The current direction (still on Google's side but missing from Firestore = 409) was the gap
Verdict: "the same pattern may exist elsewhere" -- a [Recurrence] decision matrix "horizontal expansion required" case

Instead of a quick patch, it implemented the same-direction self-healing symmetrically to the opposite-direction fallback that was already there:

Made createMeetSubscription idempotent
If POST returns 409, extract the existing Subscription name from the response and call patchMeetSubscriptionTtl
The caller writes the return value back into Firestore, so the next renewal converges to the normal PATCH path (self-healing)
Per the existing graph/no-silent-catch lint, JSON.parse failures are also logger.warn + serializeError for structured logging
Three tests added

This is what "Self-Healing pushing all the way to root cause and rolling the fix out horizontally" looks like in practice. "Close the recurrence class, don't just suppress the symptom" (the spirit of recurrence-prevention.md) executed autonomously by the AI.

Strengthening -- Guides (lint + guidelines) grow automatically

This is the layer that keeps Self-Healing from being just "auto-repair."

In Fowler's Guides / Sensors terms from Part 1, the Strengthening layer is the place where Guides grow -- i.e. the pre-execution controls that prevent AI from deviating in the first place. cortex's Guides come in two flavors:

Machine-read Guides: lint / type / CI guard / coverage thresholds / Prettier -- enforced at commit / CI time
Human-and-AI-read Guides: guidelines like recurrence-prevention.md, severity.md, ai-antipattern.md, etc. -- used as decision criteria by auto-review

The 9 lenses, severity contract, and no-downgrade rules from Part 3 are the latter; the auto-added lints in Part 4 are the former. Together they form the Guides surface. Lints are "formalized guidelines," guidelines are "lints that haven't been formalized yet."

The Sensors side -- Self-Healing and auto-review -- grow these Guides every time they run:

Self-Healing's root-cause investigation finds "the same pattern exists elsewhere" -> demands horizontal expansion + a new lint (= new Guide)
Auto-review's [Recurrence] lens blocks PRs that fix without adding lint
Both depend on cpg to see impact scope across the codebase

cpg is what lets the AI ask "where else does this trap exist." Self-Healing and auto-review (= the Sensors side) share cpg as a substrate, and each run thickens Guides by one notch.

What happens every time Self-Healing runs (the recurrence-prevention-first flow)

Every fix PR Self-Healing opens is checked for [Recurrence] by auto-review. The decision matrix:

Situation	Required action	Form
Same trap stepped on 2+ times	Lint required (custom ESLint rule / type constraint / CI guard)	Machine (new Guide)
Pattern may exist elsewhere	Horizontal expansion required (cpg traversal for similar nodes, fix all of them in this PR)	Investigation + fix
Cannot be machine-checked but worth formalizing	Add to an existing guideline	Guideline entry
One-off, no value in formalization	Nothing (bug fix only)	--

When the "stepped on 2+ times" situation applies, the fix PR can't merge without a new lint included. So every Self-Healing run produces:

Horizontal expansion via cpg -- not just the immediate fix target, every similar node enumerated
A new Guide added in the same PR -- ESLint custom rule / type constraint / CI guard / guideline entry, one of the four
All existing violations cleared in the same PR -- no warn-as-deferral, error on first introduction
Auto-review -> auto-merge -> auto-redeploy -- the regular Part 3 pipeline
Going forward, writing the same pattern gets mechanically rejected by CI / lint -- the recurrence class is structurally closed

"Add the guard while you fix the bug" runs as a self-sustaining loop driven by Self-Healing.

"We'll do it later" and "introduce as `warn`" are banned

A couple of important contract clauses from the guidelines:

"Plan to lint later," "lint when we refactor," "another PR will handle this" -- all banned. If it can be addressed in this PR, it must be
"Existing violations remain, so introduce as warn and promote to error later" -- not accepted. This is deferral in disguise. The responsibility for the warn->error promotion goes nowhere and the rule rots
If you add a lint rule, fix all existing violations in the same PR and ship at error

These extend the no-downgrade rules from Part 3 -- preempting the typical escape hatches.

The "step on it, mechanize it" lineage

Custom Guides currently piled up in cortex:

graph/no-silent-catch (ESLint) -- the source of the "inflated number" mentioned in the intro. Bans catch blocks that swallow exceptions
Stacktrace-preservation guideline (codified as a Major violation in observability.md, caught by auto-review) -- forbids logger.error(err.message) style logs that drop the stack and keep only the message string. Forces the err field to hold serializeError(error) so name / message / stack are preserved as structured fields. Observability is everything here, so logs that drop stack info are treated as inherently broken
cortex-quality/require-fetch-timeout (oxlint -- a Rust-implemented JS/TS lint that runs ESLint-compatible rule sets, dozens of times faster than ESLint due to the Rust impl. cortex uses oxlint for the standard ruleset and ESLint for custom rules that need AST-level work) -- mandates signal: AbortSignal.timeout(...) on external fetch calls. Born from a case where a no-timeout fetch hung indefinitely and triggered a Cloud Tasks redelivery storm
graph/no-bq-string-timestamp-param (ESLint) -- from a case where passing TIMESTAMP as a string to a BigQuery query parameter NULLed the value out through a serializer bug and silently failed every INSERT
graph/require-firestore-ignore-undefined (ESLint) -- forces ignoreUndefinedProperties: true on new Firestore(). From a case where a single NULL row caused a 100% failure rate in a sync batch
check-otel-env-injection (CI guard) -- the recurrence prevention for the Cloud Run OTel env injection case below
TypeScript type tightening (type level) -- tighter function signatures, branded types for ID disambiguation, exhaustive discriminated unions, etc. Patterns that can't be lint-caught but are catchable at the type level get closed from the type side

These aren't textbook-learnable rules -- they're "stepped on once, then mechanized." The number of traps the organization has stepped on translates directly into the number of Guides piled up (across ESLint / oxlint / CI guard / types).

How does the AI write a lint rule without breaking it?

Three structural things keep this sane:

Existing rules are the template: packages/eslint-plugin-graph/src/rules/ already holds 26 custom rules, each as .ts + .test.ts pairs. New rules follow the same shape, so the AI never has to write the AST-walking boilerplate from scratch
Tests first: violation / pass fixtures go into .test.ts first, implementation fills in TDD-style. Coverage threshold (90% statements + branches) is gated by the Part 3 auto-review, so a lint without tests cannot merge
lint / type / CI guard sit in the same "mechanize" bucket: the decision matrix in recurrence-prevention.md groups lint / type constraint / CI guard together as the "lint-required" row, and leaves the choice within that bucket (write it as a lint? express it at the type level? add a separate CI guard?) to the AI based on how much AST work is involved and whether runtime semantics matter. Traps that need AST inspection but actually hinge on runtime behavior usually end up as a type constraint (branded type / discriminated union / signature tightening) rather than a custom lint

So "AI writes a lint rule" is supported by existing rule corpus + the test harness + the mechanize-bucket selection criteria -- three together. The path where the AI hand-rolls raw ESLint API and bricks something is structurally closed.

A concrete example: Cloud Run OTel env injection -> promoted to CI guard

Multiple services hit this trap: when a Cloud Run Service / Job is defined in Pulumi, forgetting to inject OTEL_EXPORTER_OTLP_ENDPOINT and GRAFANA_CLOUD_API_KEY via secretKeyRef causes OTel init to be skipped in production, no trace/log reaches Grafana, and incidents become silently invisible.

The normal response would be "we'll be more careful next time." At cortex:

Incident surfaces -> Self-Healing opens a fix PR (adds the env injection to the affected service)
Auto-review's [Recurrence] decides "same trap stepped on -> lint required"
The same PR adds scripts/check-otel-env-injection.ts (CI guard) -- mechanically asserts OTel env injection across all Cloud Run resource definitions under infra/
All other existing services get their env injection added in the same PR
Merge -> deploy -> any future write of the same kind gets rejected by CI

That's what "the guardrails grow every time Self-Healing runs" looks like in practice. The trap is "stepped on -> mechanically checked from then on."

Where Guides stand right now (in numbers)

Snapshot of cortex's Guide inventory:

Category	Count	Notes
Custom ESLint rules (`@cortex/eslint-plugin-graph`)	26	`no-silent-catch` / `require-firestore-ignore-undefined` / `no-bq-string-timestamp-param` etc.
CI guards (`scripts/check-*.ts`)	13	`check-otel-env-injection` / `check-cloudscheduler-oidctoken-audience` etc.
Standard oxlint rules (set to `error`)	183	Base config ships everything at error
TypeScript strict gates (baseline)	9	`strict` / `noImplicitAny` / `strictNullChecks` / `noUncheckedIndexedAccess` etc.
TypeScript type tightening (per-recurrence)	grows over time	branded type / discriminated union / function-signature tightening etc. Patterns that can't be lint-caught but can be type-caught are closed from the type side
Test coverage thresholds	statements + branches 90%	Uniform across all packages
Prettier	1 config	Format auto-fix
Guidelines	the entire review-guidelines repo	Used as the decision basis by auto-review

The first two categories plus the type-tightening row -- Custom ESLint, CI guard, type tightening -- are the part that compounds over time through the [Recurrence] lens every time Self-Healing or auto-review runs. The guardrails grow with time. That's the substance of the Strengthening layer.

The whole loop, from the top

When you compose the three layers:

[production anomaly] -> Observation layer (OTel/Loki/Grafana) -> Alert firing
                                              ↓
                                       Event Relay -> SSE
                                              ↓
[Self-Healing mode script]
   - claude -p in worktree
   - root cause via cpg + Loki + git blame
   - commit fix
   - (if applicable) add new lint / type gate too
   - gh pr create
                                              ↓
[Auto-review (Part 3)] -- 9 lenses in order, especially [Recurrence] forces
                         recurrence-prevention action (lint / horizontal expansion / guideline entry)
                                              ↓
                          APPROVE + CI green
                                              ↓
[auto-merge -> Turborepo build -> Pulumi parallel deploy]
                                              ↓
[production recovered + same anti-pattern mechanically rejected from now on]

The loop completes without human intervention. Not just repair, but the quality gates that grow with every repair -- that's the "auto-recovery + auto-strengthening" substance at cortex.

That said, as the front of the article spelled out, the loop is only viable because cpg and Observability exist. cpg makes horizontal expansion possible; Observability turns production anomalies into structured data. With those two in place at the foundation, AI can stand on the side that does Repair and Strengthening. Self-Healing is not a standalone mechanism. It's a Sensor riding on top of cortex's Guides (cpg + Observability + lint + guidelines). That's the single most important framing in this post.

Self-Healing by the numbers

Breaking the headline down further.

Main firing categories

What kicked off Self-Healing in the past 30 days (with the mapping back to the front-of-post 2 buckets):

Category	Bucket
Service Error Log Detected (most frequent)	Production-runtime (61 side)
Pipeline Failure -- data pipeline failing a configured number of times in a row	Production-runtime (61 side)
Generator Failure -- AI generation jobs (embedding / annotation etc.) failing	Production-runtime (61 side)
Deploy Failed -- deploy step failures (Pulumi up / Cloud Run revision failed)	Deploy step (54 side)

Alert-firing to production-recovery time

Median 30 minutes to 1 hour. Roughly:

Alert firing -> AI investigation start: under 1 minute (Event Relay + SSE)
AI investigation + fix + PR open: 3-8 minutes
Auto-review (including the Part 3 10.8 review-fix iterations on average): 20-45 minutes
Auto-merge + deploy: 3-10 minutes

Many of these finish before anyone wakes up (alert fires early morning -> by the time people come in, there's just a ✅ in Slack).

What changed / Bridge to Part 5

We've now covered the cortex picture across Parts 1-4:

Part 1: the cortex big picture and harness-engineering framing
Part 2: Product Graph (cpg) -- the AI's "brain"
Part 3: auto-review -- defending quality at the PR stage
Part 4 (this post): Self-Healing + Observability + auto-added guardrails -- defending quality in production while growing the quality gates themselves

The engineering role has shifted, over the last half-year, from "write, review, fix, merge, deploy, incident-respond" -- all of that -- toward looking at the whole system from above and tuning it. human-on-the-loop, working at the Policy layer.

Part 5 covers the harness reaching the "who writes the code" layer. The center of it is domain experts (business-side managers, PMOs — non-engineers) opening PRs to production, with a concrete walk-through of a +1,742 line / 41 file feature PR that landed with zero human reviewers in the loop. What guarantees the quality is the harness stack built across this series — "whoever writes, the harness owns the quality gate" is the Part 5 framing.

The toC service expansion gets a brief mention at the end for direction, but the full implementation discussion lives in a separate post.

The actual series wrap-up is Part 6. The center of it is the underlying philosophy -- why I picked this design, what I gave up, what I kept. Alongside that, since the series so far has been mostly "what's working," I want to look back at the failures and dead ends behind that surface, and the gap between the philosophy and the implementation. A retrospective for myself, and -- hopefully -- a reference for anyone starting down a similar path.

Human-on-the-Loop: AI Reviewing AI PRs at cortex -- 769 PRs/month while raising the quality bar (Series Part 3)

Ryosuke Tsuji — Tue, 26 May 2026 14:35:43 +0000

Hi, I'm Ryan, CTO at airCloset.

Disclaimer: "cortex" in this article is the internal codename for an AI platform built in-house at airCloset. It is unrelated to existing commercial services like Snowflake Cortex or Palo Alto Networks Cortex.

In Part 1 (intro) I covered the high level -- AI driving both PR reviews and incident response on top of cortex. In Part 2 (Product Graph) I went deep on cpg, the unified knowledge graph that fuses code, docs, DB schemas and infra into a single business-aware index.

This post is about the automated PR review pipeline -- AI reviews the PR, a separate AI applies the fixes, and the system merges automatically once policy gates pass. The usual critiques of AI-assisted development ("the reviewer becomes the bottleneck" and "AI code drops the quality bar") don't really apply here. The rest of this post unpacks why.

Series

#	Theme	Key scene	Article
1	Series intro: cortex harness	PRs merging unattended / incidents fixed before anyone notices	ai-harness-intro
2	Product Graph (cpg)	Code / docs / DB / infra unified into one graph	cortex-product-graph
3	Auto PR review	webhook -> AI review -> auto-fix -> squash merge	This article ← you are here
4	Self-Healing + observability + auto-added guardrails	Alert -> AI investigates -> fix PR + new lint/type gate -> auto redeploy + same-pattern writes get auto-rejected	cortex-self-healing
5	Democratizing the maintenance phase	Domain experts open PRs to production; the harness owns the quality gate	cortex-non-engineer-prs
6	Series Final	The underlying philosophy plus a retrospective on the failures and lessons	cortex-philosophy

Start with last month's numbers

769 PRs merged.

Median time to merge: 31 minutes.

Human review involvement per PR: near-zero.

That's a typical 30 days on cortex (Apr 21 -- May 21).

Every one of those 769 PRs had an AI reviewer as the first reviewer, with an average of 10.8 review-fix loop iterations per PR (max 56). 1 in 5 merged within 10 minutes, roughly half within 30 minutes. What humans do now is look at review outcomes and tune the review prompt and the guidelines themselves -- this is human-on-the-loop, not human-in-the-loop. Humans operate on the policy layer, not the execution layer.

Past 30 days
PRs merged	769
AI reviewer coverage	100%
Avg review iterations / PR	10.8
Max review iterations	56
Per-PR human review	~0%
Median time-to-merge	31 min
Merged within 10 min	20%
Merged within 30 min	49%

This is a typical month on cortex now.

The common refrain -- "AI speeds up writing but reviews still bottleneck" and "AI-written code lowers quality" -- is something cortex absorbs through a pipeline where neither failure mode can take hold. Let me break it down.

How the review bottleneck stops forming

The conventional wisdom: the reviewer becomes the bottleneck

As AI writes faster, the load on whoever reviews the output grows proportionally. Anthropic's internal blog (How Anthropic teams use Claude Code) reports the same pattern -- the bottleneck has shifted from writing to reviewing, and senior engineers' work has moved from writing code toward integrating and reviewing AI output.

cortex hit exactly this. The moment we ran Claude Code at full throttle, writing speed jumped by an order of magnitude or more. Meanwhile the human time available to read and approve PRs only grew linearly. If the reviewer (=me) took a day off, the whole org stalled -- a classic single point of failure.

cortex's answer: move the reviewer role to AI as well

Part 1 and Part 2 kept asking the same recurring question: "how far do you push the harness?" cortex went all-in: the AI writes the code, the AI reviews the code. What humans keep their hands on is "tuning the prompts and guidelines themselves" -- not making decisions inside each individual PR, but watching the system from above and adjusting.

Three conditions had to hold for this to work:

The AI reviewer has enough context

A generic AI reviewer only sees the PR diff. The diff alone hides business meaning, upstream/downstream dependencies, and prior incident history. cortex feeds the Product Graph (cpg) from Part 2 -- a knowledge graph that fuses code, docs, DB schemas, and infra into one structure, with each node carrying business role and upstream/downstream dependencies -- into the AI reviewer, so it can trace impact into code that the PR didn't even touch. It catches:

- Missed upstream/downstream fixes
- Missed doc updates
- Tests that should have been updated but weren't

Diff-only AI review can never reach this territory.

Reviews are not improvisational

If reviews shift day to day, the team gets confused, and the AI can't be told what "correct" looks like. We enforce this by passing an explicit review-guideline document as the mandatory citation source for every review (we open-sourced a snapshot, see below).
False positives don't blanket-block merges

Treating every false positive as Critical breaks the workflow. We control this with a severity hierarchy (Critical / Major / Minor / Nit) plus strict no-downgrade rules.

So: the cpg from Part 2 solves "what context the AI sees," the review guidelines solve "what the AI should do" as Guides (pre-execution control), and the severity ladder + no-downgrade rules solve "what the AI must not do" as Sensors (post-execution control). This maps cleanly onto Martin Fowler's Guides / Sensors taxonomy (introduced back in Part 1).

One more upstream layer: before any of those three kicks in, a 500-lines-per-file lint keeps every file in any PR small enough to fit in a single AI session. That alone keeps AI review from breaking down, and unlike a human reviewer, the AI doesn't lose focus. There are plenty of other lints in front of the AI reviewer too, but the full picture belongs to Part 4 (Self-Healing + observability + auto-added guardrails).

How the auto-review system is wired

The implementation is a script running on each developer's machine. GitHub webhooks land on an in-house Event Relay server, get persisted to Firestore, and each developer's machine subscribes as an SSE client. On reconnect, Last-Event-ID replays anything missed -- zero event loss, single webhook registration. Reviewer-mode machines stay always-on, so any incoming review fires immediately. Author mode runs in the background on the PR author's own machine, alongside their normal dev work.

How we ended up with Event Relay

The current setup wasn't the original design.

First: GitHub webhook → smee.io → each machine
Then: GitHub webhook → Cloudflare Tunnel → each machine
Now: GitHub webhook → in-house Event Relay with Firestore persistence → SSE to each machine

Both smee.io and Cloudflare Tunnel ran into connection drops and missed deliveries, which caused real misses for us. Switching to the in-house Event Relay brought event loss to zero (Firestore persistence + Last-Event-ID replay), and the relay turned into a general-purpose layer we could reuse.

The webhook ingestion for Self-Healing (covered in Part 4) actually goes through the exact same Event Relay. GitHub, Grafana, and other webhook sources get consolidated through one relay, and each machine's SSE client subscribes to whichever events it cares about. Having a single general-purpose webhook relay is a piece of infra that keeps paying off in unexpected ways -- worth investing in early.

When the reviewer's machine receives an event, the script spawns claude -p and walks through 9 dimensions (Graph / Architecture / Security / Test / Doc / Impact / Observability / AI-Antipattern / Recurrence) sequentially, then reads the verdict marker the AI emitted at the end and posts APPROVE or REQUEST_CHANGES via gh pr review.

A few notes:

Modes split the role -- the same script started with --mode reviewer becomes the reviewer process; with --mode author it becomes the PR-author response process. The machine of whoever is assigned as reviewer runs reviewer mode; the machine of whoever opened the PR runs author mode. Event Relay multicasts the events, and each machine reacts in a distributed way.
Per-PR worktree isolation -- author mode merges origin/main into a fresh worktree before spawning the AI. Multiple PRs can be handled in parallel without file state contaminating across them.
9 dimensions checked sequentially in one session -- not parallel sub-agents. A single claude -p session walks the 9 dimensions while keeping context shared, which also catches cross-dimension contradictions.
Review guidelines: public snapshot -- air-closet/cortex-review-guidelines (JP/EN). The live guidelines are inside cortex (private repo) and evolve daily; the public repo is a snapshot extracted for reference.

:::message alert
Guidelines alone scale only to projects in the tens-of-thousands-of-lines range. At cortex's scale (over 1M lines of code), the knowledge graph from Part 2 (cpg) is a hard prerequisite. Porting the guidelines without cpg won't reproduce the same review quality -- the AI reviewer simply can't navigate the codebase fast enough to reason about impact.
:::

Why sequential single-session review, not parallel sub-agents

We initially tried splitting the 9 dimensions across parallel sub-agents. Three problems emerged: cpg / guidelines / PR diff got injected 9 times (token cost balloons), cross-dimension findings couldn't reference each other (a [Test] issue rooted in a [Graph] violation gets dropped in isolation), and aggregating 9 outputs into a single verdict required its own machinery.

A single sequential session fixes all three: one cpg/guideline load, earlier findings stay in context for later dimensions (cross-dimension consistency comes for free), and one verdict marker at the end is the entire aggregation step.

We also swap CLAUDE.md to a review-specific version at startup. The default CLAUDE.md is dense with development-time context (Product Graph ops, prod-data safety, MCP ordering) -- noise for a reviewer. The review-specific version centers on severity, no-downgrade, and the verdict marker spec, keeping AI attention on the review task.

Cutting wasted context lifts judgment precision and token cost at the same time.

Operational knobs

A few filters and toggles we apply in actual use:

Draft (WIP) PRs are excluded. GitHub Draft state is received but skipped; review starts firing once the author flips it to Ready for Review.
Specific PRs can be targeted manually. The webhook is the normal trigger, but you can also kick off a review against a specific PR number from the CLI -- useful after a CI failure or for re-checking a single PR.
Auto-merge is the PR author's call. Whether the pipeline runs through to auto-merge after APPROVE + CI green is set by the PR author. Default is on; for changes that go directly to prod, the author can flip it off and hit merge themselves.

Output structure: tags and severity

Every auto-review comment is structured as tag + severity + concrete example.

Tags (dimensions)

Tag	Dimension	Primary target
`[Graph]`	Product Graph integrity	`@graph-*` JSDoc, node dependencies, doc consistency
`[Doc]`	Doc consistency	Doc updates that should follow code changes, doc placement
`[Impact]`	Impact analysis	Missed upstream/downstream fixes, `via:` field inconsistency
`[Security]`	Security	Auth, input validation, secrets
`[Architecture]`	Composable Architecture	app/package boundaries, dependency direction
`[Test]`	Test quality	Coverage, matchers, naming
`[Observability]`	Observability	Structured logging, no-truncate rules
`[AI-Antipattern]`	AI-generated code traps	Hallucinated APIs, fallback overuse, dead code
`[Recurrence]`	Recurrence prevention	Bug-fix triage (lint / horizontal rollout / new guideline)

Severity

Severity	Criteria	Action
Critical	Security, data corruption, prod-risk, doc inconsistency, missing `@graph-*`, quality-bar relaxation	`REQUEST_CHANGES`
Major	Spec violation, Composable Architecture violation, missing tests	`REQUEST_CHANGES`
Minor	Naming, maintainability, light refactor	`REQUEST_CHANGES` (must be resolved)
Nit	Style preference, minor inconsistency	`APPROVE` (comment only)

The single most important rule is "no downgrade":

"Following existing patterns" is not a valid reason to downgrade (existing violations are a separate problem to fix, not justification for new violations).
"Will be addressed in a separate PR" or "incrementally" is not a valid reason to drop Critical/Major to Nit.
"Leave a TODO/FIXME" is not a valid deferral path.

This is spelled out in severity.md, and the AI literally cites it back when it issues REQUEST_CHANGES.

Example: PR migrating the embedding model from gemini-embedding-001 to gemini-embedding-2

A real review comment is more useful than abstract description. Here's a typical one. This is a feature PR (feat(meet): dual-write embeddings to new 'embedding' column (v2)) that was merged on 2026-05-19, after 6 review-fix iterations across 1.5 hours.

First review (07:35:25):

The dual-write approach, the idempotent migration script, and the Promise.all parallelization all look sound. Please address the 3 Critical and 2 Minor findings below before merging.

Critical

[Graph] Missing @graph-business tag (x3)

graph-integrity.md requires @graph-business on every top-level declaration under the app layer (severity: Critical).
All 3 new declarations below are missing it (detailed comments in separate threads).
Existing code is also missing it, but "following existing patterns" is not a valid downgrade reason (severity.md).

generateEmbeddingV2 (v2 embedding generation function)

EMBEDDING_MODEL_V2 (model name config)

EMBEDDING_LOCATION_V2 (region config)

[Graph] embedMeetContent's @graph-connects doesn't reflect generateEmbeddingV2

The JSDoc on embedMeetContent has @graph-connects generateEmbedding [calls] Generate embedding, but no corresponding @graph-connects line has been added for the newly introduced generateEmbeddingV2 call.
The graph will be missing an edge to generateEmbeddingV2.
   * @graph-connects generateEmbedding [calls] Generate embedding
+  * @graph-connects generateEmbeddingV2 [calls] v2 embedding generation (dual-write)
   * @graph-connects insertMeetChunks [calls] Insert chunks into BQ
[Doc] Corresponding BigQuery schema doc is not updated

The "BigQuery schema" section in the related doc is missing the new embedding column.
Both graph-integrity.md and severity.md define doc inconsistency as Critical.
 | `created_at`  | TIMESTAMP   | Created at                              |
+| `embedding`   | FLOAT64[]   | Embedding vector (v2: gemini-embedding-2) |
Minor

[Test] textEmbeddingV2 value is not asserted

objectContaining allows extra fields, so the test still passes even when the v2 value is never set.
         textEmbedding: [0.1, 0.2, 0.3],
+        textEmbeddingV2: [0.1, 0.2, 0.3],
[Test] No isolated scenario for "v2 returns null"

generateEmbeddingV2: mockGenerateEmbedding reuses the v1 mock, so the case "v2 returns null while v1 succeeds" is not independently verified.



The takeaway is the precision of the details.

File + line numbers are concrete.
Suggested fixes are in diff format (copy-paste ready).
Source guideline (graph-integrity.md / severity.md) is cited explicitly.
The typical excuse ("existing code has the same problem") is pre-emptively closed.
The trailing  is a machine-readable verdict marker -- the trigger that moves the PR into REQUEST_CHANGES state.

After this, the PR author (= usually another AI running on the author's machine) pushes a fix, the reviewer re-reviews. The next review confirms all 3 Criticals are actually resolved, raises the next Major / Critical, and so on. 6 iterations in 1.5 hours, finally APPROVE, auto-merge.

Plotted on a timeline:

With a human reviewer, this is "Critical x3 -> wait until tomorrow for the fix -> re-review the day after" -- 2 to 3 days per PR. cortex closes it in 90 minutes.

The difference between human review and auto review is not just speed. A single AI session walks all 9 dimensions in order and cites the guideline each time, which makes it much harder to miss the "deep" findings humans drop because their attention drifted -- doc consistency, recurrence-prevention judgments, weak matchers. Side-by-side comparison:

This is why the review bottleneck never forms here.

Evolving the guidelines: catching the moments AI gets it wrong, then fixing the rules

The review guidelines I've been referring to are not a static document. Running this in production surfaces recurring patterns where the AI mis-judges a specific class of issue. Each time that happens, we don't add a comment to the individual PR; we rewrite the guideline so the AI behaves correctly next time -- this is the meta-layer humans actually operate on.

A few concrete failures we hit on cortex, and how we closed each one by changing the rule, not the PR.

1. AI was downgrading because "existing code has the same issue"

Early on, immediately after flagging a violation the AI would add "however, since existing code has the same violation, I'm downgrading this to Nit" and self-downgrade. The result: violations on newly added code kept dropping to Nit, and the system kept emitting Approve.

We closed this by adding the no-downgrade rule to severity.md:

"Following existing patterns" is not a valid downgrade reason: if existing code violates a guideline, new code following that pattern still gets flagged at the same severity. Deferral language like "consider during the next refactor" is not accepted.

That wasn't enough on its own. Over time other excuse patterns surfaced -- "will be addressed in a separate PR," "will be addressed in the next session," "out of scope," "incrementally" -- so we added those as forbidden downgrade categories too. We also explicitly forbade deferring via TODO/FIXME comments in code. The mindset is: close every typical excuse path preemptively.

2. The final verdict had 3 options, and "comment-only" left PRs in limbo

The final verdict at the end of every review was originally APPROVE / REQUEST_CHANGES / COMMENT (approve / request changes / comment-only). When the AI picked COMMENT -- for example when only Minor issues existed -- the script took no action, the PR sat in review-pending forever, and ultimately someone had to manually pick it up. Classic anti-pattern, and it kept happening.

We collapsed the verdict to 2 options. Anything Minor or above is REQUEST_CHANGES, a missing verdict marker defaults to REQUEST_CHANGES (safe side), and only Nit-only or no findings (with CI passing) yields APPROVE. The principle: "if the judgment is ambiguous, fail-safe by defaulting to the blocking side (REQUEST_CHANGES)." Going all-in on that design eliminated the stuck-PR class entirely.

3. Checklist items had no severity, so the AI's judgment kept drifting

Originally, each guideline (graph-integrity.md, testing.md, etc.) was just a bulleted checklist. Items like "Is the test name descriptive?" or "Are mocks minimized?" were listed, but without per-item severity. As a result, the same violation could land as Major in one PR and Nit in another, depending on the session.

We converted every guideline's checklist into a severity / scope / criterion table:

Severity	Scope	Criterion
Critical	All PRs	Missing `@graph-business`
Major	App layer only	Missing tests
Minor	Shared packages only	More than 3 function args
Nit	All PRs	Naming inconsistency

The scope column is a machine-decidable filter for which paths a check applies to, so the AI reviewer doesn't trigger irrelevant items on PRs outside that scope. Just putting it in a table -- the judgment reproducibility jumped significantly.

4. The existing guidelines didn't catch AI-specific traps

After running this for a while we noticed AI-generated code has its own cluster of antipatterns -- calling APIs that don't exist (hallucinated APIs -- something like user.findOrCreate() that looks plausible but isn't actually defined), swallowing errors and returning fallback values (e.g., silently returning an empty array when an upstream API fails), leaving unused functions (a refactor adds the new function but doesn't delete the old one, leaving dead code), expanding the modification scope beyond what was asked (you ask it to change one function and it reformats the whole file), adding unnecessary backward-compatibility code (creating a deprecated alias for an internal-only function) -- and security.md / testing.md couldn't catch these. There's a distinct class of "mistakes only AIs make."

We added a dedicated ai-antipattern.md for this. Reviews now pick these up explicitly under the [AI-Antipattern] tag. Reviewing AI output requires designing around AI-specific traps -- you don't get there just by porting human review heuristics onto an AI.

5. The AI tries to relax "the standard itself"

The last and most important pattern. When the AI was writing fix PRs, occasionally instead of fixing the guideline violation it would write a PR that relaxes the guideline. For example:

Lower the test coverage threshold to avoid writing more tests
Narrow the in-house lint rule's scope to make the violation go away
Soften the guideline doc language from "recommended" to "preferred" to weaken the binding constraint

And the AI builds a formally-coherent justification: "existing code already violates this, so let's adjust the standard to match the implementation." Left unchecked, the AI gradually walks the quality bar down.

We closed this by adding "quality-bar relaxation" as a Critical in severity.md:

A PR that relaxes the quality bar -- guideline doc, lint rule, coverage threshold -- must not be Approved by the AI reviewer. It is sent back with REQUEST_CHANGES. A human reviewer's approval is required. "Existing code already violates this" is not a valid justification for relaxation.

This is the one explicit boundary where we deliberately do not give the AI autonomous Approve authority. Whether the standard itself moves is a human decision. It's the meta-level safety valve for the "AI reviewing AI" architecture.

Evolving the guidelines is the meta-layer humans actually operate on

The common thread: "when the AI gets it wrong, don't override the individual PR -- rewrite the guideline so the fix propagates forward."

AI escapes via "existing code has the same issue" -> add no-downgrade rule
AI picks "comment-only" and PR stalls -> collapse to 2-option verdict
AI's judgment drifts -> add severity / scope columns to every item
AI falls into its own traps -> add the AI-Antipattern category
AI tries to relax the standard -> classify standard-relaxation as Critical, require human Approve

As long as this loop turns, the guideline is a living document that absorbs the failure patterns AI produces in production. Don't try to write the perfect guideline up front. Catch the moment AI gets it wrong, and write the rule for that moment. That's the actual mechanism behind "quality doesn't drop even when humans aren't inside the loop."

And one more thread. Right now, the trigger for "AI got it wrong, time to rewrite the guideline" is still mostly a human judgment, but parts of that maintenance are gradually becoming automatable too. Self-Healing (Part 4 next time) -- where AI investigates production incidents, opens a fix PR, runs it through auto-review, and auto-redeploys -- requires every fix PR to write one of {add lint, add guideline, horizontal rollout} under the [Recurrence] lens. So the AI is increasingly participating in the maintenance of its own review criteria, with humans still in the loop on adoption. I'll come back to this in Part 4.

Auto-fix: a separate AI applies the changes and pushes

Once REQUEST_CHANGES lands, the same script running on the PR author's machine, but in author mode, picks up the event and starts working.

[REQUEST_CHANGES detected]
   | SSE push via Event Relay
[Author mode boots on PR author's machine]
   | Merge origin/main into a worktree
   |  (lockfile resolved up front, remaining conflicts handled by AI)
   | Read the auto-review comment as context
   | Run claude -p inside the worktree
   | Commit + push the changes
   | New SHA is delivered back to the reviewer's machine via Event Relay -> re-review

Two design choices matter here.

Reviewer and author run on different machines in different sessions -- reviewer mode and author mode are the same script, but they run on different machines in different processes. "Is the original critique correct?" is judged independently. Unlike a single AI fixing its own complaints, the judgment passes between two separate sessions.
All iteration stays inside the same PR -- we don't spawn a new PR. The "fix the root cause, no deferrals" rule from Part 2 and the review guidelines kicks in here: if the AI tries to escape via TODO/FIXME or by splitting work out into a separate PR, the next review rejects it.

Auto-merge + parallel deploy

Once auto-review returns APPROVE and CI is fully green, the auto-merge script runs and squash-merges the PR.

[Auto review APPROVE + CI green]
   |
auto-merge script
   | squash merge to main
   |
[main updated]
   |
Turborepo build (affected packages only)
   |
Pulumi up (multiple stacks in parallel)
   |- API services
   |- pipeline services
   |- MCP servers
   `- infra
   |
[Deploy complete]
   |
cpg index rebuilt (only changed nodes regenerate embeddings -- see Part 2)

pulumi up <stack1> <stack2> ... runs in parallel, so deploying 9 stacks at once finishes in about 8-12 minutes. End to end, merge-to-production is averaging 10-15 minutes.

This compounds nicely with Self-Healing PRs. Incident alert -> Self-Healing identifies root cause -> opens a fix PR -> auto review pass -> auto merge -> auto deploy runs as a single closed loop without human involvement (covered in Part 4).

The numbers, in more detail

Unpacking the headline numbers a bit further.

Depth of the review-fix loop

Across 769 PRs in 30 days, the average per PR was 10.8 review iterations, max 56. The fact that the average is past 10 means the first review almost always surfaces at least one finding.

The embedding-model migration PR shown earlier needed 6 iterations to merge, and that's representative of the average PR. What would take a human reviewer days, cortex resolves in minutes.

What the auto reviewer typically flags

The most common findings out of the first review:

[Graph] Missing @graph-business -- a prerequisite cpg leans on (from Part 2). The classic finding on newly added declarations.
[Doc] Doc inconsistency -- code changed but the corresponding docs/ section was not updated.
[Test] Weak matchers -- objectContaining weakening value assertions, single-property checks via toBe.
[Observability] Unstructured error logs -- event field or required keys deviating from the structured-log spec.
[Recurrence] No recurrence-prevention action -- a bug-fix PR description not declaring which of {lint / horizontal rollout / add guideline / nothing} applies.

These are categories human reviewers frequently miss in practice, especially doc consistency and recurrence-prevention checks. The AI reviewer applies them mechanically on every PR.

Actual false-positive rate

It's not zero. A few times a month we get "this is Nit, not Major" type misjudgments. The fix path is the one described above -- not a comment on the individual PR, but a guideline edit that corrects the judgment for all subsequent reviews.

What changed / Bridge to Part 4

Over the past six months, the engineer's role on cortex shifted from "writer" and "reviewer" to "operator" -- the human running the system, not acting inside each individual decision.

AI writes the code (Claude Code)
AI reviews the code (auto review)
A different AI applies the fixes (author mode running on the PR author's machine)
AI decides when to merge (auto-merge script)
Deploys go in parallel (Turborepo + Pulumi)

What stays in human hands: "what to build at all (product / requirements)," "is this direction actually right (architectural judgment)," "which guideline to add and where," and "look at the reviews and adjust prompts and guidelines accordingly." High-abstraction work -- not individual decisions, but watching the whole system from above and steering. From human-in-the-loop to human-on-the-loop, you could say.

The widely-reported phenomena -- "AI lowers quality," "the reviewer becomes the bottleneck" -- happen when the harness is extended on the writer side only, and the reviewer side is left to humans. If writing speeds up and reviewing doesn't, of course it bottlenecks. Of course things get missed.

cortex is the opposite. We extended the harness on the reviewer side first, before fully extending it on the writer side. Anthropic's observation that the bottleneck shifts from writing to reviewing is exactly right -- which is precisely why "move the reviewer role to AI as well" is the answer cortex chose.

"The AI writes the code, the AI reviews the code." That's the core of cortex's auto-review pipeline. Quality drop and review bottleneck are functions of how far you extend the harness -- they are not inherent to AI-assisted development.

Up next in Part 4 — Self-Healing + Recurrence Prevention: a pipeline where a production alert (observed via OTel/Loki/Mimir/Tempo/Faro) triggers AI investigation, an AI-authored fix PR plus a new lint/type gate, auto-review, auto-merge, and auto-redeploy. The fix and a recurrence-prevention guardrail land together, so the same class of incident structurally can't fire again. If auto review protects quality at PR time, Part 4 protects it at production time, while growing the quality gates themselves.

The headline number above includes Self-Healing PRs (production alerts that AI investigates, fixes, and auto-deploys). For certain classes of incidents, the fix is already merged before anyone has time to react — that's where cortex sits today.

The Heart of the AI Harness: A Knowledge Graph of the AI, by the AI, for the AI (Series Part 2)

Ryosuke Tsuji — Tue, 19 May 2026 14:16:20 +0000

Hi, I'm Ryan, CTO at airCloset.

Disclaimer: "cortex" and "cortex-product-graph" referenced in this article are internal code names for an AI platform developed in-house at airCloset. They are unrelated to existing commercial services such as Snowflake Cortex or Palo Alto Networks Cortex.

In Part 1 (Series Intro), I wrote about how AI handles PR reviews and incident response on top of a platform we call cortex. At the center of that flywheel is the Product Graph (implementation name: cortex-product-graph, or cpg) — a unified knowledge graph of code, docs, DB schemas, and infrastructure definitions, queryable through semantic search.

In Part 1, I described cpg at a high level: "all of cortex is indexed in one graph." This post goes deeper — how it's built, why we landed on this design, and what actually changed once it was in place.

Series Index

#	Theme	Key scene	Article
1	Series intro: cortex's harness	PRs auto-merge / incidents self-heal before you notice	ai-harness-intro
2	Product Graph (cpg)	Code, docs, DB, infra unified into one graph	this post ← you are here
3	AI PR review	webhook → AI review → auto-fix → squash merge	cortex-auto-review
4	Self-Healing + observability + auto-added guardrails	Alert → AI investigates → fix PR + new lint/type gate → auto redeploy + same-pattern writes get auto-rejected	cortex-self-healing
5	Democratizing the maintenance phase	Domain experts open PRs to production; the harness owns the quality gate	cortex-non-engineer-prs
6	Series Final	The underlying philosophy plus a retrospective on the failures and lessons	cortex-philosophy

Start with One Scene

"I want to change the calculation logic behind the 'bug rate' KPI on the dashboard. Where is it, and what might break?" — imagine that question comes up before you touch any code.

When you ask an AI this directly, with no function name and no file path given, it hits cpg with a semantic search and pulls the relevant nodes in one shot. What comes back isn't just functions — it includes BigQuery tables and API endpoints alongside the code. And at the end of the response, there's a "next action candidates (Runbook)" block that tells the AI to re-probe starting from the BQ table with the most reads and writes flowing through it.

The final answer looks like this:

Calculation site: calculateRatePer100pt / calculateBugCount — both pure functions with no I/O side effects; safe to change in isolation
Writers (upstream): syncKpiMetrics / writeKpiMetrics / backfillKpiMetrics all write to the kpi_bug_rate_per_100pt table; these are the real aggregation batch jobs
Readers (downstream): BigQueryKpiRepository.getSummaryByDate reads via BigQuery → /kpi/bugs API → KPI dashboard page
Related docs: docs/generator/kpi.md defines bug rate; updating the code without updating docs would leave them stale

"Update the docs together, and schedule the deploy when the aggregation batch isn't running" — that's a decision you can make with confidence.

I personally know all this — I wrote it. But that's exactly the problem: anyone else who wanted to touch this had to track me down. Three months ago, "finding out where something lives and what would break" meant finding me. Now, this same investigation is done by PMO members (non-engineers) using cpg on their own. grep didn't get them there; documentation didn't get them there. One natural-language question did.

What makes that possible is cpg — a graph where you can follow "what you want to do" in plain language to the relevant nodes in one or two hops, even when you don't know the function name. The Runbook structure — where the tool's return value itself contains the next tool call to make — is what lets the AI re-select its starting point and drill deeper on its own.

That's the setup. Now let me explain how it's built.

What Static Analysis Alone Couldn't Do

cortex has a separate system that graph-analyzes the production codebase using static analysis (I'll write about this in its own post — just touching it here). It parses JS/TS code with AST analysis across our external-facing production repos, automatically extracting function call graphs, API endpoints, DB access patterns, and event pub/sub relationships.

This works well for what it does, and we still use it actively in the production repos. But when we tried applying the same approach to cortex itself, it didn't get us where we wanted to go.

Three specific gaps:

No context — nodes exist but carry no meaning. "What is this API for?" "Why does this column exist?" isn't in the graph. Ask "where is the code that calculates the KPI bug rate?" and you'll miss unless the function name happens to look like it.
No entry point — you already have to know the file path or function name before search can start. "Let me go find it" doesn't work.
Explosion after 1–2 hops — starting from any node, related nodes multiply exponentially within a couple of hops, far exceeding what an AI can process in one context window. Trace results become too long to use.

The summary: mechanically accurate, but no semantic weighting. To be genuinely useful to AI, you need one more layer: "what matters, and why things are connected."

Meanwhile, DB Graph Was Working

Around the same time, a different approach — the DB Graph MCP we'd built — was working exactly as intended.

DB Graph is an MCP server with access to 15 schemas and 991 tables inside cortex, supporting semantic search over tables and columns with AI-generated descriptions. A natural-language query like "tables related to return processing confirmation" would find semantically connected nodes even when the table name doesn't contain those words.

After thinking about why this worked, the answer became clear: DB Graph has a business-context description attached to every node, and that description is what feeds into the embeddings. That semantic weight is what "finding by meaning" actually runs on.

Static-analysis code graph had none of that. Type relationships and call graphs exist — but "why this function exists" was never written anywhere.

The Hypothesis — Bring DB Graph's Essence into the Code Graph

The hypothesis was simple:

"A business-context description on every node, loaded into embeddings" — if that's the core of why DB Graph works, then doing the same thing for the code graph should structurally overcome the limits of static analysis.

The problem was: where do you put the "business context"?

All the options:

Location	Example	Problem
External docs	Design docs / wiki / Notion	Separate from code. Drifts instantly. Nobody maintains it.
External metadata	Sidecar YAML / `*.meta.json`	Dual-management. Breaks on rename.
Dedicated graph DB	Write annotations directly into Neo4j / Neptune	Dual-management again. Doesn't show up in PR diffs — unreviewable.
TypeScript decorator	`@GraphNode({...})` in code	Lives in the transpiled output = runtime dependency. Can't be extracted by AST alone.
DSL file	Custom `.graph` file format	High learning cost. No editor support out of the box.
JSDoc comments	`@graph-business` / `@graph-connects`	Physically co-located with the code. Extractable by AST alone. Zero runtime dependency.

The choice of JSDoc over decorators was intentional:

Zero runtime dependency: decorators survive into the transpiled output and can affect runtime behavior. JSDoc has no executable runtime semantics; with production builds that strip comments, it leaves no runtime artifact.
Generalizes beyond TypeScript: the same @graph-* syntax can extend to Pulumi definitions in infra/ and Markdown frontmatter in docs/. Decorators are locked to TypeScript syntax.
Single AST pass: ts-morph can walk declarations and extract JSDoc in one scan. Decorators sometimes require type resolution, which slows builds.
Shows up naturally in PR diffs: JSDoc sits directly above the code it annotates, so when code changes, the JSDoc diff appears in the same file. Reviewers can't miss it.
Doubles as documentation for both humans and AI: JSDoc already serves as IDE hover text and AI-readable context. Putting @graph-business there means it simultaneously explains the declaration to a human reading the code, and gives a coding AI semantic context about the surrounding functions. Graph metadata that also functions as inline documentation.

Note that the essence of this design is using parseable annotations co-located with code as the SSoT — TypeScript / JSDoc is just one implementation. The same pattern works in any language with comparable comment + AST primitives: Python docstrings + ast, Go comments + go/ast, Rust /// + syn. What matters isn't where you write the annotations, but the invariant: "physically co-located with the code, extractable by AST alone."

Same goes for the monorepo: this pattern doesn't depend on cortex being a monorepo. If anything, its real value shows when repositories are split and AI can't easily follow code across them. In a monorepo, the AI can still grep / read files across the whole tree; in a multi-repo, the cross-repo calls and data flows are the hard part to follow. Run the same build per repo, emit nodes / edges, aggregate into a central graph, and those cross-repo connections become reachable in one hop. We actually run a parallel knowledge graph over our external-facing production repos (multi-repo) using the same pattern — more on that in a separate post.

The Approach — Abandon Code Inference, Make JSDoc the SSoT

The code graph's problem was no meaning. The answer is simple: embed the meaning directly in the code.

For cortex's own code graph, we completely abandoned the approach of inferring graph structure from code. Instead:

Every declaration — function / class / method / API / Page / Cron / etc. — gets a dedicated JSDoc tag. The graph is assembled from those.

This means the SSoT (Single Source of Truth) for business context becomes the code itself. There's no gap between docs and code, because the JSDoc in the code is the authoritative source. The structural problem of "AI makes mistakes because docs are stale" is resolved at the level of where the data lives.

Placing the two side by side — "a graph from code inference alone" versus "a knowledge graph with JSDoc as SSoT" — makes the difference in what's carried on each node immediately visible:

Here's a concrete example of the tags (from cpg's own source):

/**
 * Set embeddings on nodes in place.
 * Compares textForEmbedding against existing BQ data; only re-generates
 * for nodes where the text has changed.
 *
 * @graph-stack product-graph
 * @graph-domain Engineering
 * @graph-business Compares hash of textForEmbedding against existing BQ nodes; re-generates
 *   embedding only for nodes where text has changed. Unchanged nodes reuse BQ embeddings.
 * @graph-connects cortex.product_graph_nodes [queries, via:id] read existing embeddings
 * @graph-connects vertex-ai-embedding [calls] generate embeddings for changed nodes
 */
export async function generateEmbeddings(
  nodes: ProductGraphNode[],
  options: { force?: boolean } = {},
): Promise<void> { ... }

What each tag does:

Tag	Role
`@graph-node`	Explicitly declares node type (defaults to Function)
`@graph-stack`	The infra stack this declaration belongs to
`@graph-domain`	Business domain (comma-separated, multiple allowed)
`@graph-business`	What this declaration specifically does — the body of the embedding input
`@graph-connects`	Connection targets (multiple allowed; `via:` for parameter-level tracking; `none` to explicitly declare no connections)

The key is that @graph-business feeds directly into the embedding input. It's not the node name — it's a natural-language sentence that carries semantic weight into search. In practice, almost all of these sentences are written by AI: during the normal flow of writing code in cortex, the AI writes the JSDoc alongside the code (and thanks to the ESLint enforcement below, it doesn't forget).

Making Omissions Physically Impossible

This design collapses the moment someone leaves a tag out. One function without @graph-business = that function is invisible to semantic search. One without @graph-connects = the data flow through that function is absent from the graph.

So we built enforcement that makes omissions physically impossible:

5 ESLint plugins — tag presence validation, syntax validation, naming convention enforcement (stack / domain allowlists), @graph-connects required, @graph-connects none misuse detection (flags when none appears on code that calls external services)
Automated PR review (Part 1 ③) — tags missing are flagged as [Graph] Critical; docs inconsistency is flagged as [Doc] Critical

The result: "write a declaration → business context is always written with it" holds as an invariant. Add a function → its meaning and connections are necessarily in its JSDoc.

One honest note: forcing "5 JSDoc tags on every declaration" on humans would blow up in code review within three days. Writing a @graph-business sentence per function, enumerating @graph-connects exhaustively, checking the naming allowlists — that's genuinely tedious at scale.

This works because AI writes the code. Writing four required JSDoc tags (plus optional @graph-node when the default Function type isn't enough) is rounding error on top of writing the code itself. With ESLint and automated review in the feedback loop, the AI doesn't miss tags — and human reviewers only need to check "is this tag factually correct?" not "is it there?"

:::message
This design is one that can't realistically be maintained when humans write code, but becomes viable the moment AI does. It's an AI-first design. The premise of AI-first development is what lets business context be fixed in code as the SSoT.
:::

Where Hallucination Happens Shifts

Viewed from another angle, what's going on here is that the location of hallucination shifts. Where you contain hallucination is, I think, fundamental to AI harness design.

As I wrote elsewhere, when you combine AI with a graph system, "hallucination doesn't disappear — it just changes location." For cpg, here's where it lands:

Graph build / query phase: No fresh LLM generation. Once reviewed metadata lands in the graph, the ts-morph AST pass, the BigQuery MERGE, and the MCP query responses are all deterministic.
JSDoc writing phase: This is the entry point for hallucination. Whether @graph-business is factually accurate, or whether @graph-connects is exhaustively listed — these can go wrong since the AI is writing them.

But the entry point is locked down by automated PR review. Missing tags get [Graph] Critical; factual drift gets [Doc] Critical. When something's wrong, either the AI that wrote the code or another reviewer AI catches it and fixes it.

The result: once data lands in the graph, it can be treated as deterministically sourced from reviewed code, not as a fresh generated answer that might hallucinate on every query. AI agents calling cpg don't have to guard against "this might be a generated lie" on every returned node or edge. The tools can be designed as "return facts only" without compromise.

Build — AST to Graph via ts-morph

Once JSDoc is established as the SSoT, the rest is mechanics: extract it and assemble the graph. The implementation:

AST-analyze JS/TS with ts-morph — walk every declaration (function / class / method / type / enum / variable / expression statement / export default / etc.)
Extract @graph-* tags from JSDoc — collect the four required tags plus optional @graph-node and normalize into a ParsedGraphTags structure
Generate nodes — use qualifiedName = "<filePath>:<name>" as the node ID
Generate edges — one edge per @graph-connects entry, with via: / cardinality and other metadata preserved
Generate embeddings — send @graph-business text to Vertex AI Embedding (gemini-embedding-2) and vectorize it
Load into BigQuery — MERGE all nodes / edges into cortex.product_graph_nodes / cortex.product_graph_edges

Because @graph-business goes directly into the embedding input, querying "code that calculates the KPI bug rate" in natural language returns a hit based on semantic proximity of the description — even when the function name contains neither "bug" nor "rate."

The overall flow: the three tracks (apps/ / infra/ / docs/) each go through their own parser, are merged into a single node set by the generator, and only nodes whose text has changed are sent to Vertex AI before being stored in BigQuery:

Build Cost Is Effectively Zero

The build runs automatically on push to main via GitHub Actions, using a differential embedding approach:

Compare textForEmbedding of each BQ node against the new text
Unchanged nodes reuse their existing BQ embeddings
Only changed nodes go to Vertex AI

A typical push changes a few dozen nodes, so cost is under $0.001. Full regeneration (for recovery, triggered via workflow_dispatch) is ~$0.075 for 8,000+ nodes.

Why BigQuery, Not a Graph Database

When people hear "knowledge graph," they often imagine a dedicated graph DB (Neo4j, Neptune, Memgraph, etc.). cortex runs on just two BigQuery tables (product_graph_nodes / product_graph_edges). Three reasons:

Different cost structure — dedicated graph DBs set a floor of "always-on cluster cost"; for the current implementation, BQ is storage + on-demand queries only. Even with continuous AI traffic, it's clearly cheaper than running a server 24/7.
Vector search / cosine similarity / SQL in the same place — BQ has VECTOR_SEARCH and ML.DISTANCE, so semantic search over @graph-business embeddings, filter by node properties, and adjacent-node JOINs can all live in one query. That matters when "semantic search + property filter + neighbor JOIN" is the standard access pattern.
Migration-ready for GQL once BQ Graph goes GA — BQ already has Graph in BigQuery in Preview; once it ships GA, you can put a graph view over the existing tables and likely shift to MATCH (n)-[e]->(m) queries in GQL. The current table design is already migration-ready.

In short: get the graph DB's future strength (GQL) while running on plain BQ tables today. Compared to adding a graph DB on top of a generic RAG stack (pgvector / Pinecone / etc.), fewer systems to operate and lower learning curve.

The Core Part Is Available as an Open-Source Sample

The "parse JSDoc annotations with AST analysis and output a graph" part is small enough to reproduce cleanly, so I published it as a working sample:

🔗 graph-jsdoc-extractor

It's a ~500-line library that extracts @graph-* and outputs ndjson of { kind: "node", ... } / { kind: "edge", ... } objects. Comes with a pnpm run example that runs end-to-end. For those who just want to see the output format without cloning, the built ndjson is checked in: examples/sample/output.ndjson.

This is intentionally just the "turn code into a graph" part. The real value in cortex starts when docs and DB schemas land on the same graph — that's the next section.

Connections — Landing Docs and DB on the Same Graph

Looking at the sample ndjson, a @graph-connects users [reads_from, via:id] entry has users stored as a raw string in targetId. Leaving that as-is means it's just a string. Resolving users into a rich node carrying column definitions, partition info, and per-column descriptions — that's where the resolution power of search takes a real step forward.

cortex does this in three directions.

1. DB Schemas as Nodes in the Same Graph

cpg ingests not just code but cortex's DB schemas in the same build. A @graph-connects users [queries, via:id] on the code side gets resolved at build time into a rich Table node carrying column definitions, partition metadata, and descriptions (if the same-named stub exists, its internals are replaced while its ID and all inbound edges survive).

The key point: table and column descriptions aren't AI-generated annotations attached after the fact — they're pulled directly from the description fields in the Pulumi schema definitions. Here's what that looks like (excerpt from cpg's own table definition):

export const productGraphNodesTable = new gcp.bigquery.Table('cortex-prod-product-graph-nodes', {
  datasetId: 'cortex',
  tableId: 'product_graph_nodes',
  description:
    'Product Graph nodes — unified knowledge graph of code + DB + docs. ' +
    'Auto-generated from JSDoc @graph-* tags',
  schema: JSON.stringify([
    { name: 'id', type: 'STRING', mode: 'REQUIRED',
      description: 'Unique node ID (graphId:nodeType:filePath:name format)' },
    { name: 'nodeType', type: 'STRING', mode: 'REQUIRED',
      description: 'Node type — ApiEndpoint, BigQueryTable, Function, Module, Document, etc.' },
    { name: 'qualifiedName', type: 'STRING',
      description: 'Fully qualified name — filePath:exportName format' },
    // ...
  ]),
});

Both the table-level and column-level descriptions become the embedding input for semantic search directly from the Pulumi definition. The same philosophy as cpg's JSDoc — "write the description at the place the thing is defined" — runs all the way through the DB layer. Fix a Pulumi description → semantic search improves. Same mechanics as fixing a JSDoc.

2. Docs Auto-Promoted to Nodes via Directory Convention

Markdown files under docs/ also land in the graph. The mechanism is simple: the directory structure is conventionalized so that which stack and domain each doc belongs to is deterministically resolvable:

docs/{category}/{name}.md

Examples from cpg itself:

docs/product-graph/README.md → stack: product-graph, domain: Engineering
docs/code-graph/README.md → stack: code-graph, domain: Engineering
docs/mcp/db-graph/README.md → stack: mcp-db-graph-server, domain: Engineering

Each file is ingested as a Document node in the graph, and a documented_by edge is auto-generated from code nodes whose @graph-stack matches the doc's stack. Code under apps/graph/product/ all carries @graph-stack product-graph, so it's automatically linked to docs/product-graph/README.md. Change code → related docs are already linked.

This means an AI reviewer can answer "did this code change leave related docs stale?" in one graph hop (that's the source of the [Doc] Critical comments from Part 1).

3. Infrastructure Definitions as Nodes

@graph-* tags go on Pulumi code in infra/ too. An example from cortex's own graph infrastructure:

/**
 * @graph-node {CronSchedule}
 * @graph-stack code-graph
 * @graph-domain Engineering
 * @graph-business graph-boundary-daily: runs cross-repository boundary analysis at 7:00 AM JST
 *   daily (auto-detecting API, DB, and Event connections across repos)
 * @graph-connects graph-index-job [triggers] trigger Cloud Run Job
 */
new gcp.cloudscheduler.Job(`${prefix}-graph-boundary-schedule`, { ... });

This becomes a CronSchedule node in the graph, connected to the target CloudRunJob node by a triggers edge. The Pulumi definition is itself a graph entry point — "what code runs in this cron?" is now answerable by graph traversal.

Result: Four Layers on One Graph

Adding the three together, the node types in the graph look like this:

Node type	Source
Function / Class / Method	Code (JSDoc)
ApiEndpoint / Page	Code (JSDoc `@graph-node`)
BigQueryTable / FirestoreCollection (stub)	Code `@graph-connects` targets
Table / Column / Schema (rich)	Schema files defined in Pulumi
Document	Directory parser over `docs/`
CronSchedule / PubSubTopic / CloudRunService	`infra/` JSDoc

Edge types correspondingly:

Edge type	Role
calls / queries / reads_from / writes_to / publishes / triggers	code → other nodes (`@graph-connects`)
documented_by	code → Document (auto-generated on stack match)
HAS_TABLE / HAS_COLUMN	Schema → Table → Column (DB side)
shares_topic	Between boundary nodes sharing a topic

Code ↔ DB ↔ docs ↔ infra — all reachable in one hop on the same graph. This is what "Product Graph" means: cortex's unified knowledge graph.

Here's an actual visualization of a slice of cpg itself. Starting from generateEmbeddings (code), you can see cortex.product_graph_nodes (BigQueryTable) with its columns, the Pulumi table definition resource, docs/product-graph/README.md, external services like Vertex AI, and a separate layer's graph-boundary-daily (CronSchedule) — all connected by edges on the same node set:

Where the Sample Stops

graph-jsdoc-extractor intentionally leaves out:

Resolving @graph-connects targets to real node IDs (cortex uses a seven-stage resolver; the rules are project-specific)
Same-name merging (cortex promotes DB-schema-side rich nodes to replace stubs; the merge source is project-specific)
The docs directory convention parser (cortex's docs/{category}/{name}.md convention is cortex-specific)
Embedding generation (Vertex AI setup is up to you)

These are parts where the right answer differs per project — naming conventions, where docs live, which embedding model to use, when to promote a stub to a rich node. Baking one answer into the sample library would make it harder to use, not easier. The sample draws the line at JSDoc → graph structure, and this article's job is "here's how we did it in cortex — translate it to your project's context."

MCP Tool Design and the Runbook Pattern

The graph is now assembled. Next: how AI uses it.

cpg runs as an MCP server (cortex-product-graph). From the AI's side, three tools are visible, applying the three-layer tool design (search / detail / traverse) from the Agentic Graph RAG MCP post directly to cpg:

Tool	Role
`search_product_graph_nodes`	Find entry points (vector search + name search)
`get_product_graph_node_detail`	Deterministically fetch detail by ID
`trace_product_graph_connections`	BFS subgraph traversal (`via_filter` for parameter-level tracking)

Three layers only shows you what's in the graph. For jumping from graph nodes to the actual data they point to, supplementary tools live in the same MCP:

Supplementary tool	Role
`read_file`	Pass a node's `path` property directly to fetch source (Function / Class / Method / ApiEndpoint / Document — any code-origin node carries `path`)
`grep_code`	Pattern search across the repository
`git_blame`	Last author, commit, and timestamp per line
`query_product_graph_bq`	Direct SQL against BigQuery. Find a BQTable node in the graph, then jump to its live data (executed via user OAuth, so BQ IAM applies as-is)
`read_firestore` / `write_firestore`	Read/write Firestore collections. Find a FirestoreCollection node in the graph, then go to the live documents (Firestore access follows the same user / environment permission boundary; cpg provides the entry point, not a bypass around IAM)
`list_product_graph_stacks` / `list_product_graph_domains`	Lists all stack / domain names present in the graph; useful for orienting before a search

In other words, cpg's MCP is a two-tier design: the three-layer structure for graph traversal + supplementary tools for descending into live data (source code / BQ / Firestore). The AI can do "search by meaning → traverse by structure → pull live data" entirely within one MCP server.

Runbook Pattern — Return Values Contain the Next Action

Every MCP response ends with a "related nodes (next action candidates)" block. For example, after a search returns:

3 nodes found:
- apps/generator/kpi/src/kpi-calculator.ts:calculateBugCount (Function)
- backlog_no_embedding.kpi_bug_rate_per_100pt (BigQueryTable)
- /kpi/bugs (ApiEndpoint)

## Related nodes (next action candidates)

### 🛠 Code (1)
- apps/generator/kpi/src/kpi-calculator.ts:calculateBugCount
  → `get_product_graph_node_detail("apps/generator/kpi/src/kpi-calculator.ts:calculateBugCount")`

### 🗄 DB tables (1)
- backlog_no_embedding.kpi_bug_rate_per_100pt
  → `trace_product_graph_connections(start_node: "backlog_no_embedding.kpi_bug_rate_per_100pt", direction: "backward")`

### 🌐 API (1)
- /kpi/bugs
  → `get_product_graph_node_detail("/kpi/bugs")`

Copy-pasteable tool calls are lined up by node type, showing exactly what to call next. The AI gets new options on every call, so it never has to figure out "what should I do now?"

Here's the AI ↔ MCP loop in diagram form. The MCP bundles next action candidates into every search response; the AI picks one and makes the next call, repeating:

`usecase` Parameter — Switching the Runbook

Every tool accepts a usecase parameter where the AI declares what kind of investigation it's doing:

usecase	Strategy (summary of what cpg optimizes for)
`general`	Basic investigation with unknown entry point. Default.
`design`	Understanding existing feature structure. Read business / connections via `get_product_graph_node_detail`. Deep trace is unnecessary; Document nodes take priority.
`impact`	Trace upstream and downstream impact deeply. Hit `trace_product_graph_connections` with `direction=both` / `max_depth=5`. Code + DB + infra + schedules are all on the same graph, so one traversal covers a wide area.
`test-create`	Test design. Fetch detail to read parameters and connected DB / called functions.
`test-review`	Compare existing tests against implementation coverage. Cross-check branch structure of target Function / Method against test case count.
`code-review`	Check impact of changes and detect `@graph-business` violations. Trace impact → detail to check business / source.
`bug`	Deep trace from error origin. `direction=both` / `max_depth=5` for upstream callers + downstream data flow.

The same search_product_graph_nodes call with usecase: "code-review" returns next action candidates optimized for "verify the change's impact first." With usecase: "bug" it returns candidates optimized for "trace deep from error origin + fetch logs." The Runbook switches to match the declared intent.

This matters because having the AI declare "what kind of investigation I'm doing" yields different angles from the same graph. Auto Review internally fires with code-review; Self-Healing fires with bug — the flywheel elements from Part 1 each run a different Runbook.

CLAUDE.md Convention — Forcing AI to Always Hit cpg First

Throughout this post I've said "the AI uses cpg," but AI doesn't spontaneously choose cpg. Claude Code defaults to grep / glob / file read as its first instinct. To flip that, the root CLAUDE.md in cortex opens with:

Product Graph MCP (cortex-product-graph)

This is the single most important asset in this repository. cortex-product-graph MCP indexes all code, DB schemas, docs, and infra into a unified knowledge graph with business context. It knows everything about this repository.

Always query Product Graph MCP first before grep/glob/file reads. It returns richer, contextualized results.

If Product Graph MCP is unavailable (auth expired, server down) and you are NOT in autonomous/auto mode, stop all work immediately and ask the user to authenticate. Do not proceed with degraded grep-only investigation.

Two things matter here. First, the explicit ordering — "cpg first, grep only as fallback." Second, fallback to grep is explicitly forbidden if cpg is unavailable. Without that second clause, the AI happily degrades to "cpg seems down, I'll just grep" and proceeds with stale context and wrong assumptions. With it, cpg unavailability is a hard stop, not a graceful degradation.

One clause in CLAUDE.md, and Claude Code's first move on any code investigation is pinned to cpg. Article writing, Auto Review, Self-Healing — all follow the same convention, so the entry point is always unified.

A Live Example — Investigating cpg with cpg

Enough abstraction. Let me walk through a real cpg query: using cpg to investigate cpg's own builder core — the meta-example.

Step 1: Semantic search for "the code that extracts graph source data from code annotations"

No function name assumed. Just the intent in plain language:

search_product_graph_nodes(
  query: "code that extracts graph source data from annotations written in code",
  search_mode: "semantic",
  usecase: "design"
)

Top 5 results:

- apps/graph/product/src/parsers/jsdoc-parser.ts:applyGraphTag (Function)
- apps/graph/product/src/parsers/jsdoc-parser.ts:extractTagsFromNode (Function)
- packages/eslint-plugin-graph/src/utils/jsdoc-utils.ts:extractGraphTags (Function)
- apps/graph/product/src/parsers/jsdoc-parser.ts:parseJSDocExports (Function)
- packages/eslint-plugin-graph/src/utils/jsdoc-utils.ts:getGraphTagValue (Function)

The query contained neither "JSDoc" nor "@graph-*" nor "parser" — yet the intent found the right nodes via the @graph-business embedding. grep cannot do this.

Step 2: Trace downstream from that node (`usecase: "design"` prioritizes Documents)

trace_product_graph_connections(
  start_node: "apps/graph/product/src/parsers/jsdoc-parser.ts:parseJSDocExports",
  direction: "forward",
  usecase: "design"
)

Edges returned:

- parseJSDocExports --calls--> extractDeclarationsFromFile
- parseJSDocExports --calls--> extractTagsFromNode
- parseJSDocExports --reads_from[via:filePath]--> filesystem
- parseJSDocExports --documented_by--> docs/product-graph/README.md (Document)

The last one — documented_by — is the point: the edge from code to the Document node was auto-generated. Following it with read_file retrieves docs/product-graph/README.md — and with it, the background, design rationale, and tag specification for this implementation, all in one hop.

Step 3: The meta-structure — this article itself is written with cpg

This article was drafted by Claude Code, not by me — I provided direction and review. That Claude Code has cpg MCP connected, so every time I said "show a real example from cpg's own code" or "use a cpg-related infra example," Claude queried cpg to pull actual function names, JSDoc, Pulumi definitions, and docs structure, then embedded them in the text.

In other words: the generateEmbeddings JSDoc, the Pulumi productGraphNodesTable description, the graph-boundary-daily cron annotation, the auto-link to docs/product-graph/README.md — none of these came from my memory. Claude queried cpg and found the real artifacts. My role is only the review judgment: "this is right / this is wrong."

This is the pattern repeating across all of cortex. Humans set the direction; AI uses cpg to verify and generate implementations / text / reviews. Part 1's ③ Auto Review and ④ Self-Healing run on the same structure. Article writing isn't a special case — as long as cpg exists, AI-driven work always takes this shape.

What Changed / Bridge to Part 3

That covers the inside of cpg. A closing summary of how it affects cortex as a whole:

1. I stopped running grep

Without knowing file names or symbol names, I can get the relevant code back by just describing what I want to do. The combination of 120+ apps and a team of one works because of this, more than anything else.

2. Auto Review produces context-grounded comments

The [Graph] / [Impact] / [Doc] / [Security] level comments Part 1's ③ Auto Review produces all stand on cpg. The substance is review carried out with the entire codebase as context — that's the real benefit of the cpg integration.

3. Self-Healing can trace from error origin to root cause

Part 1's ④ Self-Healing can hop from a Grafana alert → code → dependent tables → related docs in one graph traversal because cpg exists. It fires with usecase: "bug" and takes the shortest path from error to root cause.

4. The static-analysis code graph is working somewhere else

I said "we abandoned code inference" at the top, but that was specifically for cortex itself. For the external-facing production repositories (the core of the business), a different approach supplies context, and static analysis continues to run there. More on that in a separate post.

Most AI coding setups try to make the AI better at reading an unchanged repository. cpg takes the opposite approach: change the repository's information structure so AI has a first-class semantic map to read. That's the line between "another GraphRAG" and what cpg actually is.

In that sense, Product Graph is literally a knowledge graph of the AI, by the AI, for the AI: generated alongside AI-written code, maintained through AI review, and consumed by AI agents as their primary map of the product.

Coming up in Part 3 — automated PR review: the full pipeline of automated PR review built on top of cpg — from GitHub webhook ingestion through AI review / automated fix / automated merge / parallel deploy. What happens when Auto Review fires with usecase: "code-review", how [Graph] Critical comments are generated, and the worktree mechanism that lets AI apply fixes and push back.

Minimal post (test fixture)

Ryosuke Tsuji — Sun, 17 May 2026 16:48:43 +0000

Minimal test fixture used by $slug.test.tsx. No headings, no tags — covers the
null branches of TOC rendering and tag-list rendering in routes/posts/$slug.tsx.

Slugs prefixed with _ are excluded from /posts listing (production publishing
surface) but remain reachable via direct getRenderedPost(slug, lang) (the
virtual:rendered-posts lookup that backs /posts/$slug) so test fixtures can
be SSR'd without polluting the index.

Building a Real AI Harness: Auto-Reviewed PRs, Self-Healing Ops, and Non-Engineer Contributors (Series Intro)

Ryosuke Tsuji — Tue, 12 May 2026 16:34:39 +0000

Hi, I'm Ryan, CTO at airCloset.

In my previous posts I've introduced the full picture of our 17 internal MCP servers, an MCP server that searches 991 internal tables in natural language, a custom Graph RAG for measuring initiative impact, and the Sandbox MCP that lets non-engineers publish AI-built apps safely.

All of those run on top of an internal AI development platform we call cortex. This post is the first in a series about cortex itself — the platform, the design choices, and the operational experience.

Series Index

#	Theme	Key scene	Article
1	Series intro: cortex's harness	PRs auto-merge / incidents self-heal before you notice	this post ← you are here
2	Product Graph (cpg)	Code, docs, DB, infra unified into one graph	cortex-product-graph
3	AI PR review	webhook → AI review → auto-fix → squash merge	cortex-auto-review
4	Self-Healing + observability + auto-added guardrails	Alert → AI investigates → fix PR + new lint/type gate → auto redeploy + same-pattern writes get auto-rejected	cortex-self-healing
5	Democratizing the maintenance phase	Domain experts open PRs to production; the harness owns the quality gate	cortex-non-engineer-prs
6	Series Final	The underlying philosophy plus a retrospective on the failures and lessons	cortex-philosophy

Two Scenes, Up Front

Scene 1: PRs merge themselves

Monday morning. An engineer implements a feature locally, pushes a branch, opens a PR.

A few minutes later, the AI reviewer comes back with REQUEST_CHANGES. Multiple comments:
- "This data formatting duplicates formatRow() in the shared package. Please consolidate."
- "You changed an API response type, but the related docs (docs/api/...) still describe the old shape."
A separate AI agent spawns a worktree, applies the fixes, pushes a follow-up commit
Re-review comes back as APPROVE
Auto squash-merge
GitHub Actions detects only the changed stacks and deploys them to Cloud Run / Cloudflare Pages

No human touched any of this. The engineer refreshes the PR tab and notices it's already merged.

Scene 2: Incidents fix themselves before you notice

7 AM. A Grafana alert fires: "BQ pipeline failed 3 times in a row."

An AI receives the webhook, fetches the error logs from Loki via the Grafana MCP
Walks the Product Graph (implementation name: cortex-product-graph — a unified knowledge graph of the codebase, docs, DB schemas, and infrastructure definitions; covered later in this post and in Part 2) to trace the pipeline's code, dependent tables, and related docs, identifying the root cause
Opens a fix PR
AI reviewer APPROVE → auto squash-merge → automatic redeploy

By the time the engineer logs in at 9 AM, Slack already shows: "pipeline patched." The only incidents engineers personally handle are the ones AI genuinely can't crack.

What's behind both scenes is the dev environment described in the rest of this post.

Industry Context — "Harness Engineering"

Before I get to cortex, one paragraph of context. Over the past six months, the practice of building proper foundations for AI agents in production has crystallized into a recognized industry trend.

"Harness" itself isn't a new word. In AI specifically, it traces back to EleutherAI's lm-evaluation-harness (2020) — the LLM evaluation framework that put the term in active use. What changed in the past six months is its elevation into an engineering discipline for LLM agents in production:

Feb 2026: OpenAI published "Harness engineering: leveraging Codex in an agent-first world", describing how a small internal team led by Codex shipped 1 million lines in 5 months
A few days later, Mitchell Hashimoto (HashiCorp co-founder, Terraform creator) distilled it into the formula Agent = Model + Harness
April 2026: Martin Fowler (author of Refactoring, ThoughtWorks Chief Scientist) published "Harness engineering for coding agent users", establishing the Guides (proactive controls) / Sensors (reactive controls) framing
Same month: Anthropic and Cursor each published their own harness write-ups

The catchphrase that's gone viral: "2025 was the year of agents. 2026 is the year of harnesses."

The framing is: the model itself is rapidly commoditizing (the gap between Claude / GPT / Gemini is narrowing from the user side). Where you actually get differentiation is how you design the harness — the foundation that lets AI run in production.

cortex is most cleanly read as a real attempt to build that "harness" inside a real company. In this post I'll organize cortex using Fowler's Guides / Sensors framing.

From here, I'll show how the "harness beats model" thesis takes concrete shape on cortex.

Who Builds the Code

For the first few months, I built 100% of cortex by myself. The accurate framing isn't "without a harness, others can't safely PR" but rather "without a harness, no one — including me with extra hands — could ride this thing."

Even back then, between our Google Meet recording pipeline (Japanese), about half of the 17 MCP servers, and a long tail of unpublished features, roughly 50 loosely-coupled applications were already running. Each one had its purpose, background, and data flow documented carefully. But the volume was such that even with AI in the loop, you couldn't realistically have it read all the relevant docs and absorb the whole picture for any given change. The codebase had outgrown what a person — or an AI given pieces — could hold in their head at once.

Recently, with the harness in place, non-engineers (business-side managers, PMOs, etc.) have started shipping PRs to cortex too. As of writing, the cumulative commit ratio is ~91% me, ~9% other recent contributors.

If you imagine non-engineers opening PRs against a production repo, "can quality really hold?" is the obvious question. In cortex, the answer is yes, because AI review and automation own the quality gates:

PRs missing annotations, tests, or lint cleanliness get REQUEST_CHANGES from the AI reviewer
A separate AI agent applies the fixes
Until everything is satisfied, nothing merges

So whoever writes a PR — engineer or not — at the moment it merges, the same quality bar is met. The key point: it's not "you can write freely," it's "you can write inside rails that don't let you derail." The author's job stops at "communicating the intent precisely"; the harness owns code correctness.

The shift is from "X could write that because they're X" to "X can write that because of cortex." That property only emerges once the harness is built — and it's the core of cortex's design.

What's Running

cortex consists of microservices, jobs, MCP servers, web frontends, Cloudflare Workers, and so on. As of writing, there are 123 apps. The features I've already covered in past posts are each composed of multiple apps — but even adding them up by feature, only about 10% of cortex has been written about. The remaining 90% hasn't appeared in a post yet. A few examples:

A unified product UX measurement web app — UX metrics, screen analysis, funnels, and error analysis in one place
A dev-org portal web app — KPIs (bug rate, etc.), per-member GitHub Activity, QA evaluation results, plus an AI chat that answers natural-language questions about KPIs via Agentic RAG
A family of Slack bots for operational support:
- A config bot that lets you manage job configurations (DBs, attendance SaaS, Google Drive, etc.) directly from Slack
- An accounting-assist bot that takes invoice OCR and drafts payment requests / expense filings in our accounting SaaS
- In-channel knowledge search, issue/request management, meeting creation; a BigQuery cross-table RAG bot; a Google Drive cross-corpus RAG bot
- A marketing bot that returns insights (trend, creative analysis) from BigQuery marketing data
An APM auto-analysis agent that runs daily on monitoring-SaaS APM data, detects performance issues, and opens tickets in our issue-tracking SaaS
An AI-bot auditor bot that runs E2E tests against the Slack bots above and detects spec drift

…and so on. Each will get its own dedicated post later in the series.

Scale at a glance:

	Count
apps (microservices, jobs, MCP servers, web, etc.)	123
packages (shared libraries)	66
MCP servers	19
Pulumi stacks	110
TypeScript (implementation)	~630K lines
Tests	~560K lines
Markdown documentation	~110K lines / 389 files
Duration	~5 months (intensive development: ~4 months)
Merged PRs	~790

The 4-Element Flywheel — cortex's Harness

What lets "~4 months of intensive dev, mostly solo" coexist with "non-engineers shipping into the same repo" is a harness design that delegates quality to AI and automation across every layer.

cortex's harness is structured as a flywheel of 4 elements, mapped to Fowler's Guides (proactive) / Sensors (reactive) split, that mutually reinforce one another.

① Product Graph (Guides — supplying the right context)

All of cortex — code, documentation, DB schemas, infrastructure definitions — is indexed in real time as a single unified graph. It's queryable via MCP through semantic search.

"Where is the code that calculates this KPI?" → "Which BQ tables does that code touch?" → "What are those tables' column definitions?" → "What docs are related?" — all of these can be answered from a single query traversal. That graph becomes the context source for everything the AI does.

This is the foundation that "structurally reduces how often the AI gets confused." Where grep tells you "where the string appears," the Product Graph tells you "what is connected, why, and how." Implementation details come in Part 2.

② Lint / Quality Gates (Guides — physically blocking deviations)

eslint-disable / oxlint-disable are forbidden anywhere in the repo. In hand-written code, occurrences of : any / as any / TODO / FIXME are 0 (excluding generated files and unavoidable external-library cases). Type checking (using tsgo — Microsoft's Go port of the TypeScript compiler, ~10× faster than tsc; we use it to keep CI time down) runs on the entire codebase in CI.

On top of that, test coverage is enforced at ≥90% for statements / branches / functions / lines. Lowering the threshold to pass is forbidden — you write tests instead.

With every escape hatch sealed, even when the AI writes wrong code, it doesn't merge. This is also what stabilizes AI review judgments downstream.

③ Auto Review (Sensors — auto-fixing until the bar is met)

Scene 1 above is exactly this. The implementation-side note: AI review here isn't "lint with extra steps" — every comment is grounded in Product-Graph traversal of the actual impact. That's where it earns its keep. To give you a feel, comments that actually fire fall into categories like:

[Graph] Critical — missing annotation that breaks an edge in the graph
[Impact] Critical — a BQ MERGE statement referencing a column not present in the existing target table; would fail in production
[Doc] Critical — code change that left related docs stale
[Security] Minor — execSync doing string interpolation on an env var, opening a command injection vector

What you might mentally classify as "AI review" — surface-level — isn't this. Comments here are produced with the entire codebase carried as context, which is what the Product Graph integration buys you.

The only PRs that actually need a human are "AI review hits a hard case." Day-to-day PRs go from push to merge without anyone touching them.

④ Self-Healing (Sensors — re-injecting production anomalies into the loop)

Scene 2 above is exactly this. Starting from a Grafana alert, the AI traces the root cause through Product Graph + Loki + git blame, opens a fix PR, and pushes it through ③ Auto Review until it's auto-merged. Re-injecting anomalies into the loop is the essence of Sensors. Details in a later post.

What Makes It a Flywheel

These 4 elements mutually reinforce one another:

① Product Graph exists, so ③ Auto Review can comment with real impact awareness
② Lint enforces the ground rules, so ③ Auto Review can assume "everything in the codebase meets the bar"
③ Auto Review exists, so new code lands in ① Product Graph with correct semantic annotations
④ Self-Healing's incidents loop back through ③, maintaining the quality bar all the way back to ①

The harness's effectiveness scales with the size of the codebase, not against it.

Supporting Foundations

Three foundations make the 4 elements possible (covered in detail in Part 4):

Tests and coverage: ~630K lines of implementation, ~560K lines of tests (impl : test ≒ 1.13 : 1)
Documentation: ~110K lines / 389 files, written for both humans and AI, also ingested as Document nodes in the Product Graph
Observability: Frontend = Faro, backend = OTel, infrastructure and CI logs all consolidated in Grafana. The AI sees the same data humans see. Gemini API token usage and cost are tracked separately in Prometheus.

Technical Foundation

cortex is a full-TypeScript monorepo.

Layer	Stack
Applications (`apps/`)	TypeScript (Hono, TanStack Router, Vite, etc.)
Shared packages (`packages/`)	TypeScript
Infrastructure (`infra/`)	TypeScript (Pulumi)
Edge (`worker/`)	TypeScript (Cloudflare Workers)
Lint plugins	TypeScript
Doc scripts	TypeScript (tsx)

Having everything in one language is a much bigger win when viewed from the AI's side than from a human's. Specifically:

You can feed the AI ASTs and type definitions directly as context — no language boundary fragments the picture
Refactors don't cross language boundaries — one ESLint plugin can inspect and auto-fix apps/, packages/, and infra/ together
Edges don't break in the Product Graph — for example, a Cloud Run service definition (infra/, TS) connects in a single graph to the Hono route (apps/, TS) it actually invokes

When you ask the AI "what does this change affect?", the reason it can hop infra → apps → packages and answer in one round-trip is that all of this is one language.

Build is parallelized via Turborepo and pnpm workspaces. Deploys go through GitHub Actions, which detects only changed stacks and applies them in parallel via Pulumi.

Numbers (snapshot at time of writing)

	Value
Duration	~5 months (intensive development: ~4 months)
Commits	~4,000
Merged PRs	~790
% of commits authored by me	~91%
apps	123
packages	66
MCP servers	19
Pulumi stacks	110
TypeScript (implementation)	~630K lines
TypeScript (tests)	~560K lines
Markdown documentation	~110K lines / 389 files
`as any` / TODO / unjustified lint-disable in hand-written code	0 (excluding generated files / unavoidable external-library cases)
Coverage gate	90% (statements / branches / functions / lines)

The PR-flow Switch That Multiplied Throughput

Up until April, I was AI-assisted reviewing every change carefully on my own machine and then committing directly to main. The review bar was unchanged, but throughput was bottlenecked on my hands.

In April, switching to fine-grained, PR-based operation (auto review → auto fix → auto merge) dramatically changed the per-month merged-PR count:

Month	Merged PRs
2026-02	10
2026-03	23
2026-04	518
2026-05 (through the 10th)	235

A ~22× jump between March and April. Total commits actually went down (because committing directly to main was replaced by going through PRs), so this isn't "I wrote more code." This is "the manual review step got replaced by the harness, and the throughput ceiling moved." The 22× is exactly the moment a human reviewer was swapped for Auto Review — clean evidence of the flywheel property where the harness's effectiveness scales with codebase size.

What's Required for These Numbers to Hold

These numbers are not explained by "we use AI" alone. The prerequisites:

Full TypeScript monorepo — code, tests, infrastructure, scripts all under one static-analysis system
Composable Architecture — packages/ holds reusable parts; apps/ compose them. Direct imports between apps/ are forbidden — everything routes through packages/. This is what guarantees components don't interfere with each other.
Strict quality gates — lint / coverage / annotations are run "no lowering, no working around"
Unified graph — code, docs, DB, infrastructure on a single graph as the foundation that lets the AI act with context
Auto PR review / auto fix / auto merge / auto self-healing — the harness that swaps the rate-limiting manual step for AI
Unified observability — humans and AI see the same data (OTel + Faro + Prometheus)

The design has to be in place first, and AI runs on top of it. That's what makes both volume and quality possible at the same time.

Composable Architecture in particular is what drives the headcount-of-one production. Because components don't interfere, multiple Claude Code sessions can run in parallel on different parts of the codebase. In practice, I've run up to ~10 sessions in parallel at peak — this multiplies with the harness's effectiveness.

It's system design, not magic. Each piece will get its own deep-dive in this series.

Some Honest Caveats

If you've read this far, it might sound like everything runs perfectly on autopilot. It doesn't. Three things I want to be upfront about:

1. High code quality doesn't prevent bugs.

What the harness protects is "correctness of the code" — not "correctness of the spec." Even when implementation is clean, getting the spec interpretation wrong still ships bugs. AI review can catch "code contradicts the documented spec," but if the spec itself is wrong, the issue sails right through. That part is still a human responsibility.

2. The work is split deliberately.

New pipelines that connect to external APIs, and anything touching secure data, are handled by engineers. Non-engineers mostly work on modifications to features that already exist (peeking at our business-side members' PRs makes it concrete pretty quickly). "Non-engineers can develop too" means "the harness provides rails they can't derail from, so they can safely modify in maintenance mode" — not "anyone can build anything from scratch."

3. This level of automation works because it's an internal platform.

Yes, cortex's full-auto deploy works partly because Composable Architecture cleanly separates apps and infrastructure. But honestly, a big part of it is that this is an internal-only platform. If something breaks, only employees are affected, and we can roll back fast. The same approach can't be applied directly to consumer products or systems where downtime is immediately critical (warehouse management, for example). We've started moves to close that gap on the consumer side too, but that's a separate post.

Series Roadmap

The series is planned as 6 parts.

Part 1: Series Intro (this post)
The big picture of what cortex is and why it works in "harness" form. The map to the rest of the series.

Part 2: Product Graph — code, docs, DB, infrastructure as one unified graph ★ recommended next
The implementation side: how the unified graph is built and maintained. What happens when you take the design principles from the Agentic Graph RAG MCP post and apply them to the entire cortex codebase.

Part 3: AI reviews, fixes, merges, and deploys PRs
GitHub webhook → AI review → on REQUEST_CHANGES, AI fixes via worktree → auto squash merge → changed-stack detection → parallel deploy: the full pipeline.

Part 4: Incidents self-heal, guardrails self-strengthen
Grafana alert → AI investigation (Loki + Product Graph + git blame) → fix PR + new lint/type gate → auto merge → automatic redeploy: the auto self-healing system. Also covers the full OTel + Loki + Mimir + Tempo + Faro stack, Gemini cost tracking, and how the quality gates are designed to be "non-loweriable, non-bypassable, and self-growing."

Part 5: Scaling the harness from cortex to toC services
The first half covers how business members can already open PRs directly to cortex -- and where that breaks (additions to existing pipelines work; new pipelines and architectural changes still need humans in the loop). The second half is the roadmap and the thinking behind scaling cortex's harness across the whole product org (multiple services, multiple infra stacks, multiple teams).

Each post stands on its own, but Part 2 (Product Graph) is the foundation for the others, so the recommended reading order is Part 1 → Part 2 → any.

Cadence: Tuesdays or Thursdays, 8–10 AM JST.

Closing

Building cortex, what's struck me is that in an AI-era dev environment, "absorbing everything that comes after the writing" wins over "reducing the burden on the writer". Tests, lint, types, coverage, code review, incident response — instead of "these get in the way, let's reduce them," the choice that worked was "have the AI do all of them, without compromise." The counterintuitive result is that quality and dev speed both go up at the same time.

And it expands two things — how much one engineer can ship, and how much non-engineers can participate — well beyond what was possible before. That's the texture of the "harness" we've built on top of cortex.

In subsequent parts, I'll walk through the individual mechanisms that make this work.

→ Part 2: Product Graph — code, docs, DB, infrastructure as one unified graph

Graph RAG Isn't a One-Shot Anymore — The Case for Agentic Graph RAG MCPs

Ryosuke Tsuji — Thu, 07 May 2026 09:57:32 +0000

Hi, I'm Ryan, CTO at airCloset.

Over my last few posts, I've introduced internal MCP servers we've been building: DB Graph MCP, the full picture of our 17 internal MCP servers, Biz Graph, and Sandbox MCP.

DB Graph is built from ORM parsing. Biz Graph extracts initiatives from meeting slides and uses a hand-designed Week node structure. Sandbox MCP is an app deployment platform. The purposes and implementations are completely different — but as I was writing each piece, I noticed that the design ideas at the root are the same.

This post is about that root. Agentic Graph RAG — a design frame we keep coming back to whenever we build graphs across different domains.

If you've heard "Graph RAG" before — maybe Microsoft's open-source project — wait a moment. The same words mean different things in the era when retrieval was assumed to be a single shot versus the era when AI agents are everywhere. The optimal design changes completely. This post is about the latter — a new way to think about Graph RAG in a world where Claude Code, Codex, and friends are doing the orchestration.

What Is RAG, Really?

Quick refresher. Skip if this is familiar.

RAG (Retrieval Augmented Generation) is the umbrella term for any technique that retrieves related information from external data and mixes it into the prompt before the LLM generates an answer.

Why was this needed? In the early days of generative AI — late 2022 and through 2023 — we ran into three problems:

Tiny context windows: GPT-3.5 had 4K tokens, early GPT-4 had 8K. You couldn't fit your internal docs in there.
Stale model knowledge: The model didn't know anything past its training cutoff. It certainly didn't know your internal data.
Hallucination: It would confidently fabricate answers when it didn't know.

The RAG idea was: every time the user asks something, fetch the relevant chunks from external data and feed them in before generation.

Vector RAG — The First Practical Answer

The earliest RAG implementation that actually caught on was Vector RAG.

The recipe is simple:

Split documents into small chunks (say, 500 tokens each)
Embed each chunk with a model (e.g., 1536-dim vectors)
Store them in a vector DB (Pinecone, Weaviate, pgvector...)
Embed the user's question with the same model, retrieve the top-k closest by cosine similarity
Stuff those chunks into the prompt and call the LLM

For its time, this was a great invention. Because:

Search is fast: tens to hundreds of milliseconds
No training needed: feed it docs, it's instantly searchable
Domain-agnostic: works for legal documents, medical charts, internal wikis — the same machinery
Rides model improvements: better embedding models, better recall

And critically, agent technology was still immature. OpenAI's Function Calling shipped in June 2023, was unstable for a while, and running a meaningful agentic loop of multiple tool calls was both slow and expensive. So RAG was designed around the assumption: one retrieval has to fetch everything you need. Vector RAG was perfectly tuned for this constraint.

The Limits of Vector RAG

But anyone who runs Vector RAG in production discovers the same thing fast: it can't follow relationships.

Take a question like:

"How did last month's SNS ad campaign affect new member signups?"

Vector search returns chunks that are textually similar to the question. The campaign description might come up. But:

When was the campaign actually running?
What were the new-member numbers during that same period?
What happened with previous similar campaigns?

These aren't textual similarity — they're structural traversals across data. Embedding maps "spring SNS ads" and "spring promotion initiative" close together, but it cannot start from "ran from March 1 to March 31" and reach "new member counts in that same period". That's not a similarity problem; that's a join problem.

On top of that:

Chunk boundaries kill context: related info gets split across chunks
Top-k cliff: critical info at rank 11 is invisible
Granularity mismatch: questions like "summarize the whole thing" can't be answered by collecting chunks

Vector RAG nailed "fetch text similar to the question in one step." It's weak at "follow data through structural relationships." That's the gap that Graph RAG was born to address.

Graph RAG — Search That Follows Relationships

The basic idea of Graph RAG: extract entities (people, organizations, concepts) and relationships (belongs-to, affects, references) from your documents, store them as a graph, and at query time traverse the graph to gather information across multiple hops.

This handles questions like our SNS-ads-and-new-members example — anything that requires multi-hop reasoning.

Classical Graph RAG — Built for the One-Shot Era

The most well-known implementation right now is Microsoft's GraphRAG, released in 2024. The papers are well-written and I have a lot of respect for it. But the design philosophy is squarely from the one-shot retrieval era.

Roughly, Microsoft GraphRAG does this:

Entity extraction: feed the entire corpus through an LLM to extract entities and relationships
Community detection: find graph clusters (communities) using the Leiden algorithm (a community detection method)
Hierarchical summarization: have the LLM summarize each community. Then summarize groups of communities into higher-level summaries
Query time: pick the relevant community for the user's question, dump its summary into the prompt, answer in a single shot

Why is the preprocessing this heavy? Because of the assumption underneath: "calling tools many times at query time isn't realistic". Function calling loops were slow, expensive, and unstable. So you preprocess the entire corpus with an LLM, build community summaries, and front-load the work to make query-time retrieval a single hop or two.

This wasn't a design failure — it was the rational answer for that era. LangChain's RetrievalQA, LlamaIndex's query engines — all of them were built on the same premise: "retrieval is single-shot, generation is one-turn."

What Classical Graph RAG Solved, and Didn't

What it solved:

Relationship-aware search (community summaries even cover "the big picture")
Multi-hop questions like "the relationship between Sam Altman, OpenAI, and Microsoft"

What it didn't solve cleanly:

Construction is expensive: extracting entities from a large corpus via LLM costs real money
Schema is at the LLM's mercy: the entities and relationships extracted are whatever the LLM thinks. This works fine for public-knowledge corpora (papers, news, etc.), but for domains that lean on internal tacit knowledge, the extracted units don't always match what's meaningful for the business
Updates are heavy: every new document means recomputing communities
Sometimes off-target: community summaries get over-abstracted, and the specific information you actually need falls out

Honest disclaimer: I haven't seriously run classical Graph RAG in production myself. By the time I started building graph-based MCPs in our company, Claude Code was already running on my laptop, and I started from a world where agents calling tools many times was the default. As a result, I never actually needed the heavy "compress the answer ahead of time" preprocessing of community summaries. If AI can re-fetch as many times as needed, the graph just has to hold the facts accurately.

The flip side: if I had been doing this in 2023, I likely would have ended up on the same path as community summaries. The problems classical Graph RAG was solving are real — the underlying assumptions just changed faster than the design.

Things Changed — The Agentic Era

From late 2024 through 2025, the landscape shifted:

Production-grade agents arrived: Claude Code, OpenAI Codex — agents that can run long tasks while orchestrating their own tool calls
MCP (Model Context Protocol) landed: tool descriptions became a standardized contract the model can read
Tool-use accuracy from Sonnet/Opus-class models: "pick the right tool from 20" became reliable
Long context windows + prompt caching: stacking many tool calls in a session is now economically reasonable
stop_reason: tool_use as a natural loop: the model itself decides "I have enough info" or "I need to look more"

When all of these line up, the assumption "we can't afford retrieval as a loop" no longer holds. Five tool calls per session, ten, twenty — that's now the norm.

The constraint Microsoft GraphRAG was designed against — "loops are expensive at query time" — has dissolved.

This isn't to say Microsoft GraphRAG is "outdated." It was the right answer for its constraints. The constraints just changed, and so does the optimal answer.

Agentic Graph RAG — Deterministic Retrieval, AI-Driven Orchestration

Here's the thesis. In one line:

Each retrieval step is deterministic. Only the orchestration is AI.

For context: "Agentic Graph RAG" isn't a term I coined. Neo4j's NODES AI 2026 featured a session titled "Agentic GraphRAG," and O'Reilly is publishing Agentic GraphRAG by Anthony Alcaraz and Sam Julien in November 2026. The industry as a whole is pivoting from "one-shot Graph RAG" toward "agent-driven Graph RAG." This article is my attempt to put words around the design we'd been arriving at independently inside our company.

That said, when "Agentic GraphRAG" is used in public contexts, the dominant framing centers on agents automating the graph construction itself (Neo4j's talk above is in that lineage). What this article takes from that broader idea is specifically the query-side agentic pattern. We still hand-design the graphs because the domains we target (internal DB schemas, initiatives × KPIs, codebases) lean heavily on internal tacit knowledge — for now, hand-designing produces better results in practice. We aren't rejecting auto-construction in principle; we're applying the query-side concept to graphs we still build by hand.

Vector RAG had probabilistic retrieval. Embedding cosine is an approximation, and it sometimes misses. Hallucination starts at the retrieval layer.

Classical Graph RAG runs retrieval once at query time. Heavy preprocessing prepares "the answer itself" in advance, and at query time you just look it up.

Agentic Graph RAG sits between these two.

The graph is designed by humans. Our domains lean on internal tacit knowledge, so humans deciding "this is the granularity I want to slice the data with" produces better results.
Each tool call is deterministic. Pass an ID and you get the connected nodes and edges. There's no embedding wiggle.
The AI only judges which tool to call next, what ID to pass in, and when to stop.

The result: errors get localized. Retrieval itself is deterministic, so the only places to be wrong are "AI picked the wrong starting point" or "AI stopped too early." The data in the response is the truth.

Tool Return Values Become a Runbook

The most important design move in Agentic Graph RAG: the tool's return value tells the AI what to do next.

This is different from a regular API. Regular APIs answer the question they were asked. MCP tools are in conversation with an AI. The other side of the conversation needs not just an "answer" but candidates for the next move.

Concrete example.

When the AI calls DB Graph MCP's search_tables tool, it gets:

5 tables matched (vector similarity ranked):

warehouse.return_package_table (postgresql) (distance: 0.2557)
warehouse.receipt_record_table (postgresql) (distance: 0.2720)
inventory.receipt_confirmation_table (mysql) (distance: 0.2921)
warehouse.receipt_record_detail_table (postgresql) (distance: 0.2951)
app.return_status_change_history_table (mysql) (distance: 0.3170)

※ Schema and table names are anonymized — they map to internal system names.

Notice that the response itself contains the next tool's argument. The qualified name warehouse.receipt_record_table is exactly what get_table_detail(table_name: "warehouse.receipt_record_table") expects. If the AI decides "let me look at the details," it just copy-pastes.

The get_table_detail response is even more direct:

# warehouse.receipt_record_table
DB: POSTGRESQL / ORM: typeorm / Repo: warehouse-api

## Columns (9)
- id: int [PK, AI, NOT NULL]
- shipping_order_id: varchar [NOT NULL]
- status: enum [NOT NULL, default=IN_PROGRESS]
- ...

## References (2)
- shipping_order_id → warehouse.shipping_order_table.id (explicit)
- operator_id → warehouse.user_table.id (explicit)

## Enum / Status Definitions (2)
- Status: COMPLETE = received, IN_PROGRESS = in progress
- Type: RENTAL_RETURN = rental return, ...

This response implicitly tells the AI:

"The meaning of status is in the Enum definition" → don't guess, read it
"There are FK references" → if needed, you can follow them with trace_relationships
"There's no direct FK to the app schema" → you'll need a different path

In other words, the tool's response is a runbook for the AI. The AI reads it and assembles the next move on its own.

Now look at the response from sql_query_database:

**app** (staging) — 1 row

| id     | status   | warehouse_order_code |
|--------|----------|----------------------|
| 98765  | RETURNED | SO-2026-00012345     |

> **Table**: Manages the full lifecycle of delivery orders...

### Column descriptions
- **status**: Delivery status (1=awaiting shipment, 2=ready, 3=delivered, 4=returned, ...)
- **warehouse_order_code**: Link code to the warehouse-side shipping order

### Related tables
- → **app.member_table** (user_id → id)
- → **app.plan_master** (plan_id → id)
- ← **app.order_history_table** (delivery_id → id)

Column descriptions and related tables are auto-attached below the query result. This is composed dynamically from the graph data we cached in BQ. Reading that "warehouse_order_code links to the warehouse side," the AI immediately decides "next, look up the warehouse table by this code."

Nobody had to tell the AI "now look at warehouse." The response itself is the instruction.

DB Graph in Action — A Production Investigation in 4 Steps

Here's the full flow (also shown in the DB Graph MCP article).

The scenario: a CS agent asks, "This member shows 'returned' in the app, but did the warehouse actually confirm receipt?"

Step 1: Find tables in natural language (vector-similarity entry-point search)

search_tables(query: "return processing confirmation", search_type: "semantic")
→ warehouse.receipt_record_table, warehouse.return_package_table, ...

Step 2: Look at the details (deterministic detail retrieval)

get_table_detail(table_name: "warehouse.receipt_record_table")
→ status=COMPLETE means "warehouse received it"
→ shipping_order_id connects to warehouse.shipping_order_table

Step 3: Find the path to the other schema (deterministic graph traversal)

trace_relationships(table_name: "warehouse.shipping_order_table", direction: "both")
→ from the app side, connection goes through an intermediate table
search_tables(query: "warehouse linkage")
→ app.warehouse_linkage_table (warehouse_order_code maps to warehouse.shipping_order.code)

Step 4: Verify against real data (deterministic query execution)

sql_query_database(database: "app", sql: "SELECT ... WHERE user_id=12345 AND status='RETURNED'")
→ warehouse_order_code = "SO-2026-00012345"

sql_query_database(database: "warehouse", sql: "SELECT ... WHERE code='SO-2026-00012345'")
→ receive_status = COMPLETE → confirmed by warehouse

The crucial part: the AI built this 4-step flow autonomously. The human only asked the original question. Each step's response carried "look here next" inside it, so the AI could keep composing the next call correctly.

And each step's retrieval is deterministic. The enum definitions for status in warehouse.receipt_record_table are facts pulled from the graph — not values the AI invented. warehouse_order_code = SO-2026-00012345 is real data — not an ID the AI fabricated.

This is a different experience from both Vector RAG and classical Graph RAG. Vector RAG is "return all the text in one shot," but hallucinations slip in. Classical Graph RAG is "return the community summary in one shot," but specifics get lost in summarization. Agentic Graph RAG is "fetch as many times as you need, but every fetch returns nothing but facts."

The Same Pattern, Across Many Graphs

This pattern — what we adopt: human-designed graph + deterministic retrieval tools + responses that double as AI runbooks — isn't limited to DB Graph and Biz Graph. We use it across many MCP servers internally.

Including the ones I mentioned by name in the 17 internal MCP servers post, the lineup looks like this:

Graph	What it covers
DB Graph	991 tables × 15 schemas across the company
Biz Graph	5,000+ initiatives × 4,000+ KPIs
Code Graph	Functions, APIs, events across all repos
Cortex Product Graph	Code + DB + docs + infra unified for the cortex repo
Service Product Graph	API → DB dependencies per service

The structures are all different. DB Graph from ORM parsing. Biz Graph from meeting-slide extraction plus hand-designed MetricDomain. Code Graph from static analysis. Product Graph from JSDoc annotations on top of everything else. Different sources, different assembly.

But the shape from the MCP-tool side is identical:

Entry-point search: vector or substring to find "around here" (the only place fuzziness is allowed)
Detail retrieval: pass an ID, get facts (deterministic)
Relationship traversal: jump from ID to ID along edges (deterministic)
Embed next-step hints in responses: related IDs, enum definitions, annotations, links

This 3+1 template is the universal Agentic Graph RAG shape. Different graph internally, identical surface. From the AI side, they all feel the same — Claude Code uses DB Graph and Code Graph and Product Graph with the same "search → drill down → traverse" rhythm.

Of the graphs above, only DB Graph and Biz Graph have dedicated deep-dive posts so far. Code Graph and the Product Graph family will get their own writeups; for this post, they're listed as fellow examples of the pattern.

A Designer's Checklist

For implementers. Below are the six things I always keep top of mind when adapting Agentic Graph RAG to a new domain.

Things I keep top of mind when building an Agentic Graph RAG:

1. Choose the graph-construction method based on the domain

If the domain leans on internal tacit knowledge, humans deciding the nodes and edges produces better results. Sometimes you intentionally design a structure that doesn't exist naturally — Biz Graph's "Week node" and "MetricDomain" are examples. The design is what determines quality.

Conversely, when the domain is mostly public knowledge (papers, news, public docs), having agents automate construction is a strong option (the Neo4j talk lineage). This article assumes the former.

2. Make retrieval deterministic

The entry-point search may use vector similarity (to accept natural-language queries). After that, "get details by ID" and "follow relationships from this ID" must always return definite values via graph traversal. Using similarity here lets hallucination back into the retrieval layer.

3. Tool granularity: search → detail → traverse

Don't pile everything into one giant tool. Split into search-style entry points, detail lookups, and traversal/data tools. The AI understands the difference and uses them appropriately.

4. Tool descriptions are AI runbooks

Write tool descriptions as execution guides for the AI, not human documentation. "If you see this kind of response, call this tool next." "In this situation, format the argument like this." As I mentioned in the Sandbox MCP post, this directly determines how smart the agent appears.

5. Embed "next move candidates" in responses

Don't just return data. Return:

Related IDs: where to traverse next (FK targets, similar initiatives, parent commits)
Enums and definitions: so the AI can interpret values without guessing
Annotations and warnings: DEAD flags, deprecation marks, PII (personally identifiable information) redaction notes

At a granularity where the AI can read "this is what I should do next" out of the response.

6. Let the AI do the summarization

Don't pre-bake "community summaries" or similar on the server. The AI assembles facts case by case at the right granularity. Return facts. Let the AI interpret.

Limits and Caveats

Heads up. This approach has clear weak spots. If you're considering adopting it, read this section before you start designing.

Agentic Graph RAG is not a silver bullet. To be honest:

Quality depends entirely on graph design. If the schema doesn't carve up the domain correctly, no number of tool calls will reach what you want. And in tacit-knowledge-heavy domains, the call about which nodes/edges to include is one only someone deeply familiar with the domain can make.
If the agent picks the wrong entry, it falls into a deep hole. Miss at the first search_* and the rest of the graph traversal goes sideways. Entry-point quality matters.
Cost is tool-call-count × context length. 10–20 tool calls per session add up tokens straightforwardly. Prompt caching and progress reporting via MCP help, but you have to keep an eye on it.
Hallucination doesn't disappear — it relocates. From the retrieval layer to "entry point selection" and "stop judgment." But it's much narrower territory, so debugging and evals get easier.

The first item is the one designers should worry about most. In tacit-knowledge domains specifically, graphs aren't found — they're designed. I wrote this in the Biz Graph post too, and for these domains I don't think it can be overstated.

Summary

The three eras of RAG, in one table:

Era	Representative	Retrieval	Orchestration
Early days	Vector RAG	Probabilistic (cosine)	None (one-shot)
Function-calling era	Classical Graph RAG	Pre-summarized	Light, mostly one-shot
Agent era	Agentic Graph RAG	Deterministic (graph traversal)	AI assembles in many steps

Vector RAG made "search and dump some context" work. Classical Graph RAG packaged "follow relationships" into a single-shot lookup. Agentic Graph RAG separates "tools that return only facts, accurately" from "AI agents that orchestrate them in multiple steps."

The graphs we've built internally — DB Graph, Biz Graph, Code Graph, Product Graph family — they're all from the same lineage. The contents and construction differ, but in our domains they all share the same shape: "give Claude Code a human-designed graph through deterministic tools." Which is why, from the AI side, they all feel the same.

If you're building AI-native internal infrastructure, give this perspective a try. Don't hand the AI an answer. Hand it a map. It walks much further than you think.

And the quality of that map comes down to how deeply you understand the domain — at least for the domains where the relevant knowledge sits as tacit understanding inside people's heads. In those domains, the best AI systems are still built by the people who know the problem space best. Domain expertise hasn't lost value in the AI era — it's gained it. That's been my strongest takeaway from two years of building graphs across our company.

Cutting Self-Built MCP Server Token Usage by 90% — The Parking Pattern

Ryosuke Tsuji — Fri, 01 May 2026 01:10:27 +0000

Hi, I'm Ryan, CTO at airCloset.

In my previous posts I introduced the full picture of our 17 internal MCP servers, an MCP server that lets you search 991 internal tables in natural language, a Graph RAG MCP for measuring initiative impact, and the Sandbox MCP that lets non-engineers publish AI-built apps safely.

This time I want to share something that came out of running those in production — a small trick we use to cut token consumption on self-built MCP servers.

The Annoyance: MCPs Eat More Tokens Than You'd Think

The first surprise when extending an AI agent with MCP is that token consumption is higher than expected.

An MCP tool call is, at the end of the day, JSON-RPC over HTTP. Both the arguments the AI sends and the result the tool returns land directly in the conversation context. If you implement things naively:

Sending whole files as arguments → thousands of lines of source code stick to the context
Returning all DB query rows → a multi-thousand-row × multi-column table sticks to the context

A single tool call can easily consume tens of thousands of tokens, putting the Claude Code session straight into compaction.

It's worse than just inefficiency: above a certain row count, the response simply fails to come back at all because it exceeds MCP's payload size limit.

When we were ramping up our internal MCP fleet, this little mismatch was reliably making the tool experience worse.

The Pattern: Park the Big Stuff Elsewhere, Pass Only a Key

The fix is embarrassingly simple:

Take the parts that tend to grow and move them off the MCP wire. Pass only a reference key (or URL) through MCP itself.

Both the request side and the response side benefit from the same idea.

Direction	What to remove	Where to park it
Request	Large files / source code	GitHub, Drive, or any object store
Response	Large list data / query results	Spreadsheet / GCS / BigQuery

Two examples from airCloset.

Example 1: Lighter Requests — Sandbox MCP × Self-Hosted Git Server

Last time I wrote about Sandbox MCP, the platform that lets non-engineers publish AI-built apps internally. The first iteration was fully MCP tool-driven file uploads.

sandbox_write_file(app_name: "todo-app", path: "index.html", content: "<html>...")
sandbox_write_file(app_name: "todo-app", path: "app.js", content: "import ...")
sandbox_publish(app_name: "todo-app")

The moment apps got slightly bigger, this collapsed:

Constant chunking: hitting the payload size limit, the AI looped through "first half of file A → second half → first half of file B → ..."
Tokens going up in flames: full source code landed in the conversation context — a single deploy of a few-thousand-line app could burn tens of thousands of tokens
Retries made it worse: the AI would "verify after sending" by re-reading the same file with sandbox_read_file. Write → read → write loops

So we changed the contract: MCP only returns a URL; the actual content moves over git push.

# 1. MCP returns a git URL — no payload involved
sandbox_init_repo(app_name: "todo-app")
# → https://mcp-sandbox.example.com/git/sandbox/ryan/todo-app.git

# 2. AI runs git in the background — MCP isn't involved
git init && git add . && git commit -m "init"
git remote add sandbox <returned URL>
git push sandbox main

# 3. Only the deploy command goes through MCP
sandbox_publish(app_name: "todo-app")

git push gives us:

No file size limit
Differential transfer — second-time pushes are fast
Source code never lands in the MCP conversation context

From the AI's point of view, it's just "I got handed a git URL; I push to it." Fundamentally different in token economics.

By the way, we don't use GitHub Organizations here. Issuing GitHub seats for every employee wasn't worth the cost or operational overhead, and we already had a self-hosted Git Server on GCE for a different purpose, so we just added one repo (sandbox-apps). The "park" doesn't have to be something you build from scratch.

Example 2: Lighter Responses — DB Graph MCP × Spreadsheet

DB Graph MCP is the MCP that lets us search and query 991 internal tables in natural language.

The annoying-but-common case here is "give me everything"-style queries:

SELECT * FROM service_main.user WHERE created_at >= '2026-01-01'

When the result is several thousand to tens of thousands of rows, you get either:

A multi-million-token response that triggers immediate session compaction
An MCP error because the payload exceeds the size limit

Or both. The "right" AI behavior is to do LIMIT 100 and analyze a sample — but if the user actually wanted the full list as a CSV, that doesn't help them.

So we built a "export to spreadsheet, return only the URL" mode into DB Graph MCP. You can opt in explicitly, but the MCP also auto-falls back to this mode whenever the result exceeds a row-count threshold. Even if the AI forgets to add a LIMIT and the query is about to return 10,000 rows, the server decides "this is too big to return inline," exports to a spreadsheet, and hands back the URL.

// Conceptual call (the real shape is documented in the tool description)
sql_query_database({
  query: "SELECT * FROM ...",
  output: "spreadsheet"  // ← explicit export mode
})

// Without `output`, the server still auto-falls back over a threshold (e.g. 500 rows)
sql_query_database({
  query: "SELECT * FROM ..."
})
// → server detects row count → spreadsheet export + URL response

// Either way, the response shape is the same
{
  url: "https://docs.google.com/spreadsheets/d/{...}/edit",
  rows: 12483,
  columns: ["id", "email", "created_at", ...],
  exported_reason: "row_count_exceeded"  // set on auto-fallback
}

The response is just a URL plus metadata. The real data never enters the context. "Light if you're careful" becomes "light even when you're not" — and that's what makes it feel safe in day-to-day operation.

This pattern works because a surprisingly large fraction of real use cases are just "I want this data somewhere I can use it later" — not "let's analyze this in chat with AI." Things like:

Save it to a spreadsheet I can stare at later
Share it with another team
VLOOKUP it against another sheet

For those, MCP's job ends at "write the query, drop the result somewhere." That's enough.

If the user genuinely does want AI-side analysis, you do still need the data in context. The standard workflow becomes a two-step: LIMIT 100 for sample analysis, then output: spreadsheet for the full export once the conclusion is clear.

How Much Did It Save?

Every MCP we run logs every tool call. After rolling these patterns out, total token consumption across all tools dropped 70–90%.

Bonus: Google Workspace OAuth Pairs Beautifully With This

A note on choosing where to "park" data: if your MCP authenticates via Google Workspace OAuth, this whole design becomes much easier.

The reason is that you get two things from a single OAuth flow — two birds with one stone:

Authentication for MCP itself — figuring out who's using the tool
Authorization for Workspace apps — scoped access to Spreadsheet / Drive / Gmail / Calendar

Once the user has logged into the MCP, you don't have to ask for any additional permissions to write to the park location. Which means you can:

Use the operating user's own permissions
To save files to that user's My Drive
Without the MCP itself owning a write-anywhere service account

Files end up in the user's drive, not on a shared service account. "Accidentally world-readable" or "visible to people who shouldn't see it" stops being a realistic accident — it's structurally prevented.

You also dodge the operational cost of issuing a separate GCP service account, storing its key safely, and managing its IAM policy out of band. The safety property genuinely comes for free.

There's one catch though:

The AI agent has to be able to read the spreadsheet URL it got back.

Returning a URL alone doesn't help the AI access the underlying data. Stock tooling in Claude Code can't read a Spreadsheet directly, so you need a separate Workspace-operating MCP.

At airCloset we run a dedicated MCP that wraps the Google Workspace APIs (Drive / Sheets / Gmail / Calendar). Combined with the export pattern above, it gives us a clean flow: "drop results into a spreadsheet → call into the Workspace MCP later if the AI wants to actually read them."

DB Graph MCP → exports to Spreadsheet → returns URL
                                          ↓
              Workspace MCP ← invoked when the AI decides it needs to read the data

From the user's side, this naturally produces the rhythm of "dump it into a spreadsheet first, ask AI to analyze only when needed."

Wrap-Up

A few small tricks for keeping self-built MCP server token consumption under control:

Move the parts that tend to grow off the MCP wire
Park them somewhere — Git server, Spreadsheet, GCS — and only pass keys/URLs through MCP
Pick a park that pairs well with Google Workspace OAuth — you get safety almost for free
If you want the AI to read parked data later, run a Workspace-style MCP alongside

It's an unflashy design move, but the difference in MCP usability before and after is dramatic.

If you're running self-built MCP servers internally and feeling the token squeeze, give it a try.

Bridging 'I Want to Build' and 'I Want to Publish Safely' for Non-Engineers — Sandbox MCP

Ryosuke Tsuji — Mon, 27 Apr 2026 23:04:57 +0000

Hi, I'm Ryan, CTO at airCloset.

In my previous posts, I've introduced our internal MCP servers: an MCP server for natural-language search across all our databases, the full picture of our 17 internal MCP servers, and a custom Graph RAG that lets AI answer "Did that initiative actually work?".

This time I'm covering something a bit different: Sandbox MCP — a platform that lets non-engineer employees deploy apps they built with AI to a safe, internal-only URL with a single command.

The pitch is simple: "If Claude Code can build an app, why not publish it directly?" The hard part is making "directly" mean safely.

The Problem: Building Got Easy. Publishing Safely Did Not.

The arrival of Claude Code and other AI coding agents is reshaping how work happens inside our company.

"Building an app" used to be an engineer's job. You had to do requirements, design, frontend, backend, database, CI/CD, production deploy — all in one head.

Now PMs, designers, and customer-success folks are talking to Claude Code with "build me a screen that does X" and getting working mockups on the spot. Inside airCloset we're seeing more and more:

Mockups for new project proposals
Interactive reports that visualize research findings
KPI dashboards used only by a single team
Small tools for everyday operational improvements

These non-engineer outputs are growing fast. People are even saying "let's just run with this in production for a bit."

That's where the wall hits.

Easy to Build. Hard to Publish Safely.

Anyone can build something that runs locally now. Spin up python -m http.server 8000, view it on your Mac — five minutes max.

But the moment it becomes "I want my team to see this" or "I want others to actually use it," the difficulty curve goes vertical.

Where do you run it? Cloud means GCP/AWS accounts, IAM, billing.
What URL? Domain registration, DNS, SSL certificates, Cloudflare.
What about auth? If it touches confidential info, you need employees-only. OAuth implementation, domain restriction.
And the data? Is localStorage enough, or do you need a real DB? If a DB, who manages the password?
How do you deploy? Can you write a Dockerfile? Cloud Run config, env vars, service accounts, IAM.
What about security? What if the AI-written code has a vulnerability? An auth bypass?

You could "let the AI write all of it." But the result is left to the AI. Cloudflare misconfigured and exposed to the world. Auth bypassed. A service account with production database write access slipped into the code. The more code AI writes, the higher the risk of these accidents.

When a non-engineer says "I want to try building this," we need to clearly separate what the builder is responsible for from what the platform must guarantee by default.

There's also a quieter problem.

UI Inconsistency and Data Sprawl

When non-engineers build apps independently:

One person uses React, another Vue, another raw HTML
Buttons look and behave differently
Some store data in localStorage, some in Google Sheets, some in Firebase

After 10 or 20 such apps, internal tooling becomes chaos. Users wonder "wait, who built this one?" and "why does this button work differently?"

Even for internal tools, you need a baseline of consistency — both in design and in where data lives.

Sandbox MCP — Standing Between "Build" and "Publish"

That's why we built Sandbox MCP.

A non-engineer just says "build this" to Claude Code, and:

An app is generated using a unified UI Kit
They can verify it works locally
A single command deploys it to https://sbx-{nickname}--{app-name}.example.com/
Self-hosted OAuth on the Cloudflare Worker enforces internal-only access
Data is stored, isolated, in a dedicated Firestore database

— all of this completes within a single chat session with the AI.
The builder is only responsible for functionality. Security, data isolation, domain & SSL, authentication are all handled by the Sandbox MCP platform by default.

Scale

Resource	Details
MCP tools	10 (publish, status, schedule, list, delete, write_file, read_file, list_files, init_repo, unschedule)
Supported runtimes	Python (Flask + gunicorn), Node.js, static HTML/SPA, custom Dockerfile
URL	`sbx-{nickname}--{app-name}.example.com` (covered by Universal SSL, no ACM)
Authentication	Self-hosted OAuth on a Cloudflare Worker (Google Workspace)
Data	Firestore named DB `sandbox`, namespaced per nickname × app
Infrastructure	Self-hosted Git Server (GCE) + Cloud Run + Cloudflare Worker + KV
Deploy time	Typically 2–5 minutes (git push to public URL)

Let's walk through the internals.

What It Does — Web, API, DB, and Cron

Sandbox MCP supports four app shapes so it can cover almost any "I want to ship something internally" use case.

Type	Detected by	Use cases
Python	`.py` files present	Flask + gunicorn for APIs, analysis tools with a UI
Node.js	`package.json` present	Express APIs + UI; Bun also works
Static HTML/SPA	only `.html` files (no Python/Node)	nginx-served, React/Vue dist supported
Custom	includes a `Dockerfile`	Any runtime — Go, Rust, Bun, anything

Pick any of these and sandbox_publish deploys it with no extra config.

There's also sandbox_schedule for scheduled batch apps via Cloud Scheduler. Things like "post a risk summary to Slack at 9 AM every morning" become one-line cron setups.

sandbox_schedule(
  app_name: "risk-alert",
  schedule: "0 9 * * *",
  path: "/api/cron",
  timezone: "Asia/Tokyo"
)

Cloud Scheduler now hits the app's /api/cron every morning at 9. No need to open the scheduler UI or translate cron syntax into IaC.

Frontend — Unified Design via sandbox-ui-kit

Even apps built by non-engineers should feel consistent as a tool family. That's the job of the sandbox-ui-kit repo.

It lives on mcp-sandbox.example.com/git and provides:

File	Contents
`sandbox-ui.css`	Design tokens + glass-morphism component styles (dark/light)
`sandbox-ui.js`	Theme switcher, modals, toasts, generic JS utilities
`sandbox-db.js`	SandboxDB client SDK (more below)
`index.html`	Storybook-style component catalog
`README.md`	Full API documentation

The key: it's designed for AI to read and use.

The sandbox_publish tool description literally says:

When building an app, first read README.md with read_file and use the UI Kit.

When Claude Code builds a new app, it read_files this README, learns which CSS/JS to load and which component names to use, then generates code accordingly. Instead of a human walking the AI through UI guidelines, we centralized the "how to use" in one place targeted at the AI.

The result: apps built by anyone (with AI) end up with consistent buttons, modals, and forms.

Backend — Auto-Generated Dockerfile + Cloud Run

"I don't want to write Docker." "I don't want to think about runtime configuration." Classic non-engineer requests.

Sandbox MCP inspects the source files and generates a Dockerfile automatically.

// apps/mcp/git-server/src/sandbox/tools.ts
if (hasPy) {
  dockerfile = generatePythonDockerfile(hasRequirements);
  // Auto-create requirements.txt if missing
  if (!hasRequirements) {
    await writeFile('requirements.txt', 'flask\ngunicorn\n');
  }
} else if (hasPackageJson) {
  dockerfile = generateNodeDockerfile(true);
} else if (hasHtml) {
  dockerfile = generateStaticDockerfile();
}

For example, a Python app gets:

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
ENV PORT=8080
CMD ["python", "-u", "$(ls *.py | head -1)"]

If requirements.txt is missing, flask + gunicorn get added automatically. AI can write from flask import Flask and the dependencies will resolve — no missing-package surprises.

Deployment uses gcloud run deploy --source, with Cloud Build handling the image build. App authors can write a Dockerfile, but they don't have to. No Dockerfile gets the standard, with one customizes — friendly to both non-engineers and engineers.

Database — Transparent Fallback Between localStorage and Firestore

"I want to save data. I don't want to set up a database."

The SandboxDB SDK handles that. The same code uses localStorage locally and Firestore once deployed.

<script src="https://mcp-sandbox.example.com/api/db/sdk.js"></script>
<script type="module">
  const db = new SandboxDB({ token: googleOAuthAccessToken });

  // Save (storage location auto-detected from hostname)
  const { id } = await db.collection('items').add({ name: 'test' });

  // List
  const items = await db.collection('items').get();

  // Get / update / delete
  await db.collection('items').doc(id).update({ name: 'updated' });
  await db.collection('items').doc(id).delete();
</script>

The SDK internals:

this._isLocal = location.hostname === 'localhost'
              || location.hostname === '127.0.0.1';

async add(data) {
  if (this._db._isLocal) return this._localAdd(data);  // localStorage
  return this._req('', 'POST', data);                  // Firestore REST API
}

When running on localhost, it uses localStorage. The moment it's deployed under sbx-*.example.com, it switches to Firestore. No code changes required.

This dramatically improves the experience of building apps with AI:

Local: no network, no auth, all features work
Deployed: same code runs, data is properly persisted
Development data never leaks into systems outside Sandbox (it physically can't reach them)

Firestore Namespace Isolation

Once deployed, data paths are strictly isolated:

sandbox_data/{nickname}--{app}/{collection}/{docId}

nickname: user identifier resolved via OAuth
app: Sandbox app name
_createdAt / _updatedAt: auto-attached by the SDK

Data from different apps is physically unreachable from each other. Even apps built by the same person live in different paths.

The most important point: we use a dedicated sandbox named database. It's a completely separate Firestore database from the (default) DB used by other internal systems. No matter how badly an app's code misbehaves, it can never touch data outside Sandbox.

Infrastructure — Wildcard DNS + Cloudflare Worker + Self-Hosted Git Server

Now for the infrastructure highlights.

How URLs Are Determined

The public URL takes the form:

https://sbx-{nickname}--{app-name}.example.com/

nickname is automatically pulled from the MCP OAuth session. When a user logs into Sandbox MCP via Google, the email is looked up in a Firestore users collection to resolve the nickname. Users never have to repeat "I am ryan" each time.

r.tsuji@air-closet.com → users[r.tsuji@air-closet.com].nickname → "ryan"
                                                       ↓
                                  sbx-ryan--todo-app.example.com

Note: The users collection is kept in sync from a separate internal pipeline (a daily batch that pulls from our HR system and Google Workspace directory). Sandbox MCP just reads from it — no need to maintain its own employee master.

The benefit: you can tell whose app it is just by reading the URL. When someone says "go look at ryan's todo-app," reading the URL aloud naturally communicates ownership.

Instant Publishing via Cloudflare Worker

Normally, publishing a new subdomain requires:

Adding A/CNAME DNS records
Issuing an SSL certificate (15–30 minute wait with ACM or Let's Encrypt)
Configuring a load balancer or DomainMapping

Sandbox MCP skips all of this with a Cloudflare Edge Router Worker.

DNS is fixed as *.example.com wildcard + Cloudflare proxy, with Universal SSL automatically covering every subdomain. The Cloudflare Worker receives all *.example.com/* traffic and routes by subdomain.

The logic is three-tier:

// apps/worker/edge-router/src/index.ts
export async function handleRequest(request, env) {
  const url = new URL(request.url);

  // ① sbx-* prefix → Sandbox routing
  const sandboxSub = extractSandboxSubdomain(url.hostname);
  if (sandboxSub !== null) {
    return handleSandboxRequest(request, url, sandboxSub, env);
  }

  // ② KV route:{subdomain} registered → Cloud Run proxy
  const subdomain = extractSubdomain(url.hostname);
  if (subdomain) {
    const proxyResponse = await handleCloudRunProxy(request, url, subdomain, env);
    if (proxyResponse) return proxyResponse;
  }

  // ③ Otherwise → fetch(request) passthrough
  return fetch(request);
}

When sandbox_publish finishes, all it does is write a route:{nickname}/{app} key into Cloudflare KV. That single write makes the new subdomain routable instantly.

await kvPut(`route:${nickname}/${appName}`, serviceUrl);

No DNS setup. No waiting for SSL issuance. No IaC deploy. Everything completes within the MCP tool execution.

Self-Hosted Git Server for Larger Apps

This setup actually started out without git at all.

Since the primary users were going to be PMs and CS folks, we figured "git concepts are too high a bar — let's keep everything inside MCP tools." Write files via sandbox_write_file, deploy via sandbox_publish. That should be enough, we thought.

The approach hit two walls quickly.

Wall 1: Constant chunking

MCP tool calls travel over HTTP, with a payload size limit. React/Vue build bundles, SPAs with images, business tools with dozens of files — they don't fit in a single call. We added an append mode to sandbox_write_file for chunking, but every "first half of file A → second half of file A → first half of file B → ..." sequence triggered error recovery and retries. Deployments became flaky.

Wall 2: Massive token consumption

This was the real killer. When you tell the AI "deploy this app," it sends the entire source as MCP tool arguments. The file contents land in the conversation context, and a few-thousand-line app burns through tokens fast. A single deploy easily consumed tens of thousands of tokens, and Claude Code sessions hit compaction quickly.

Worse, the AI tends to "verify after sending" — re-reading the same file via sandbox_read_file. Write → read → write loops, with tokens going up in flames.

So we pivoted to using git push as well. With git push:

No file size limit
Differential transfer — second-time pushes are fast
Source code stays out of the MCP conversation context (no AI tokens consumed)

We never expected business-side employees to run git push by hand. But if Claude Code runs git commands in the background, it's not a barrier. The user just says "build this and publish it" — the AI runs git init && git push on its own when needed.

Why a Self-Hosted Git Server?

Once we adopted git push, the next question was: where do we host the repos? We considered using GitHub Organizations but ruled it out.

Issuing and managing GitHub accounts for every employee — including non-engineers — wasn't worth the cost or the operational overhead. Paying for a GitHub seat just to ship one app is overkill.

Fortunately, we already operated a self-hosted Git Server on GCE for a different purpose: hosting an internal "read-only Git MCP for code investigation." A VM with repositories cloned under /mnt/repos/.

We just added a Git Smart HTTP Protocol endpoint and one new repo (sandbox-apps) to it. The VM was already running, so the marginal cost was near zero. Authentication piggybacks on the existing Google OAuth setup. Repository management is just OS directory operations. Borrowing space on the existing internal Git Server was vastly simpler than spinning up new infrastructure.

Actual Usage Flow

# 1. Get the git URL from the MCP tool (nickname is automatic)
sandbox_init_repo(app_name: "my-app")
# → https://mcp-sandbox.example.com/git/sandbox/ryan/my-app.git

# 2. Local commit (the AI does this in the background)
cd ~/my-app/
git init && git add . && git commit -m "init"
git remote add sandbox <returned URL>

# 3. Push
git push sandbox main
# Username: oauth2accesstoken
# Password: $(gcloud auth print-access-token)

# 4. Deploy
sandbox_publish(app_name: "my-app", description: "...")

Auth uses a Google OAuth token as the Basic Auth password (same pattern as GCP Source Repos). Only @air-closet.com accounts pass. No GitHub account required — any employee can push.

The remote repo is configured with receive.denyCurrentBranch=updateInstead, so the working tree updates server-side on push. Cloud Run uses that directory as --source, so there's no extra step between push and publish.

For small apps (a few files, hundreds of lines each), sandbox_write_file still works fine. Switch between MCP-only and git push depending on app size.

Security — Four Independent Gates

That covered the "convenient to build" side. Now the "safe to publish" side.

As I noted at the start, exposing AI-generated code in front of users is risky. So Sandbox MCP layers four independent safety mechanisms that don't depend on the app's own implementation.

① Public-Facing Gate — Self-Hosted OAuth on the Cloudflare Worker

sbx-*.example.com sits behind a self-hosted OAuth gate built into the same Cloudflare Worker that handles routing. When someone visits, the Worker first checks the cortex_session cookie; if it's missing or invalid, it redirects to a Google Workspace SSO entry point (auth.example.com/__edge/auth/start). Without an @air-closet.com account, requests never reach Cloud Run.

This is independent of the app's implementation. Even if the AI didn't write a single line of auth code, the Worker stops the request first. "Accidentally public" is physically impossible.

Why we migrated from ZeroTrust Access to self-hosted OAuth

The first iteration used Cloudflare ZeroTrust Access. You just configure the @air-closet.com domain restriction in the Cloudflare dashboard and you're done — no auth code at all. As a starting point it was ideal.

The catch: ZeroTrust's free tier caps at 50 users. As headcount grew and Sandbox MCP usage spread, we approached the cap, and switching to pay-as-you-go (~$7/user/month) wasn't trivially cheap. On top of that we wanted to share the same auth foundation with internal apps in production (KPI dashboards, inventory tools, etc.), so we decided to consolidate everything into a self-hosted OAuth with no user limit.

Conveniently, the Cloudflare Worker already in front of every *.example.com request — the routing layer Sandbox MCP relies on — was perfectly positioned for this. A small extension gave us:

auth.example.com/__edge/auth/start to kick off Google OAuth 2.0
auth.example.com/__edge/auth/callback to exchange tokens, persist the session in Upstash Redis, and issue a cortex_session cookie scoped to Domain=.example.com
Worker-level gating for sandbox + internal-app subdomains, injecting X-Cortex-User-Email and friends into the Cloud Run request when authenticated

All of this fits inside the existing Worker — no extra Cloud Run, no extra VM. Workers do have a CPU-time budget, but OAuth flows and cookie checks complete in single-digit milliseconds, so latency is indistinguishable from ZeroTrust.

Net result: the user cap is gone, anyone with @air-closet.com can use Sandbox out of the box, and the auth implementation is fully visible in our own codebase.

② Deploy Gate — MCP OAuth

Operations like sandbox_publish and sandbox_delete enforce Google OAuth on the MCP server side. Sandbox MCP implements RFC 8414 (/.well-known/oauth-authorization-server), so Claude Code runs the OAuth flow automatically on first connection.

The strongest guarantee is "you can't accidentally update or delete someone else's app."

When multiple people share a Sandbox MCP, an AI accident like "wait, I overwrote a coworker's app while updating mine" would be devastating. To prevent that, the AI doesn't get to decide whose app is being touched. The server injects nickname automatically from the OAuth session.

// Strip the `nickname` property from the MCP tool schema and have
// the server force-inject the logged-in user's nickname.
function injectNickname(tool: McpTool, userNickname?: string): McpTool {
  const { nickname: _, ...restProperties } = tool.schema.inputSchema.properties;
  return {
    schema: { ...tool.schema, inputSchema: { ...tool.schema.inputSchema, properties: restProperties } },
    execute: (args, ctx) => tool.execute({ ...args, nickname: userNickname }, ctx),
  };
}

From the AI's perspective, the nickname input doesn't exist. Even with a prompt injection like "delete ryan's app," there's no mechanism to do so. "You can only touch your own apps" is enforced at the API spec level.

On top of that, inputs are validated strictly against /^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$/, rejecting shell-injection and path-traversal patterns (.., /).

③ Data Gate — SandboxDB Namespace Isolation

As mentioned earlier, data lives at:

sandbox_data/{nickname}--{app}/...

Per request, the SandboxDB API resolves the path server-side:

Browser (OAuth): resolve email → users → nickname, take app from the Origin header
Backend (SA token): take nickname/app from the X-Sandbox-App header (required — missing returns 400)

The client cannot spoof the path.

We deliberately do not use the K-Service header (the Cloud Run-injected service name). That's a client-spoofable header, and another implementation that relied on it had a "read another app's data" vulnerability disclosed. Requiring X-Sandbox-App keeps the only valid route through an explicitly server-validated path.

The clincher: a dedicated named database for Sandbox. Instead of the (default) DB (which contains data from other systems), we use an independent Firestore database called sandbox, and the Cloud Run SA gets an IAM Condition that allows access only to the sandbox DB.

// From infra/mcp/git-server/index.ts
// IAM Condition on roles/datastore.user:
//   resource.name == "projects/.../databases/sandbox" ||
//   resource.name.startsWith("projects/.../databases/sandbox/")

No matter how badly the AI-written code goes wrong, it physically cannot reach data outside Sandbox.

④ Execution Gate — Cloud Run SA + IAM

All sandbox-* Cloud Run services run under a single shared SA (e.g. sandbox-run). The permissions on that SA are minimal.

roles/logging.logWriter (write its own logs)
roles/bigquery.jobUser + bigquery.dataViewer scoped to the sandbox_logs dataset only (its own access logs, nothing else)
roles/datastore.user (IAM Condition limiting to sandbox DB)

What it does not have:

Access to the (default) Firestore that holds data from other systems
Access to BigQuery datasets used by other internal systems
Direct access to Secret Manager
Permission to manage other Cloud Run services

In other words, even if a Sandbox app goes completely rogue, the blast radius is limited to sandbox_data and sandbox_logs. Nothing outside Sandbox is affected.

Logging — Apps Can Query Their Own Access Logs

Sandbox apps eventually want to look at logs too. "How many views did this page get?" "Who hit that error?"

We forward Cloud Run request logs to BigQuery via a Logging Sink:

// From infra/mcp/git-server/index.ts
const sandboxLogSink = new gcp.logging.ProjectSink('sandbox-logs-sink', {
  destination: `bigquery.googleapis.com/projects/${projectId}/datasets/sandbox_logs`,
  filter: [
    'resource.type="cloud_run_revision"',
    'resource.labels.service_name:"sandbox-"',
    'logName:"run.googleapis.com%2Frequests"',
  ].join(' AND '),
  bigqueryOptions: { usePartitionedTables: true },
});

The sandbox_logs dataset is locked down with project-owner-only ACLs (it contains PII like remoteIp and User-Agent), and the Sandbox SA gets a tightly scoped bigquery.dataViewer to it.

This lets apps query their own access logs from BigQuery. "Post last week's user count for this app to Slack" can be done entirely inside Sandbox.

Tool Design — Making AI Use Tools Correctly

Let me close with a note on tool definitions. I personally think this is where MCP design really makes or breaks.

Sandbox MCP exposes 10 tools:

Tool	Purpose
`sandbox_publish`	Start deploy (async)
`sandbox_deploy_status`	Check deploy status
`sandbox_init_repo`	Initialize git push repo
`sandbox_write_file`	Write file (overwrite/append)
`sandbox_list`	List apps
`sandbox_delete`	Delete app
`sandbox_schedule`	Configure Cloud Scheduler
`sandbox_unschedule`	Remove Cloud Scheduler
`sandbox_read_file`	Read source code
`sandbox_list_files`	List files

Whether the AI picks the right tool at the right moment is almost entirely determined by what's written in the tool description.

For example, the description for sandbox_publish covers not just functionality but also:

Supported app types and required files (Python / Node.js / static HTML / custom)
Startup command and PORT requirement per type
When to use write_file vs git push
How to use SandboxDB (with SDK code samples)
How to use the UI Kit (explicit instruction to fetch README.md via read_file)

With this in place, the AI can autonomously do:

User says "build me a tool that displays Slack emoji scores"
→ Reads sandbox_publish description and sees "first read the UI Kit README"
→ Calls read_file on sandbox-ui-kit/README.md
→ Generates HTML/CSS/JS following the guidelines
→ Sees the SandboxDB SDK usage in the description and integrates persistence
→ Calls sandbox_publish

— without asking the user a single follow-up question. Writing not just "what it does" but "what to do with it" into the tool definition is the secret to AI-friendly design.

If you write tool definitions tersely, the AI keeps coming back asking "what should I do next?" The description is less of a human-facing doc and more of an AI-facing runbook. That framing helps a lot.

Wrap-Up

Sandbox MCP exists to answer two challenges of building internal tools in the AI era:

Building is now possible for anyone, thanks to AI
Publishing safely remains hard

To close that gap, we:

Standardized every layer on the platform side: frontend / backend / DB / infra / auth / domain / SSL
Embedded a runbook into tool descriptions so the AI naturally uses things correctly
Layered four access gates (Worker-level OAuth / MCP OAuth / namespace isolation / IAM) so safety doesn't depend on the implementation being correct

Building this, what struck me again is that the role of platforms in an AI-powered development era is shifting. Platforms used to optimize for "easy for humans." Now they also need to optimize for "used correctly by AI." Tool descriptions are AI-facing docs, and safety must be designed assuming AI will write incorrect code.

At the same time, by limiting what the builder is responsible for, we drastically lower the barrier to "let me just try something." That's the entry point that turns a non-engineer's "I want to build this" into actual operational improvements.

I hope this is useful for anyone designing internal platforms.

Still Measuring Initiative Impact Manually? How We Used Graph RAG + MCP to Make It Explorable

Ryosuke Tsuji — Mon, 20 Apr 2026 15:27:35 +0000

Hi, I'm Ryan, CTO at airCloset.

In my previous posts, I introduced an MCP server that lets you search all company databases in natural language and showed the full picture of our 17 internal MCP servers. This time, I'm diving deep into what I briefly mentioned as "Biz Graph."

This is the story of how we represented the relationship between business initiatives and KPIs as a graph structure, enabling AI to answer "Did that initiative actually work?"

Why Graph RAG?

To get more value from AI, what matters is not just feeding it data — it's conveying the relationships between data.

If your data volume is small enough, tools like NotebookLM can deliver great results. But you can't fit all your business data into a context window. Initiative reports, KPI spreadsheets, marketing weekly reports, logistics daily metrics — you simply cannot dump all of that into a prompt.

That's why I believe the best available option right now is Graph RAG: making the right data searchable at any time, along with its relationships. When AI is asked "What metrics are related to this initiative?", it can traverse the graph and extract only the information it needs — because that structure was built in advance.

But there's a catch.

Making Non-Graph Data Into a Graph

Many of you have heard of "knowledge graphs" and "GraphRAG." But when you actually try to build one, most people hit the same wall:

Business data doesn't naturally form a graph.

With our DB Graph project, things were different. Tables had foreign keys. ORMs had @JoinColumn and belongsTo. Relationships already existed in the data — we just had to parse and convert them.

But the relationship between "initiatives" and "KPIs" has none of that.

A meeting slide says "SNS ad campaign launched"
A spreadsheet records "This week's new members: 1,234"
There's no FK between these. No join key.

"The SNS campaign affected new member signups" — that relationship exists only in someone's head. It's nowhere in the spreadsheet.

This is what "business data doesn't form a graph" means. The relationships between entities aren't self-evident — you have to design the graph structure itself.

The Problem: "Did That Initiative Actually Work?"

Every week, our company reports initiative progress in all-hands meetings and group-level standups.

"We launched the spring SNS ad campaign"
"We improved the recommendation engine"
"We're raising our CS SLA achievement rate"

— Dozens of initiatives reported weekly. Hundreds per year. Over 5,000 total.

Meanwhile, a separate spreadsheet tracks 200+ metrics daily and weekly: member count, new signups, retention rate, satisfaction scores, acquisition CPA...

The problem: these two worlds are completely disconnected.

"How much did last month's SNS campaign contribute to new member acquisition?"

Answering this requires:

Confirm the initiative's execution period (which slide was that again?)
Find KPI data for that period (which sheet, which tab?)
Align timeframes and compare numbers (week-over-week? month-over-month? year-over-year?)
Check if other initiatives were running simultaneously (confounding factors?)

This manual analysis takes 30-60 minutes, happening every week for multiple initiatives. Realistically, most initiative effectiveness reviews end with "it probably worked, I think."

Biz Graph: The Big Picture

We built Biz Graph to solve this.

Scale

Note: The numbers below differ from actual values but convey the order of magnitude. In any case, this is far too much data to fit in an LLM's context window.

Resource	Count
Nodes	~10,000 (14 types)
Edges	~71,000 (22 types)
Initiatives	~5,000
KPI Metrics	~4,000 (members/signups/retention/satisfaction/UX/marketing/logistics)
Marketing Channels	~100 (SEM/LINE/email/CRM etc.)
Data Sources	9 tables/spreadsheets

Three Components

Biz Graph Transformer — Weekly graph rebuild from all data sources (Cloud Run Job, every Friday 22:00)
Biz Graph MCP Server — Graph search + time series analysis accessible from AI (Cloud Run)
Biz Data Loader — Daily auto-import of marketing/logistics data (Cloud Run Job, every morning 6:00)

The Core Design: The Week Node

Here's the heart of this article.

How do you connect "initiatives" and "metrics" in a graph? The obvious first thought is direct edges:

Initiative("SNS campaign") ──AFFECTS──→ Metric("new_members")

This design breaks down. Three reasons:

Edge explosion: 5,000 initiatives × 4,000 metrics = up to 20 million edges
Causal uncertainty: "SNS campaign affected new members" is a hypothesis, not a fact. Direct edges make it look like a confirmed relationship
Missing temporal info: There's no way to express when the impact occurred

Instead, we designed Week nodes as shared anchors for indirect connections.

Initiative("SNS campaign")     ──ACTIVE_DURING_WEEK──→  Week:2026-03-03
Metric("new_members")          ──HAS_DATA_AT──→         Week:2026-03-03
QualityMetric("avg_rating")    ──HAS_QUALITY_DATA_AT──→ Week:2026-03-03
MarketingChannel("SEM brand")  ──HAS_MARKETING_DATA_AT──→ Week:2026-03-03

Initiatives and metrics aren't directly connected — they're indirectly linked through the same week.

Why This Works

1. Prevents edge explosion

Initiatives only connect to "weeks they were active." Metrics only connect to "weeks that have data." Instead of a cross-product, each connects independently to Week nodes — edge count grows linearly.

2. Expresses co-occurrence, not causation

"Initiatives that were active the same week as metric fluctuations" — this isn't asserting causation, it's a structure for discovering causal candidates. It leaves room for human or AI judgment.

3. Edge types distinguish data sources

Same Week node, but HAS_DATA_AT (business KPIs), HAS_QUALITY_DATA_AT (service quality), HAS_UX_DATA_AT (UX metrics), HAS_MARKETING_DATA_AT (marketing), HAS_LOGI_DATA_AT (logistics) — "what kind of data" is embedded in the edge type itself.

4. Time series traversal is natural

Week nodes are connected by NEXT_WEEK edges. "How did metrics change in the 3 weeks before and after initiative start?" can be expressed as graph traversal.

MetricDomain: Bridging Worlds Without Join Keys

Week nodes tell us "what happened the same week," but not which metrics are relevant to a given initiative. There's no point looking at logistics data when analyzing an SNS ad campaign.

However, there's no join key between initiative categories ("Marketing (Advertising)") and metric groups ("New Acquisition"). The knowledge that "ad initiatives relate to new acquisition" is tacit — it exists only in people's heads.

MetricDomain (6 domains) structuralizes this tacit knowledge.

Domain	Meaning	Connected metric types
acquisition	New acquisition	Marketing channels, new member count, registration CV
retention	Retention / churn prevention	Member count, churn rate, plan transitions
service_quality	Service quality	Satisfaction, ratings
operations	Operations	Selection, shipping, returns, logistics KPIs
ux	UX experience	Sessions, funnels
revenue	Revenue / purchases	Purchase CV, upsell

These 6 domains aren't fixed — they can be freely added or split as the business grows and the organization evolves. Domain definitions are just mapping tables in code, so the cost of expansion is nearly zero.

By humans defining the mapping between initiative categories and MetricDomains, and between metric groups and MetricDomains, we enable "automatically show acquisition-related metrics when viewing a marketing initiative."

Category("Marketing ads") ──CATEGORY_IN_DOMAIN──→ MetricDomain("acquisition")
                                                           ↑ IN_DOMAIN
                                                  MetricGroup("New Acquisition")
                                                  MarketingChannel("SEM brand")
                                                  UxMetric("registration_completed")

Result: Pass domain: "acquisition" to compare_metrics, and the initiative overlay automatically filters to acquisition-related initiatives only.

SIMILAR_TO: AI Answers "Have We Done Something Like This Before?"

Another unique design element: SIMILAR_TO edges.

Initiative text (title + description) is vectorized to 768 dimensions using Vertex AI's gemini-embedding-001, then BigQuery's VECTOR_SEARCH auto-detects similar pairs with cosine similarity >= 0.75.

SELECT base.id, query.id, distance
FROM VECTOR_SEARCH(
  TABLE cortex.biz_graph_nodes,
  'embedding',
  (SELECT id, embedding FROM cortex.biz_graph_nodes WHERE node_type = 'Initiative'),
  top_k => 6,
  distance_type => 'COSINE'
)
WHERE base.id != query.id AND distance <= 0.25  -- distance <= 0.25 = similarity >= 0.75

Currently ~13,000 SIMILAR_TO edges exist. Up to 5 similar initiatives are pre-computed for each one.

"Didn't we run a similar SNS campaign last summer? How did that one perform?" — traverse similar initiatives on the graph instantly, then compare KPI changes during weeks those initiatives were active.

Real Usage Examples

Here's how exploration works via MCP tools.

All tool execution examples below run through MCP from an AI coding agent. The response format matches the real system, but numbers are dummy values and content is simplified.

"Find marketing initiatives that drove acquisition"

search_initiatives({
  "query": "SNS advertising for new acquisition",
  "domain": "acquisition",
  "dateFrom": "2025-10-01",
  "dateTo": "2026-03-31",
  "limit": 5
})

Response (excerpt):

5 initiatives found (by vector similarity):

1. SNS Ad Spring Collection Campaign (2026-03-09)
   Category: Marketing (Advertising)
   Similarity: 892/1000

2. Instagram Reels Ad Test (2026-02-23)
   Category: Marketing (Advertising)
   Similarity: 845/1000
   ...

"Show me the impact of that initiative"

get_initiative_context({
  "initiative_id": "Initiative:2026-03-09:SNS Ad Spring Collection Campaign",
  "metric_window_days": 30
})

Response (excerpt):

## Initiative Context

Title: SNS Ad Spring Collection Campaign
Execution Period: 2026-03-01 to 2026-03-31
Category: Marketing (Advertising)
Target Domain: acquisition

## Similar Initiatives (SIMILAR_TO)
- Instagram Reels Ad Test (similarity: 0.82)
- 1-Month Free Trial Campaign (similarity: 0.78)

## KPI Changes During Initiative (30-day window)
| Metric | Pre-avg | Post-avg | Change |
|--------|---------|----------|--------|
| new_regular | 50 | 60 | +20.0% |
| new_lite | 30 | 35 | +16.7% |
| monthly | 1,000 | 1,050 | +5.0% |

## Service Quality Metrics
| Metric | Before | After | Change |
|--------|--------|-------|--------|
| avg_rating | 3.50 | 3.60 | +2.9% |

## UX Metrics
| Metric | Before | After | Change |
|--------|--------|-------|--------|
| total_sessions | 10,000 | 12,000 | +20.0% |
| registration_completed | 100 | 130 | +30.0% |

This is the power of the Week node design. Identify the weeks an initiative was active, then automatically pull all metrics (KPIs, quality, UX, marketing, logistics) from those same weeks.

"Visualize new acquisition YoY with initiative overlay"

compare_metrics({
  "metrics": ["new_regular", "new_lite", "new_monthly"],
  "dateFrom": "2025-10-01",
  "dateTo": "2026-03-31",
  "granularity": "weekly",
  "overlay_initiatives": true,
  "domain": "acquisition"
})

Time series data with acquisition-domain initiatives overlaid on the same timeframe. KPI spikes become instantly attributable to "that initiative's timing."

The Build Pipeline: 9 Phases

The graph is constructed in 9 phases:

Phase	Content	Output
1	Initiative nodes + Category/Business/Team	Initiative, Category, Business, Team
2	Daily KPIs (50 metrics)	Metric → MetricGroup (10 groups)
3	Business KPIs + Departments	Department → Metric (DEPT_TRACKS)
4	Week nodes (shared anchors)	HAS_DATA_AT + ACTIVE_DURING_WEEK + NEXT_WEEK
5	Service quality metrics (~50)	QualityMetric → Week
6	UX metrics (~40)	UxMetric → Week
7	Marketing channels (~100)	MarketingChannel → Week
8	MetricDomain (semantic bridge)	6 domains + IN_DOMAIN + TARGETS_DOMAIN
9	Logistics KPIs (~10 categories)	LogiMetric → Week

Phases 4 and 8 are the key design points. Other phases simply "turn data into nodes" — these two "structuralize relationships that don't exist."

Phase 4: Week Node Generation

// Convert initiative execution period to ISO weeks, generate ACTIVE_DURING_WEEK edges
for (const initiative of initiatives) {
  const weeks = getISOWeeksBetween(
    initiative.executionStartDate,
    initiative.executionEndDate
  );
  // Cap at 52 weeks (guard against long-running initiatives)
  for (const week of weeks.slice(0, 52)) {
    edges.push({
      edge_type: 'ACTIVE_DURING_WEEK',
      source_id: initiative.id,
      target_id: `Week:${week}`,
    });
  }
}

// Generate HAS_DATA_AT edges for weeks that have metric data
for (const metricWeek of metricWeeks) {
  edges.push({
    edge_type: 'HAS_DATA_AT',
    source_id: `Metric:${metricWeek.metric}`,
    target_id: `Week:${metricWeek.week}`,
  });
}

// NEXT_WEEK edges for time series traversal
const sortedWeeks = [...allWeeks].sort();
for (let i = 0; i < sortedWeeks.length - 1; i++) {
  edges.push({
    edge_type: 'NEXT_WEEK',
    source_id: `Week:${sortedWeeks[i]}`,
    target_id: `Week:${sortedWeeks[i + 1]}`,
  });
}

Phase 8: MetricDomain Generation

// Category → Domain (semantic mapping defined by humans)
const CATEGORY_TO_DOMAINS: Record<string, string[]> = {
  'Marketing (Advertising)': ['acquisition'],
  'CRM / Retention': ['retention'],
  'Quality / Service Improvement': ['service_quality'],
  'Operations Improvement': ['operations'],
  'New Feature': ['ux', 'revenue'],
  // ...
};

// Initiative → TARGETS_DOMAIN (main business only — limited to where KPI data exists)
for (const initiative of initiatives) {
  if (initiative.business !== MAIN_BUSINESS) continue;
  const domains = CATEGORY_TO_DOMAINS[initiative.category] ?? [];
  for (const domain of domains) {
    edges.push({
      edge_type: 'TARGETS_DOMAIN',
      source_id: initiative.id,
      target_id: `MetricDomain:${domain}`,
    });
  }
}

Why Not a Dedicated Graph DB or OSS Libraries?

We implemented the graph using BigQuery alone, without Neo4j, Amazon Neptune, or OSS like Microsoft's GraphRAG.

Why not a dedicated graph DB?

Aspect	Dedicated Graph DB	BigQuery
Graph traversal	Fast (native)	Fast enough (~10,000 node scale)
Vector search	Requires separate service	VECTOR_SEARCH built-in
Time series analysis	Weak	Native (window functions)
Operating cost	Always-on instances	Serverless (pay per query)
Joining other data	ETL required	Same project, instant JOIN

For Biz Graph, "graph structure + time series analysis + vector search combined" matters more than "deep graph traversal." BigQuery handles all three in one engine.

Additionally, BigQuery has announced Graph capabilities — once GA, native graph queries on node/edge tables will be available. Currently we traverse with SQL JOINs, but we expect to migrate to faster, more intuitive queries in the future.

Why not OSS libraries / SaaS?

OSS like Microsoft GraphRAG and various Graph RAG SaaS products focus on automatically extracting entities and relationships from text documents. Great for research papers or news articles, but not for our use case.

The reason is simple: we need to design the graph structure itself.

The concept of Week nodes as "temporal anchors" doesn't exist in generic tools
MetricDomain "semantic bridging" reflects our specific business structure
The Initiative → Week → Metric indirect connection pattern won't emerge from LLM entity extraction

Generic tools "auto-generate graphs from text." What we needed was "design the graph schema ourselves and integrate heterogeneous data sources." Fundamentally different problems.

Internal query example (get_initiative_context):

-- Get weeks the initiative was active
WITH active_weeks AS (
  SELECT target_id AS week_id
  FROM cortex.biz_graph_edges
  WHERE source_id = @initiative_id
    AND edge_type = 'ACTIVE_DURING_WEEK'
),
-- Get metrics that have data in those same weeks
co_occurring_metrics AS (
  SELECT e.source_id AS metric_id, e.edge_type, w.week_id
  FROM cortex.biz_graph_edges e
  JOIN active_weeks w ON e.target_id = w.week_id
  WHERE e.edge_type IN (
    'HAS_DATA_AT', 'HAS_QUALITY_DATA_AT',
    'HAS_UX_DATA_AT', 'HAS_MARKETING_DATA_AT'
  )
)
SELECT * FROM co_occurring_metrics

Graph traversal and time series data retrieval complete in a single SQL query. With a dedicated graph DB, you'd need to pass traversal results to another service for time series queries — an extra hop.

Initiative Data Ingestion: Auto-Extraction from Meeting Slides

Graph quality depends on source data quality. Initiative data comes from all-hands and group meeting slides.

Source	Format	Frequency
All-hands	pptx in Drive → Slides conversion → text extraction	Weekly
Group standups	Google Slides (cumulative, latest week appended)	Weekly

Text is extracted from meeting slides and structured by AI into the initiative table.

interface InitiativeRow {
  meetingDate: string;       // Meeting date
  source: string;            // Source (all-hands / group standup etc.)
  business: string;          // Business unit
  category: string;          // Marketing (Ads), New Feature, ...
  title: string;             // Initiative title
  description: string;       // Detailed description
  team: string;              // Executing team
  executionStartDate: string; // Execution start date
  executionEndDate: string;   // Execution end date
  metrics: string;           // JSON format numeric metrics
  status: string;            // planned / in_progress / retrospective
}

Critical: executionStartDate / executionEndDate. The meeting date (meetingDate) differs from when the initiative actually runs. "We started the SNS campaign last week," reported on 3/9, means executionStartDate is 3/1. This distinction is essential for accurate Week node connections.

Operating Cost

Resource	Cost
Vertex AI Embedding (weekly)	~$0.05/run
Claude Code (initiative extraction)	Within monthly plan
BQ storage	A few GB (negligible)
Cloud Run Jobs	Nearly free (1x weekly + 1x daily)
MCP Server	Nearly free (Cloud Run min-instances=0)

A few dollars per month to maintain a 10,000-node, 71,000-edge graph.

Comparison With Typical Knowledge Graphs

Let's take a step back and see how this design differs from conventional approaches.

Aspect	Typical Knowledge Graph	Biz Graph
Node design	Entities mapped directly to nodes	Deliberately designed temporal anchors ("Week")
Edge semantics	Relationships described as-is	Edge types encode data source classification
Intermediate nodes	Taxonomies for classification	MetricDomain as semantic bridge (structuralized tacit knowledge)
Graph construction	Relationships extracted from existing data	Deliberately designed graph from data with no inherent relationships
Use case	Primarily search and navigation	Goes further into causal candidate exploration for initiative impact
Similarity search	Text-based search	Pre-computed SIMILAR_TO edges via Embedding

In one sentence:

Our DB Graph "made existing relationships discoverable." Biz Graph "designed and created relationships that didn't exist."

The former is an analysis problem. The latter is a design problem — designing the graph structure from scratch and integrating heterogeneous data sources (meeting slides, spreadsheets, BQ tables) into a single explorable structure. That's the essence of Biz Graph.

Why Graph RAG Over Flat RAG

Let's revisit the "why Graph RAG?" question from the introduction.

For initiative effectiveness analysis, consider what happens with standard vector search (flat RAG). Ask "What was the SNS campaign's impact?" — flat RAG returns text chunks similar to the initiative description. You get info about the initiative itself.

But it won't return concurrent KPI changes. It won't return results from past similar initiatives. It won't return related domain metrics.

These are information connected "through the graph," not by "text similarity." You can only reach them by traversing Week nodes. This "need to follow relationships" use case is exactly where Graph RAG has a clear advantage over flat RAG.

Design Honesty: Not Asserting Causation

One thing I was conscious of in this design: not asserting causation.

Many BI tools and AI analyses want to declare "this initiative impacted this KPI." But in reality, there's no such certainty. Multiple initiatives may have been running simultaneously, it could be seasonal, it could be external market changes.

Week node indirect connections simply "lay out what happened in the same period." Causal judgment is left to human or AI reasoning. I believe this is a statistically honest approach.

"A structure for discovering causal candidates" — not "a structure for asserting causation." This distinction matters.

Limitations: The Designer's Tacit Knowledge Is the Bottleneck

Let me be honest about the weaknesses of this approach.

MetricDomain mappings ("Marketing Advertising → acquisition domain") are hardcoded by humans. If this design is wrong, the entire graph's exploration results are skewed.

This is simultaneously the answer to "why build it yourself." Off-the-shelf graph tools can't reflect your business structure — which initiative categories relate to which metric groups. Structuralizing this tacit knowledge requires someone who knows the business.

Going forward, we're considering having AI propose these mappings with humans reviewing them. Full automation is hard, but an "AI suggests, humans approve" workflow could reduce the maintenance cost of domain knowledge.

Summary

Turning business data into a graph is more of a design challenge than a technical one.

There's no FK between "initiatives" and "KPIs." No join key. But by deliberately designing two structures — temporal axis (Week nodes) and semantic domains (MetricDomain) — it becomes an explorable graph.

Week nodes: Indirect connections via "same week" instead of direct initiative-metric edges. A structure for discovering causal candidates
MetricDomain: Semantic bridge between initiative categories and metric groups. Structuralized tacit knowledge
SIMILAR_TO: Pre-computed similar initiatives via AI Embedding. Instant answers to "have we done this before?"

As a result, questions like "Did that initiative work?", "Find initiatives that drove acquisition", "Show metrics YoY with initiative overlay" — AI can now autonomously explore the graph to answer these.

Graphs aren't something you "find" — they're something you design. Especially for business data.

DEV Community: Ryosuke Tsuji

AI Isn't Something to Trust — It's Something to Design (Series Final)

Series Index

Origin — What I Was Thinking About in 2025

Wall 1: The Context Window Limit

Wall 2: Don't Lean on Learning Either

The Way Out — GraphRAG + MCP

I Don't Trust AI to Begin With

So I Build Harnesses to Hold AI to Determinism

What "Don't Make AI Infer, Lean on Determinism" Actually Means

Trial and Error That Got Me Here

I Spent Two Months on Static-Analysis code-graph, Then Threw It Out

Setting Coverage 90% as a Solo Target Broke the Implementations

Parallel Sub-Agent → Sequential Evaluation

What This Section Is Really Saying

Closing the Series

Afterword — Where Engineering Careers Are Heading

Part 5 ("The Author Doesn't Have to Be an Engineer") has been generating sharp comments. Worth a read for the thread alone.

The Author Doesn't Have to Be an Engineer: How the Harness Holds Quality (Series Part 5)

The Author Doesn't Have to Be an Engineer: How the Harness Holds Quality (Series Part 5)

Series

Start with one scene

When you need a change, you can make it yourself

Two non-engineer PRs that recently shipped

#1573 -- a deep root-cause fix

#1557 -- a big feature build on top of an existing stack

What works, what doesn't

The principle: standing up a stack is hard, building on top of one isn't

What's left for engineers: laying the rails -- the stack and the harness itself

Why this works for non-engineers

① The knowledge graph pulls relevant code from "what you want to do"

② Auto Review enforces quality at the gate

③ Self-Healing catches what slips through to production

④ Recurrence Prevention keeps the trap count from growing

Next -- carrying the pattern to consumer-facing services

Wrap-up

Fixed Before Anyone Notices, Stronger After Every Fix: Self-Healing + Recurrence Prevention (Series Part 4)

Start with last month's numbers

Series

Big picture -- the three layers: Observation, Repair, Strengthening

Observation -- where do the alerts come from?

What Observability can't catch, Self-Healing can't fix either

Repair -- the Self-Healing flow

What happens when the AI judges "this is not fixable in code"

Deduplication

Self-Healing and Part 3 auto-review -- "the fixer AI" and "the reviewer AI" are independent

A concrete example: meet subscription's 409 ALREADY_EXISTS

Strengthening -- Guides (lint + guidelines) grow automatically

What happens every time Self-Healing runs (the recurrence-prevention-first flow)

"We'll do it later" and "introduce as warn" are banned

The "step on it, mechanize it" lineage

How does the AI write a lint rule without breaking it?

A concrete example: Cloud Run OTel env injection -> promoted to CI guard

Where Guides stand right now (in numbers)

The whole loop, from the top

Self-Healing by the numbers

Main firing categories

Alert-firing to production-recovery time

What changed / Bridge to Part 5

Human-on-the-Loop: AI Reviewing AI PRs at cortex -- 769 PRs/month while raising the quality bar (Series Part 3)

Series

Start with last month's numbers

How the review bottleneck stops forming

The conventional wisdom: the reviewer becomes the bottleneck

cortex's answer: move the reviewer role to AI as well

How the auto-review system is wired

How we ended up with Event Relay

Why sequential single-session review, not parallel sub-agents

Operational knobs

Output structure: tags and severity

Tags (dimensions)

Severity

Example: PR migrating the embedding model from gemini-embedding-001 to gemini-embedding-2

Critical

[Graph] Missing @graph-business tag (x3)

[Graph] embedMeetContent's @graph-connects doesn't reflect generateEmbeddingV2

[Doc] Corresponding BigQuery schema doc is not updated

Minor

[Test] textEmbeddingV2 value is not asserted

[Test] No isolated scenario for "v2 returns null"

"We'll do it later" and "introduce as `warn`" are banned

[Graph] Missing `@graph-business` tag (x3)

[Graph] `embedMeetContent`'s `@graph-connects` doesn't reflect `generateEmbeddingV2`

[Test] `textEmbeddingV2` value is not asserted

`usecase` Parameter — Switching the Runbook

Step 2: Trace downstream from that node (`usecase: "design"` prioritizes Documents)