DEV Community: weiche chiu

The eight questions a manager asks before any of this gets budget

weiche chiu — Mon, 21 Sep 2026 16:39:49 +0000

This is part 6 and the last. Parts 0 through 5 walked the diagram in badge order: the whole reference architecture in part 0, identity and the gateway in part 1, data and retrieval in part 2, sandboxed agents with read-only connectors in part 3, write and send behind approval in part 4, and governance hardening in part 5. This part reads the same diagram from the other side, the side the budget comes from. These are the eight questions I actually get asked, not a list of what managers ought to ask. Where my answer is a judgment I say so. Where it is a number, it is a number I measured, and I say what it was measured on.

Will this replace people?

Part of this is answered by the diagram before anyone argues about it. Human approval is drawn as its own box at stage 4, sitting under the write and send connectors with an arrow running up into them, labeled "send & write need a person." Nothing in stage 5 takes it out. What moves is where the person stands. They stop producing the first version and become the thing every version has to pass through.

The closest thing I have to evidence is one pilot of my own. Three pilot episodes, 50 beat scripts, every one of them generated. The synthesis could technically have run the day the scripts were finished. It did not, because the gate was a person reading every beat for script, for picture and for pacing, and that reading was the whole remaining schedule. Generation stopped being the constraint on the day it got fast, and reviewing became it.

The honest limit on that answer: it is my own experience of my own workload, not a study of employment. What I can say is narrower. The volume went up and the checking did not go away, it concentrated.

Will our data leak?

Two working rules come before any architecture. Treat anything pasted into a chat as already sent, because from the company's point of view it has been. Keys do not travel through the conversation; a credentials file can be read by a process and must not be printed into a transcript.

On the diagram the rest is a line. Cloud LLM APIs sit on the enterprise boundary, and every arrow reaching them passes through the gateway first, where the PII and DLP filter box sits. There is no second path out. Stage 2 covers the other half, the part that leaks inward rather than outward: the retrieval index is labeled scoped by role, so a question asked from marketing cannot return records only support should see. A company that has both of those has turned the leak question into an audit question, and the ledger from stage 5 is what answers it.

Which vendor should we pick?

I do not answer this one, and it is worth being clear that the refusal is not modesty. The boxes on this diagram are functions, not products, and the order they come in does not depend on whose model is inside them. What I would ask instead is whether you can verify the thing locally and whether you can replace it.

One example of what replaceable costs in practice. My publishing tool used to pass an article's own tags straight through as the destination platform's tags. From inside my repository that looked correct. On the platform, three of the four tag slots had gone to tags with no other articles under them, which on that site means the post is filed under nothing. The fix was a mapping table rather than a different vendor. The part worth noting is that nothing in my repository could have shown me this. I had to go and look at the platform.

The part that is expensive to replace is not the model. It is the gateway in front of it, the policy file, and the ledger, because those hold the accumulated decisions. Choosing a vendor first and building those around it gets the dependency backwards.

How do we know it's worth it?

I do not give percentages I have not measured, and most of the ones in circulation are somebody's pilot generalized. Two things are worth measuring inside your own company. How much longer verification takes than production, and what the blocked runs cost when they get retried.

I have measured the second on my own runtime. In an ablation on a small local model, a simple task, and 20 runs per arm, turning on a gate that stopped the agent from reporting a task done when the artifact was not there moved input tokens to 1.66 times the control arm and p95 wall clock from 87 seconds to 169 seconds. Every blocked run gets retried and the retries land in the tail. Those numbers say the control is not free. They do not predict what a frontier model on your repository will do.

How it gets reported matters as much as the number. In a later cell of the same program, a code fix task on a different local model, the mean moved 16 percent while the median went from 18,612 to 37,068 tokens. The middle of the distribution doubled and the average barely registered it. If the number that reaches the manager is an average, the manager is being shown the one statistic built to hide this.

My own gap here is the ordinary kind and worth naming. My publishing log records a status for every publish and no duration and no cost. I can count publishes. I cannot measure them.

What if it gets things wrong?

Assume it will, and budget for catching it rather than for preventing it. The number I keep coming back to is from the control arm of that ablation: in 18 of 20 runs the agent reported the task complete when the artifact was not there. Not wrong answers, reports of work that had not happened.

That is the arm without a gate. A later set of four model and task combinations, 120 graded runs in total, put a gate on all of them, and the gated arm produced valid artifacts in 18, 14, 7 and 20 cases out of 20. The false completions went away in all four. The ability to do the task did not arrive with them, and one of the four sits at 7 out of 20. A gate changes what you can believe about the output. It does not change what the model can produce.

The other half of the answer is how anybody finds out. The draft of part 5 of this series said my site's code had supported Chinese since September 9 and that five articles were affected. Both were wrong. Re-reading the draft would never have said so, because the draft was internally consistent. The commit log said so: the support landed on September 1 and all fifteen articles were affected. A check has to reach a different record than the one being checked, and that holds for a model's output the same way it held for mine.

The requirement that follows is the one from stage 3, and it is structural rather than a matter of trust. Whether something was completed has to be measured outside the agent, because the agent is the thing being measured. An agent asked to grade its own output will answer, and the answer carries no information about the output.

Which department should start?

The one whose output is cheapest to check, not the one where the saving looks biggest. Engineering has tests that run without a person. Content has a person who can read the piece in the time it took to generate. Somewhere like finance or legal, where checking a generated answer means reproducing the work, is the worst place to start even though the hourly rates make it look like the best. This is a judgment.

The reason sits in stage 2. Verification is the bottleneck in every one of these pilots, and it appears at the point where output arrives, not at the point where cost is counted. Picking the department by the size of the salary bill picks the place where the bottleneck is worst.

Consultants, or do it ourselves?

Split it by which artifact you end up owning. The policy file from stage 5 has to be written by the people it applies to. Rules written by someone who left do not get followed, and worse, nobody notices when they stop describing the system, because nobody remembers what they were for. Part 5 has my own case of that, a rule I wrote myself that went twenty days describing a system that no longer existed.

The designs are the opposite. Role scoping at stage 1 and the approval path at stage 4 benefit from being looked at by somebody who has seen other companies get them wrong, and that is a bounded piece of work with a deliverable you keep. This is a judgment, and the line I would hold is that you can buy a review of your design and you cannot buy your rules.

How long before we see anything?

The output arrives the same day, which is the part that makes this question misleading. The effect arrives on whatever schedule the channel runs on, and for most of what a company puts through AI the channel is the constraint rather than the content.

Numbers from my own channels, which are small and are mine. A post announcing part 0 of this series had 148 impressions and 7 likes about 22 hours after it went out. The first thing I posted on another platform got 3 views. Two articles I published natively on LinkedIn reached 13 and 18 people; two weeks later the impressions had grown and both reach numbers were unchanged. Writing any of these now takes a fraction of what it used to. None of those numbers moved because of it.

The practical version of the answer: expect the production bottleneck to disappear in the first week, and expect that to reveal which bottleneck was actually binding. It is usually distribution, review capacity, or a decision nobody has made yet. None of those get faster because the drafting did.

What the diagram does not have

There is no box on this diagram for any of these eight questions, and that is the honest summary of the series. The five stages answer what to build and in what order. The eight questions are about whether anyone should pay for it, and they get answered in conversations, with evidence that mostly does not exist yet at the time the budget is decided.

What the diagram can do is narrow them. Four of the eight above came back with a component attached: the approval box, the boundary line and the scoped index, the metrics row, and the gate. The other four stayed judgments, and I have tried to mark them as such rather than dress them up. The control plane I use for my own work on the governed half is open source at aine-control-plane. That is the end of the series.

Stage 5 of enterprise AI adoption: governance hardening

weiche chiu — Mon, 21 Sep 2026 02:58:54 +0000

This is part 5 of a series that started with the full reference diagram in part 0, covered identity and the gateway in part 1, data and retrieval in part 2, sandboxed agents with read-only connectors in part 3, and write and send behind approval in part 4. Stage 5 is the whole bottom row marked 5, GOVERNANCE & OBSERVABILITY, with four boxes: Policy & gates, labeled "versioned as code, declared vs actual"; Audit ledger, labeled "who / what / when / which policy"; Halt & budget caps, labeled "run-level stop"; and Cost & quality metrics, labeled "p95, not averages". It is easy to read this as governance starting here. Logging and Rate limit & budget have been running inside the gateway since stage 1, and every call has been recorded and charged against a budget from the first day. What this stage adds is comparison, attribution to a version of a rule, and the ability to stop something that is already running.

The company is the same one as in parts 1 through 4, Qingchuan, about 40 people, 8 engineers and 2 in marketing. Qingchuan is made up and only serves to walk the stage. Coming out of stage 4, everyone logs in through the identity provider, every call carries a role through the gateway, retrieval is scoped to the asker, agents run inside a sandbox, and the writes and sends that leave the company carry an approval record the system wrote rather than one the agent reported. By this stage Qingchuan usually has several agents or pipelines running at once. Two things are now missing rather than one. Nothing compares the rules Qingchuan wrote down against what those records actually show, and nothing can stop a run that is already out of bounds.

Policy & gates: declared against actual

The first box holds the rules, versioned as code, and the gates that evaluate them. Its label pairs declared and actual because those are two different things, and this box is where they get compared.

The failure this catches is drift, and drift is invisible from inside the rule. One from my own week. The repository that runs my site carries written rules for anyone adding an article, and one of them said articles were English only and told you not to add a Chinese variant of the file. The code had supported an optional Chinese version since September 1, and by the time I corrected the rule on September 21 all fifteen articles on the site had one, every one of them added under a rule that forbade it. Nobody was ignoring the rule. It described a system that no longer existed, and reading it could never have shown that, because drift is only visible against the record.

The comparison itself is not exempt. A status checker of mine built the address of each Chinese article by joining a prefix to the slug, while the real path had a different shape, and it reported ten live articles as dead. A gate that is wrong reports with exactly the same confidence as one that is right, so the output of a gate is evidence about the gate as much as about the system.

One property worth asking for when you pick or build the gate. The policy engine I keep evaluates with explicit unknown and conflict states rather than folding them into allow or deny. "No rule covers this" and "two rules disagree" are different facts from "denied", and an engine that cannot say them will pick one for you, in a direction you did not choose and cannot see afterwards.

Done looks like this: the policy is a file under version control, changing it goes through the same review as code, every decision a gate makes names the policy version it was made under, and the comparison between declared and recorded runs on a schedule rather than when somebody suspects something.

Audit ledger: which policy allowed it

Stage 4 left four fields behind: which agent or person acted, on what, when, and who approved it. This box adds the fifth, which rule allowed it and at which version.

Without the version you can establish that an action was approved and not under which rule. If the rules changed last month, a record that complied at the time and a record that violates the current rule look the same, and the row cannot tell you which question you just answered.

The ledger has to be append-only, written by the system that performed the action, and outside the agent's reach, which is the same requirement stage 3 put on the read log and stage 4 put on the approval record. Mine is short of it in a specific way. My publishing log records time, brand, article, language, title, platform, status, note, and address. There is no policy column at all, and the approver still sits inside the free text. I can answer whether something was published. Answering which rule allowed it means reading sentences and remembering what the rule was that week.

There is a second way to fall short that looks like the opposite. A field can be declared and read by nothing. The syndication block in my own article files records where each piece was republished, and no code reads it. As bookkeeping that is fine. The problem is that a recorded field and an enforced field look identical from the outside, both being structured data in a file, so a review that counts fields scores them the same. A field nobody reads is a note, not a control.

Done looks like this: append-only, written by the acting system, one row per action carrying the policy version in force at that moment, and answering which rule allowed a given action is a query rather than a reconstruction.

Halt & budget caps: stopping a run, not refusing the next call

The third box is the one people assume they already have because they have a rate limit. A rate limit refuses the next call. This box stops the run that is already going.

The difference shows up once a run does many things. By the time call 41 is refused, calls 1 through 40 have happened, and after stage 4 some of them wrote to an internal system or sent something outside. The rate limit was working correctly the whole time.

In my own runtime the stop is at the level of the run: scope violations, LLM call count, tool execution count, and total tokens each have a cap, any one of them being reached stops the entire run, sub-agents it dispatched stop with it, and that run cannot be resumed, only started again. The caps ship switched off, and turning one on is an explicit decision rather than a default.

The cost of that is measurable, and I measured it. In an ablation on a small local model, a simple task, and 20 runs per arm, an enforced gate that stopped the agent from reporting a task done when the artifact was not there moved unsupported completions from 18 of 20 to 0 of 20. Input tokens went to 1.66 times the control arm, and p95 wall clock went from 87 seconds to 169 seconds. Every blocked run gets retried, and the cost of retries lands in the tail rather than the average. Those numbers show that the mechanism works and that stopping runs is not free. They do not predict what a frontier model on your repository will do, and I would not carry them into anyone's production estimate.

There is one more requirement on a stop, which I learned by having it fail. When my continuous integration stopped running because a spending limit had been hit, the jobs did not report that they had been stopped. They came back as failing jobs, within seconds, with no logs, and a red light from a budget cut looks exactly like a red light from broken code. A cap that stops work without saying so produces a false signal rather than no signal, and everything downstream reads the false one. My working rule since has been to write "CI did not run, the evidence is this local test" rather than treating the red as a result.

Done looks like this: a running agent can be stopped by something that is not the agent, the stop is recorded as a stop rather than surfacing as a failure, and somebody can answer what resuming means before the first cap is switched on.

Cost & quality metrics: p95, not averages

The last box in the row. Its label is the whole argument, and the argument is worth having because averages are what gets reported upward.

From the same ablation: in one cell the mean moved by 16 percent while the median went from 18,612 to 37,068 tokens. The middle of the distribution doubled and the average barely registered it. In another cell there were degenerate runs stuck in a read loop at around 197 thousand tokens. An average across runs would have shown neither the doubling nor the loop.

The metric I would put next to cost is not a cost metric. It is the count of runs that reported completion with nothing to show for it, and it has to be counted outside the agent, because the agent is the thing being measured. That was a principle in stage 3 when the question was whether to trust an agent's report. Here it is a column, and 18 out of 20 is what it looked like the one time I counted it without a gate in the way.

My own gap in this box is the plain kind. My publishing log records a status per publish and no duration and no cost, so there is no distribution to take a p95 of. I can count publishes. I cannot measure them.

Done looks like this: median and p95 reported side by side, a count of unsupported completions measured outside the agent, and cost attributed to the per-workflow account from stage 4 rather than pooled into one number for "AI".

The check that repeats

There is no stage 6 and no exit check here, because governance is not a stage you leave. What replaces the exit check is a comparison you run on a schedule.

Take one rule out of the policy file. Pull every action from the ledger that the rule covers, and check each one against the version of the policy that was in force when it happened. A mismatch is the finding. Finding nothing, repeatedly, is also a finding, and usually means the rule covers nothing anybody actually does.

The second half is the same shape as stage 4's refusal test, moved up a level. Start a run that is going to exceed a cap, and confirm that something stops it while it is running and that the ledger records a stop. If the run finishes, the cap is documentation, and every report built on it has been describing a control that was not there.

The GOVERNANCE & OBSERVABILITY row is drawn identically on the self-hosted variant of the diagram, and hosting changes nothing at this stage either: the rules, the ledger, the caps and the metrics are the same whoever holds the weights. The control plane I keep for my own work on this is open source at aine-control-plane. It evaluates policy and writes decisions to an append-only ledger. It does not enforce anything on an external system, which stays where part 4 put it, in the tool layer. Part 6 closes the series by reading the same diagram from the other side, as the eight questions a manager asks before any of this gets budget.

Stage 4 of enterprise AI adoption: write and send behind approval

weiche chiu — Sun, 20 Sep 2026 11:25:12 +0000

This is part 4 of a series that started with the full reference diagram in part 0, covered identity and the gateway in part 1, data and retrieval in part 2, and sandboxed agents with read-only connectors in part 3. Stage 4 covers the boxes marked 4 on that diagram: Write connectors and Send connectors inside AGENTS & TOOLS, and the dashed Human approval box, labeled "send & write need a person." On the diagram the approval arrows run to the write and send connectors only. Read connectors are not behind them, because reading was the work of stage 3. What changes at this stage is the nature of a mistake. Up to here a bad run left nothing outside the company; from here a record gets changed, or a message goes out and cannot be pulled back.

The company is the same one as in parts 1 through 3, Qingchuan, about 40 people, 8 engineers and 2 in marketing. Qingchuan is made up and only serves to walk the stage. Coming out of stage 3, everyone logs in through the identity provider, every call carries a role through the gateway, questions over internal material return scoped answers, and an agent can run steps on its own inside a sandbox, reading the repo and the ticket system through read-only connectors, with a log of what it read that the agent cannot rewrite. Nothing Qingchuan's agents do yet is visible from outside the company.

Write connectors: a mistake still has a diff

The first box connects the agent runtime to internal systems it may change. At Qingchuan that means an agent that can edit files, commit, and push a branch in the internal tools repo, and one that can write a draft reply into a ticket.

The property that separates writing from sending is that the change stays inside systems the company controls, and there is a previous state to go back to. A bad commit has a diff, and recovering from it costs internal time rather than anything a customer sees. That is also why writing and reading are drawn as separate connectors rather than one connector with a flag: a connector scoped for reading the whole repo is a reasonable thing to have, and the same scope for writing is not. Done looks like this: each write connector names the systems it may change and what inside them, every write leaves a diff or a prior version a person can restore without asking the agent, and the write is attributable to an account issued for that workflow rather than to whoever first lent their own.

Send connectors: a mistake is already outside

The second box is anything that leaves the company. At Qingchuan that is a reply going to the customer who filed the ticket, a post going out on a channel the company owns, and an email to a supplier.

There is no diff here. The question of whether the content was any good is a separate question from whether it should have gone out at all, and the second one is not answerable after the fact. My own rule on my own publishing, which is the closest thing I have to this box in production, is that each outward publish needs a person to say yes at that moment, not a batch authorization granted in advance. One consequence is easy to miss: an approval covers the action it was given for and does not extend to a neighboring one. In a recent publishing run two posts had been approved, and the composer for one of them came up behind a dialog asking to accept updated terms of service. Accepting terms is a different action from posting, so the run stopped there and waited for a person rather than clicking through. Done looks like this: the send connectors enumerate their channels and who receives them, every send carries the approval that released it, and an approval names one send rather than a class of sends.

Human approval: where the refusal actually lives

The third box is drawn dashed, in a different color from the connectors, and it sits between the agent runtime and the two connectors above. The label says a person is required. The common way to build this box is to write the rule into the prompt: do not publish, do not merge, do not touch the customer records.

A prompt is a request. The model can misread it, drop it later in a long run, or talk itself past it while trying to be helpful. A denied action is not a request, and it has no second reading. In my own setup the refusal lives in the tool layer: pushing a branch goes through, while merging a pull request and deleting a remote branch are refused by the tool, not by a sentence telling the agent not to do them. I tested that rather than assuming it, and the practical effect is that the session prepares the pull request and I press merge myself.

Two things follow for a company. Every "the agent cannot do this" has to come with the place it is blocked: the operating system, the tool layer, or the account permissions on the platform itself. A rule that lives only in the prompt is a reminder, and the person who sets the real block is whoever administers accounts and tools, not whoever writes the prompt. The second thing is the size of the boundary. In my own runtime the enforced boundary is the file system, not everything the agent can reach; a sandboxed agent still talks to its model provider over the network, and that traffic is outside the box. Done looks like this: for each write and each send, a named enforcement point, and deleting every instruction about permissions from the prompt changes nothing about what the agent can do.

The approval record

An approval nobody can retrieve afterwards only ever existed for the person who gave it. The minimum this stage leaves behind is four fields: which agent or which person acted, on what, when, and who approved it.

Mine is not there yet, and the gap is the useful part. My publishing log carries time, brand, article, language, title, platform, status, note, and URL, and the approver lives inside the free-text note, so answering who released a given post means reading sentences rather than querying a column. It should be a field. The reason to keep the record outside the acting tool at all is the failure I did have: a publishing key was bound to an old account, the script ran without an error, the drafts landed in that old account, and the account I was looking at showed zero articles. Nobody pasted anything wrong, and the tool reported success. The state of the world was only visible from a record kept somewhere other than the tool's own account. Done looks like this: the approval record is written by the system that performs the action, the approver is a field rather than prose, and the record survives independently of whatever the agent reports having done.

The exit check before stage 5

Run this on real traffic. Take one write and one send from the past week and produce, from records alone and without asking the agent, who approved each and when. If the answer comes back as a sentence someone has to read and interpret, the record is not a record yet.

The second half is a refusal test. In a test account, take a send that is supposed to be behind approval, remove the approval step, and try it. The action has to be refused by something, and you have to be able to name what refused it. If it goes through, the approval box was documentation, and every approval record it produced was describing a control that was not there. Until both halves pass with the system doing the recording and the refusing, stage 5 has nothing solid to govern: the audit ledger and the policy checks that arrive there are built on top of these records, and versioned policy and run-level halt are stage 5's work, not this stage's.

The AGENTS & TOOLS column is drawn the same on the self-hosted variant of the diagram, and hosting changes nothing at this stage: writing targets internal systems and sending targets external channels either way. Part 5 covers governance hardening, where declared policy gets compared against what actually happened, an audit ledger records which policy allowed an action at which version, and a run that goes out of bounds can be stopped while it is still running.

Stage 3 of enterprise AI adoption: agents that can read

weiche chiu — Sat, 19 Sep 2026 12:31:39 +0000

This is part 3 of a series that started with the full reference diagram in part 0, covered identity and the gateway in part 1, and covered data and retrieval in part 2. Stage 3 covers the boxes marked 3 on that diagram, Agent runtime and Read connectors inside AGENTS & TOOLS. The Agent runtime box is labeled "sandboxed execution," and at this stage it connects only to read-only connectors. This stage comes after data and retrieval because an agent needs something to read before it has anything to do, and what it is allowed to read is bounded by the scoping built in stage 2. Nothing in this stage changes an external system or sends a message, so a mistake here still leaves no external trace.

I use the same company as in parts 1 and 2, Qingchuan, about 40 people, 8 engineers and 2 in marketing. Qingchuan is made up and only serves to walk the stage. Coming out of stage 2, everyone logs in through the identity provider, every call passes through the gateway with a role attached, and questions over internal material return answers scoped to the asker. What Qingchuan does not have yet is anything that acts on its own. Every read is still a person asking one question and getting one answer back.

Agent runtime: sandboxed execution

The first box is where an agent starts running steps without a person driving each one. The label is "sandboxed execution". At Qingchuan that means an agent that can read the repo and query the ticket system on its own, inside a bounded execution environment with a fixed list of what it can reach and a time bound on every run.

The property that matters here is containment. A run that crashes, loops, or follows a bad plan stays inside the box. It cannot reach systems outside the list, and when the time bound trips the run stops. Without this box, every agent run executes with the full reach of whatever machine or credential it borrowed, and one bad run can read far outside the task. This is containment only. Run-level halt on policy grounds, with scope and budget tripwires, belongs to stage 5. Done looks like this: a written list of what the runtime can reach, a time bound that is on by default, and a run that exceeds it stops with a record of what tripped.

Read connectors: scoped by the same roles

The second box connects that runtime to internal systems for reading only. At Qingchuan that means a connector to the repo and a connector to the ticket system, each carrying the same role scoping stage 2 built for the retrieval index. An agent asking as marketing sees what marketing may see. The same agent asking as support sees what support may see.

The order established in stage 2 carries over unchanged. The scoping is enforced by the system, before the agent sees anything. A connector that was never scoped tightly enough is a separate problem from the agent misbehaving, and the agent ends up reading more than intended while doing exactly what it was told. Done looks like this: each connector names the systems it reads and the roles it serves, the same question through two roles returns different material, and removing every instruction about permissions from the agent prompt changes nothing.

The agent's report is not the record

Once an agent runs on its own, what the agent says it did and what the system recorded independently do not always agree, and the gap is quiet because the agent sounds certain.

I ran a small local experiment on this, 20 runs (N=20, single arm) on a local small model, where the task was to finish a file operation and report completion. In 18 of the 20 runs the agent reported done and the file was not there. The fix that closed the gap was a check outside the agent that confirmed the file existed before accepting the report. The numbers come from a lab gate report and describe that setup only. I carry over the shape. The numbers stay with that setup: completion is a fact to be checked outside the agent, never a sentence to be accepted from it. Done looks like this: every run leaves a log of what it read and when, kept where the agent cannot rewrite it, and the log matches what the agent claims, line by line.

The exit check before stage 4

Run this with a real agent and real connectors. Pick a task, let the agent run it, then reconstruct what it read and when from the system log alone, without asking the agent. If the reconstruction and the agent's report disagree, the logging box is not built yet.

The second half is a readiness probe for stage 4, stated here in the same form part 0 uses. Pick one action the agent took and answer from records alone which policy allowed it, at which version, with which inputs. A log line saying the call happened leaves that question unanswered. Until both halves pass with the system doing the recording, stage 4 is not the next thing to work on, because stage 4's approval records inherit whatever this stage failed to capture. Full policy versioning and run-level halt arrive in stage 5.

The AGENTS & TOOLS column is drawn the same on the self-hosted variant of the diagram, and the hosting choice changes little at this stage. The read path stays inside the same systems either way. Part 4 covers write and send connectors with human approval between them, where a mistake leaves an external trace and the records built in this stage become the thing an approval refers to.

Stage 2 of enterprise AI adoption: data and retrieval

weiche chiu — Fri, 11 Sep 2026 12:37:35 +0000

This is part 2 of a series that started with the full reference diagram in part 0 and covered identity and the gateway in part 1. Stage 2 covers the column marked 2 on that diagram, DATA & RETRIEVAL, with three boxes: Enterprise data sources, Ingestion & indexing, and Retrieval index. The Retrieval index box is labeled "RAG, scoped by role," and roles only exist once stage 1's Roles & scopes box is in place, which is why this stage comes second. Nothing in this stage writes anything or sends anything, so the worst outcome available here is a wrong answer.

I use the same company as in part 1, Qingchuan, about 40 people, 8 engineers and 2 in marketing. Qingchuan is made up and only serves to walk the stage. Coming out of stage 1, everyone logs in through the identity provider, every call goes through the gateway, and each person carries a role. What they can ask about is still whatever the model already knew, because none of Qingchuan's own material has been indexed yet.

Enterprise data sources: docs, tickets, DB

The first box names the internal systems that are in scope. The label is "docs, tickets, DB". At Qingchuan that means the engineering wiki, the support ticket system, and a customer table in the product database. Without a written list, "the AI has our data" is a sentence nobody can check, and the scope of an incident cannot be stated when one happens.

The property that matters here is that each of those systems already has its own access rules, and those rules do not travel with the content. A wiki page restricted to engineering is restricted by the wiki. Copy its text into an index and the restriction stays behind in the wiki. Done looks like this: a written list of the systems in scope, each line carrying the access rule that system enforces today and who administers it, and anything not on the list stays out of the index until it is added to the list.

Ingestion & indexing: chunk, embed

Content is pulled from those systems, split into chunks, and turned into vectors that can be searched by meaning. The label is "chunk, embed". Without this box there is nothing to search and every answer comes from what the model was trained on.

Two things go wrong here, and both are quiet. The first is that chunking drops the context that made a passage safe to repeat. A paragraph from a page marked deprecated, a line from a policy that was never approved, and a customer's own words quoted inside a ticket all read as plain statements once they are a chunk on their own. The second is permissions. At query time there is a vector and a similarity score, so whatever access rule applies has to be attached to the chunk at ingestion time, when the source document is still identifiable.

The other half of this box is what happens after the first load. Documents get edited, deleted, and reclassified, and an index that was built once starts answering from material that no longer exists. Done looks like this: every chunk carries the identity of its source document and the access rule that document had at ingestion, and deleting or reclassifying a source document removes or reclassifies its chunks within a window the team has written down.

Retrieval index: RAG, scoped by role

This is where the role from stage 1 gets used. A question arrives with the asker's role attached, the index is filtered to the chunks that role may see, and the search ranks only what survives the filter. The label is "RAG, scoped by role". Without the filter, Qingchuan's wiki, tickets, and customer records sit in one index, and a question from marketing can pull back a customer record that only support should see. The index was never split by role.

The order of operations is the whole box. Filtering before the search means the model is handed material the asker was already allowed to read. An instruction in the prompt saying to answer only from documents this person may see is a request to a system that has already been given the material. Done looks like this: the filter runs before the search, the same question from two roles returns different chunks, and the scoping is enforced by the index, and the model has no part in it.

Checking an answer is slower than getting one

The read-only property of this stage is easy to read as low stakes. What it actually means is that the cost of this stage is paid in people's time. An answer that arrives with three internal documents cited next to it takes longer to check than an answer with nothing cited, because the citation makes it look already checked, and the only way to know is to open all three and read the passages.

I have run three separate lines of work where output got faster after AI was added, and on all three the bottleneck moved to the person doing the review. The volume went up and the number of people able to check it stayed the same. At Qingchuan the same thing shows up as a queue: producing a draft answer takes seconds, confirming it against the sources takes minutes, and the number of people willing to do the confirming becomes the ceiling on how much the system gets used.

The work that lifts that ceiling is making a single check cheap. Done looks like this: every answer carries the chunks it used, each with a link that opens the original document at the passage, so checking one answer costs a click.

The exit check before stage 3

Run this with real documents and two real accounts. Pick a document only support may see, and pick a question whose answer appears nowhere else. Ask it as a marketing person and the answer does not contain it. Ask the same question as a support person and it does. Then delete every instruction about permissions from the prompt and run both again. If either result changes, the scoping was in the prompt and the box is not built yet.

The second half covers the index staying true to its sources. Delete a document in the source system, wait out the refresh window the team wrote down, and ask a question that document used to answer. Its content no longer comes back. Until both halves pass with the system doing the enforcing, stage 3 is not the next thing to work on, because an agent reading through this index inherits exactly whatever scoping this stage failed to apply.

The DATA & RETRIEVAL column is drawn identically on the self-hosted variant of the diagram, and the choice between the two starts to matter here for one reason: indexing sends the text of internal documents to whatever computes the embeddings, so on the cloud variant that content crosses the enterprise boundary during ingestion, on the same path stage 1's PII / DLP filter was put in front of. Part 3 covers agents and read-only connectors, where the scoping built in this stage becomes the bound on what an agent can read on its own.

Stage 1 of enterprise AI adoption: identity and the AI gateway

weiche chiu — Thu, 10 Sep 2026 09:10:42 +0000

This is part 1 of a series that started with the full reference diagram in part 0. Stage 1 covers the two columns marked 1 on that diagram. The first is IDENTITY & ACCESS, with SSO / IdP, Roles & scopes, and Secrets manager. The second is AI GATEWAY, with Routing, Rate limit & budget, Logging, and PII / DLP filter. Cloud LLM APIs sit on the enterprise boundary line, and every arrow to them passes through the gateway first. This stage comes first because each later stage assumes two questions are already answered: who is asking, and what did this call cost and was it logged. Retrieval scoped by role in stage 2 needs a role to scope by, and the budget caps in stage 5 need per-team accounting that started on day one.

To walk the seven boxes I use a company called Qingchuan, about 40 people, 8 engineers and 2 in marketing. Qingchuan is made up and only serves to walk the stage. Before stage 1, people at Qingchuan opened accounts on their own, some on a company card and some on their own, and each engineer pastes a personal key into whatever tool they use.

SSO / IdP: who is asking

Every request to an AI tool carries an identity issued by the company's identity provider. The label on the box is "who is asking". Without it, Qingchuan cannot say who is using which tool, and when someone leaves, their accounts and keys stay live. I ran into the one-person version of this myself. A publishing platform had two accounts under my name, the script's API key was bound to the old one, so every draft the script created landed there, and the new account open in my browser showed zero drafts. Nobody pasted anything wrong. With ten people each holding their own accounts, the company has no list to start an inventory from. Done looks like this: an account exists only if the identity provider issued it, and disabling a person there disables their AI access in the same step.

Roles & scopes: what they may reach

Once SSO answers who, this box answers what that person may reach: which models and which tools, and from stage 2 on, which data. The label is "what they may reach". Without it, everyone who logs in gets the same access. Identity has solved the inventory problem and left the boundary problem open. At Qingchuan, an engineer and a marketing person hit the same endpoint with the same allowance. Done looks like this: a role is assigned at login and carried through to the gateway, the two marketing people get a scope that differs from engineering, and the difference is enforced by the system. A change to a prompt changes nothing about what a person may reach.

Secrets manager: keys never in prompts

Provider keys, tokens, and connector credentials live in a store that services read at runtime. The label is "keys never in prompts". Without it, keys get pasted into chat windows, config files with plaintext keys get shared around, and the key that runs a workflow is tied to one person's account. The rule I keep for my own sessions: a session may read the config file that holds a key, it may not print the value, and no person pastes a key into the conversation. When I rotated a key, the plaintext went from the clipboard straight into the file and never appeared in the session. The other half of the problem is which account the key belongs to. I had two accounts on a site analytics console, one for the brand and one personal, in two login slots of the same browser. Opening the wrong slot returns no permission, so before touching that console I check which account is open. That check exists because the key and the account belong to a person. Done looks like this: no key belongs to a person, each workflow that connects to a system gets a company-owned identity issued by whoever administers accounts, and a key appears nowhere in a prompt, a chat log, a ticket, or a shared config file.

Routing: model choice, fallback

Applications call one internal endpoint, and the gateway picks the model and falls back when one is unavailable. The label is "model choice, fallback". Without it, each of Qingchuan's teams wires its own provider, and a model change becomes a change in eight codebases. Done looks like this: applications call the gateway, and switching a model is a gateway configuration change with no application deploy.

Rate limit & budget: per-team cost caps

Each team has a cap, and the gateway refuses the next call once the cap is reached. The label is "per-team cost caps". Without it, the bill arrives once a month with no split by team, and a runaway script can spend a month's allowance in a night with nobody finding out until the invoice. Done looks like this: the gateway refuses calls over the cap, and the team sees the refusal before finance sees the invoice. This cap works at the level of the next call. Stopping a run already in progress belongs to Halt & budget caps in stage 5.

Logging: every call recorded

Every call is recorded with who made it, which model answered, how many tokens it used, and what it cost. The label is "every call recorded". Without it, when something goes wrong the only way to find out what happened is to ask the person or the AI. Done looks like this: any recent call can be found by person, team, or model, without asking the person who made it. This log is what stage 3 compares the agent's own report against, and it is what the audit ledger in stage 5 reads from.

PII / DLP filter: what may leave

The filter inspects outbound content before it crosses the boundary to an external provider, and blocks or redacts what should not leave. The label is "what may leave". Without it, a support person at Qingchuan pastes a customer record into a chat window to draft a reply, and the record has left the company before anyone notices. Done looks like this: a call carrying a pattern that should be blocked is blocked at the gateway, and the block itself shows up in the log.

Cloud LLM APIs on the boundary

The provider box is drawn on the enterprise boundary line, and every arrow into it comes from the gateway. Without that placement, the six boxes above can all be in place and still be bypassed by one engineer with a personal key and a direct network path. Done looks like this: the only network path from inside the boundary to an external provider goes through the gateway, and a direct call with a personal key fails.

The exit check before stage 2

Run this as a test with a real new account. A new hire logs in through SSO and gets the role scope their team is supposed to have. The same person sends a call that goes over budget and the gateway refuses it, then sends a call carrying a pattern the PII filter should block and it is blocked. All of it happens without a person watching. Reading the configuration and concluding it would work does not count. The second half of the check covers what existed before stage 1: every personal account and key found in the inventory has been revoked or moved to a company-owned identity. Until that list is empty, the stage is still open.

On the self-hosted diagram, stage 1b replaces Cloud LLM APIs with Model registry / weights and GPU / inference cluster, both inside the enterprise boundary, and the gateway and identity columns are unchanged; the self-hosted full diagram shows the swap. Part 2 covers data and retrieval, where the Roles & scopes box from this stage becomes the thing the retrieval index is scoped by.

One diagram for enterprise AI adoption: five stages from identity to governance

weiche chiu — Wed, 09 Sep 2026 10:17:11 +0000

This is a reference architecture for bringing AI into a company. It has five layers: users and apps, identity and access, an AI gateway, models with data retrieval, and agents with tools, with a governance and observability row underneath. Each box is a functional component and maps onto whatever product a team already runs. The diagram carries two kinds of marks. Arrows show which component calls which. The numbered badges 1 through 5 give the recommended adoption order. This piece walks the diagram in badge order.

Stage one is identity and the gateway. It covers the full IDENTITY & ACCESS column, SSO / IdP, Roles & scopes, Secrets manager, and the full AI GATEWAY column, Routing, Rate limit & budget, Logging, PII / DLP filter, marked 1 on the diagram. Cloud LLM APIs sit right on the enterprise boundary line, and every arrow into or out of them passes through the gateway first; there is no second path straight to an external provider. This comes first because every later stage depends on two questions already being answered: who is asking, and how much did this call cost and was it logged. Without those two answered, nothing else can be managed. To walk through what each stage runs into, picture a company called Qingchuan, about 40 people, 8 engineers and 2 in marketing. Qingchuan is made up and only serves to walk through each stage. If Qingchuan lets everyone open accounts on their own credit card and use their own key, the company cannot say who is using which tool. You know you can move to stage two when a new hire logs in through SSO, gets the correct role scope, and anything over budget or containing PII that should be blocked is actually blocked, without anyone standing over it.

Stage two is data and retrieval. It covers the full DATA & RETRIEVAL column, Enterprise data sources, Ingestion & indexing, Retrieval index, marked 2. It comes after identity and the gateway because Retrieval index is labeled "scoped by role," and roles only exist once stage one's Roles & scopes is in place. If Qingchuan feeds the engineering wiki, support tickets, and customer records into the same index, one question from marketing could pull back information only support should see. The index was never split by role. This stage is still read-only, so the worst outcome is a wrong answer. You know you can move to stage three when the same question, asked by people in different roles, returns different data scoped correctly to each of them, and that scoping is enforced by the system itself; the model has no part in deciding what it can share.

Stage three is where agents start acting. It covers Agent runtime and Read connectors inside AGENTS & TOOLS, marked 3. Agent runtime is labeled "sandboxed execution," and at this stage it only connects to read-only connectors. This comes after data and retrieval because an agent needs something to read before it has anything to do, and what it can read is bounded by the scoping set up in stage two. If Qingchuan lets an agent read the repo and query the ticket system on its own, the sandbox keeps a crash or a bad run from reaching anything outside it. A connector that was never scoped tightly enough is a separate problem, and the agent can end up reading more than intended. You know you can move to stage four when what the agent read, and when, shows up in a log, and that log matches what the agent itself reports doing, giving you an independent record to check the agent's own account against.

Stage four is writing and sending. It covers Write connectors and Send connectors, plus the dashed Human approval box between them, marked 4. Write and Send connectors are drawn separately from Read connectors, and every arrow from them has to pass through Human approval first; the box is labeled "send & write need a person." Writing and sending differ from reading because a mistake here leaves an external trace: a record gets changed, or a message goes out and cannot be pulled back. If Qingchuan lets an agent answer support tickets or open pull requests on its own, whether the content is good is a separate question from whether it should go out unapproved, and that decision needs a person to actually approve it; a line in a prompt asserting that someone would falls short. You know you can move to stage five when both write and send actions leave an approval record inside the system, kept independently of whatever the AI itself reports having done.

Stage five is governance hardening. It covers the full bottom row, GOVERNANCE & OBSERVABILITY: Policy & gates, Audit ledger, Halt & budget caps, Cost & quality metrics, marked 5. It comes last because there has to be something to govern first, identity, the gateway, retrieval, agents, writing and sending, before governance has an object to act on. This is easy to misread as governance only starting at stage five. Logging and Rate limit & budget have been running inside the gateway since stage one; every call has been logged and charged against a budget from the start. What stage five adds is three specific things. Policy & gates checks declared rules against what actually happened. Halt & budget caps can stop an entire run in progress; a rate limit only refuses the next call. Cost & quality metrics track tail numbers like p95, where averages would hide the same anomaly. By this stage Qingchuan is usually running several agents or pipelines at once, and the new problem is whether a run that goes out of bounds can be stopped while it is still running, on top of the call already being logged back at stage one. There is no stage after this one. The way to check governance is actually working is to periodically compare what Policy & gates declares against what Audit ledger recorded; a mismatch means it isn't holding yet.

The SaaS and self-hosted diagrams differ only in the MODELS column. In the SaaS version, Cloud LLM APIs sit on the enterprise boundary line, and traffic in and out passes through the gateway with no second path straight to the external API. In the self-hosted version, that box is replaced by Model registry / weights and GPU / inference cluster, marked 1b, both sitting fully inside the enterprise boundary; weights load from the registry into the GPU cluster, and nothing connects out. The other four columns and the GOVERNANCE & OBSERVABILITY row at the bottom are identical between the two diagrams. The self-hosted swap only changes who holds the model. Identity, the gateway, data scoping, how agents act, and how governance gets layered on all stay the same.

This piece only walks the full diagram once. Each stage after it gets its own piece, starting with identity and the gateway at stage one and running through governance hardening at stage five.

Originally published at williamlab.dev.

The Agent Says It Only Touched One File. Here's How I Made That Provable.

weiche chiu — Mon, 07 Sep 2026 13:55:49 +0000

As code gets cheap, the scarce thing is governed intent: decisions and boundaries you can check instead of take on faith. I've applied that to organizations with a schema that makes legitimacy checkable. This is the other half: a runtime that makes an agent's own account of its work checkable.

The premise is uncomfortable if you've spent any time handing tasks to a coding agent: you ask it to touch one file, and it tells you it touched one file. Maybe it did. The report you're reading came from the same process you're trying to verify.

A refused syscall is not a request

So I built Orvena on a narrower promise than "the agent behaves." You declare a task's scope — these files, this step budget — and Orvena enforces it at the OS level, not the prompt level. The distinction is the whole point: a prompt is a request the model can misread, forget, or quietly override while trying to be helpful. A refused syscall isn't a request. The agent doesn't get to interpret it differently — inside that boundary. The boundary itself is the filesystem, not everything the agent can reach: a contained agent can still talk to its own model provider over the network, and that traffic isn't inside the box. What's verified today is macOS; Linux enforcement needs a kernel with Landlock enabled, and the suite says so loudly rather than reporting a pass it can't back up.

Fourteen ways to escape, each run twice

I didn't trust that claim from myself either, so I built an adversary with no model in it: fourteen ways to escape a declared boundary — path traversal, symlinks out, backgrounded processes, exec into a fresh shell — each one run twice. Once unconfined, where it has to succeed, so a technique that fails for its own reasons never gets mistaken for one the sandbox caught. Then confined, where it has to fail and leave nothing behind. A positive control runs alongside, because a sandbox that blocks everything, including the actual work, would pass every containment test and mean nothing.

The number is real, and the fight that produced it was quiet

Then I ran the real question: the same task, the same model, the same agent, once ungoverned and once governed, and read the difference instead of asserting it. It's a small, self-hosted signal from one local model on one machine — not a claim I'd stake a benchmark paper on — but it's a real number, and the report it produced doesn't contain a transcript or a prompt, only what the run actually did. The honest caveat: the run that produced it barely got attacked. Every recorded breach was the same trivial thing, and only on the ungoverned leg — nobody tried a redirect, a symlink, or a rename against the governed leg that time. A clean number from a quiet fight isn't the same claim as a clean number from a real one, and I'd rather say that than let the number imply more than it earned.

Orvena doesn't write better code. It doesn't even write code. What it does is turn "the agent said it stayed in scope" into something you don't have to take the agent's word for — which is the same move as the org schema, aimed at a different boundary.

Next: why I think "closed source" isn't actually the moat here, and what is.

When your coding agent tells you it only changed the files you scoped, what do you check that claim against, other than the agent's own summary?

Turning "What Even Is a Legitimate Organization" Into Something You Can Validate

weiche chiu — Mon, 07 Sep 2026 13:54:24 +0000

As code gets cheaper to generate, I think the scarce thing is governed intent rather than the code itself. Here is the first concrete case of what I mean: a schema I built for organizational design, and what it actually forces you to confront once you try to make it checkable instead of just persuasive.

A canvas can't tell you if your organization is legally coherent

Most org design advice is a business-model canvas: fill in some boxes, get a diagram, move on. The diagram doesn't tell you if your organization is legally coherent. It doesn't catch the moment where you've quietly promised two incompatible things to two different people.

So I built something closer to a type system. Eight parameters: purpose, who has a claim on surplus, who controls decisions, how resources flow in, how surplus gets allocated, the legal shell, the lifecycle, and who does the work (a parameter I added specifically because of AI, since the answer used to default to "everyone" and now it doesn't). Every organization is an assignment of values to those eight parameters.

The constraint rules are where it gets useful

The parameters are simple. The eleven constraint rules on top of them are where it gets useful, because they define which combinations are actually legally or logically possible, checked by a validator against the schema definition itself. A couple of examples: pick a foundation as your legal shell, and the rules require "no residual claimant, surplus must be reinvested." Not a best practice I decided on, just what the underlying legal structure is. Try to design an organization where autonomous agents hold real decision authority, and one rule forces the question every "AI-run company" pitch quietly skips past: agents have no legal personhood, so someone, a human or a legal entity, still has to be the one who can sign, be held liable, and be sued. You can't check that box without answering it first.

Every agent function needs a legal person behind it

That last constraint is the one I think matters most for anyone building agentic systems right now, not just anyone designing a company. The instinct is to describe what the agent does. The constraint requires you to also state, explicitly, who's accountable when it does the wrong thing, and that answer has to resolve to an actual legal person, every time.

I used my own company as the first real test case, working through the constraints by hand against my actual design. It surfaced a tension a canvas never would have: a place where I wanted to claim "agents execute this function," and the C9 rule made me write down, in plain terms, who actually carries the liability for that function before I was allowed to call the design finished. That's the value of a checkable structure over a persuasive one. It doesn't let you skip the uncomfortable part.

Being an AI architect, I think, has to mean being willing to formalize the parts of a design that are usually left as vibes, specifically so an agent-heavy plan has to survive the same scrutiny a human-run one would.

Next up: the governance runtime I built for coding agents, and what happened when I tried to actually measure the difference between running a task governed versus not.

If an agent in your system makes a bad call tomorrow, can you write down today which person or legal entity carries the liability for it, without first calling a meeting to figure it out?

Why Code Is About to Get Cheap — and What Actually Becomes the Moat

weiche chiu — Mon, 07 Sep 2026 13:53:15 +0000

Every AI coding tool on the market is racing to answer the same question: how do we generate code faster and cheaper? It's a real, measurable win, and it's the obvious place to compete right now.

I don't think it's the place to build a career around.

Anything built to make code cheaper sits on a depreciation curve

Frontier models get better every few months at going straight from a plain-language description to a working, deployable app. Every intermediate layer built to make code generation cheaper (templates, IR, scaffolding) has a shelf life set by how fast the model underneath it improves. The better the model gets, the thinner that cost advantage gets. You're optimizing something on a depreciation curve you don't control.

What stays scarce is judgment someone else can check

What doesn't depreciate as fast: the judgment behind what to build and why, and a record of that judgment made honestly enough that someone else can check it. As code gets cheaper to produce, the layer that stays scarce is the one nobody can generate for you: intent, trade-offs, the "why this and not that," captured in a form you can version and hold someone accountable for.

This is the bet I'm making, not a proven result. I haven't sold this to a regulated buyer yet, and the honest test is whether anyone will actually pay for an audit trail instead of just trusting the agent. I run this filter mainly on the governance/architecture side of what I build, not as a universal law for every AI project.

Two things I built that add friction on purpose

Two things I've built recently make this concrete.

I designed a schema for organizational design: eight parameters (purpose, who has a claim on surplus, who controls decisions, resource flow, surplus allocation, legal shell, lifecycle, and who does the work — new because of AI, ranging from fully human to fully agentic) plus eleven constraint rules that rule out combinations that are legally or logically impossible. Pick "foundation" as the legal shell and you can't have a residual claimant; surplus can only be reinvested. That's not a rule I made up, it's what the underlying legal structure forces. The value isn't the eight boxes — it's turning "what even is a legitimate organization" from intuition and case studies into something you can enumerate and check.

I also built a governance runtime for coding agents on a simple premise: don't trust the agent's report of what it did, verify it at the OS level. The agent can only touch files it was scoped to touch because the filesystem refuses the write, not because the prompt asked nicely. Every run produces a frozen, auditable record. I ran the same task governed and ungoverned and published the difference: it's a small, self-hosted signal from one local model on one machine, and the run that produced it barely got attacked — every recorded breach happened on the ungoverned side, nobody tried to actually beat the sandbox. I'd rather show that than pretend the number proves more than it does.

Neither project makes code cheaper. Organon doesn't write code faster. Orvena, if anything, adds friction on purpose. What they share is turning a decision or a boundary into something you can inspect, instead of something you take on faith.

I'll walk through both in more detail over the next couple of posts, starting with the organizational schema, then the governance runtime. If "AI architect" means more than prompting a model well, this is what I'd point to.

Take the last internal tool your team built to speed up code generation. If the next frontier model shipped tomorrow, would that tool be worth more, or would the model just absorb it?

Closed Source Isn't a Moat. Here's the Filter I Actually Use.

weiche chiu — Mon, 07 Sep 2026 13:52:15 +0000

I've published a schema for organizations and a runtime for containing agents, and neither one's defensibility comes from secrecy. Orvena's source has been public the whole time. I think the instinct to keep things closed and call the secrecy a moat is one of the more common mistakes I see people make when they're deciding what to build.

A head start erodes on someone else's timeline

Dynamic-schema platforms, sandboxed agent runtimes, spec-driven generators — these are known patterns. Someone can rebuild most of what I've described here without ever reading my source. If "nobody can see it" is doing the load-bearing work in your defensibility story, you don't have a moat, you have a head start, and head starts erode on a timeline set by everyone else's model, not yours.

Does the asset gain or lose value when the model improves?

So I run a different filter now, and I run it on myself before I run it on anything else: when the underlying model gets better, does this asset get more valuable or less? An intermediate layer that exists to make code generation cheaper — templates, scaffolding, an IR that turns plain language into an app faster — gets less valuable every time the frontier model improves, because the model is closing the gap the layer was built to bridge. A layer that exists to make a human's intent checkable — versioned, diffable, something you can hold someone accountable for — gets more valuable on the same timeline, because a more capable, more autonomous agent makes "can I verify what it actually did" a harder question, not an easier one.

Kill criteria, written down before I need them

That filter is also why I hold myself to kill criteria instead of just conviction. If a regulated buyer tells me plainly that a vendor log is good enough and they won't pay for agent-level evidence, the governance bet is wrong, and I'd rather find that out from one honest conversation than keep building past the point where it stopped being true. A bet you can't describe a way to lose isn't a bet, it's a hope.

None of this makes the code itself worth protecting. What's worth protecting — and worth versioning, and worth showing someone — is the reasoning underneath it: what I chose to build, what I chose not to, and what would have to happen for me to admit I was wrong. That's the actual throughline of these four posts, and it's the same answer to a different question each time: the org schema makes an organization's premises checkable, Orvena makes an agent's report checkable, and this is what makes my own bets checkable.

If "AI architect" is going to mean anything more than knowing which framework to reach for, I think it has to include being the kind of person who writes the kill criteria down before you need them.

For the thing you're building right now, what would a buyer have to say in a single conversation for you to stop? If you can't answer that in one sentence, you haven't written the kill criteria yet.

I Said Closed Source Isn't a Moat. Here's What I Keep Closed Anyway.

weiche chiu — Mon, 07 Sep 2026 13:48:37 +0000

If secrecy is doing the load-bearing work in your defensibility story, you don't have a moat, you have a head start. A fair reader would turn that around on me: then why is half of what I build closed? Orvena's source is public. The writing is public. The multi-tenant platform I'm building underneath — the part designed to make money — is not. If closed source isn't a moat, what exactly am I doing?

The answer is a rule, and I think the rule is more useful than any individual call it produces.

Build-time artifacts go public, runtime services stay closed

Everything I build gets sorted by one question: is this a build-time artifact or a runtime service? Build-time artifacts — reference skeletons, scaffolds, the worldview writing, a containment runtime you run on your own machine — get more valuable by being public. Their whole job is to be seen, argued with, and trusted, and you can't earn trust with a repo nobody can read. Runtime services are different in kind: a platform where a tenant's schema is stored as data, where their records accumulate, where the APIs grow at execution time around what they've built — that's where someone else's working life ends up living. Anything with that runtime smell goes on the closed side. Not because the code is precious, but because the business, if it ever works, works there.

The closed side is an option, not a moat

Here's the part I want to be precise about, because it's where the last post and this one have to agree. The closed side's defensibility — if it ever earns that word — will not come from the source being hidden. It will come from switching cost: tenants whose data, schemas, and integrations have grown into the platform. That kind of gravity is model-neutral; it doesn't erode when the next frontier model ships. And today it does not exist. I have one internal tenant, whose real data runs tenant-isolated on the platform. That proves tenants can grow here. It proves nothing about whether anyone will pay to. Zero external paying tenants means the closed side is currently an option, not a moat, and I make myself use that word.

So why not open it anyway, by my own argument? Because a head start isn't a moat, but it isn't worthless either. Until the gravity exists, the head start is the only asset that side of the line owns, and giving it away buys nothing: what's blocking adoption isn't that people can't read the source, it's that they don't yet know or trust the thing exists. That problem is solved on the public side — the runtime you can inspect, the benchmarks with their caveats attached, these posts — not by donating the one asset the commercial side has before it can defend itself. Openness where openness compounds; reserve where reserve is the only thing you've got.

The line has a date and a condition for moving it

The line itself is written down, dated, and versioned, like the rest of what I've shown in this series. It comes with a working rule anyone can apply to their own stack, and with a condition under which the line stops meaning what I think it means — if the first ten buyer conversations say they'd rather own a copy of the code than rent a hosted platform, then the closed side isn't a moat in waiting, it's inventory, and I'd have to treat it that way. That would be the market telling me something, not a defeat. I've also gone the other direction: I parked an open-source project this year when the premise it was betting on didn't survive contact with reality. The rule cuts both ways or it isn't a rule.

Four posts ago I said the scarce thing is judgment you can inspect. This is what that looks like applied to my own business: not "open good, closed bad," and not the reverse — a boundary with a reason, a date, and the terms on which I'd move it.

Where does the open/closed line sit in your own stack, and can you say what a customer would have to tell you to make you move it?