Back in March I wrote that in an AI-native company, processes stop being folklore and become measurable artifacts — versioned, evaluated, improvable. That was a claim. This is the receipt.
Seven days in August. A harness I built drained a queue of 37 task packets. Each one implemented by a model in its own git worktree, verified by tests the harness ran itself, reviewed against invariants the packet declared by name, squash-merged onto the feature branch only if it survived all three.
172 files changed. 29,360 lines added. The test suite it was building went from 81 passing tests to 422. Direct cost: $3.89.
Thirty-seven packets, thirty-seven merges, and I did not read a single diff before it landed.
Where this came from
This didn't start as an idea. It started as a habit I'd been demoing to my team for the better part of a year.
The workflow was manual and it was simple: plan first, read the plan yourself, ask for the implementation, then open a fresh model with no memory of writing the code and ask it to assess what changed. Separately, git worktree so several agents could work different changes in the same project without stepping on each other. Nothing clever. It just worked better than letting one long conversation both write the code and reassure me about it.
I ran that by hand for months. Then the honest question: if the loop is always plan → implement → assess-with-a-clean-model, and I'm the one clicking through it forty times, what exactly am I contributing?
The answer was git worktree, a SQLite file, and about a week.
What a packet is
The unit of work isn't a prompt. It's a file in git.
+++
id = "B01"
slug = "adrh-loader"
goal = "add ingest/adrh_load.py — the ADRH income CSV, all levels, all years"
tier = "standard"
gate = "auto"
surface = "api"
spec_commit = "504abb9dca9e050fda5e85a665f5bd2b66c2944b"
spec_path = "docs/IMPLEMENTATION-S3-reporting-baseline.md"
requires = ["A08", "A10"]
max_attempts = 3
forbidden_paths = ["api/ingest/vintage.py", "api/tests/conftest.py", ...]
[[invariants]]
id = "thousands-separator"
critical = true
assert = "`16.429` loads as 16429, not 16.429. The dot is a thousands separator."
[[invariants]]
id = "empty-total-loads-as-null"
critical = true
assert = "An empty `Total` cell loads as NULL, never 0 and never a skipped row. 15,480 rows in the source have one."
+++
Two fields do most of the work.
spec_commit pins the packet to the exact revision of the design document it was cut from. If the spec moves, the packet is stale, and the harness refuses to run it rather than building something the design stopped asking for.
[[invariants]] matters more. Each one has an id, and the reviewer must return a verdict on every id — held, violated, or unverifiable — with a file:line behind it. A review that quietly skips one is rejected by the contract. Silence is not a pass.
Now look at what those two invariants actually say. Not "parse the CSV correctly." A specific dot in a specific number. A count — 15,480 rows — that somebody went and measured in the real file.
That precision isn't style. It turned out to be the entire difference between packets that worked and packets that burned every attempt they had. More on that next time.
The loop
ready ──claim──▶ running ──▶ gating ──green──▶ reviewing ──accept──▶ merging ──▶ done
▲ │ │ │
└────── needs_work ◀─────────┴── red ──────┴── revise ──┘ awaiting_human
│
└── attempts exhausted ──▶ blocked ──▶ the loop halts
Per task: claim it, check the spec hasn't moved, create a worktree, measure the test baseline, plant the supplied test files and commit them, implement, gate, review, merge.
Two exits are not done.
blocked halts the whole queue instead of skipping ahead. A task that failed three times has usually revealed something wrong with its packet, and running the next twenty on the same misunderstanding wastes more than it saves.
awaiting_human is where a packet goes when I marked it gate = "human" at cut time. Five of the 37 carried that flag, on one rule: the failure is silent and the blast radius is the whole stage. A migration that would need undoing on live data. A cache that never hits — every assertion still passes, only latency changes. A chart that draws backwards while the numbers printed next to it stay correct.
An automated green gate cannot certify an invariant that fails green. Pretending otherwise just merges it.
The one idea
Everything above is mechanism. The actual claim is one sentence:
The harness never believes the agent.
The implementer returns a verification block saying its tests pass. That block is recorded and ignored. ruff check, pytest, npm run lint and npm run build are run by the harness, in the worktree, and its parse of the output decides whether the work is even eligible for review.
Same principle at every layer. Test files are planted into the worktree and committed before the model starts, then declared off-limits — so git diff against that commit proves the model made the test pass instead of making it agree. The reviewer's accept is downgraded automatically over a red gate, a violated critical invariant, or a blocker finding, and the reason is written to a column instead of living in my head.
That's the difference between letting a model write your code and having a pipeline. Not the model. Not the prompt. Refusing to accept any actor's report of its own work as evidence about that work.
The first real run
Then I ran it for the first time, end to end, on one small task.
The model did its job. All five invariants held, 96 tests passing. And the harness could not get the work into git.
The first run turned up a cluster of bugs, every one of them mine. My favourite: the loop regenerates its status board after each merge and leaves the file modified — and the loop refuses to start on a dirty tree. It was dirtying its own working copy. It would have blocked its own next invocation.
A week building a governance system for other people's agents, and the first thing it governed into a corner was itself.
What it cost me
The work didn't disappear. It moved.
The harness is 5,741 lines of stdlib-only Python, plus 2,168 lines of tests for itself — 161 of them. No framework, no dependencies, runs on a bare python3. That's a real week of engineering to avoid reading 37 diffs.
Which is a terrible trade for 37 diffs. It's a good trade for a process that states its own failure conditions, records what every run cost, and can be pointed at the next thirty-seven.
And I got the economics wrong going in. My assumption was that a cheap model does the work and a stronger one checks it, so the checking is the rounding error. The first task I measured properly: $0.11 to implement, $0.18 to review. The review was the expensive half. I had built the whole thing on the opposite premise and found out on task one.
It's public now, Apache 2.0: github.com/frozer/factory
One caveat before someone finds it for me. It was built for — and has only ever run against — a repository with exactly two surfaces, an api/ and a webapp/. Point it anywhere else and it reports an environment fault. The toolchain is narrow and the README says so.
But the toolchain was never the interesting part. What generalises is the failure modes: what goes wrong when you let a model plan work for other models, and why the obvious safety net — just retry it — catches none of them.
I gave every task three attempts, because three attempts felt like obvious hygiene.
Four tasks in that queue burned all three. Every one of them failed identically each time.
The retry budget caught nothing.
Top comments (0)