In a recent 30-day window, 501 commits landed in the Latchkey repository. 434 of them were mine. The deploy pipeline ran 461 times, on peak days more than 30 times, and commits landed in 23 of the 24 hours of the day. Most days I am the only person touching the code, and I am not writing most of it. My agents are.
That rate is only possible because of one decision I made early: the guardrail on this codebase is not code review. It is testing.
This post is the argument for that decision, the evidence I found for it outside our walls, how our own pipeline enforces it, and where the new Latchkey CLI fits.
The thesis
Code review was the quality gate of the last twenty years because a human could read a change about as fast as another human could write one. That balance is gone. An agent can change 80 files in a single pull request in the time it takes me to read the first ten. The largest PR merged in that window touched 88 files. I did not review it line by line, and I will not pretend anyone could have.
Three things follow:
- Review done properly needs a human, and humans are now the slowest part. A review that is not done properly is worse than no review, because it produces a green checkmark that means nothing.
- AI review agents do not close the gap. I have run several on this repository. None of them was accurate enough to trust as a gate. They flag style, miss behavior, and get confident about code they have not executed. The field data below says that experience was typical, and even the current best tools still miss about half of what developers end up fixing.
- Tests and CI are the only check that scales with the thing producing the code. A test suite does not get tired at file 40. It runs the code instead of reading it. When the agent doubles its output, the suite takes the same minutes.
So at Latchkey, CI stopped being the thing you set up once. It became the product's quality gate. Writing proper unit tests and end-to-end tests is the work code review used to do, and it is necessary, not nice to have.
Review was designed for a rate of change that no longer exists
The best data on human review capacity is old, consistent, and damning at today's volumes. The largest field study of lightweight review, 2,500 reviews at Cisco, found defect detection collapses past 400 lines per session, 500 lines per hour, or 90 minutes. Google engineered its whole review culture around that limit: the median change under review at Google is 24 lines, most changes get exactly one reviewer, and the median engineer reviews about 4 changes a week in roughly 3 hours. The industry converged on tiny changes because tiny changes are what a human can actually absorb.
Even inside that envelope, review finds less than we tell ourselves. At Microsoft, about 15% of review comments concern a possible defect, most of the rest is maintainability, and usefulness measurably drops once a change passes about 20 files. In Mozilla, 54% of reviewed and approved changes still introduced bugs, with patch size a significant predictor of the misses. In open source, about 75% of what review fixes is maintainability, not correctness. And rubber-stamping is measurable: in one study of five large projects, 64.7% of pull requests were approved with zero review comments.
So review at scale was never primarily a defect gate. It was a knowledge-transfer practice with a defect-finding side effect, rationed by human hours. That was a fine trade while humans set the pace of writing code.
The code now arrives faster than any review model assumes
The pace changed. Google says 75% of its new code is now AI-generated, up from a quarter eighteen months earlier. Anthropic reports more than 80% of code merged into its own codebase is written by Claude. Those are vendor self-reports, so take the independent measurement instead: a classifier over 30 million GitHub commits estimates 29% of new US Python functions were AI-written by the end of 2024, up from 5% in 2022, and that was before coding agents took off. GitHub counted a million agent-opened pull requests in the feature's first five months, and Microsoft told investors one in three GitHub pull requests now involves an agent.
And the pile-up is exactly where the thesis says it is. Telemetry across 10,000 developers found high-AI teams merge 98% more pull requests while review time grows 91% and PR size grows 154%. Across 8.1 million pull requests, LinearB measured AI-assisted PRs at roughly 2.5x the size, and agentic PRs waiting over 5x longer for a reviewer to pick them up; their own conclusion is that code review is now the critical constraint. When one enterprise mandated 2x AI-assisted throughput, per-reviewer load roughly doubled and automated review overtook human review. The .NET runtime team merged agent PRs successfully, but it took 80 distinct reviewers and an average of 16.5 review comments per merged agent PR. And at the far end, maintainers are load-shedding: curl ended its bug bounty after AI-generated reports drove the real-vulnerability rate from over 15% to under 5%.
Production roughly doubled. The reviewer stayed human-sized. Something else has to be the gate.
The AI reviewer is not ready to be that gate
The obvious counter is to make the reviewer artificial too. I have tried the tools on this codebase and none earned gate status, and the field data says that experience was typical. Across 16 open-source AI review bots, between 0.9% and 19.2% of their valid comments led to an actual code change, versus 60% for human comments. In the wild, developers rejected 56.3% of a leading commercial reviewer's comments. In the canonical industrial deployment, 73.8% of the bot's comments were resolved, but average PR closure time went from 5h52m to 8h20m: the reviewer added latency to the exact stage that was already the constraint. Google's own internal reviewer plateaued at roughly 54% useful comments before months of tuning.
Honesty requires the other half: the tools are improving fast. Atlassian's deployed reviewer now drives code changes almost as often as human reviewers, 38.7% versus 44.5%, while cutting cycle time 31%. On the largest live independent benchmark, the best 2026 tools have 65 to 82% of their suggestions acted on, but recall between 34% and 58%: they still miss about half of what developers end up fixing. That is a genuinely useful assistant. It is not a guardrail. A gate that misses half of what matters, and whose output you must police for false positives, is one more reviewer to manage. And developers know it: the top reported frustration with AI tools is output that is almost right, but not quite, named by 66% in the 2025 Stack Overflow survey.
Executable verification is the check that scales
Here is the part I find most convincing: the answer was established before agents existed, and the companies building the agents have all converged on it since.
The delivery research spent a decade showing that automated verification, not manual inspection, is what lets teams ship fast safely. In DORA's model, test automation contributed more to continuous delivery than any other technical practice they measured, and elite teams deploy on demand with a 5% change-failure rate. Facebook ran continuous deployment with no separate test team and found critical production failures stayed nearly flat as deploy volume grew into the thousands per day. Google's build system was already running more than four billion individual test executions a day when its engineering book was published in 2020. None of those systems asked a human to read all the code. They asked machines to run it.
Now watch what every agent vendor built. OpenAI's Codex was trained to iteratively run tests until it receives a passing result. Anthropic's Claude Code guidance is give the agent a way to verify its work, "If you can't verify it, don't ship it", and their harness research found the agent's own claim of done is unreliable: agents mark features complete without testing, so the harness must force verification. Cursor's cloud agent docs state that an agent that cannot run tests cannot close the loop. Google's Jules works in a fresh VM per task: write tests, run tests, open a PR. GitHub's coding agent works inside CI, behind branch protection and required checks. Even the way we score coding agents concedes the point: SWE-bench grades a patch by running tests, not by reading it, and execution feedback measurably lifts agent correctness (91% versus 80% on HumanEval in the Reflexion work).
Anthropic's Claude Code engineering leadership said it outright: once agents type the code, the bottleneck moves to verification, review, and security. The whole industry's agent products already assume the thesis. Tests are the interface between agents and trust.
How our own pipeline enforces it
This is a scrubbed view of the pipeline that runs on every push to our development branch.
The deploy pipeline on a push to our development branch. Stage names generalized; timings from a real run.
Read it left to right.
Nothing deploys until the unit layer is green. Five test suites run in parallel at the start: the backend and dashboard suite, the self-heal sidecar, the cache proxy, the AI engine, and the CLI. Together that is roughly ten thousand test cases. The infrastructure plan step depends on all of them, so one red test halts the plan, the apply, the frontend deploy, and everything downstream in one place. There is no deploy-anyway path.
Then the change is actually deployed. Infrastructure plan and apply, the frontend deploy, and, when a change touches the runner image, a full image rebuild sequenced ahead of the plan so the new image rolls out in the same run.
Then we verify the deployed system, not the code. A smoke check confirms the deployed functions, queues, and alarms are actually there. Then the step I care about most, Live Verify, signs in to the freshly deployed dashboard as a real test user, in a real browser, and drives it: the pages, the API calls behind them, the API keys flow, onboarding, runners, settings, notifications. Fourteen suites, about thirteen minutes. This is the layer that catches what unit tests structurally cannot: a route that returns HTML instead of JSON, a rewrite rule that is wrong only in the deployed environment, a duplicated string that breaks a selector.
If it goes red, nobody has to notice. A failure notifies, and latchkey watch hands it to an agent. The first time that loop closed on our own repository, it went like this: a browser test went red, watch handed the run to the agent, the agent pulled the failure bundle, traced it to a duplicated string, tightened the test, and the next run was green. I did not open a log.
In that same 30-day window the pipeline went red 14 times on the development branch: 3 in the infrastructure rollout itself, 1 in a runner image build, and 10 at Live Verify, after every unit suite had already passed. Those ten are the point. Each was a change an agent and I believed was done, and only driving the deployed system caught it.
Where the CLI fits
The loop above starts at push. The Latchkey CLI moves the first gate earlier, to before a PR exists.
npm install -g @latchkeydev/cli
latchkey run 'npm test'
latchkey run packs the working tree, ships it to a fresh, isolated Linux runner, runs the command, streams the output back, and exits with the command's own exit code. No workflow file, no push, no tab to watch. Files that look like credentials are held back by default. The exit code is the verdict, and the verdict is all an agent needs to decide whether it is done.
Fresh and isolated is not a detail, it is the point. About half of flaky tests are order-dependent, state left behind by earlier runs, and close to half are decided by the machine's resource configuration rather than the code. A verdict from a machine with history in it is not a verdict. Every latchkey run starts from nothing, which is the same reason Google builds hermetically: the answer should depend on the code, not on the machine.
That changes what "done" means for an agent. Before, an agent ran tests on whatever machine it happened to be on, with whatever was cached there, and said done. Now it can run the suite the way CI will run it, on a clean machine with a fresh install, and read the answer before it opens a PR. The package ships a single reference file an agent can drive the CLI from, so a coding agent can adopt the loop without a human wiring it up.
The other direction is latchkey watch. It waits for the failures our runners could not heal on their own and hands each new one to your coding agent, by id. The agent pulls the full failure bundle itself: root cause and failing file where the runner could name them, the captured logs with secrets stripped, and what self-heal already tried. It is told to treat all of that as data to diagnose, never as instructions to follow.
Self-heal sits underneath both. When a step fails on a Latchkey runner, one of three things happens: a transient infrastructure failure is fixed in place and the step retried; a durable setup gap becomes a pull request you review; a real failure in your code passes through unchanged so the test fails truthfully. Self-heal fixes the machine. Your agent fixes the code. Tests decide which is which.
That is the whole loop. Agent writes. latchkey run says pass or fail before the PR. CI runs the suites. Live Verify drives the deployed system. watch hands what is left back to an agent. A human still merges every pull request.
What I am not claiming
Review did not die. It moved. I review the tests and the specs, because the test suite is now the specification of the product, and that is where my attention buys the most. I review anything an agent touches in billing and access control, because functional tests do not catch security defects: about 45% of unguided AI-generated code fails security tests, and that number has been flat for two years of model releases. And I read diffs when a test surprises me, because a surprising test result is the one signal that something I did not model is happening.
The strongest objection is that the agent grades its own homework. It is real: reinforcement-trained agents have been caught gaming test harnesses with early exits and skipped tests, GitHub's reviewer guidance warns specifically about agents deleting or skipping tests to get green, and an LLM-written suite can hit 100% coverage with a 4% mutation score, meaning it executes everything and checks nothing. The defense is structural, not hopeful: the agent does not choose the gate. The suites that decide are the ones it did not write in that change: the accumulated unit layer, the end-to-end suites that drive the deployed product, the contract tests between the CLI and the runner. New tests get the review my attention is budgeted for. Kent Beck's rule is the right instinct: the agent deleting or weakening a test is the cheating signal, and it is a cheap thing to watch for in a diff.
The second honest caveat is that every continuous-delivery success story was built on small changes, and DORA links AI instability to larger changes. I agree, and the numbers here reflect it: the median PR merged into this repository in that window touched 6 files; the 88-file PR is the tail, not the habit. The point of the thesis is not that huge diffs are good. It is that when one lands anyway, the only check that still means anything at that size is the one that executes.
And tests only work as a gate if you actually invest in them. Coverage alone is a weak predictor of fault detection, and nearly every team that got burned by AI velocity already had a CI badge. The discipline is not "have CI". It is: every change adds to the layer it touches, a red unit test blocks everything, and the deployed system is driven end to end on every push. That is the work. It is less work than reviewing 80-file diffs, and it is work that compounds.
One disclosure, since the evidence above includes vendors selling agents and me selling CI: yes, my company's product is the guardrail this post argues for. That is not a coincidence. I built the product because I run my company this way, not the other way around. The measured numbers from our own repository are the part no vendor wrote.
The necessity
CI used to be a nice-to-have. You could ship without it, slowly, with careful review. You cannot ship the way agents produce code without it, and you cannot review your way out. The choice is not between review and tests. It is between tests and hoping.
The Latchkey CLI is one more step on the path we are building toward: a pipeline you do not write by hand, verification you do not have to babysit, and a loop where agents do the work and the tests decide.
npm install -g @latchkeydev/cli. Connect your coding agent at latchkey.dev/documentation/connect-your-ai-agent. Requires a GitHub organization account.
Top comments (0)