A repo crossed my feed this week — window-ml — where the code is written and maintained by agents, and the author says plainly that he doesn't read it. Not "I skim it." Not "I review the important parts." He doesn't read it.
The replies did what replies do. Clutched pearls, invoked the sanctity of review, implied the man is one bad merge from deleting prod.
I think he's right, and I think he's also one lazy week away from being very wrong. Both things.
Nobody was reading the code anyway
Let's be honest about what code review actually was on a large service. You read the diff. You read the parts that touch auth, money, and the migration. You skimmed the rest and told yourself the tests would catch it. On a 40k-line repo, "I reviewed it" meant "I sampled it and I trust the sampler."
Agents didn't invent sampling. They changed the ratio. The volume of code went up by an order of magnitude and the reading stayed flat, so the sample rate collapsed. That's the actual news here, and it happened to most teams quietly, without anyone announcing it. The researcher in question just said the quiet part at normal volume.
The review didn't disappear, it moved up a level
Here's the part I'd defend in a design review. When code is cheap to produce, the code stops being the artifact. The artifact is the harness — the eval set, the fixtures, the assertions, the scorer, the seeds, the thing that decides whether the output was good.
That's where the judgment lives now. And it's a real place to put judgment, because the harness is smaller, denser, and it's the only thing that can tell you if any of the generated code did what it claimed.
So "I don't read the code" is not a complete sentence. It's a sentence with the second half missing. The defensible version is: I don't read the code, I read the harness. The indefensible version is just the first clause, and it means you have no review at all — you have a deploy button and a prayer.
What I actually check now
I've been doing this long enough to have a routine, and it's not glamorous.
Break the thing and watch the test go red. Every agent-written PR that claims a fix gets the same treatment: I revert the fix, run the test, and see if it fails. If it stays green, the test is decoration. I've caught more fake tests this way than I've caught real bugs, which tells you something about the base rate.
Check whether the eval set contains the case that broke. If the bug that prompted the PR isn't in the eval, the PR fixed one instance and taught the system nothing. The eval is the memory. Without it you're re-fixing the same bug with different variable names every three weeks.
Look at the blast radius, not the logic. I don't care how the function is written. I care that the diff added a network call, a new dependency, an env var, or a file write outside the sandbox. Agents are extremely good at adding a dependency that duplicates one you already have, and extremely bad at mentioning it.
Ask whether I could delete it and regenerate it. If the answer is yes and nothing changes, the code was never the asset. The harness was. That's not an argument against reading code — it's an argument about which code deserves the reading.
Where this falls apart
Two places, and I want to be specific because the hot take version of this post would just say "agents are the future, old man."
First: agent failure modes are boring and invisible to accuracy evals. Silent exception swallowing. A retry loop that turns one 500 into a bill. A test that passes because it asserts nothing. A scorer that returns 1.0 for an empty string. None of that shows up in a benchmark that measures whether the output was correct. So the harness has to include the unglamorous telemetry — cost per run, wall clock, outbound call count, diff size over time. I log token spend and tool calls per run, and that log has caught more regressions than any eval I've written.
Second, and this is the one that actually worries me: the harness is also code. Someone wrote the scorer. If the scorer was agent-written and unread, you've built a mirror. You're checking generated output against generated judgment and calling the agreement a pass. That's not review, that's a confidence trick you played on yourself.
So I read the scorer by hand. Always. It's usually the smallest file in the repo and it decides everything. Same for the fixture loader. Everything else can be agent-written and skimmed, and I'll sleep fine.
The question worth asking
The pearl-clutching replies are asking the wrong thing. "Did you read the code" is unanswerable at scale and was already a polite fiction. The question is: what did you read instead, and would you bet the deploy on it?
If the answer is the eval suite, the fixtures, and the cost log — good. That's a real review process, and it's probably stricter than what most teams were doing before, because it's written down and it runs on every commit instead of living in someone's head.
If the answer is nothing — if the code is agent-written, the tests are agent-written, the scorer is agent-written, and nobody has read any of it — then you don't have a new workflow. You have an unowned system with a CI badge.
The researcher shipping window-ml gets away with it because he's explicit about where his attention went. That's the whole trick. Not that he stopped reviewing. That he moved the review to the place that can still catch him.
Most teams haven't moved it anywhere. They just stopped.
Top comments (2)
Reading the scorer by hand is the part I'd copy. A scorer that returns 1.0 for an empty string will pass every review that only looks at process, so the only real defense is a check on what it outputs. Even a few planted bad answers that must score low would show whether the mirror is cracked. Do you keep a set of known-bad outputs next to the fixtures, or is the hand read the only guard?
I make Tokenade, and I'd read the run cost next to the diff: tokens spent per change that ships, and which files the agent opened to get there.