Last month I let AI write 100% of my code for 30 days. The single loudest lesson wasn't "AI is amazing" or "AI is useless." It was one sentence: the thing that writes the code can never be the thing that reviews it. A model grades its own homework and it always passes.
So this month I did the obvious next experiment. If the author can't be the reviewer — fine. Make a second AI the reviewer. Author agent writes the feature. A separate skeptic agent tries to tear it apart. No human in the review loop at all, on purpose, to find out how far the structure alone could carry me.
It worked far better than I expected. Right up until the one moment it mattered most.
The setup
Two agents, deliberately given different jobs — because I'd already learned the hard way that "independence" is not a second prompt to the same model asking "is this correct?" It just agrees with itself in a calmer voice.
- The author got the normal brief: build the feature, make the tests pass.
- The skeptic got an adversarial brief, never a blessing brief. Not "review this diff." Instead: "Assume this is broken. Produce the input that loses a customer money. Find the thing that already exists that this reimplements. Find the state nobody designed for."
Different frame, different objective. A grader looks for reasons to say yes; a skeptic hunting for the failure looks for the one input that breaks it. That gap is the whole reason the second agent is worth its tokens.
I logged everything for 30 days. Every issue the skeptic caught, every issue it missed that I caught in my own final read, every false alarm. 41 real issues that a competent reviewer should have caught. Here's the honest scoreboard.
What the skeptic caught — 38 of 41, and some of them impressed me
This is the part that surprised me, so let me be fair to the machine first.
Architecture drift, gone. The author wrote a second formatCurrency because it didn't know the first existed. The skeptic, pointed at the whole diff and told to "find what this reimplements," caught it in one pass. Same with an inline auth check that duplicated my middleware, and a subtly-different User type in a new module. The stuff that compiles, passes, and quietly rots your codebase — the skeptic was genuinely good at it, better than a tired human at 6pm.
Swallowed errors, flagged. The author's instinct is to wrap everything in try/catch and move on. The skeptic's adversarial frame — "what failure does this hide?" — caught the catch blocks that logged and continued as if nothing happened.
A real race condition. Two requests, one counter, no lock. The author never sees it because it works every time in testing. The skeptic reasoned about concurrent callers because I'd told it to assume the worst input, and the worst input is two of them at once.
If I'd stopped here I'd be writing the "you don't need human reviewers anymore" post. Eight of nine break-classes from last month's experiment, caught by a machine, for pennies. Then I looked at the three it missed.
The three it missed were the same species
All three survivors were the same kind of bug: a silent data-integrity failure on the unhappy path. The flagship was one you've heard me tell before, because it keeps happening and it keeps being the one that matters.
The author wrote a Stripe webhook handler that acknowledged the event before persisting it. Return 200 to Stripe, then write the row. Works flawlessly in every test. In production, one database blip between the ack and the write = a paying customer with access to nothing and no record they ever paid. Stripe considers it delivered. Your DB never heard about it.
I fed that exact handler to the skeptic, with its adversarial brief, and asked it to find the input that loses a customer money.
It approved it. Confidently. It even praised the "clean early acknowledgement to keep webhook latency low."
Why it missed it — the part that changed how I build
Here's the thing I actually want you to take away, because it's not "the AI wasn't smart enough."
The author and the skeptic are the same model family. Same training distribution, same instincts, same idea of what "clean webhook code" looks like. The author thought ack-before-persist was fine. And when I handed that code to the skeptic, the skeptic shared the exact mental model that thought it was fine. It didn't refute the blind spot. It re-derived it, in a more confident voice, and called it a strength.
That's not a review. That's an echo with better manners.
"Author ≠ reviewer" is necessary. It is not sufficient. If both agents reason from the same prior, the second one isn't a check — it's the first one wearing a lab coat. Point them both at a bug that lives inside their shared blind spot and they will agree, twice, and hand you a green checkmark over a landmine.
Then a human caught it in five minutes
I gave the same handler to a senior engineer. No special prompt, no framing. She read it, went slightly pale, and said: "it acks before it writes — I got paged for exactly this in 2021, it's a nightmare to reconcile."
Five minutes. And notice why. It wasn't that she was smarter than the model, or reasoned more carefully. She'd been burned. She had scar tissue the model doesn't have and can't get, because you can't train the memory of a 2am reconciliation into a context window. The model has read a million descriptions of the dual-write problem. She has lived one. Those are not the same knowledge, and the difference is the entire margin.
That's the missing rung again, by the way — the one I keep coming back to. The reason she could catch it is that early in her career she shipped something like it and paid for it. If AI does all the entry-level work, nobody accumulates that scar tissue, and then nobody can catch the bug that the two agents will keep waving through. The skeptic and the senior aren't interchangeable. One pattern-matches on everything it's read; the other remembers what hurt.
What actually buys independence
So the second agent isn't useless — it caught 38 of 41, and I'd never ship without it now. But the experiment taught me exactly where its ceiling is, and how to raise it. Independence isn't a second seat. It's divergence, and you can buy it in three places:
- A different model family. Different training distribution, different blind spots. The single cheapest way to stop the reviewer from sharing the author's delusion. This one change would have caught the webhook — a model trained on different data doesn't necessarily think ack-before-persist is clean.
- A different frame. Don't ask "is this correct?" (it agrees). Give it a job it can only do by finding the failure: "assume this loses money; produce the transaction that does it." A skeptic hunting a specific failure beats a grader blessing the happy path even from the same weights. This is what got me to 38.
- A human with scar tissue on the merge button. Not to out-code the machine — the machine out-codes the human. To hold the one thing neither agent has: the memory of having been burned by exactly this before.
Stack all three and the blind spot has nowhere to hide, because a bug has to survive a different model, an adversarial objective, and a human who's paid for it before. Any one of those alone, and the landmine ships with a green check on top.
The structure, earned instead of asserted
Last month I asserted this structure. This month I have the log that proves it:
- The thing that writes the code can't be the thing that reviews it.
- The thing that reviews the code can't share the writer's mental model, or it reviews nothing — it re-derives.
- And a human owns the merge, because the one bug that survives two agents is precisely the one that requires having lived it.
That's not a prediction about which model wins. It's a structure, and it holds whether the author is today's model or something we haven't built yet — because it doesn't depend on the model being good. It depends on the model being checked by something that fails differently than it does.
It's also, not coincidentally, exactly how we build xenition: an agent that does the work, a different agent that tries to tear it down, and a person who owns the merge. I didn't arrive at that from a thesis. I arrived at it from 30 days of watching two AIs agree with each other over a bug a human spotted in five minutes.
The two-agent setup caught 38 of 41. The 39th is why there's still a person on the merge button — and why there needs to keep being one who's been burned.
Four questions I'd genuinely like answered in the comments:
- If you run an AI reviewer, is it the same model as your author? Have you ever measured what it catches versus a different family?
- What's your "I got paged for exactly this in 2021" bug — the one you'd catch in five minutes that a model waves through every time?
- Has anyone A/B'd "review this diff" against an adversarial frame ("find the input that loses money") on the same code? I'd love numbers.
- Where does the next reviewer's scar tissue come from, if the entry-level work that used to grow it is the first thing we automated?
Top comments (6)
This is a fascinating experiment and a great reminder that AI capabilities and human expertise are not always measured in the same way. AI can review large amounts of code quickly, identify patterns, and catch many common issues, but human intuition, context understanding, and experience still play a huge role in finding the unexpected problems.
I think the future of software development is not about choosing between humans and AI, but about creating a stronger workflow where AI handles repetitive analysis while humans bring critical thinking and deeper judgment.
Really interesting example of why collaboration between developers and AI will be more valuable than replacement.
Thanks — and I agree with the collaboration framing over replacement. I'd just add one refinement the experiment pushed me toward, because "AI does the repetitive analysis, humans bring the judgment" is almost the lesson but not quite.
The webhook bug wasn't hard because it needed deep human judgment in some general sense. It was hard because the reviewer shared the author's blind spot — same model, same training, same wrong instinct about what "clean" looks like. The human didn't win by being smarter. She won by being different — she'd been burned by that exact bug before, so she failed in a different direction than the model did.
That reframes the division of labor slightly. It's not "humans handle the deep stuff, AI handles the volume." It's that you need at least one reviewer that fails differently than the author — and that can be a human with scar tissue, or just a second model from a different family. The human is the most reliable source of that divergence today, but the principle is about diversity of blind spots, not humans-vs-AI.
So: collaboration, yes — but the thing the human contributes isn't "judgment" in the abstract. It's a different failure mode. That's the part I'd bet stays valuable even as the models get better.
The adversarial brief is the actual finding here. Asking the same model "is this correct?" gets you agreement in a calmer voice; giving a second model "assume this is broken, produce the input that loses a customer money" changes the search, not just the reviewer. 38 of 41 is a strong scoreboard, and the interesting question is the shape of the 3 it missed - if they cluster around context only the human carried (the state nobody designed for, the thing that already exists elsewhere), that tells you exactly where the human read still earns its five minutes.
The adversarial brief is the finding, agreed — "is this correct?" and "produce the input that loses a customer money" are two different searches, and only the second looks under the happy path. That got me most of the 38.
On the shape of the 3, though, the useful part is that they didn't cluster the same way:
Two were exactly what you describe — context only the human carried (the thing that exists elsewhere, the state nobody designed for). Good news: those are fixable without a human. Feed the reviewer a conventions file / the existing-exports list / the schema, and the adversarial frame catches them. Not a hard ceiling — a context-plumbing problem.
The webhook was a different, scarier species. The reviewer had everything — handler, schema, brief. It missed it because it shared the author's prior that ack-before-persist is clean. That's not a fact the model lacked; it's judgment it had backwards, in a direction it can't see. More context can't fix it, because it isn't missing information — it's confidently wrong, and a same-family reviewer is wrong the identical way.
So where the human read earns its five minutes splits too: for the carried-context misses, it's replaceable — plumb the context in and a reviewer earns them instead. For the shared-prior miss, it's replaceable only by divergence — a different model that was never trained to think that's clean, or a human who got paged for it in 2021.
Context solves "the reviewer didn't know." Divergence solves "the reviewer was wrong the same way." The second is the smaller bucket, and the one that loses you the customer.
"Echo with better manners" is a better framing than anything I've seen. I ran into the same thing from the retrieval side: enrich a corpus with LLM-generated descriptions, retrieve with the same model family, and you've just amplified the model's existing prior back at itself rather than gotten a second opinion. The scar tissue distinction is the one worth keeping. She didn't reason to the bug; she'd been paged for it in 2021, and you can't train that into a context window.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.