Last month I let AI write 100% of my code for 30 days. The single loudest lesson wasn't "AI is amazing" or "AI is useless." It was one sentence: th...
For further actions, you may consider blocking this person and/or reporting abuse
You asked question 3 for numbers and I can't give you that one honestly - a review-frame A/B across model families needs model access I don't have here, so I won't gesture at it. What I can do is measure the scoreboard, because that one is arithmetic and I ran it. Three things about 38/41.
The 41 is a count of what somebody found, so 92.7% is a ceiling, not a recall. The denominator came from the same process being measured: you found the 41 by reviewing, so any bug nobody found was never eligible to be missed. Every row below is consistent with your log, and nothing in it chooses a row:
Your design actually forbids measuring it, and that is fixable. The human read exactly the skeptic's leftovers ("every issue it missed that I caught in my own final read"), so the two reviewers' catch sets are complements by construction - the overlap is 0. Plug your numbers into the standard two-source abundance estimate and you get 155 bugs, i.e. 24.5% recall:
Chapman = (38+1)(3+1)/(0+1) - 1 = 155. At zero overlap the estimator's variance is unbounded, which is the formal way of saying the log puts no upper bound at all on the misses - not "a small number", none. Two defensible-sounding numbers, 92.7% and 24.5%, from one log.The fix is to randomise who reads what. Give the human a random share of all the diffs rather than the skeptic's rejects, and the overlap becomes measurable. Simulating that against your published structure (60 true bugs, 15% in the shared-blind-spot class, skeptic catches 95% of everything except that class, human catches 60%/90%):
The truth is 60, so this recovers it and puts a real interval around it - which is the number you'd want next to "38 of 41". Note the last column: at a small sample the estimate fails outright rather than degrading politely.
And report it per class, because your headline hides the finding. Your own three survivors are all one class, so the per-class table is 38/38 ordinary and 0/3 for the class the post is about. Same reviewer quality, different bug mixes, aggregate anywhere from 26.8% to 92.7%:
So the aggregate measures the mix, not the reviewer - which matters for your questions 1 and 2, because a single number can't travel between codebases. The class-level numbers do.
One consequence for the article's own claim: "a different model family would have caught the webhook" is plausible but the log can't show it, since the class is 0/3 for both reviewers you ran. The honest version of the finding is stronger anyway - you have a measured 0% recall in the class where the money is, which is why the human keeps the merge button. That number you earned. What you don't yet have is the ceiling on how many more of them there are, and one randomised read is what buys it.
This is the comment I hoped the post would get and half-feared it wouldn't. You didn't argue with the number — you audited how it was manufactured, and you're right on every load-bearing point.
Concession first, because you earned it: 92.7% is a ceiling, not a recall. The denominator is contaminated — the 41 came out of the same reviewing process I was grading, so a bug nobody found was never eligible to be counted as a miss. "Found by the process being measured" is exactly the trap, and I walked straight into it.
The part I hadn't seen until you spelled it out is the zero-overlap-by-construction. The human read the skeptic's rejects, not a fresh sample — "every issue it missed that I caught." So the two catch sets are complements by design, overlap is structurally 0, and any recapture estimate has unbounded variance. My log doesn't say "few misses." It says "no upper bound." That's a design flaw, not a data point, and it's mine.
The fix is almost embarrassingly cheap and I'm running it next month: randomize the read. Give the human a random share of all diffs instead of the skeptic's leftovers, and the overlap stops being 0-by-fiat and becomes measurable. Your simulation recovering 60 with a real interval is the experiment I should have designed the first time.
But the line I'm actually going to steal is the per-class one. 38/38 on ordinary bugs, 0/3 on the blind-spot class — and the aggregate just measures the mix, not the reviewer. That's not a footnote to the post; it's a cleaner statement of the whole thesis. A blended number can't travel between codebases because it silently reports your bug distribution. The class number is the one that generalizes. I led with the wrong figure.
And you're right to hold my feet to the fire on "a different family would have caught the webhook." The log can't show that — the class is 0/3 for both reviewers I ran, so it's a hypothesis wearing a conclusion's clothes. The honest version is stronger anyway, exactly as you say: a measured 0% recall in the class where the money lives. That's why the human keeps the merge. That number I earned. The ceiling on how many more landmines are down there, I didn't — and one randomized read buys it.
Genuinely, thank you. This moved the finding from "a number I asserted" to "a number I understand the shape of."
I took the randomized read seriously enough to simulate it against your published structure (60 true bugs per 100 diffs, 15% in the blind class, skeptic at ~95% ordinary / ~0 blind, human at 60% ordinary / 90% blind), because "a random share" leaves the one number that matters open: how big the share has to be. Two results, one for the aggregate and one against the class.
The aggregate works, at ~20%, and the estimator is easy to get wrong. With a random read the estimate is
N_hat = s*h/m:s= the skeptic's catches,h= the human's catches inside the read window,m= the overlap. The trap isn2: if you plug in the number of diffs read instead of the human's catches, a full read returns exactly your diff count — 100 in my run — which is a perfectly plausible number with no content in it.So the budget line is not "a random share" but "about a fifth of the diffs": at 10% the estimator stops failing outright and starts lying politely — a nominal 95% interval that misses the truth a third of the time — and somewhere between 10% and 20% the overlap stops disappearing. Both of those are the same mechanism, and they are what your five-minute worst case (m = 0) looks like from the inside: not an outlier, the low end of a small-sample estimator.
What randomization does not buy is the class, and this one can't be fixed by sampling. Run the class-level version of the same estimator at any fraction and the class overlap is zero in 100% of runs — including reading every single diff. The reason is different from the one you fixed: the skeptic's catch probability in that class is ~0, so there is no second set for a random read to overlap with. Your plan replaces "overlap is zero by the design of who reads what" with "overlap is zero because one of the two readers never enters that set". Better sampling frame, same structural zero, so the ceiling on how many more landmines are down there is still unbounded for the class where the money lives.
What the randomized read does give you for the class is a plain one-reader density estimate,
b*D/k(b= class bugs you caught in the window,D= all diffs,k= diffs read): at 10% read you see a class bug in 59% of runs and the estimate is 10 (90% interval 10–30); 20% → 84% of runs, 10 (5–20); 30% → 95%, 7 (3–17); 50% → 100%, 8 (4–12); full read → 8 (6–9) against a truth of 9. Note the direction — a thin read cannot produce a density below one item's worth, so it overstates, and it only tightens when you read half the set. If the class is the finding, the class is where the reading budget goes: sample the aggregate thinly, stratify the rest toward the diff types that carry that class.And if you want a ceiling for the class rather than a point estimate, the second reader has to be able to fail inside it — different family, or the scar tissue. Otherwise the nicer sampling frame just re-creates the structural zero. That is your own thesis one level down: the reviewer of the class cannot be the reviewer that shares its blind spot.
One number in your concession, since you said you'd write it down. "A measured 0% recall in the class where the money lives" — 0 of 3 is a point estimate of 0 with a 95% upper bound of 63% (rule of three,
1 - 0.05^(1/3)); at 0 of 10 it is 26%, at 0 of 20 it is 14%. So what the log earned is "0 of 3", and the merge-button conclusion does not need more than that: the machine approved the handler confidently, and the class exists. It is the recall figure that is unmeasured above a very high ceiling, and ten more class items is the cheapest way to lower it.If you want the simulation run against your real month rather than my assumed structure, the two inputs are the total diffs and how many the human actually caught — the estimator is a two-liner and the class table above is where it usually gets interesting.
The adversarial brief looks like the real lesson. Two agents agreeing with each other is not much stronger than one agent agreeing with itself if both are optimizing for the same framing. A reviewer gets more useful when its job is to break a specific assumption, invariant, or failure mode.
Yeah — the frame is the lever, and I think it's because a frame changes the objective, not just the second opinion. "Is this correct?" is answered by pattern-matching against what correct code usually looks like, which is exactly the author's own prior. "Produce the transaction that loses money" can only be answered by actually finding one. One rewards agreement; the other rewards a counterexample. That gap is the whole reason the second agent earns its tokens.
The one caveat I'd add — and it's the bruise in the post — is that the frame can only break assumptions the model is able to hold. Point an adversarial brief at a failure mode that lives inside the model's blind spot and it'll hunt hard and still come back empty, because it doesn't believe that thing is a failure. That's what happened with the webhook: great frame, wrong prior. "Assume this loses money" didn't help when the model was convinced ack-before-persist was clean.
So I've landed on: the frame is what makes a same-family reviewer worth running at all, but it can't reach past the shared distribution. For that you need a different failure surface entirely — a different model, or a human who's been paged. Frame gets you the 38. It structurally can't get you the 3.
Yep, I think that's exactly it. A second opinion only helps if it's actually looking for something different. If both miss the same thing, they can agree all day and still be wrong. That's where I still want a person involved when the downside matters.
"Agree all day and still be wrong" — that's the whole thing. Two reviewers only count as two if they fail differently; same model family is just the first opinion twice in a calmer voice. So the real question isn't "do we have a reviewer" but "does the reviewer fail the same way the author does?" Curious where you draw the "downside matters" line for pulling in a person — dollars, auth, data loss, or gut call?
the ack-before-persist example is the sharpest part of this. that pattern isn't some rare mistake, it's basically the reference implementation in half the stripe webhook tutorials out there. two labs training on the same public code could both call it clean, not because they share weights but because they share source material. did you actually test a model from a different lab on this exact bug, or is that still a hunch? curious if divergence needs to be forced or if two labs already disagree here without you doing anything extra.
You put your finger on the exact thing I can't yet back up, so straight answer: it's a hunch. I did not run this handler through a different lab's model during the logging month. The class is 0/3 for both reviewers I actually ran, which means "a different family catches it" is a hypothesis I dressed up as a conclusion. Guilty.
But the mechanism you're proposing is better than mine, and it's the part that worries me. I was implicitly assuming the blind spot comes from shared weights. You're pointing out it can come from shared source material — and that's worse, because it doesn't require the same lab at all. If half the Stripe webhook tutorials on the public internet ack before they persist, then every model trained on that internet inherits the same "clean" prior independently. Different weights, different labs, same landmine — because they all read the same teacher. That would mean a different family is not automatically a different failure surface. Divergence might have to be forced (adversarial frame, property-based tests, an actual spec) rather than assumed to come free with a new logo.
So I owe the experiment, not just the claim. Next round I'm running this exact handler through models from two different labs, cold, no adversarial framing, and logging whether either one flinches at the ack-before-persist. If they both bless it, your shared-source-material theory is the real story and "just use a different family" is weaker advice than I gave. I'll post the result either way — this is too good a question to leave as a hunch.
This is a fascinating experiment and a great reminder that AI capabilities and human expertise are not always measured in the same way. AI can review large amounts of code quickly, identify patterns, and catch many common issues, but human intuition, context understanding, and experience still play a huge role in finding the unexpected problems.
I think the future of software development is not about choosing between humans and AI, but about creating a stronger workflow where AI handles repetitive analysis while humans bring critical thinking and deeper judgment.
Really interesting example of why collaboration between developers and AI will be more valuable than replacement.
Thanks — and I agree with the collaboration framing over replacement. I'd just add one refinement the experiment pushed me toward, because "AI does the repetitive analysis, humans bring the judgment" is almost the lesson but not quite.
The webhook bug wasn't hard because it needed deep human judgment in some general sense. It was hard because the reviewer shared the author's blind spot — same model, same training, same wrong instinct about what "clean" looks like. The human didn't win by being smarter. She won by being different — she'd been burned by that exact bug before, so she failed in a different direction than the model did.
That reframes the division of labor slightly. It's not "humans handle the deep stuff, AI handles the volume." It's that you need at least one reviewer that fails differently than the author — and that can be a human with scar tissue, or just a second model from a different family. The human is the most reliable source of that divergence today, but the principle is about diversity of blind spots, not humans-vs-AI.
So: collaboration, yes — but the thing the human contributes isn't "judgment" in the abstract. It's a different failure mode. That's the part I'd bet stays valuable even as the models get better.
"Echo with better manners" is a better framing than anything I've seen. I ran into the same thing from the retrieval side: enrich a corpus with LLM-generated descriptions, retrieve with the same model family, and you've just amplified the model's existing prior back at itself rather than gotten a second opinion. The scar tissue distinction is the one worth keeping. She didn't reason to the bug; she'd been paged for it in 2021, and you can't train that into a context window.
The RAG version is the same disease with different symptoms, and I hadn't connected them until you said it. Generate the descriptions with one family, retrieve with the same family, and the retriever doesn't find what's relevant — it finds what's phrased the way that family phrases things. You've built a closed loop that scores its own vocabulary as truth. Same structure as author-reviewing-author: the second stage shares the first stage's prior, so it can only confirm, never contradict. Independence isn't a second pass; it's a different failure surface. Nice to see it show up on your side of the stack too — makes me think it's a property of the loop, not of code.
And yeah, the scar tissue line is the one I'd defend hardest. She didn't out-reason the model — the model has read more about the dual-write problem than she ever will. She'd paid for it once. The model has the description; she has the memory of the 2am reconciliation, and only one of those flinches when it sees ack-before-persist. You can pour a million tokens of "dual writes are dangerous" into the context and it still won't develop the flinch. That gap is the entire margin, and it's exactly why the entry-level-work question keeps me up.
The adversarial brief is the actual finding here. Asking the same model "is this correct?" gets you agreement in a calmer voice; giving a second model "assume this is broken, produce the input that loses a customer money" changes the search, not just the reviewer. 38 of 41 is a strong scoreboard, and the interesting question is the shape of the 3 it missed - if they cluster around context only the human carried (the state nobody designed for, the thing that already exists elsewhere), that tells you exactly where the human read still earns its five minutes.
The adversarial brief is the finding, agreed — "is this correct?" and "produce the input that loses a customer money" are two different searches, and only the second looks under the happy path. That got me most of the 38.
On the shape of the 3, though, the useful part is that they didn't cluster the same way:
Two were exactly what you describe — context only the human carried (the thing that exists elsewhere, the state nobody designed for). Good news: those are fixable without a human. Feed the reviewer a conventions file / the existing-exports list / the schema, and the adversarial frame catches them. Not a hard ceiling — a context-plumbing problem.
The webhook was a different, scarier species. The reviewer had everything — handler, schema, brief. It missed it because it shared the author's prior that ack-before-persist is clean. That's not a fact the model lacked; it's judgment it had backwards, in a direction it can't see. More context can't fix it, because it isn't missing information — it's confidently wrong, and a same-family reviewer is wrong the identical way.
So where the human read earns its five minutes splits too: for the carried-context misses, it's replaceable — plumb the context in and a reviewer earns them instead. For the shared-prior miss, it's replaceable only by divergence — a different model that was never trained to think that's clean, or a human who got paged for it in 2021.
Context solves "the reviewer didn't know." Divergence solves "the reviewer was wrong the same way." The second is the smaller bucket, and the one that loses you the customer.
A human caught the bug in five minutes because they had context the AIs did not. That is why I always keep a human in the loop for the final audit before any commit ships.
Exactly — and I'd push it one step further: it's not just keeping a human in the loop, it's which human and where. Put a human on every line and they start rubber-stamping (the same 6pm fatigue that makes any reviewer miss things). The leverage is a human who's actually been burned by that class of bug, sitting on the merge button — the last gate, not a second pair of eyes on everything.
Because the failures that survive two agents are precisely the ones that need lived memory, not more reading. Genuinely curious about your setup: on that final audit, is it whoever's free, or do you deliberately route the risky changes — payments, auth, anything touching money or state — to whoever's been paged for that exact thing before?