TL;DR: We didn't set out to run an experiment — AI agents just kept eating more of our sprint over six months until they were writing roughly half...
For further actions, you may consider blocking this person and/or reporting abuse
"Clean naming, consistent style, plausible logic triggers the same 'this looks fine' pattern-match reviewers use for competent human code." That's the sharpest description I've seen of the exact trap, and it's the one I spend most of my time defending against as a non-engineer who builds with AI. Broken code I can catch — it errors. The fluent-but-wrong version sails past me precisely because it looks like the competent code I was hoping for.
Your fix being "rebuild how we review, not whether we accept" is the part I wish more people led with. I landed on the low-budget version of your agent-percentage labels: any change to auth, or anything that touches real patient-facing behavior in my tools, doesn't get to close on "it looks right" — it has to close on a check that ran and passed. The reviewer's "this looks fine" and the system's "this actually works" come from different places, and AI-written code is exactly where those two quietly diverge.
"Ask them what broke, not what got faster" is going in my notebook as both a hiring filter and a self-audit. Fluency is what got faster; the whole discipline is refusing to let fluency stand in for correctness. Great retrospective — the kind that's honest about the wins and still names the wreckage.
Really glad the "check that ran and passed" framing landed — that's exactly the line I'd draw too, and I think you've actually generalized it better than I did. I was mostly talking about auth in my own retrospective; you're right that it applies to anything touching real user-facing behavior, patient data being the sharpest version of that.
One thing I'd add from the other side of the table: this gets harder, not easier, once you're working with an outsourced or dedicated team instead of writing the code yourself. When the person reviewing isn't the person who prompted the AI, "this looks fine" gets a second layer of trust stacked on it — the reviewer isn't just trusting the model, they're trusting that whoever prompted it caught the obvious stuff first. That's usually where the fluent-but-wrong code actually gets through, not at the first pass.
The fix ends up being the same one you landed on, just enforced contractually instead of personally: no merge on sensitive paths without a named check that ran, full stop, regardless of who — human or AI — wrote the diff. Doesn't matter how good the vendor's reputation is; "trust me" and "here's the test output" can't be treated as equivalent inputs to a merge decision.
Appreciate you sitting with the piece long enough to push back on it usefully — that's rarer than the praise.
The double-stacked trust is the part I hadn't seen, and it's the same bug I've been chasing all week, wearing an org chart. When the reviewer isn't the prompter, "this looks fine" quietly inherits a second assumption — "and whoever prompted it caught the obvious stuff" — which is just a self-report from the prompter that the reviewer is now grading as if it were evidence. That's the exact shape of a monitor reading a service's own "I'm healthy": the auditor stands one layer outside and still ends up trusting a report the audited party wrote. You've found the human-org version of it, and you're right that it's where the fluent-wrong code actually lands — not the first pass, the handoff.
And "believe me" and "here's the test output" not being equivalent inputs to a merge is the whole thing in one line. They feel interchangeable in the moment because both arrive as confidence — but one is a private variable that dies with the person holding it, and the other is an artifact anyone downstream can re-run. A merge that treats them as equal is pricing a promise at the same rate as a receipt. Contractual enforcement is just the honest admission that personal enforcement doesn't survive a handoff to someone who owes your caution nothing.
The clinical mirror is almost too on-the-nose: this is the shift-change problem. "Patient's fine" from the outgoing nurse is believe-me; the actual vitals on the chart are the test output. Hospitals learned the hard way that verbal handoffs lose people, so they force structured ones — the value, not the verdict, crosses the boundary. Same fix, same reason: the boundary between two people is exactly where confidence stops being transferable and only an artifact makes it across. Thanks for pushing the piece into the place it actually gets hard — the code was never the risk, the handoff was.
The "trust me vs. here's the test output" line is worth stealing outright — that's the cleanest way I've seen the actual failure mode stated. It also points at something worth naming explicitly: on a dedicated team, that check needs to be visible to you, not just to the vendor's internal QA. We've seen contracts where "tests run before merge" is technically true, but the client never sees the output — they're trusting the vendor's internal process the same way the vendor is trusting the model. The fix isn't just "require a named check," it's making sure the check's output is part of what gets handed to the client, not just part of the vendor's internal gate.
Practically that's meant asking clients to request CI output or coverage reports on sensitive paths as a standing deliverable, not a one-off audit — the same way you wouldn't accept "we reviewed it" from a contractor without the actual review artifact. Cheap to ask for, and it collapses that second layer of trust you're describing back down to one.
"The check needs to be visible to you, not just to the vendor's internal QA" is the upgrade, because it catches the version I'd have missed: a check can run, pass, and still be a self-report if its output never crosses the boundary to you. "Tests run before merge" with nothing shipped is just "trust me" wearing a process diagram — you're not seeing the artifact, you're trusting that the artifact exists. Same failure, one level more sophisticated: the receipt was written and then kept in the drawer of the party being audited.
The standing-deliverable-vs-one-off distinction is the part that does the most work, and it lines up with something I keep hitting from the reliability side. A one-off audit only proves behavior at audit time — the vendor performs when watched and relaxes when not, same as anyone. A standing deliverable removes the "when watched" entirely: if the CI output ships with every sensitive change, there's no moment the check isn't observed, so there's no gap left between "performs under audit" and "performs." You've quietly converted an audit into a property of the system instead of an event that happens to it.
The clinical mirror is the medical record. "We reviewed it" is the drawer; the chart is the standing deliverable — the value is recorded every time, and the external regulator's power is precisely that they can pull any chart, not that they scheduled a visit. Hospitals didn't get safer by auditing harder; they made the artifact mandatory and continuous, so the auditor became the format, not a person who shows up. Requesting CI output as a standing deliverable is the same move: cheap to ask for, and it turns "did you check" from a question you have to remember to ask into a thing that's just always on the table. Collapsing two layers of trust to one by making the artifact travel — that's the whole discipline in a contract clause.
"The receipt kept in the drawer of the party being audited" — that's the whole vulnerability in one image, and it's sharper than anything I'd have gotten to. The medical-chart analogy earns its keep too: the auditor becoming the format rather than a visitor is the actual mechanism, not just a nice metaphor. One thing worth guarding against once you make the artifact mandatory — it can start getting gamed for shape rather than substance, the way charts sometimes get filled out to satisfy the audit rather than to reflect what happened. Making the check a standing deliverable removes the "were they watched" question, but it opens a new one: is anybody still reading it, or has it become a box that gets checked because the box exists?