The bug that doesn't look like a bug
There's a specific kind of bug that has started showing up more often over the last two years, and it doesn't behave like the bugs engineers are used to. It doesn't look sloppy. It doesn't have a typo in it. It reads like someone who knew exactly what they were doing wrote it, and it's still wrong.
That's the part worth taking seriously about AI-written code. Not that it's bad. Most of the time it isn't. The problem is that when it's wrong, it's harder to catch than a human's mistake, because it doesn't come with any of the usual warning signs.
A senior backend engineer who writes under the name Devrim spent a year going back through every production bug his team shipped, specifically comparing AI-written code against human-written code. His conclusion wasn't that AI code had more bugs. It was that it failed differently. Human bugs tend to look uncertain: a missing edge case, a comment saying "not sure this is right," a function that clearly wasn't finished. AI bugs pass review and pass tests just as often as human code does. What's different is where they hide.
Full writeup: I Traced Every Production Bug We Shipped for a Year
That distinction, not "more bugs" but "differently shaped bugs," is the actual argument for why engineering judgment matters more now, not less.
Why the gap exists
AI models don't reason about a system the way an engineer reasons about it. They predict what's statistically likely to come next based on everything they've seen before. Most of the time that's close enough to real understanding, because most code follows patterns that have already been written thousands of times.
Real engineering judgment is knowing when this specific case is the exception, when the textbook answer breaks because of a constraint nobody documented anywhere a model could have seen it. AI has no way to know what it doesn't know. It keeps generating with the same confidence either way.
A developer named Alex Carter wrote about what happened when he shipped AI-generated code under deadline pressure without giving it a proper review. The interesting part of his account isn't the bug he already suspected. It's what he found once he went back and reviewed the rest of it properly: a database query that worked fine in every normal case and was quietly vulnerable to injection under a specific set of inputs nobody had tested. His point was that a human who doesn't understand something usually leaves a trace of that uncertainty. AI just implements the wrong thing with total confidence, and the confidence is what makes it hard to catch.
Full writeup: I Shipped AI-Generated Code Without Reviewing It. Here's What Broke.
What happened to curl
curl is a small piece of software that runs quietly inside billions of devices: cars, TVs, phones, a large share of the servers running the internet. For six years it ran a bug bounty program through HackerOne, and it worked well. Over that time it paid out more than $100,000 and confirmed 87 real vulnerabilities, with a confirmed-vulnerability rate north of 15%.
In 2025, that rate collapsed to below 5%. Not because researchers got worse at finding bugs, but because AI made it trivially cheap to produce reports that sound like real vulnerabilities: technical language, specific function names, plausible-sounding attack scenarios, and none of it real.
By January 2026, the project's maintainer Daniel Stenberg described a single 16-hour window where curl received seven submissions. His team spent real hours on each one: reading it, trying to reproduce the claimed exploit, tracing the referenced code paths. The result was zero actual vulnerabilities. He shut the whole program down. His own line at the time was that the goal was to remove the incentive for people to submit reports that weren't properly researched, "AI generated or not."
- Coverage of the shutdown: Curl shutters bug bounty program to stop AI slop
- Fuller account of the numbers: AI slop vulnerability reports drowned curl's security team
That last phrase from Stenberg is the whole argument, honestly. It was never really about whether AI wrote the report. It was about confident-sounding output with nobody checking whether the confidence was earned.
What happened at Amazon
The same pattern shows up at a much larger scale inside Amazon. Between December 2025 and March 2026, the company had at least four Sev-1 production incidents, the most severe internal classification, including one six-hour outage tied to an estimated 6.3 million lost orders.
Internal documents reportedly grouped these under the label "Gen-AI assisted changes" and flagged them as high blast radius, which led Amazon to require additional senior engineer review specifically for AI-assisted production changes going forward.
Amazon has since pushed back on some of the framing. A Fortune report noted the company's position that only one of the incidents involved AI tooling directly, and that the root cause there was an engineer acting on bad advice an AI had inferred from an outdated internal wiki page, not faulty generated code itself.
Either version of the story lands in the same place: serious enough that one of the largest engineering organizations on earth changed its approval process because of it.
- Detail on the incidents: Amazon Vibe Coding Failures: 4 Sev-1s in 90 Days
- Broader roundup of the data: AI-Generated Code Incidents: What the 2026 Data Shows
What the numbers say
The scale behind all of this is larger than most people assume. A 2026 survey covering more than 1,100 professional developers found AI now accounts for roughly 42% of all committed code, with developers expecting that to reach around 65% by 2027.
A separate benchmark from CodeRabbit, covered by The Register, found AI-generated code was 2.74 times more likely to introduce cross-site scripting vulnerabilities and 1.88 times more likely to mishandle passwords compared to human-written code. Cross-site-scripting prevention was the single worst category in that benchmark, with an 86% failure rate, and that number hasn't moved meaningfully even as the underlying models have gotten better, holding around a 55% overall pass rate since 2023.
Summary of that data: Vibe Coding Goes Wrong: What AI-Generated Code Actually Breaks in Production
The study that measures perception, not code
There's a controlled study from METR worth knowing about, because it doesn't measure code quality at all. It measures perception.
Experienced open-source developers were given AI coding tools and asked to work on real issues in their own repositories. Before starting, they expected to be about 24% faster with AI assistance. Measured afterward, they were actually 19% slower. Even after finishing the tasks, those same developers still believed AI had made them roughly 20% faster: a nearly 40-point gap between what they felt and what the clock actually recorded.
Researchers call this the effort heuristic. Typing less feels like working less, and working less feels like working faster, whether or not it actually is. This matters because it means a developer's own sense of whether AI is helping them is one of the least reliable instruments available.
Google's own DORA research found something adjacent to this at the team level: AI adoption correlates with close to a 10% increase in code instability. Teams generating more code with AI were, on average, deploying less stable software. More output, proportionally more failure surface.
One engineer's own account of chasing this down in her own codebase: Why My AI-Generated Code Kept Breaking in Production
What this doesn't mean
None of this is an argument against using AI to write code. It's genuinely good at what it's good at: boilerplate, a first draft of something you already know how to evaluate, syntax you've forgotten, turning a clear spec into working code faster than typing it yourself. Every engineer cited in this article is still using the tools daily, including the ones writing detailed postmortems about what broke.
What's changed is what the job actually is now. For most of the history of this profession, the scarce resource was producing working code. If you could do that quickly and cleanly, you were valuable almost regardless of what you were building. That scarcity is gone, structurally, for the first time.
What hasn't gotten any cheaper, what's arguably gotten more expensive, is the judgment to know whether the code that got produced is the right code. Whether it accounts for the constraint nobody wrote down anywhere a model could read it. Whether the well-commented, syntactically clean function that passed every test is quietly wrong in a way that won't show up until it costs something.
That judgment doesn't come from watching AI generate code faster than you could type it. It comes the way it's always come: from shipping something, watching it fail, tracing the failure back through the system until you actually understand why, and building the instinct that lets you catch the next one before it ships. AI can't do that part for you. It can only skip you past it, and skipping past it isn't the same as having it.
If you're early in your career
If you're worried AI has made your skills irrelevant before you finished building them, the honest answer is that the skill itself hasn't changed. The path to it has.
Read what the AI hands you the way you'd review a fast, talented, occasionally reckless coworker's pull request. Ask it why it made the choice it made. Try to break it on purpose before something else does. That friction is where judgment actually gets built, and there's no shortcut that skips it without costing you later.
What's the closest call you've had: AI-written code that looked completely fine until it wasn't? I'd like to hear the specific bug, not just the general feeling. If you've got a postmortem or a writeup of your own, drop the link. This list is worth building out.
Top comments (1)
The review bottleneck is really an observability bottleneck. For AI-assisted changes, I would require the diff to state the assumed invariant and the evidence that exercises it; reviewers can then challenge a concrete claim rather than reverse-engineering intent from polished code. That also creates a useful post-incident trail when a normal path was correct but the system constraint was missing.