We know AI builds fast. We also know that things happen inside the AI black box that no one plans or intends. We set out to open the box and look at what AI produced---not in a demo, not in a blog post about how great Cursor is---the real code in real repositories that real teams are shipping.
We examined vibe-coded GitHub repositories identified by a "fingerprint" (projects built primarily with AI coding assistance) across a range of frameworks, languages, and team sizes to find patterns, failure modes, and recurring structural problems we hypothesized were there.
We expected some variation. Different tools, different teams, different codebases. Instead, we got the same handful of patterns, over and over, regardless of any of that.
Recurring patterns
| Pattern | Prevalence |
|---|---|
| Compiled build artifacts committed to source control | Extremely common |
Untyped or any-typed parameters erasing type safety |
Pervasive |
| God-object files exceeding 1,500 lines | Common |
| Silent fallback chains masking errors | Pervasive |
| Hardcoded design values instead of token references | Nearly universal |
Security violations (eval, innerHTML, hardcoded secrets) |
Frequent |
None of these are exotic. They're boring, structural, and completely invisible in a demo. The code compiled. The tests passed (when there were tests at all). The linters were clean. And underneath all of that, errors were being swallowed silently, missing data was getting papered over with defaults, and the codebase was quietly rotting from the inside.
That's the part that makes this worth writing about: no one would guess any of this was happening. Nobody wants to look.
A closer look at two patterns
Silent failures. This is the most dangerous pattern we found, and also the most common. An AI assistant writes a try/catch, the catch block logs the error (or doesn't), and execution continues as if nothing happened. The app looks like it's working. The user sees no error. But the data is wrong, an operation got skipped, and the system is now in a state nobody accounted for.
The fix isn't complicated; it's just a habit AI doesn't have by default: every error needs to be visible in three places at once: the console, the UI, and as a thrown exception that actually halts execution. If an error only shows up in a log nobody's watching, it isn't handled; it's hidden.
Phantom correctness. This is the quieter cousin of the above. Code that compiles, passes the linter, and operates on the wrong data or a fabricated value without anyone noticing. Two of the biggest contributors: any types used to make a type error go away instead of actually resolving it, and hardcoded values standing in for something that should have come from a real data source. Both look completely fine in a diff. Neither one throws an error. They just quietly mean the code isn't doing what it looks like it's doing.
Why this keeps happening
None of this is really an intelligence problem. LLMs are very good at producing code that looks correct. That's what they're optimized to do. The gap is that "looks correct" and "is correct" are different claims, and nothing in the default generation loop checks the second one. An AI assistant will also frequently declare a task done once the obvious, cheaper, high-priority, "big win" issues are resolved, leaving warnings and info-level findings "for later," which in practice means never, because nothing forces a return trip.
Where this runs out
Guardrails (rules, skills, etc.) are genuinely useful, and they work; but they all act at one specific moment: when the code is being written. They're authoring-time discipline.
There's a second category of problem that shows up only after the code executes: the happy-path trap in async code (what happens on the second concurrent click, not the first), library lock-in from importing a vendor SDK directly across dozens of files, tests that assert implementation details instead of actual behavior and break on every refactor. None of these show up in the demo. They show up in production, or six months later, when someone who wasn't there when the AI wrote it has to touch the code.
And even with perfect authoring-time discipline, there's a problem no amount of prompt engineering solves: LLMs are probabilistic, so the code they generate will drift from design intent over time regardless;
- Renaming a prop in a way that's technically valid but breaks a contract
- Approximating a spacing value instead of reading it from a token
- Restructuring a component in a way that works but no longer matches the design system.
Clean code that no longer matches the design is still drift. That's a structural problem, not a discipline one, and it's a separate problem from anything a rules file can catch.
What we did about it
We turned these findings into a governance framework: six principles and a set of forbidden/required controls organized by the class of problem they prevent, not by technology. It's built to drop directly into whatever you're already using: a system prompt layer, per-tool rules for Cursor, Copilot, or Claude Code, and a set of on-demand skills for the failure classes that need more than a one-line rule.
It's free — no signup required, just an email so the pdf and zip file land in your inbox. If you're shipping a meaningful amount of AI-generated code, it's worth fifteen minutes to see how many of these patterns are already sitting in your own repo. And how you prevent future drift.
We also have other expansion packs in the pipeline. If we have your contact info, we can send them to you.
If you've run into other failure patterns we didn't cover here, I'd genuinely like to hear about them. Ping me on X or LinkedIn.
Top comments (2)
The recurring categories are believable, especially swallowed errors and fabricated fallback values. The next useful step would be publishing enough methodology to distinguish a measured pattern from an anecdotal one.
I'd want to see:
Some categories also need semantic confirmation.
innerHTMLis not automatically exploitable without an attacker-controlled source and an unsafe sink; committed build artifacts can be intentional; a secret-shaped string is different from a live credential. Reporting those separately would make the security numbers much more actionable.A small public, anonymized fixture set would help too: one true positive and one lookalike negative for each pattern, with the expected classification. That would let others reproduce the claims and test whether their own checks are detecting the same thing.
Most of this I can answer, @peterbuildssecure.
The set: GitHub repos with commits attributed to a coding agent, pulled from GH Archive between Jan 2024 and Oct 2025, filtered to 100+ stars and Python/JS/TS. The JS/TS half is what I scanned — 2,855 repos, ~204M lines, split TypeScript 2,252 / JavaScript 603, and about 88% of them carry React or JSX. Heavily Copilot (1,782 repos), then Claude, Gemini, Cursor, Devin.
Forks, archived repos and templates were recorded but not excluded. The 100-star floor drops most forks incidentally, which isn't the same as deduplicating, and repos that read as templates or starters are about 1% by name — a name match, not a real check. Build output and minified files are skipped at scan time.
The prevalence words are doing two different jobs — how many repos have the thing, and how hard it hits the ones that do:
any/ unsafe type assertions"Pervasive" is the top two by reach. "Nearly universal" is the top by density — a repo that hardcodes design values does it ~735 times. "Frequent" is half the set, single digits each. "Common" is one in three: 986 repos, largest file 26,995 lines. Build artifacts is the row the scan doesn't cover, since it skips generated files — that one's from reading repos.
Two limits. The fingerprint is an inclusion filter, not a classifier — it means "this repo has AI-attributed commits," not "an AI wrote most of it," so there's no accuracy number to report on it; where I can measure the share, most repos are well under 1%. And I don't have a false-positive rate across the whole set, only hand-checked samples on the noisiest individual rules. Not going to pretend otherwise.
You're right on the security semantics: secret-shaped strings aren't verified credentials, and eval isn't exploitable without a bad source. We do separate unvalidated innerHTML sinks from plain usage; the other two we don't.
The fixture set is the best idea in your reply — I have the positive cases, the lookalike negatives are the gap. Worth publishing both.
And you're right to be suspicious of a prevalence table: it can't separate "an AI wrote this" from "this is a big JS/TS repo." Dose can. As the share of AI-attributed commits rises, drift rises with it, and reliability problems roughly 2.3×. On a smaller hand-labelled set, lines carrying at least one finding go 1 in 23 human-led, 1 in 16 AI-assisted, 1 in 10 AI-led. Small numbers, directional.
That's the next post ;)