Empirical research from ReWeaver AI. 42 identical prompts, across 5 tools and 8 production dimensions, compared to human baseline. One metric that changes how you see drift.
Everyone knows AI-generated code has quality issues. What’s less understood is that the way most teams measure those issues — by how often they occur — systematically understates the risk.
We ran a controlled study to find out how badly. The answer surprised us, particularly in one dimension.
What We Did
We gave five leading AI coding tools (Cursor, Claude Code, Lovable, Figma Make, and VS Code with Copilot) 42 identical prompts: realistic single-component builds — buttons, forms, dashboards, navs, modals, auth surfaces. We scanned every output with ReWeaver, our deterministic drift-detection engine, across eight production readiness dimensions:
User Experience
Security & Privacy
Accessibility
Design Consistency
Reliability
Maintainability
Architecture
Testability
We also scanned six human-authored open-source repositories as a reference baseline.
For each dimension, we calculated two things:
Drift frequency — the percentage of lines containing at least one drift occurrence. Counts what went wrong.
Production Drift Ratio (PDR). The PDR is a metric that weights frequency by estimated remediation cost on a 0–1 scale. A PDR of 0.30 is roughly 45 minutes of cleanup per component; 0.70 is about 2.5 hours.
The Finding That Stopped Us
In Security & Privacy, AI tools produced 3× the human drift frequency. That looks manageable — a meaningful gap, but not alarming.
The PDR was 22× the human reference.
Not 22% more. 22 times more costly to fix.
The frequency gap makes Security & Privacy drift look like a minor concern. The PDR reveals it’s the most expensive problem in the dataset. AI-generated security drift (client-side authorization gates bypassable in DevTools, raw PII and credentials passed through props without tokenization) is syntactically identical to safe code. It passes review, but the fixes are harder to find and remedy.
This is the core argument of our study: frequency counts what went wrong. The PDR quantifies what it will cost to fix it.
The Full Results
Here’s how much more relative drift frequency and severity AI produced across all eight dimensions:
| Dimension | Frequency multiplier | PDR multiplier |
|---|---|---|
| Security & Privacy | 3.4× | 22× |
| User Experience | 4.5× | 6.5× |
| Accessibility | 1.7× | 5.2× |
| Design Consistency | 1.7× | 4.1× |
| Reliability | 1.4× | 2.5× |
| Testability * | 0.61× | 2.0× |
| Architecture * | 0.55× | 1.7× |
| Maintainability * | 0.52× | 1.5× |
* These three dimensions showed lower AI frequency than the human reference, likely due to a corpus maturity effect, not an AI advantage. Our human reference draws from mature production repositories carrying accumulated technical debt; the AI corpus is fresh greenfield components. The PDR gap remains positive even here.
In every dimension, AI-generated drift is more expensive to remediate than human-authored drift, even where humans produce more of it.
The Statistics
We ran Wilcoxon signed-rank tests comparing AI tool PDR scores against the human reference.
Global test (n = 40 paired observations; 5 tools × 8 dimensions): AI tools produced significantly more costly drift than the human baseline (z = −5.43, p < .001). Of 40 comparisons, 38 showed AI PDR above the human reference. (One was lower. One tied.)
The same test applied to drift frequency was not significant (z = −0.585, p = .559).
That asymmetry is the finding. The same code, measured two ways, tells two very different stories.
Per-dimension note: with n = 5 tools per dimension, the minimum attainable exact p-value is 0.0625, which doesn’t clear the conventional p≤.05 threshold. We report these results as directional evidence, not formally significant findings, supported by effect sizes (r = 0.90–0.91 in six of eight PDR dimensions) and unanimous positive ranks.
What the Drift Actually Looked Like
This is where the data gets concrete. The drift we found wasn’t just malformed code. It was absent code: components that satisfied the prompt and omitted the production context the prompt didn’t ask for.
Security & Privacy: Role gates enforced only in the UI, bypassable in DevTools. Raw credentials passed through props without tokenization. The model treated the browser as a trusted environment.
Accessibility: Focus escaped modals and was never returned. Interactive elements built without semantic markup. Keyboard users left stranded.
User Experience: Containers missing overflow containment. Forms that fail silently — errors that identify the problem but not the fix. Lists with no empty state. Constraint hints shown before the user has touched the field.
Design Consistency: Models named UI primitives from memory without checking they existed in the design system. Inline styles bypassed design tokens. Placeholder content shipped live.
Testability: Document and window reached synchronously in component bodies — unmockable in test environments. The silent killer: components emitted with no tests alongside them.
What This Means in Practice
Drift is endemic. Every file we tested (both human and AI-generated) produced drift across every dimension. In terms of tool performance, no tool performed better across all eight dimensions. Switching tools hoping to improve on drift only changes where it shows up and how. No tool avoids drift.
Frequency misleads. A Security & Privacy gap that looks like 3× is actually 22× when you account for what fixing it costs. Teams relying on frequency-based metrics are systematically underestimating their production readiness risk, most severely in the dimensions that matter most.
Tool selection isn’t the answer. The actionable conclusion isn’t which tool to use. It’s that any tool requires a verification layer capable of catching what generation leaves behind.
Try It
The Playground at reweaver.ai/playground lets you paste React/TypeScript code and get your own PDR score. Free, no login, code scanned in memory and never stored.
The full research report is at https://info.reweaver.ai/drift-research-report
Top comments (0)