On August 6, 1Password's new security research team, Off-by-1 Labs, published the results of its inaugural study: what happens when frontier models generate patches for recently disclosed, complex vulnerabilities. The team generated 6,480 patches across six CVEs using OpenAI's ChatGPT-5.5 and Anthropic's Opus 4.8, removed 400 patches where the model was caught retrieving information about an existing fix, and analyzed the remaining 6,080. The headline result: 53.9% of the patches either failed to fix the vulnerability, introduced a new one, or both.
FLAWED — Fix-Like Artifacts With Embedded Defects — is 1Password's term for an AI-generated patch that looks like a fix but does not survive verification: it leaves the vulnerability unresolved, introduces a new one, or both. In the study, that described 53.9% of patches analyzed.
The six targets were chosen for recency, so the fixes were unlikely to appear in either model's training data: a Linux privilege escalation (CVE-2026-31431, "Copy Fail"), an ActiveMQ RCE (CVE-2026-34197), a Chrome use-after-free (CVE-2026-8512), an Exim unauthenticated RCE (CVE-2026-45185), a SpringAI SpEL RCE (CVE-2026-22738), and a Gemini CLI RCE (GHSA-wpqr-6v78-jr5g). Just 26.0% of patches fully resolved the vulnerability without materially changing application behavior. Another 20.1% fixed the flaw but altered how the application worked — swapping allow-list logic for deny-list logic, or reimplementing a file-local parser along the way.
Why do AI-generated patches fail this often?
The study's design explains part of it. 1Password deliberately targeted vulnerabilities disclosed too recently to be in training data, then hypothesized models would still succeed more than 67% of the time on open-source code they had seen. The result came in far lower and far more uneven — which suggests models are pattern-matching toward fix-shaped output rather than reasoning to a verified fix.
The fragility finding makes the same point more sharply. More than 33% of the patches that did fix the vulnerability were rated fragile from a security standpoint: they guarded against the specific malicious input in the proof of concept rather than addressing the underlying vulnerable code. On the SpringAI CVE, both models repeatedly generated patches that escaped particular characters in user input — blocking the demonstrated exploit string while leaving the root cause untouched. A patch like that passes a glance and fails an attacker.
Guidance quality compounds the problem. According to the research paper, as reported by The Register, models given correct initial guidance fixed the flaw 65.0% of the time, models with no guidance 50.4% — and models given incorrect guidance dropped to 15.2%. Human developers have a decent chance of noticing that the advice they were handed is wrong; models mostly do not.
Anthropic, whose model was one of the two studied, gave 1Password feedback that names the structural gap: patch generation has outpaced patch verification, and verification needs to become "execution-grounded rather than inspection-based," with domain experts staying on as final reviewers at current model capabilities. Meanwhile the review habit is eroding in the other direction: developer-habits data from Cursor, cited in Dark Reading's coverage, puts 36% of changes as auto-accepted without manual review — a share of change volume, not of developers, but the direction is the problem. Generation is scaling; verification is not.
What should teams check before trusting AI-generated patches?
First, map where model-authored changes enter your codebase. Auto-remediation bots, coding agents opening pull requests, IDE agents with auto-accept enabled — each is a path by which a FLAWED patch can land without a qualified human reading it.
Second, make verification execution-grounded. The fragile third of "successful" patches is exactly the class that code inspection misses. Reproduce the exploit before the patch, rerun it after, and run a behavior-regression suite — 20.1% of working patches changed application behavior in ways a vulnerability-only test would never catch.
Third, benchmark on your own code. 1Password released its tooling, datasets, and patch corpus so teams can run the same evaluation against previously patched vulnerabilities in their own repositories, and find out where models are strong and where they are dangerous before trusting them with anything live.
Fourth, treat "requires human review" as a control, not a norm. If an agent-authored change can reach your default branch without a recorded approval from a named person, the requirement is a wish. That is a quality-gate problem, and it deserves the same enforcement rigor as the rest of your gates.
How Waxell handles this
The study's recommendation is a human-review requirement. The recurring failure mode with such requirements is that they live in a wiki while the pressure to burn down a vulnerability backlog lives in the sprint — the same dynamic 1Password's earlier survey on agent access documented, where adoption ran ahead of governance. An unenforced review rule erodes one auto-accepted merge at a time.
Waxell puts the enforcement where agent-driven changes actually happen: the tool call. When a coding agent pushes a commit or merges a pull request through an MCP tool, the Waxell MCP Gateway evaluates that call against your tenant's policy rules before the upstream sees it. A call that requires approval is parked: the gateway holds the connection open so the agent doesn't time out, a reviewer approves and the call resumes upstream, or denies and the agent receives a structured error it can recover from. A merge by an agent becomes contingent on a recorded human decision — with approval fatigue managed by policy scope, not by reviewing everything. Each governed call is resolved to a real user identity, and the audit log keeps the call, the decision, and the rules that fired — durable and exportable, so "who approved this agent's merge" has an answer. Policy rule changes propagate within 30 seconds.
One scope note the incident demands: the gateway governs calls that traverse it. An agent holding a direct GitHub credential, or a developer pasting a model's diff into their own branch, is outside that path — those paths are closed by repo-side branch protection, not by a gateway.
For agents you build in Python, Waxell Observe applies the same logic inside the run: among its 50+ policy categories, Quality policies score outputs, flag low-confidence responses and block inadequate results, while Control policies define approval gates and escalation paths — so an agent's output can be escalated to a human before the next step executes, not examined after it shipped.
Waxell does not judge whether a patch is correct. The study is clear that this takes execution-grounded testing and a domain expert. What policy enforcement adds is the guarantee that the expert's review happens before the change lands — and a record that it did.
FAQ
What did 1Password's FLAWED study find?
Off-by-1 Labs generated 6,480 patches across six recently disclosed CVEs using ChatGPT-5.5 and Opus 4.8, and analyzed 6,080 after removing flagged attempts. 26.0% fully fixed the vulnerability without changing application behavior, 20.1% fixed it but altered behavior, and 53.9% failed to fix it, introduced a new vulnerability, or both.
What does FLAWED stand for?
Fix-Like Artifacts With Embedded Defects: patches that have the shape of a fix but leave the vulnerability unresolved or introduce a new one. The term is 1Password's, coined in the research paper accompanying the study.
Are AI-generated patches cheaper than human-written ones?
Per attempt, yes: the study measured an average of $2.11 per attempted patch-and-validation cycle for ChatGPT-5.5 and $2.81 for Opus 4.8. But the clean-success rate was roughly 1 in 4, and the paper concludes that a fully LLM-generated patch with no human review carries an expected value that is "net-negative by a considerable margin."
How can teams enforce human review of AI-generated code changes?
Move the requirement from convention to control: require a recorded approval before an agent-authored change can merge, gate the agent's write-path tool calls behind approval policies, and keep an attributed audit trail of which changes were made by agents and who approved them. Branch protection covers direct human paths; a governed gateway covers the tool calls agents make.
Does Waxell verify that an AI-generated patch is correct?
No. Patch correctness takes execution-grounded testing and domain expertise, which the study shows cannot be skipped. Waxell enforces that the review happens: approval policies park an agent's merge-type tool calls until a human decides, quality policies block inadequate outputs mid-run, and the audit log records the decision.
Sources
- 1Password (Keith Hoodlet), "Off-by-1 Labs: Why AI-generated vulnerability patches still require expert human review", August 6, 2026
- The Register (Thomas Claburn), "AI struggles to patch vulns without adult supervision", August 6, 2026
- Dark Reading (Robert Lemos), "AI-Generated Patches Fail Half the Time", August 7, 2026
Originally published on the Waxell blog.
The 53.9% figure is an argument for making review non-optional where agents act. Start free with the Waxell MCP Gateway — one governed endpoint, with approval policies on the tool calls your agents already make.
Top comments (0)