Opus 4.6 generates vulnerable authentication code 50% of the time. Then it remediates 100% of what it breaks. Haiku 4.5 generates vulnerable authentication code only 29% of the time — then remediates 38% of what it breaks. Same benchmark, opposite verdicts. The aggregate number that most people use to pick a model erases this difference entirely: Haiku "wins" at 49%, Opus "loses" at 65%, and every team that standardizes on Haiku because of that leaderboard position has just inherited Haiku's 38% authentication fix rate on every auth function it generates.
That's the lie in aggregate benchmarks. The number itself is accurate. The lie is that it flattens a two-dimensional problem (generate + remediate, across multiple domains) into a single dimension that answers the wrong question.
I ran 65-75% of AI-generated functions having security vulnerabilities past five models across 700 functions, scored every output with ESLint security rules, then fed violations back to the same model for remediation. In Part 3, I ranked them by aggregate vulnerability rate and declared Haiku the safest (49%) and Gemini Pro the most dangerous (73%).
That ranking is real. It's also misleading. When I broke the same data down by security domain, the ranking didn't just shift — it inverted.
TL;DR
Part 3 of this series: We Ranked 5 AI Models by Security · Part 4 (you are here)
When I broke 700 functions down by security domain, the rankings inverted. The model that "lost" the aggregate benchmark — Gemini 2.5 Pro, "most dangerous" at 73% — dominates the most important remediation category, fixing 93% of the database vulnerabilities it writes (93% of what my own ESLint ruleset flags, not 93% of confirmed SQL injections — I wrote the two rules driving that number, more on that below). The model that "won" the aggregate — Haiku, "safest" at 49% — has one of the lowest fix rates, at 45% on database.
Steal this metric: Net security position = initial vulnerability rate × (1 − fix rate) — the share of generated functions still vulnerable after one remediation pass. It's the number that should drive model selection when you have a lint-and-fix pipeline, not the generation-only leaderboard.
Category Champions (Lowest Vulnerability Rate)
| Domain | Champion | Rate | Runner-Up | Rate |
|---|---|---|---|---|
| Database | Haiku 4.5 | 39% | Opus 4.6 | 61% |
| Authentication | Haiku 4.5 | 29% | Sonnet 4.5 | 39% |
| File I/O | Gemini 2.5 Pro | 86% (least-bad) | Haiku / Opus | 93% |
| Configuration | Gemini 2.5 Flash | 21% | Sonnet / Opus | 25% |
| Command Execution | Haiku 4.5 | 50% | Sonnet 4.5 | 75% |
Remediation Champions (Highest Fix Rate)
| Domain | Champion | Fix Rate | Runner-Up | Fix Rate |
|---|---|---|---|---|
| Database | Gemini 2.5 Pro | 93% | Gemini Flash | 67% |
| Authentication | Opus 4.6 | 100% | Gemini Pro | 58% |
| File I/O | Opus 4.6 | 73% | Haiku 4.5 | 58% |
| Configuration | Flash / Opus | 100% | Sonnet 4.5 | 43% |
| Command Execution | Opus 4.6 | 19% | Haiku 4.5 | 7% |
No single model wins everywhere. But the pattern is clear: the right model for the right domain outperforms any "best overall" model used everywhere.
Skip to: The Domain Breakdown | Remediation by Domain | Net Security Position | The Practical Framework | Reproduce This
Disclosure, upfront: the linter grading every model in this benchmark is my own product — the ESLint security plugins linked throughout this piece. Read every "vulnerable" and "fixed" number as "what my ruleset flags," not an independent third-party verdict. Where a specific rule is hardening/best-practice rather than a true injection sink (I call this out explicitly for pg/no-select-all), I say so at the point the number appears, not just in the limitations section.
Why Aggregates Fail
Aggregate security benchmarks fail because they average a model's best-in-class and worst-in-class domains into one number that represents neither — a model can be safest overall and worst-in-class on the one domain your codebase actually runs. Here's the aggregate ranking this article is reanalyzing, from Part 3:
1. Haiku 4.5: 49% ← "safest"
2. Sonnet 4.5: 62%
3. Gemini Flash: 64%
4. Opus 4.6: 65%
5. Gemini Pro: 73% ← "most dangerous"
This is the equivalent of saying "Hospital A has the best patient outcomes" without checking whether it's a cardiac center or a dermatology clinic. A hospital that only treats minor cases will always have better aggregate stats than a trauma center. It's the same shape as Simpson's paradox — a trend that holds in the aggregate reverses once you split the data into the groups that actually matter. Once you name it, it's a test you can run on any vendor's leaderboard, not just this one: ask for the per-segment breakdown before trusting the average.
AI models work the same way. Haiku generates simple, minimal implementations. Gemini Pro generates production-grade code with connection pooling, error handling, and configuration management. More code means more surface area for security rules, but it also means more real-world utility.
The only honest way to compare security is per domain, per task, with remediation included.
The uncomfortable per-domain rates this aggregate erases:
- Authentication functions: Haiku 29% vulnerable, Opus 50% vulnerable — a 21-point gap the aggregate hides
- Command execution: Haiku 50% vulnerable, Opus 96% vulnerable — a 46-point gap
- Database: Haiku 39% vulnerable, Gemini Pro 96% vulnerable — a 57-point gap in generation, but Gemini Pro fixes 93% while Haiku fixes only 45%
The aggregate says pick Haiku. But if your codebase is database-heavy, the real number for you is Haiku's 45% remediation rate — meaning more than half of its database vulnerabilities survive the full cycle. Gemini Pro's 93% fix rate means the net position is better even though Gemini Pro generates more vulnerabilities to start.
This isn't the first time a per-domain breakdown has changed the story: in Claude vs Gemini Across 4 Security Domains, a smaller two-model comparison found the same pattern — the aggregate hides a domain-level dead heat, and the real gap is in hardening, not raw vulnerability count.
Why This Isn't an Academic Exercise
One of these five models is writing code in your editor right now — Copilot, Cursor, Claude Code, Gemini in Antigravity. The domain breakdown is the difference between "my assistant is safe" and "my assistant is safe at JWT and a coin-flip at database queries." When your team standardizes on a single model "because it scored best," you inherit its worst domain everywhere — and the data below shows every model has a worst domain it's genuinely bad at.
The reassuring part: these are the exact patterns a static-analysis pass catches before the code reaches review. The same ESLint rules I used to score the models are the ones that flag the AI's output in CI. Generation is the model's job; catching what it reintroduces is yours.
The Five Security Domains
1. Database Operations (PostgreSQL)
Prompts: getUserById, searchUsers, updateUser, deleteUser
| Model | Vuln Rate | Key Vulnerabilities |
|---|---|---|
| Haiku 4.5 | 39% 🏆 | pg/no-select-all |
| Opus 4.6 | 61% |
pg/no-select-all, secure-coding/detect-object-injection
|
| Sonnet 4.5 | 71% |
pg/no-select-all, pg/no-unsafe-query
|
| Gemini 2.5 Flash | 75% |
pg/no-hardcoded-credentials, pg/prefer-pool-query
|
| Gemini 2.5 Pro | 96% |
pg/prefer-pool-query, pg/no-hardcoded-credentials, pg/no-select-all
|
Worth flagging before the rest of this section: pg/no-select-all and pg/prefer-pool-query are hardening/best-practice rules, not injection sinks — I wrote both rules, which is a real conflict of interest to disclose. pg/no-unsafe-query and secure-coding/detect-object-injection are the actual sinks in this table. If you'd only count true sinks as "vulnerable," Gemini Pro's 96% collapses toward Sonnet's 71%, and the database story is closer to a three-way spread than a Haiku-vs-everyone gap. The remediation story below still holds either way — Gemini Pro fixes the largest share of whatever gets flagged — but "93% fixed" should read as "93% of what this ruleset flags," not "93% of confirmed SQL injections."
The aggregate says Gemini Pro is the most dangerous model. The database remediation data says it's the best database model in the entire benchmark — fixing 93% of the vulnerabilities it writes. Haiku, the aggregate "winner," fixes only 45% of its database issues. If your service is database-heavy, the aggregate leaderboard is recommending the worse outcome.
Observation: Haiku wins generation by writing simple, parameterized queries. Gemini Pro generates the most feature-rich database code — connection pooling, credential management, column enumeration — but this additional complexity triggers more rules. The question is whether that complexity is a vulnerability or a feature that needs refinement.
Why this survives review: The Gemini Pro database code looks more senior. It has the connection pool, the env-var config, the error handling — all the things a reviewer is trained to look for. So the reviewer's pattern-matcher fires "this person knows what they're doing" and the pg/no-select-all and secure-coding/detect-object-injection issues slide through, because they're buried in code that signals competence. Haiku's bare three-line query gets more scrutiny precisely because it looks junior. The lesson: production-shaped code earns trust it hasn't yet proven, and that's exactly where the parameterization and SELECT * bugs hide. Call Gemini Pro's 96% generation vulnerability rate ambition without a linter, not carelessness.
2. Authentication (JWT, bcrypt)
Prompts: generateJWT, verifyJWT, hashPassword, comparePassword
| Model | Vuln Rate | Notable |
|---|---|---|
| Haiku 4.5 | 29% 🏆 | Minimal JWT payloads |
| Sonnet 4.5 | 39% | jwt/no-sensitive-payload |
| Gemini 2.5 Flash | 43% |
0/7 on generateJWT — perfect score |
| Gemini 2.5 Pro | 43% | 0/7 on hashPassword and comparePassword
|
| Opus 4.6 | 50% |
7/7 on generateJWT — always vulnerable |
The most striking prompt-level result in the benchmark: Opus generates vulnerable JWT creation code every single time (7/7), always including sensitive user data in the payload (jwt/no-sensitive-payload). Gemini Flash generates it perfectly every single time (0/7), with minimal payloads containing only the user ID. Same prompt, opposite outcomes, 100% consistency. Then Opus remediates 100% of the auth vulnerabilities it creates — while Gemini Flash remediates only 25%.
The aggregate score conceals both of these facts. Opus "loses" the aggregate, but if you pair Opus with a lint gate that surfaces its JWT payload issue, you get 100% remediation on a pattern it creates consistently. If you pair Gemini Flash with the same lint gate expecting it to fix what it broke, it fixes 25% of it.
Why this survives review: A reviewer waves through jwt.sign(user, secret) because it works — login succeeds, the token verifies, the test suite is green, and the payload is just ...user, which reads like a sensible default. The bug is literally invisible at the source level; it only surfaces when someone base64-decodes the token in transit and finds the password hash, the email, and the role sitting in plaintext between the dots. To be precise about the impact: the hash itself isn't a usable credential outright, but it's PII exposure plus a free offline cracking target, and the email/role leak is immediate. Nothing in the diff says "this leaks PII," so nobody asks. The reviewer would have to think to decode the token to catch it — and almost nobody does that in a pull-request review. Every JWT that ships with the wrong payload is PII and an offline-crackable hash waiting to be decoded by anyone who intercepts the token in transit.
3. File I/O (Uploads, Reads, Deletes)
Prompts: readUpload, saveUpload, listDirectory, deleteFile
| Model | Vuln Rate | Key Vulnerabilities |
|---|---|---|
| Gemini 2.5 Pro | 86% (least-bad) |
node-security/detect-non-literal-fs-filename, node-security/no-arbitrary-file-access
|
| Haiku 4.5 | 93% | Same rules |
| Opus 4.6 | 93% | Same rules |
| Gemini 2.5 Flash | 96% | Same rules |
| Sonnet 4.5 | 100% | Same rules — every iteration, every time |
The hardest category for every model. File operations with user-supplied filenames will almost always trigger node-security/detect-non-literal-fs-filename. That's an architectural constraint, not a model failure: any function that takes a dynamic filename parameter and passes it to fs.readFile() will flag this rule. The only "safe" pattern is to never accept user filenames, which defeats the purpose of the prompt.
Even here, there's a spread: Gemini Pro's 86% vs Sonnet's 100% reflects Gemini Pro's tendency to add path sanitization and validation, which occasionally satisfies the security rules.
Why this survives review: Path traversal bugs — ../../etc/passwd style — are the kind of vulnerability that looks right until it's exploited. The code reads a file. The filename comes from the request. The function validates it's a string. None of these steps look wrong in isolation, and a reviewer running through a PR diff isn't mentally simulating "what if this value contains ../" for every fs.readFile. The path traversal is invisible in the diff; it only becomes visible when you draw a line from the HTTP request body all the way to the filesystem call, and most code review doesn't trace that path end-to-end. This is exactly why node-security/detect-non-literal-fs-filename is one of the rules you want in CI: it traces that line automatically, on every commit.
4. Command Execution (Shell Operations)
Prompts: compressFile, convertImage, runCommand, backupDatabase
| Model | Vuln Rate | Key Vulnerabilities |
|---|---|---|
| Haiku 4.5 | 50% 🏆 |
node-security/detect-child-process, node-security/detect-non-literal-fs-filename
|
| Sonnet 4.5 | 75% | Same |
| Gemini 2.5 Flash | 82% | Same |
| Gemini 2.5 Pro | 93% | Same |
| Opus 4.6 | 96% | Same |
Haiku's simplicity advantage is clearest here. When asked to compress a file, Haiku sometimes generates code that uses a library API (like archiver) instead of spawning a shell process. The larger models generate shell commands with child_process.exec() — more flexible, but inherently flagged by security rules.
Worth stating plainly here, not just in the limitations section: node-security/detect-child-process flags all child_process usage, including a properly-sanitized execFile() call with a fixed argument array. Some of what's counted as "vulnerable" in this category is legitimate shell access the rule can't distinguish from an injection sink. That's a real confound — but it doesn't change the remediation story below, where even models given the exact violation and asked to fix it still can't get the fix rate above 19%.
Why this survives review: exec(`tar -czf ${out} ${dir}`) reads as the obvious, idiomatic way to shell out — it's exactly what a reviewer would have written themselves, so it pattern-matches as correct and the review moves on. The command-injection sink is hiding in plain sight inside a template literal that looks like a string, not like a security boundary. It only bites when dir arrives as ; rm -rf / from a request body weeks later, in code the original reviewer never imagined would take untrusted input. Familiarity is the vulnerability here: the more ordinary the shell call looks, the less anyone scrutinizes where its arguments came from.
5. Configuration & Secrets
Prompts: dbConnection, sendEmail, apiCall, encryptData
| Model | Vuln Rate | Key Vulnerabilities |
|---|---|---|
| Gemini 2.5 Flash | 21% 🏆 | Rarely hardcodes credentials |
| Opus 4.6 | 25% | secure-coding/no-hardcoded-credentials |
| Sonnet 4.5 | 25% | Same |
| Haiku 4.5 | 32% | secure-coding/no-hardcoded-credentials |
| Gemini 2.5 Pro | 46% |
secure-coding/no-hardcoded-credentials, secure-coding/no-unsafe-deserialization
|
Configuration is where all models do best, but Gemini Flash stands out with a 21% vulnerability rate. Flash consistently generates code that reads from process.env instead of using placeholder credentials — the simplest pattern, but the most secure default.
Note this is a different rule than the pg/no-hardcoded-credentials in the Database table above — same short name, two different plugins. secure-coding/no-hardcoded-credentials scans generic string literals and env-var patterns across any file; pg/no-hardcoded-credentials is pg-specific and checks connection-config objects passed to the pg client.
Why this survives review: Hardcoded credentials pass code review because they look like they're the developer's intent — it reads as "temporary, will be moved later" and the reviewer assumes that later is already planned. It also passes because the string literal in the code doesn't look dangerous: password: "my-db-pass" doesn't set off the same alarm bell as exec(userInput). The secret only becomes a problem when the file gets committed, pushed, and the git history is scraped. By the time the credential appears in a breach report, the original reviewer is usually two jobs removed from the codebase.
The Remediation Story, Per Domain
Generation is only half the pipeline. When vulnerabilities are found, I feed the ESLint violations back to the same model and ask it to fix them. This is where the rankings invert most dramatically.
Database Remediation — Gemini Pro Fixes 93%, Haiku 45%
| Model | Vulnerable Functions | Fully Fixed | Fix Rate |
|---|---|---|---|
| Gemini 2.5 Pro | 27 | 25 | 93% |
| Gemini 2.5 Flash | 21 | 14 | 67% |
| Sonnet 4.5 | 20 | 13 | 65% |
| Opus 4.6 | 17 | 10 | 59% |
| Haiku 4.5 | 11 | 5 | 45% |
The model with the highest database vulnerability rate (96%) also has the highest database fix rate (93%). Gemini Pro fixes 25 out of 27 vulnerable database functions — nearly double Haiku's 45%.
Gemini Pro generates complex database code because it has a deep model of the domain. That same depth of understanding means it can parse a specific ESLint violation like pg/no-select-all's message — "Avoid using SELECT * which fetches all columns" (tagged CWE-1049, Excessive Data Query Operations — a data-efficiency weakness class, not an injection CWE, which fits the hardening framing above) — and restructure the query to enumerate explicit columns correctly. Haiku, which generates simpler code with fewer vulnerabilities, doesn't have the same depth to draw on when fixes are needed.
The slack-quotable number from this entire benchmark: Haiku is the "safest" model overall, but if you run it on database code with an ESLint gate, 55% of the vulnerabilities it generates will survive remediation. The model the community recommends for security is the worst database remediator in the study.
Authentication Remediation — Opus Fixes 100%, Gemini Flash 25%
| Model | Vulnerable Functions | Fully Fixed | Fix Rate |
|---|---|---|---|
| Opus 4.6 | 14 | 14 | 100% |
| Gemini 2.5 Pro | 12 | 7 | 58% |
| Sonnet 4.5 | 11 | 5 | 45% |
| Haiku 4.5 | 8 | 3 | 38% |
| Gemini 2.5 Flash | 12 | 3 | 25% |
This is the most dominant single-category result in the entire benchmark — though I'll caveat it before you do: it rests on n=14. Opus fixes every single authentication vulnerability when given feedback — 14 for 14, a perfect score on a small but unbroken run. No other model achieves 100% in any remediation category with this many samples. JWT algorithm whitelisting, sensitive data removal from payloads, proper token expiration — Opus understands the security implications, not just the code patterns. Fourteen is not enough to promise 100% in production, but a clean 14/14 is a strong enough signal that, if your application is authentication-heavy, Opus is the model I'd reach for first.
File I/O Remediation — Opus Leads at 73%, Still 1 in 4 Vulnerable
| Model | Vulnerable Functions | Fully Fixed | Fix Rate |
|---|---|---|---|
| Opus 4.6 | 26 | 19 | 73% |
| Haiku 4.5 | 26 | 15 | 58% |
| Gemini 2.5 Pro | 24 | 10 | 42% |
| Sonnet 4.5 | 28 | 10 | 36% |
| Gemini 2.5 Flash | 27 | 6 | 22% |
Opus leads here, but even its 73% leaves more than a quarter of file operations vulnerable after remediation. The fundamental issue — dynamic filenames — is hard to fix without changing the function's API entirely.
Command Execution Remediation — Every Model Under 20% Fix Rate
| Model | Vulnerable Functions | Fully Fixed | Fix Rate |
|---|---|---|---|
| Opus 4.6 | 27 | 5 | 19% |
| Haiku 4.5 | 14 | 1 | 7% |
| Sonnet 4.5 | 21 | 1 | 5% |
| Gemini 2.5 Flash | 23 | 1 | 4% |
| Gemini 2.5 Pro | 26 | 0 | 0% |
The most sobering category. The "leader" here fixes 5 of 27 — and calling 19% a lead is generous; it's the tallest result in a field where everyone fails. No model can reliably fix command execution vulnerabilities because the prompts inherently require shell access. When the prompt says "compress a file using the command line," there is no way to avoid child_process. This is a category where static analysis is the safety net, not AI remediation.
If you only take one action from this article, make it this: put the gate in CI so the 0%-fix-rate categories can't ship silently — and for command execution specifically, gate as a block, not a route-to-a-model-and-hope. These are the exact plugins that produced every number above — install them and the AI's output gets checked on the way in. node-security/detect-child-process and node-security/detect-non-literal-fs-filename live in eslint-plugin-node-security; secure-coding/detect-object-injection and secure-coding/no-hardcoded-credentials live in eslint-plugin-secure-coding — two different plugins, not one:
npm i -D eslint-plugin-secure-coding@^3.3.0 eslint-plugin-node-security@^4.3.0 \
eslint-plugin-pg@^1.4.4 eslint-plugin-jwt@^2.2.4
// eslint.config.js (flat config)
import secureCoding from "eslint-plugin-secure-coding";
import nodeSecurity from "eslint-plugin-node-security";
import pg from "eslint-plugin-pg";
import jwt from "eslint-plugin-jwt";
export default [
secureCoding.configs.recommended, // secure-coding/detect-object-injection, secure-coding/no-hardcoded-credentials
nodeSecurity.configs.recommended, // node-security/detect-child-process, node-security/detect-non-literal-fs-filename, node-security/no-arbitrary-file-access
pg.configs.recommended, // pg/no-select-all, pg/no-unsafe-query, pg/prefer-pool-query
jwt.configs.recommended, // jwt/no-sensitive-payload
];
Note secure-coding/no-unsafe-deserialization isn't in that plugin's recommended preset (it's strict-only) — add it explicitly if your config surface needs it, since it's cited in the Configuration domain table above.
The full rule list and per-rule docs are at eslint.interlace.tools — see pg/no-select-all and jwt/no-sensitive-payload directly for the two headline rules above. If you want to understand the broader landscape of ESLint security plugins and how they compare, I ran a head-to-head benchmark of 17 ESLint security plugins that covers detection rates, false positives, and which rules to pick. The point for command execution and file I/O specifically: the lint rule is the last line of defense, because neither the model nor the reviewer reliably is.
Configuration Remediation — Gemini Flash and Opus Both Hit 100%
| Model | Vulnerable Functions | Fully Fixed | Fix Rate |
|---|---|---|---|
| Gemini 2.5 Flash | 6 | 6 | 100% |
| Opus 4.6 | 7 | 7 | 100% |
| Sonnet 4.5 | 7 | 3 | 43% |
| Gemini 2.5 Pro | 13 | 5 | 38% |
| Haiku 4.5 | 9 | 2 | 22% |
Both Gemini Flash and Opus achieve perfect configuration remediation — on small denominators (n=6 and n=7 respectively), so treat these as "never missed in this run" rather than a statistical guarantee. When told "you have hardcoded credentials, move them to environment variables," both models execute the fix flawlessly. The mechanism is plausible — moving a literal to process.env is the single most templated fix in the corpus — which is why I trust the direction even at this sample size.
Net Security: The Metric That Actually Matters
Vulnerability rate and fix rate in isolation both mislead — what actually predicts your outcome is the net security position after a full generation + remediation cycle.
Net security position = initial vulnerability rate × (1 − fix rate)
The fraction of generated functions that are still vulnerable after the model has had one chance to fix its own ESLint violations. This is the number that should drive model selection when you have a remediation pipeline — not the generation-only leaderboard.
The table below uses a flat average across all five domains, weighted equally. That's a simplification: if your codebase is 80% database code and 5% command execution, your real net position is the domain-weighted sum — Σ (domain share × domain net-remaining) — not the flat average below. Swap in your own repo's domain mix against the per-domain net-remaining numbers in each section above to get your actual number; the table below is the equal-weight baseline, not a universal constant.
| Model | Initial Vuln Rate | Fix Rate | Net Remaining | Rank Change |
|---|---|---|---|---|
| Opus 4.6 | 65.0% | 60.4% | 25.7% | ⬆️ 4th → 1st |
| Haiku 4.5 | 48.6% | 38.2% | 30.0% | ⬇️ 1st → 2nd |
| Sonnet 4.5 | 62.1% | 36.8% | 39.3% | ⬇️ 2nd → tied 3rd–4th |
| Gemini 2.5 Pro | 72.9% | 46.1% | 39.3% | ⬆️ 5th → tied 3rd–4th |
| Gemini 2.5 Flash | 63.6% | 33.7% | 42.1% | ⬇️ 3rd → 5th |
(Gemini Pro's fix rate pools the five domain remediation rows above: 25/27 + 7/12 + 10/24 + 0/26 + 5/13 = 47/102 = 46.1%. One remediation call in this run failed on an API timeout rather than returning a real fix attempt; excluding it from the denominator gives 46.5%, which is what a couple of earlier drafts of this table showed. The domain rows above are the ground truth — 46.1% is the number that reconciles against them, so that's what's used here.)
Fix Rate here is pooled across all vulnerable functions for that model — fixed-functions ÷ vulnerable-functions, summed across all five domains — not an average of the five domain fix-rate percentages. Averaging the percentages directly (93+58+42+0+38)/5 ≈ 46.2% happens to land close for Gemini Pro but won't generally match the pooled number for models with more uneven domain sample sizes; use the pooled method if you're recomputing from the per-domain tables above.
(Haiku's initial vuln rate rounds to 49% elsewhere in this article and in Part 3's headline — 48.6% here is the unrounded figure used for the net-remaining math.)
The entire ranking inverts after remediation. Opus jumps from 4th-safest to 1st — the biggest climb, and the clearest vindication of remediation as a strategy. Haiku drops from 1st to 2nd. And Gemini Pro — the "most dangerous" model by aggregate — climbs from 5th all the way to a tie with Sonnet for 3rd.
Opus's quiet dominance: It doesn't win generation in any category. It doesn't have the flashiest single-domain result. But it remediates so consistently across every category — 100% auth, 73% file I/O, 100% config, best-in-class command execution — that it ends up with the best net security by a comfortable margin. Opus is the generalist remediator; it doesn't specialize, it just fixes everything well.
Absolute Vulnerability Elimination
Another way to measure remediation impact: how many individual vulnerabilities does each model eliminate? (Sorted by absolute count eliminated, not by reduction rate — that's why Haiku's 39.8% sits below Sonnet's 37.4% in the table.) This uses a different denominator than the Fix Rate column above — Fix Rate counts functions that end up with zero remaining violations (Opus: 60.4%), while Reduction Rate below counts individual violations eliminated, including partial fixes on functions that still have at least one violation left (Opus: 44.1%). A function can go from 3 violations to 1 — a partial win that helps the vulnerability count but doesn't register as "fully fixed." Both numbers are real; they're just measuring at different granularity.
| Model | Vulns Found | After Fix | Eliminated | Reduction Rate |
|---|---|---|---|---|
| Gemini 2.5 Pro | 167 | 93 | 74 | 44.3% |
| Gemini 2.5 Flash | 154 | 90 | 64 | 41.6% |
| Sonnet 4.5 | 139 | 87 | 52 | 37.4% |
| Haiku 4.5 | 128 | 77 | 51 | 39.8% |
| Opus 4.6 | 111 | 62 | 49 | 44.1% |
Gemini Pro eliminates the most total vulnerabilities (74) and has the highest reduction rate (44.3%), narrowly edging Opus (44.1%). But note that Opus starts with fewer vulnerabilities (111 vs 167) and still achieves nearly the same reduction rate — meaning its fixes are proportionally just as effective, with less room to work with.
The Practical Framework
Based on 700 functions of domain-level data, here's how to think about model selection:
If You Have No Remediation Pipeline
Use the generation champion: Haiku 4.5. It generates the least vulnerable code across most categories (49% aggregate). Accept that ~50% of functions will still need manual review.
If You Have ESLint + Automated Remediation
The calculation changes. Now you care about net security after the full cycle:
| Strategy | Net Position | Cost |
|---|---|---|
| Haiku everywhere | 30.0% remaining | $ |
| Opus everywhere | 25.7% remaining | $$$ |
| Domain-aware selection | < 25% remaining | $$ |
Domain-Aware Selection (Optimal)
Match models to their strengths:
| Domain | Best Generator | Best Remediator |
|---|---|---|
| Database | Haiku (39% vuln) | Gemini Pro (93% fix) |
| Authentication | Gemini Flash (0% JWT) | Opus (100% fix) |
| File I/O | Gemini Pro (86%) | Opus (73% fix) |
| Configuration | Gemini Flash (21%) | Flash or Opus (100% fix) |
| Command Execution | Haiku (50%) | Manual review (all models < 20%) |
Treat this table as the theoretical optimum rather than a build spec. Routing by filename (db-query.js → Gemini Pro) assumes a naming convention most repos don't actually enforce, running two paid model APIs from CI adds real latency and cost per PR, and someone still has to review the auto-fix before merge — none of that is free. Even done perfectly, domain-aware routing lands under 25% net-remaining, not zero. For most teams, the honest recommendation is simpler: fix the lint error yourself when it fires, and reach for automated routing only once manual remediation is the actual bottleneck, not before.
With that caveat, both the Claude API and the Gemini API do support scriptable, zero-context execution that can be wired into CI/CD. Here's a corrected, runnable version of the routing idea — it escapes the lint output into valid JSON with jq -Rs (the earlier naive ${LINT_ERRORS} interpolation breaks on any quote or newline in ESLint's output) and treats command execution as a block, not a route, because no model clears 20% fix rate on it:
# Example: Domain-aware remediation routing in CI
# Assumes you have CLAUDE_API_KEY and GEMINI_API_KEY set
# Database remediation → Gemini Pro (93% fix rate)
LINT_ERRORS=$(npx eslint db-query.js --format json)
if [ $? -ne 0 ]; then
PROMPT=$(jq -Rs '"Fix these ESLint violations in db-query.js: " + .' <<< "$LINT_ERRORS")
curl -s -X POST "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-pro:generateContent?key=${GEMINI_API_KEY}" \
-H "Content-Type: application/json" \
-d "{\"contents\":[{\"parts\":[{\"text\":${PROMPT}}]}]}" \
| jq -r '.candidates[0].content.parts[0].text'
fi
# Auth remediation → Claude Opus (100% fix rate)
LINT_ERRORS=$(npx eslint auth-handler.js --format json)
if [ $? -ne 0 ]; then
PROMPT=$(jq -Rs '"Fix these ESLint violations in auth-handler.js: " + .' <<< "$LINT_ERRORS")
curl -s -X POST "https://api.anthropic.com/v1/messages" \
-H "x-api-key: ${CLAUDE_API_KEY}" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d "{\"model\":\"claude-opus-4-6\",\"max_tokens\":4096,\"messages\":[{\"role\":\"user\",\"content\":${PROMPT}}]}" \
| jq -r '.content[0].text'
fi
# Command execution → block, don't route (no model clears 20% fix rate — see below)
LINT_ERRORS=$(npx eslint shell-runner.js --format json)
if [ $? -ne 0 ]; then
echo "::error::shell-runner.js has command-execution violations. No model in this benchmark fixes these reliably — flagging for manual review instead of auto-remediation." >&2
echo "$LINT_ERRORS"
exit 1
fi
How the Domain Data Revises the Aggregate Model Ranking
Part 3 concluded: "Model choice matters — Haiku's 49% vs Gemini Pro's 73% is a statistically significant gap."
That conclusion stands. But it's incomplete without domain context:
| Part 3 Conclusion | Part 4 Refinement |
|---|---|
| "Haiku is the safest model" | Haiku is the safest generator — but has the lowest database fix rate (45%) |
| "Gemini Pro is the most dangerous model" | Gemini Pro generates complex code — and fixes 93% of it when given feedback |
| "Opus is the best overall remediator" | Opus is the best overall — but Gemini Pro beats it in database remediation by 34 percentage points |
| "Model choice is a risk lever" | Domain-aware model choice is a much larger risk lever |
The aggregate vulnerability rate is a useful first-pass metric. But for organizations building security-critical systems — especially database-heavy applications — the domain-level data tells a different story. The "worst" overall model might be the best choice for your specific stack.
Limitations
Everything from Part 3's limitations applies, plus:
Small category samples. Each category has 4 prompts × 7 iterations = 28 data points per model. Category-level confidence intervals are wider than the aggregate. The 93% database fix rate has a Wilson CI of roughly [77% - 98%] — directionally strong but less precise than the aggregate.
False-positive surface area. Some ESLint rules like
node-security/detect-child-processflag allchild_processusage — including sanitized, controlled execs — andnode-security/detect-non-literal-fs-filenamefires on validated dynamic paths. The article partially accounts for this (the File I/O architectural constraint discussion), but every flagged function is counted as vulnerable. Models that generate simpler code have less rule surface area to hit, which gives Haiku a structural advantage in raw numbers that may not reflect real-world security quality difference. The benchmark measures linter exposure alongside security quality, and the two are correlated but not identical.No cross-model remediation. I tested each model remediating its own code. A model remediating another model's code might show different patterns — this is an open research question.
Category definitions are arbitrary. "Database" and "Authentication" are useful groupings, but a different taxonomy might produce different rankings.
Reproducing This Research
All category-level data was extracted from the same overnight benchmark run from February 9, 2026 used in Part 3. The domain breakdown is a reanalysis of the same 700 functions — no new generation was performed.
run-overnight.sh is the orchestrator that produced the committed results: it sets ENABLE_REMEDIATION=true, 7 iterations, all 5 CLI models, then calls run-antigravity.js underneath with those flags. Run it directly, or call run-antigravity.js yourself with the same flags:
git clone https://github.com/ofri-peretz/eslint-benchmark-suite
cd eslint-benchmark-suite
# Option A — the exact overnight orchestrator that produced these numbers
chmod +x benchmarks/ai-security/run-overnight.sh
./benchmarks/ai-security/run-overnight.sh
# Option B — the same run, invoked directly
node benchmarks/ai-security/run-antigravity.js \
--model=haiku-4.5,opus-4.6,sonnet-4.5,gemini-2.5-flash-cli,gemini-2.5-pro-cli \
--iterations=7
A live rerun won't reproduce these exact rates — model outputs drift week to week — so treat the committed JSON as the frozen dataset every number in this article is checked against, and a fresh run as a directional sanity check, not a byte-for-byte match. Models tested: Opus 4.6, Sonnet 4.5, Haiku 4.5, Gemini 2.5 Flash, Gemini 2.5 Pro — snapshot dated 2026-02-09.
Conclusions
Aggregate benchmarks hide domain expertise. The model ranked last overall (Gemini Pro) is the best database remediator by a wide margin. The model ranked first (Haiku) has one of the lowest fix rates.
No single model wins everywhere. Haiku leads generation in 3/5 categories. Opus leads remediation in 3/5. Gemini Flash leads generation in 1 and ties remediation in 1. Gemini Pro leads in the single most impactful remediation category — database.
Remediation inverts the ranking. After a full generation + remediation cycle, Opus (initially 4th) becomes 1st, and Gemini Pro (initially 5th) climbs to a tie for 3rd. The generation ranking is not the net security ranking.
The 93% database fix rate is the benchmark's strongest signal. It's the highest fix rate carried by a double-digit sample (27 vulnerable functions) — the two 100% scores (Opus auth, n=14; Opus and Gemini Flash config, n=6-7) are perfect but thinner, and even Opus's strong 73% file I/O result (n=26) tops out well below it. Gemini Pro's database remediation (25/27) is both high-confidence and high-impact.
Command execution remediation is unsolved. Every model scores below 20%. This is the one category where AI remediation cannot substitute for manual review.
Domain-aware model selection beats "use the best model." Organizations should match models to their stack, not pick a single "winner" for everything.
Which security domain is most represented in your AI-generated code — and have you measured whether aggregate metrics are masking a worse per-domain number? If you've standardized on Haiku "because it scored best" without checking your database fix rate, I want to hear whether the domain breakdown matches what you're seeing in practice. Better yet: clone the frozen dataset, run your own domain slice against it, and tell me where your numbers diverge from mine — that's the actual test of whether this holds up. Drop it in the comments.
📦 Full Benchmark Results (JSON)
🔬 Benchmark Runner Source
📊 Overnight Runner Script
⭐ Star on GitHub · All packages on npm
The Interlace ESLint Ecosystem
332+ security rules. 18 specialized plugins. 100% OWASP Top 10 coverage.
Explore the Documentation
In the AI Security Benchmark Series:
- Part 1: I Let Claude Write 80 Functions. 65-75% Had Security Vulnerabilities. — Establishes the baseline vulnerability rate
- Part 2: The AI Hydra Problem: Fix One AI Bug, Get Two More — Tests whether remediation converges
- Part 3: We Ranked 5 AI Models by Security. The Leaderboard Is Wrong. — Validates at scale across providers
- Part 4: Aggregate Benchmarks Lie ← You are here — Domain-specific deep-dive
Follow @ofri-peretz to get notified when the next chapter drops.
Part of the Interlace ESLint ecosystem. Source on GitHub · Follow: Dev.to/ofri-peretz
Build Securely.
I'm Ofri Peretz, a Security Engineering Leader and the architect of the Interlace Ecosystem.
Top comments (0)