The security firm Dam Secure ran 10 AI models over the same 10 pull requests, each seeded with one hidden access-control bug, and found that GPT-5.6 Sol caught every single one -- 100% recall at about $0.70 per review, the first model to hit perfect recall on their harness. Grok 4.5 landed on the cost frontier at $0.20 per review, while no Anthropic model reached the cost-quality frontier for this particular workload.
Key facts
- GPT-5.6 Sol: 100% recall, $0.70 per pull request -- described by the authors as "the king."
- Grok 4.5: on the cost/quality frontier at $0.20 per PR; Fable 5 (with Opus 4.8 fallback) most expensive at ~$3.61 per PR and off the frontier.
- Method: 10 models, 10 pull requests, one planted access-control bug each (IDOR, missing auth), 5 runs per model, scored on recall, precision, and F1.
- Published July 11, 2026 by Ethan Cavill, Simon Harloff, and Patrick Collins of Dam Secure. Benchmark writeup.
The hook is what an "access-control bug" actually is, because it is the most common serious flaw in real web software. It is the class of mistake where the code forgets to check whether the person making a request is allowed to -- so user A can read or change user B's data just by editing an ID in the URL. These bugs are boring, invisible in a quick read, and catastrophic in production. A tool that reliably catches them in code review is genuinely useful, which is why measuring it matters.
Dam Secure's setup is designed to avoid a trap that plagues AI benchmarks: contamination. If a model has seen the test answers during training, its score is meaningless. So they built private, unpublished synthetic pull requests from open-source repos, and separately used "reverse-replay" of real disclosed vulnerabilities -- take a known CVE, revert the fix, and bury it in genuine same-era commits so the model has to find it fresh. Each model saw the same diff and its findings were scored against the known ground truth. They also ran two different harnesses and got similar results, which helps isolate model quality from pipeline design.
The headline result is a reversal of the usual pecking order. On this task the cheapest-and-good models -- GPT-5.6 Sol at $0.70 and Grok 4.5 at $0.20 -- dominate, while Anthropic's models, often the default for coding, do not reach the cost frontier. GPT-5.6 Sol is also about 45% cheaper than its predecessor for roughly the same performance, which is the anchor efficiency stat: a frontier security-review model that got dramatically cheaper without losing accuracy.
Why it matters: automated security review is one of the most concrete, buyable uses of AI coding tools, and this is a rare head-to-head with cost attached. It also fits the day's broader theme -- efficiency and cost, not raw capability, are where the frontier fight is being decided.
The honest caveat is essential, and the authors are refreshingly explicit about it. Dam Secure sells AI vulnerability scanning, so this is a single-vendor benchmark with an obvious interest in a striking result. More importantly, they say directly: "Fable does perform well on Full Code Scans, but this workload is a PR Scan." A pull-request scan looks at a small diff; a full-code scan reads the whole repository, and Fable handed off to its Opus fallback only 10.7% of the time -- meaning most Fable runs here were the base model, not the stronger fallback. So the correct reading is narrow and true: for the specific job of catching access-control bugs in a single pull request, GPT-5.6 Sol and Grok 4.5 were both accurate and cheap, and Anthropic's models were not on the cost frontier -- on this test. It is one benchmark's result, not a verdict on which model is better at security overall.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)