Smart contracts guard over $100 billion in crypto assets. They're immutable once deployed, run autonomously, and have historically been the single fattest target in all of DeFi. The question is no longer whether AI will play a role in securing them — it's how good AI actually is at the job today.
In February 2026, OpenAI and Paradigm dropped EVMbench, an open-source benchmark that puts AI agents through three brutally practical tests: detect vulnerabilities, patch them without breaking anything, and exploit them end-to-end in a sandboxed blockchain. The results are simultaneously impressive and terrifying.
Let's break down what EVMbench actually measures, what the numbers tell us, and what it means for security researchers and protocol teams.
The Three Modes: Detect, Patch, Exploit
EVMbench isn't a toy benchmark with synthetic bugs. It draws from 117 curated vulnerabilities across 40 real audits, mostly from Code4rena competitions, plus additional scenarios from the security audit of Paradigm's Tempo blockchain (a payments-focused L1).
🔍 Detect Mode
The agent receives a smart contract repository and must identify ground-truth vulnerabilities documented by human auditors. Scoring is recall-based: did the agent find the bugs the humans found?
The catch: Agents tend to stop after finding one issue rather than exhaustively auditing the codebase. This mirrors a common human auditor anti-pattern — "first bug satisfaction" — but at machine scale, it's a fixable problem.
🔧 Patch Mode
The agent must modify vulnerable code to eliminate the exploit while preserving all original functionality. Tests must still pass. The contract must still do what it's supposed to do.
This is the hardest mode. Fixing a reentrancy guard is easy. Fixing a subtle accounting bug in a lending protocol's liquidation path without breaking the interest rate model? That requires deep semantic understanding of the code's intent, not just its flaws.
💀 Exploit Mode
The agent gets a sandboxed Anvil environment with the vulnerable contract deployed. Its job: drain funds. Grading is programmatic — did the on-chain state change in a way that indicates successful exploitation?
A Rust-based harness deploys contracts, replays agent transactions deterministically, and restricts unsafe RPC methods. Everything runs locally, against historical and publicly documented vulnerabilities.
The Numbers: 71% Exploit Rate Is a Wake-Up Call
Here's where it gets real:
| Model | Exploit Success Rate |
|---|---|
| GPT-5.3-Codex (via Codex CLI) | 71.0% |
| GPT-5 (6 months earlier) | 33.3% |
| Earlier models | <20% |
That's a 2x improvement in six months. Paradigm's Alpin Yukseloglu put it bluntly: "When we started working on this project, top models were only able to exploit less than 20% of the critical, fund-draining Code4rena bugs. Today, GPT-5.3-Codex exploits over 70%."
But detect and patch scores remain significantly lower. This creates an asymmetry that should concern everyone:
AI agents are better at exploiting vulnerabilities than finding or fixing them.
This isn't unique to AI — human pentesters also find it easier to exploit known bug classes than to write comprehensive fixes. But the speed differential is what matters. An AI that can exploit 71% of critical bugs in minutes changes the threat landscape fundamentally.
Why Patching Is the Hardest Problem
The patch mode results reveal something important about the current state of AI code understanding. Fixing a smart contract vulnerability requires:
- Understanding the vulnerability — what state transition is being abused?
- Understanding the intended behavior — what should this code actually do?
- Preserving invariants — does the fix break any downstream logic?
- Not introducing new bugs — reentrancy guards can create new DoS vectors; access control changes can lock out legitimate callers.
Current models struggle with (3) and (4). They can identify and fix the vulnerability in isolation, but smart contracts are systems of interacting components. A fix to a vault's withdrawal logic might break its integration with a yield aggregator that depends on specific callback ordering.
This is exactly where formal verification tools like Halmos, Certora, and hevm become essential complements. EVMbench + formal verification could form a powerful pipeline: AI finds candidate patches, formal verification proves they don't break invariants.
What This Means for Security Researchers
The Defender's Advantage (For Now)
EVMbench is explicitly framed as a defensive tool. OpenAI is betting that if AI agents can exploit 71% of known bugs, defenders should be using those same agents to find bugs before deployment.
The practical workflow:
1. Pre-audit: Run AI agent in detect mode against your contracts
2. Triage: Have humans verify flagged issues
3. Patch: Use AI-suggested fixes as starting points
4. Verify: Run exploit mode against your patched code
5. Formal verify: Use Halmos/Certora to prove invariants hold
This doesn't replace human auditors. It makes them faster and gives them better coverage of the "long tail" of vulnerability patterns.
The Attacker's Reality
The uncomfortable truth: attackers don't need 71% accuracy. They need to find one exploitable bug. And they don't need to patch anything — just drain and bridge.
The time from vulnerability discovery to exploit execution is collapsing. When a model can go from "here's a contract" to "here's a working exploit" in minutes, the window between code deployment and potential exploitation shrinks to near-zero.
This makes pre-deployment auditing more critical than ever, and it strengthens the case for:
- Bug bounties with competitive payouts
- Formal verification as a standard requirement, not a premium add-on
- Monitoring and circuit breakers (Forta bots, OpenZeppelin Defender, Tenderly alerts)
- Timelocks and gradual rollouts for contract upgrades
EVMbench's Limitations (And Why They Matter)
The benchmark has several structural constraints worth noting:
- Single-chain only — No cross-chain exploits, which are increasingly common (think Wormhole, Ronin-style attacks)
- Clean state — Anvil instance, not a mainnet fork. Real exploits often depend on existing on-chain state (liquidity depth, oracle prices, governance parameters)
- Sequential transactions — No timing-dependent attacks like sandwich attacks or block-stuffing
- Known vulnerabilities — All bugs are historical and documented. Zero-day discovery is a different (harder) problem
- No mock detection — If an agent finds bugs humans missed, EVMbench can't tell if they're real or false positives
These limitations mean EVMbench likely understates the difficulty of real-world smart contract security while simultaneously understating the potential of AI to find novel vulnerability classes.
The Bigger Picture: AI Arms Race in DeFi Security
EVMbench arrives in a DeFi security landscape that's rapidly evolving:
- Q1 2026 alone has seen the Venus Protocol supply-cap bypass ($3.7M), the Solv Protocol exploit ($2.7M), the Bonk.fun domain hijack, and the Aave $50M slippage incident
- Attack vectors are shifting from pure smart contract bugs to operational security failures (compromised keys, DNS hijacks, supply chain attacks)
- The Solana ecosystem faces unique challenges with its account model that EVM-focused tools like EVMbench don't address
OpenAI is pairing EVMbench with concrete defensive investments:
- Aardvark — their security research agent (private beta)
- $10M in API credits for cyber defense, targeting open-source and critical infrastructure
- Expansion of their Cybersecurity Grant Program
Practical Takeaways
For protocol teams:
- Start integrating AI-assisted auditing into your CI/CD pipeline today
- Use EVMbench as a test suite for evaluating security tooling vendors
- Don't rely on a single audit — layer AI scanning, human review, formal verification, and runtime monitoring
For security researchers:
- Learn to use AI agents as force multipliers, not replacements
- Focus on the areas AI is weakest: cross-contract interaction bugs, economic design flaws, governance attack vectors
- Contribute to EVMbench — it's open source on GitHub
For the ecosystem:
- The 71% exploit rate means sophisticated attacks will become cheaper and more accessible
- Invest in defense-in-depth: circuit breakers, timelocks, monitoring, and incident response playbooks
- The next wave of DeFi security isn't just about finding bugs — it's about speed of response when AI-discovered exploits go from zero-day to weaponized in hours
EVMbench isn't just a benchmark. It's a signal that the AI security arms race in DeFi is accelerating faster than most protocol teams are prepared for. The 71% exploit rate is today's number. Six months ago it was 33%. Six months from now?
Start defending accordingly.
This article is part of my DeFi Security Research series. Follow for weekly deep dives into vulnerabilities, audit tools, and security best practices across EVM and Solana ecosystems.
Top comments (0)