DEV Community: Gilad Haimov

Ship the parachute in v1: why upgradeability and disaster recovery are not v2 features

Gilad Haimov — Thu, 08 Oct 2026 05:27:14 +0000

Your v1 smart contract is not done until it can survive you.

One thing that always surprises me, and it happens more and more as the industry shifts to AI dev: the ease with which new teams omit upgradeability and disaster recovery features from their v1 smart contracts.

To be clear, these features are usually on the roadmap, right next to the audit and the bug bounty and all the other things that will happen after launch. The omission is in v1 itself, the version that actually goes onchain, takes real deposits, and gets attacked first.

It’s easy to see why this happens. A v1 contract without a proxy, without a pause switch, without an owner, is simpler. It demos better. It feels cleaner, more in the spirit of immutability. And when the code is generated rather than written (the default these days for most young teams), the recovery features are the first thing to disappear, because they are invisible in the happy path. An AI model generating a vault will give you deposits, withdrawals, and a yield strategy. It will rarely volunteer a circuit breaker. The demo works without one.

The data on AI generated code is not reassuring here. Veracode’s 2025 GenAI study found that across more than 100 models, roughly 45% of AI generated code introduced a vulnerability from the OWASP Top 10. That is not a fringe tool producing bad output. That is the average of the field, on the kind of task teams now hand to an agent without a second read.

And the consequences have started to show up with names and dates attached. In February 2026, Moonwell lost about $1.78M when an oracle misconfiguration valued cbETH at $1.12 instead of roughly $2,200, and the exposed position drained in about four minutes. A security researcher who looked at the faulty pull request flagged that it appeared to be co-authored by a frontier AI model. That is one researcher’s public read, not an official post-mortem finding, and whether a model actually caused the bug is not something anyone outside the team can prove. What is not in dispute is the shape of it: the code looked right, it compiled, it got merged, and the money was gone before a human made a decision. That is the environment your v1 now ships into.

Speed is fine. I use AI in my own work on a daily basis. But it raises the stakes on the one decision that is almost free at deploy time and almost impossible to change later: whether your v1 can be fixed, paused, or unwound when, not if, something goes wrong.

The tombstones

A few stories every founder should know before choosing an immutable v1.

Parity, 2017. A user initialized the shared library contract behind Parity’s multisig wallets and then triggered its self destruct. Every wallet that depended on that library froze instantly. 513,774 ETH across 587 wallets, locked, not stolen, just permanently unreachable, which from the owners’ perspective is the same thing. A proposal to recover the funds, EIP-999, was put to the community and never adopted. The money is still frozen today. There was no pause, no upgrade path, no recovery function. There was nothing to argue about, because there was nothing anyone could do.

The DAO, 2016. The DAO lost about 3.6M ETH, roughly $60M at the time, to a reentrancy bug in its split function. It had no pause and no upgrade mechanism either. The only recovery available was to split the entire Ethereum chain in two, which is how we got ETH and ETC. When your disaster recovery plan requires a hard fork of the underlying blockchain, you do not have a disaster recovery plan.

Audius, 2022. This is the one worth getting right, because the usual retelling is wrong. It was not simply a forgotten storage gap. Audius used a custom upgradeable proxy whose admin address sat in storage slot 0, the same slot OpenZeppelin’s Initializable uses for its initialization flag. The collision made the init guard readable as already satisfied, the contracts could be reinitialized, and an attacker delegated a large block of governance tokens to themselves and pushed through a malicious proposal. About 18.56M AUDIO left the treasury, reported around $6M. Two prior audits had not caught the slot collision. The lesson is not “add a gap.” The lesson is that storage layout under a proxy is load bearing, and it fails silently.

Kinto, 2025. An exploit drained roughly $1.55M, a modest number next to the others on this list. Kinto wound down within weeks. That is the part founders underrate: a protocol does not need a nine figure loss to die. It needs a loss it cannot absorb, explain, or recover from, arriving before it has the balance sheet or the trust to survive one. Recoverability is not only about getting funds back. It is about still having a company on the far side of a bad week.

I‘ve written genesis contracts for an L1 and run protocol upgrades on a live EVM mainnet, with real value on the other side of the transaction. Genesis is v1 in its purest form: no previous version, no migration, no users to coordinate with, and whatever you bake in at block zero, the network inherits. Nobody who has operated a live protocol ships an immutable v1 on purpose. You learn exactly once what it feels like to know a mistake cannot be walked back, and you never design that feeling into a system again.

The v1 checklist

Here is what I would not launch without. Every item is cheap at deploy time. Most are impossible to retrofit.

I highly recommend your team to either implement all of them in your v1 or have a very good reason why not to do so.

1. Upgradeable from day one, UUPS, and the owner is the upgrader

Use a proxy. Specifically, use UUPS (ERC-1822) unless you have a concrete reason not to.

UUPS keeps the proxy to a minimal ERC-1967 shell and puts the upgrade logic in the implementation, where your access control already lives. There is no separate ProxyAdmin contract to deploy, fund, and keep track of, and no admin versus user selector ambiguity to reason about. OpenZeppelin has moved its own recommendation this way: its current docs describe UUPS as the lightweight, versatile default and note that the Transparent pattern carries its admin logic in the proxy and costs more to deploy. I am going to skip the exact gas-per-call numbers you will see quoted around this, because they depend on library versions that keep changing and the per-call difference against a current Transparent proxy is smaller than the folklore claims. The durable reasons to prefer UUPS are a smaller deploy footprint and one less contract in your trust surface.

There is also one option most teams miss. Because the upgrade function lives in the implementation, a later version can remove it, and the contract becomes immutable by choice after it has earned stability, instead of immutable by accident on day one. Treat that removal as a deliberate, final act. Ship it by mistake and you can no longer upgrade the contract. That is not the same catastrophe as funds frozen, the business logic keeps running, but it is a self inflicted dead end with no reason to exist.

The two rivals, briefly. Transparent proxies still work and are a defensible choice. Diamond (ERC-2535) is genuinely powerful for systems that outgrow the contract size limit, but it is a lot of machinery and audit surface, and rarely the right v1 for a small team. Beacon proxies make sense when one upgrade must hit many identical contracts at once, and that shared blast radius is also their risk.

Now the part teams get wrong. In UUPS, the gate to the whole system is one function:

function _authorizeUpgrade(address newImplementation) internal override onlyOwner {}

Force the upgrade authority to be the owner address. Not a separate upgrader key kept on a developer’s laptop, not an address nobody remembers assigning, not a role granted during testing and never revoked. The address that owns the contract is the address that can upgrade it: one key set to protect with a multisig and a timelock, one address whose compromise is already your worst case, so the upgrade path adds no new worst case of its own. Every additional privileged address you invent is an additional key that can be lost, leaked, or left behind by someone who quit. Put bluntly, v1 should not have two gods.

Two more UUPS rules that are not optional:

Never ship an upgrade that removes or breaks the upgrade function, unless that removal is the deliberate final step above. OpenZeppelin’s tooling has compatibility checks that catch most of these; keep them on.
Test the upgrade path on a fork before mainnet. An upgrade mechanism that has never been exercised is a hypothesis, not a capability.

2. The constructor trap: in upgradeable contracts, the constructor does not run

This one deserves its own section because it produces a specific, repeatable, silent bug, and generated code walks into it constantly.

When you deploy an upgradeable contract, two different things exist: the implementation, deployed once, and the proxy, which holds the state and which users actually interact with. The constructor runs exactly once, on the implementation, against the implementation’s own storage. It never runs for the proxy. Every line of setup you put in a constructor, assigning an owner, linking a dependency, recording a configuration value, happens in storage nobody uses, and the proxy starts life with all of it unset.

That is why upgradeable contracts initialize through an initialize() function guarded by the initializer modifier: it is ordinary function code, executed through the proxy, writing to the proxy’s storage.

Now the part that bites even careful developers: inline variable initialization is constructor code. When you write

uint256 public feeBps = 250;

the compiler places that assignment in the constructor. In a plain contract it works fine and reads as clean, idiomatic Solidity, which is exactly why models emit it so reliably, since most of their training examples are plain contracts. Behind a proxy, feeBps reads 0 until initialize() sets it, and nothing reverts, nothing warns, nothing looks wrong in the demo. Your fee is silently zero. Or your treasury is the zero address. Or your cap is unlimited. The one partial exception is immutable variables, which live in the bytecode rather than storage and do carry over through the proxy. They are fine for true constants and useless for anything that must differ per deployment.

The rule set is short enough to enforce in review:

In an upgradeable implementation, the constructor contains one thing only: _disableInitializers(). That call locks the implementation so nobody can initialize it directly and seize it, the hole that in 2021 put real money at risk at KeeperDAO and Rivermen and in 2022 earned a Wormhole researcher a $10M bounty, the largest ever paid at the time, for an uninitialized implementation. Note what that call does not do: it does not initialize your proxy. Locking the implementation and initializing the proxy are two separate jobs, and treating them as one is how uninitialized-proxy incidents keep happening.
All real initialization lives in initialize(), behind the initializer modifier, setting every variable explicitly, including the ones with obvious defaults.
Any non trivial logic in a constructor, wiring contracts together, seeding state, approving spenders, is a review blocking finding in an upgradeable codebase. Full stop.

3. Storage discipline: the gap, or namespaced storage

Upgradeable contracts share one storage layout across all versions. If a base contract gains a variable in v2, every variable below it shifts, and the new implementation reads garbage where user balances used to be. This failure mode is silent. No revert, just wrong data. It is the same family as the Audius collision above.

The classic defense is a reserved gap at the end of every upgradeable base contract:

uint256[50] private __gap;

Add a variable in v2, shrink the gap by the slots you consumed. The modern alternative is ERC-7201 namespaced storage, which is what OpenZeppelin Contracts v5 uses for its upgradeable variants: each module keeps its state in its own computed location, so collisions between modules are structurally unlikely. Either discipline works, and a carefully managed append-only layout can even be kept compatible without a reserved gap. What does not work is having no discipline and hoping. Pick an approach on day one, enforce it in code review, and add a storage layout check to CI so a layout breaking pull request fails the build instead of failing on mainnet.

4. Ownership changes: two-step, always

If there is one item on this list I would tattoo on the deploy checklist, it is this one, because it defuses the single most irreversible mistake in an ownership change.

A one step ownership transfer is final in the same transaction. If the new address has a typo in it, belongs to the wrong network, is a contract that cannot sign anything, or is simply an address nobody has the key to, the transfer still succeeds. The old owner is gone, the new owner does not exist in any practical sense, and every capability on this page dies with that one fat fingered transaction: no upgrades, no pause, no rescue, no parameter changes, forever. The protocol keeps running, user funds and all, with an empty chair where the admin used to be. It is the Parity freeze, self inflicted, in one line of code.

Two step ownership (OpenZeppelin’s Ownable2Step) removes that failure class. The current owner nominates a new owner. Nothing changes yet. The new address then has to actively accept ownership, from the new address, proving it exists, is controlled by someone, and can sign. Only then does the transfer complete. A wrong nomination becomes a recoverable embarrassment instead of a permanent loss. The cost is one extra transaction.

Be honest about what it does not do. Two step ownership protects you from handing control to an address nobody holds. It does nothing against a compromised owner key: an attacker holding that key can nominate themselves and accept, all in house. It is insurance against the fat finger, not against theft. Compromised-key survival is a separate design, and it is the next item.

Related, and simpler: never call renounceOwnership on a live v1. Renouncing is marketed as decentralization and finality. What it actually is, at v1, is throwing away the parachute to prove you trust the plane.

5. Every privileged address must be replaceable

Ownership is not the only privileged address in a real system. There is the guardian who can pause, the oracle that feeds prices, the treasury that receives fees, maybe a keeper or a strategist role. Every one of those addresses must be changeable by the owner, because addresses fail in two ways: keys get compromised, and keys get lost.

Run the scenarios on an address that cannot be changed. A compromised guardian key means an attacker can pause your protocol whenever they like, forever, a permanent denial of service you cannot repair. A merely lost key, nobody’s fault, no malice anywhere, produces the Parity outcome at the admin layer: capabilities that exist on paper and can never be exercised again.

So the rule is mechanical: any address the contract treats as privileged gets a setter, the setter is owner only, the setter emits an event (item 8), and for the owner itself the change goes through the two step flow (item 4). The context is not theoretical. Hacken’s data for the first quarter of 2025 put roughly $1.63B of about $2B in stolen funds down to access control failures. The admin layer is where the money actually goes, and key rotation is how you respond when one of those keys goes bad.

6. Granular pause: fast to stop, slow to restart

A circuit breaker is the difference between an incident and a post mortem. When something looks wrong, the owner, or a faster guardian, must be able to stop the funds: halt deposits so new money stops entering a broken system, and halt withdrawals or borrows so a drain in progress stops moving.

Make the switches granular. Pausing inflows and pausing outflows are different decisions with different costs. Pausing deposits during a scare costs your users a delay. Pausing withdrawals traps their money with you, so it is the nuclear option, justified when funds are actively leaving through a bug and not as a response to a rumor. A single all or nothing pause forces you to choose between doing nothing and doing the maximum. Two switches let you respond at the size of the actual problem. One caution: a pause aimed at the wrong function can block safe exits while the exploit keeps running, so pause the operation that is actually causing harm, and test that the safe routes stay open.

Split the authority by speed as well. Pausing should be fast: a dedicated guardian role that can act in one transaction, no timelock, because attackers move in minutes. Unpausing should be slow and deliberate, owner only. Two details get overlooked. The pause must carry no cooldown or usage limit that locks you out of pausing again in the middle of a messy incident. And it must be reachable by whoever happens to be awake, because a pause you cannot reach at 3am is decoration.

In November 2023, Aave’s guardian paused a set of V2 markets and froze several V3 assets after an issue with one feature was reported. The incident closed with a sentence every protocol wants to be able to write: no user funds were lost. That sentence was possible because the pause existed before it was needed.

7. A timelock on upgrades and critical parameter changes

An upgrade that executes the moment it is signed is invisible until it lands. Put upgrades and sensitive parameter changes behind a timelock, OpenZeppelin’s TimelockController or a Safe with a delay module. 48 hours is a common minimum.

The delay does two jobs. It gives your own team a window to catch a bad, or malicious, proposal before it executes. And it gives users who disagree with a change time to exit. A protocol whose users can leave before a change lands is a protocol users can trust with the next one. Choose the length deliberately, though, against your exit mechanics: if users need three days to unwind a position, a two day upgrade delay is not really an exit window, and if withdrawals can be paused, the promise needs even more scrutiny.

At CoreDAO I built a local fee market at the network layer. Live parameter systems teach humility fast: the values you launch with are wrong in ways you only discover under real load. A system whose parameters can only change instantly and silently is a system designed by someone who believes their first guess was perfect.

One pairing note: the emergency guardian from item 6 must not share this delay, and must not hold any power beyond pausing. An emergency path that can also move funds or upgrade instantly defeats the entire timelock.

8. A detailed event on every set and every fund action

Every setter, every role grant, every privileged address change, every pause, every upgrade, every fund movement the contract initiates: emit a detailed event, with the actor, the old value, the new value, and the amounts involved.

A word about the name, because the name causes real damage. “Event” is a poor name for what this is. It sounds like a notification, a convenience for the frontend team, the first thing to cut when a deadline moves. What an event actually is: a record written into the transaction log, stored on chain, that contracts cannot read back as ordinary state. It is written for the outside world, and the outside world is everyone who will ever have to understand your system under pressure. Indexers read events. Monitoring tools alert on them. Block explorers display them. Auditors reconstruct behavior from them. During an incident, when the question is what happened, in what order, initiated by whom, events are the difference between an investigation and archaeology. A setter without an event is a silent configuration change to a live financial system. A fund action without an event is a movement of other people’s money with no receipt.

Events are not your only forensic trail, and you should not treat them as proof on their own. A malicious implementation can simply decline to emit the record you expected, so monitor balances and critical state as well. But as a baseline, a contract that narrates its own privileged actions is one you can actually operate under fire.

9. A scoped way to extract excess and stuck funds

Tokens get sent to contracts by mistake. Fees accumulate where your accounting did not plan for them. Airdrops land on your contract address. Without a rescue function, that value can be locked forever, which helps nobody, including the users who will beg support to retrieve tokens they sent straight to the contract.

So include an extraction function, owner only, that can move tokens or ETH the contract holds. The scoping is the entire point. An unrestricted “send everything to the owner” function is a rug vector, and auditors and users will read it as one, correctly.

But be careful how you define excess, because a token balance is not automatically surplus. If users deposited 100 units and the contract holds 110, the extra 10 might be shareholder yield, backing for a queued withdrawal, collateral, or earned protocol fees. The balance alone does not tell you who owns it. A formula like “balance minus recorded liabilities” is only meaningful if those liabilities are complete, current, and measured in the same units, and that is a real accounting model, not a one liner. Where you can, refuse outright to touch the core asset, the way Compound’s market contracts reject a sweep of their own underlying token. Separate accidental-token recovery from fee withdrawal from emergency migration, give each its own permissions and destination rules, and log every sweep with an event so the extraction is public the moment it happens.

10. Rate limits on value outflow

Assume a bug gets exercised before anyone notices, and design so the damage is bounded: caps on withdrawals or borrows per block, per hour, or per day, sized to normal usage with headroom.

Moonwell is the cautionary detail here, and it cuts against the lazy version of this advice. When the oracle mispriced cbETH in February 2026, the team did reduce caps toward zero, but that happened after the bad debt had already formed, in a window measured in minutes. A cap that a human sets during the incident is usually too late. A limit that the contract enforces automatically, before anyone is awake, is what actually bounds the loss. A bug that can drain a contract in one transaction is a total loss. A bug that hits a standing rate limit is an incident you can pause your way out of. There is a standard for this pattern now, ERC-7265, for teams that want it off the shelf.

One small bonus, thirty seconds of work, that ends a class of confusion:

function version() external pure returns (string memory) { 
    return “1.0.0” 
}

Frontends, monitors, and your own incident channel can read it to orient fast. Just remember it is a label, not proof: a buggy or malicious implementation can return any string it likes, so the real record of what is deployed is the implementation address and its code hash, kept in your operational notes.

11. Rehearse the recovery while the contract is still empty

Everything above is capability. This item is the one that turns capability into a plan, and it is the one teams skip.

Before launch, run the whole recovery on a fork with realistic balances, queued claims, and the production permission set. Trigger a representative failure. Have the guardian pause the affected route. Rotate an operator key. Execute the upgrade through the real multisig and timelock path, not a shortcut. Run the migration. Then check that balances, user claims, and permissions all survived, that an unauthorized caller is rejected, that the rotated-out key can no longer do anything, and that a single missing signer does not make the procedure impossible.

Two things this rehearsal will teach you that no amount of reading will. First, the missing permission or the circular dependency, the one where replacing a lost key turns out to require that same key to sign, shows up here instead of during an incident. Second, do not assume rollback means pointing at yesterday’s implementation. After a migration has changed what the stored data means, the old code may no longer understand it, and recovery can require another forward upgrade. Discover that on the fork.

The test I keep coming back to is simple. Before the first real deposit, put one question to the launch review: if we find a serious bug tomorrow, show me the exact transactions we would send, who can send them, and the evidence that users’ claims survive. If that answer does not exist yet, your v1 is not finished, however well the happy path demos.

The honest counterargument

Upgradeability is a trust assumption. Admin keys are an attack surface. A compromised owner key on an upgradeable, pausable contract is genuinely dangerous, and admin security is operational, not just contractual: the largest theft of 2025, Bybit’s roughly $1.46B, came through a compromised signing interface, not a contract bug. All true.

None of it changes the conclusion, and the mitigations are standard and listed above: a multisig owner, a timelock on upgrades, guardian-only pause, replaceable privileged addresses so a bad key can be rotated out, events on everything so admin actions are public the moment they happen, and signing discipline for the humans behind the keys. An admin mechanism can itself fail or be the attack path, so it has to be designed with the same seriousness as the rest of the protocol. That is an argument for building it well, not for leaving it out.

Run the comparison honestly. The alternative to a recoverable contract is not a trustless contract. It is an unfixable one. Immutability does not remove trust. It removes your ability to respond when code written by humans, or generated by models trained on humans’ code, under deadline, turns out to be wrong. In v1, it will be wrong somewhere. The only question is whether you find out during a controlled upgrade, or during a drain, with 513,774 ETH of precedent for how the second one goes.

The rule

You pack the parachute before the jump, not after the plane develops a problem. Recovery and upgrade capabilities are not features you add once the product succeeds. They are what lets the product survive long enough to succeed.

Build them into v1.

I’m Gilad Haimov, former lead smart-contract and protocol engineer at CoreDAO. I build EVM protocols and run pre-audit reviews at SmartRev. Shipping a v1? Happy to take a look: LinkedIn

What an ERC-4626 price test actually shows

Gilad Haimov — Mon, 14 Sep 2026 07:13:00 +0000

Vaults, shares, and price

ERC-4626 gives tokenized vaults a common interface for deposits, withdrawals, and share accounting. A user deposits an ERC-20 token and receives shares representing a claim on the vault's underlying holdings. The vault might lend those assets, stake them, or put them into another yield strategy. Users can later redeem their shares and, where the vault allows it, transfer them to someone else.

In a simple proportional model, a user's shares divided by total shares, multiplied by the vault's assets, gives their claim on those assets. Actual redemption amounts also depend on the implementation, including rounding, fees, and any virtual assets and shares used in the calculation.

I called this project "Vault Price Honesty" because I wanted to examine what those exchange rates mean for the people using a vault. The name needs a little care: an unexpected price change is not enough to call the accounting wrong. A direct donation, for example, can raise the value of existing shares without harming a later depositor.

I focused on five situations:

A tiny first deposit followed by a donation. Someone acquires a small number of shares, then transfers tokens directly into the vault. The resulting exchange rate can leave the next depositor with very few shares, or none. Whether the first depositor profits needs a separate calculation.
Assets remaining when no shares exist. This can happen after the last redemption or because someone donates to an empty vault. How the next deposit is priced depends on the implementation. The two situations are worth testing separately.
Direct donations counted as assets. Including donated tokens in the vault's accounting raises the value of existing shares. That can be intended behavior; the question is what follows from it.
Preview and execution disagreeing. Previews have specific requirements under ERC-4626. A difference needs to be checked against those requirements before it is called a violation.
Reported assets and assets held at the vault address differing. A strategy may hold assets elsewhere, so the difference can be legitimate. Stale or incorrect valuation is a separate concern.

Following the tokens

I started this project intending to build a reusable, hardened ERC-4626 base contract. Working through the cases made the scope less straightforward than I expected. Donation accounting, strategy valuation, and the treatment of assets remaining after the last redemption depend on the vault's design. Shared protections are useful, but they do not settle all of those choices.

I shifted the project toward a Foundry suite for examining that behavior. One local test made the reason particularly clear.

A user deposited 50 tokens and received zero shares. Before that deposit, an attacker had deposited one base unit of an 18-decimal token, then transferred 10,000 tokens directly into the vault. This fixture uses an OpenZeppelin-based vault with a decimals offset of zero.

The depositor's loss was clear. The attacker's position needed another calculation.

Participant	Contributed	Redeemable	Net
Attacker	10,000 + 1 wei	Approximately 5,025	Approximately −4,975
Depositor	50	0	−50

These are underlying-token amounts in a local fixture, excluding gas. The attacker's initial deposit was one base unit, not one whole token.

OpenZeppelin's virtual asset and share accounting explains the result: part of the donation cannot be recovered by the attacker. In this particular sequence, causing a depositor to lose 50 tokens costs the attacker roughly 4,975 tokens.

The depositor suffered a complete loss, but this sequence did not generate an attacker profit. It demonstrates griefing. It does not establish the economics of other donation amounts, multiple victims, or a different vault implementation.

The local inflation test also includes an OpenZeppelin-based vault with a decimals offset of three. With the same initial attacker deposit, donation, and victim deposit, the victim receives shares redeemable for approximately 45.02 tokens. That avoids the zero-share outcome, but still leaves the victim almost 10% short.

A test asserting only that the victim received shares would miss that loss. Comparing raw share counts between the two vaults would also be misleading because the offset changes share precision. The useful comparison is the amount of underlying assets those shares can recover.

Another local case starts with zero shares and 500 donated tokens already in the vault. The next user deposits 1,000 tokens and receives shares redeemable for 750. Here, too, a nonzero share balance conceals a substantial loss. This setup deliberately donates assets to an empty vault; it should not be confused with a normal redemption leaving a small rounding remainder.

Interpreting the other measurements

Direct donations require more care. In the recorded fork run, transferring underlying tokens into sUSDe increased its reported assets and share value. That observation alone does not establish depositor harm. A higher price per share can mean that each share represents more assets.

To make a stronger claim, I would need to follow a specific consequence: a depositor's round trip, a fee calculation, or an integration using that price as collateral value. The donation test does not demonstrate those outcomes.

The same caution applies to totalAssets(). Assets deployed through a strategy need not appear in the underlying token balance at the vault address. A difference between the two numbers is a starting point for examining the accounting. Establishing an error requires understanding what the vault owns and how it values those positions.

Preview checks have a more explicit reference: the ERC-4626 specification. When a preview is followed by the corresponding operation in the same transaction, under unchanged conditions, deposit and redeem must return at least their previewed amounts; mint and withdraw must require no more than theirs. The standard also requires previews to approximate the operation as closely as specified. Simple equality with convertToShares or convertToAssets is not a general substitute, since previews account for factors such as fees.

These checks need evidence from execution, including balance changes. A returned number is insufficient if the corresponding assets never arrive.

What I would use the report for

The local fixtures make particular sequences easy to inspect. Fork tests add real accounting structures and operational restrictions. Their conclusions remain limited to the paths actually exercised: the live tests cover deposit previews and redeem where possible, rather than all four operations. Empty-vault scenarios stay local for the populated vaults in the sample. An unsuccessful run leaves a coverage gap; it does not identify the cause of failure.

For a developer reviewing a result, I want the report to answer three questions: what operation ran, what changed, and which conclusion those measurements support. A standards violation, a demonstrated economic loss, and a design-dependent behavior each call for a different response.

The most useful result from this work was the small table above. It shows both the depositor's loss and the attacker's cost. Those two numbers say more about this particular attack than the zero-share result alone.

The Foundry project, tests, and reports are on GitHub. The suite is a focused investigation tool, not a comprehensive audit.

False Completion Is the Real Failure Mode of Coding Agents

Gilad Haimov — Thu, 03 Sep 2026 07:22:22 +0000

A deliberately strict process for preventing false completion in autonomous software development

In an earlier article, I argued that the model should have to earn every agent in an agentic system. One agent is a valid answer. Every additional boundary has to justify its cost.

This article starts one level lower. Assume we have selected the topology. Who gets to decide that the resulting product is actually the product we asked for?

After experiencing dozens of AI-development efforts and reviewing the results of many more, I no longer think the main danger is that the development procedure will manifestly fail. A crashed build, a red test or an explicit error report is relatively easy to deal with. The system has admitted that it has a problem.

The far more serious failure is that of FALSE COMPLETION: the AI system believes it has accomplished the development goal, reports success with complete confidence, and delivers a product that has little to do with what the system design had in mind.

The code may compile and the interface may look polished. Yet core flows are missing, persistence is half-wired, or the implementation has quietly solved a simpler problem than the one it was given.

And this is not merely anecdotal. An analysis of over 20,000 coding-agent sessions identified inaccurate self-reporting and misreading developer intent as recurring forms of developer-agent misalignment. Most visible resolutions still required explicit human correction. The agent’s confidence was not evidence that the job was done. The study is worth reading.

My response is a deliberately strict development process. It costs more tokens and wall-clock time than letting one agent run until it declares victory. I deliberately pay that cost. For substantial autonomous builds, this is now the only way I work. A failed process is visible. A successfully completed wrong product may survive until a user discovers it.

1. Make the product book executable

Before any code is written, the product book, call it a PRD, specification or product contract, must be reviewed by an AI agent whose task is not to improve the prose. Its task is to attack the design.

Can every important requirement be converted into observable behavior? Are the user flows, data rules, failure behavior and non-functional requirements clear? Is anything important hiding behind words such as “fast,” “intuitive” or “secure” without a measurable meaning?

The objective is not a specification that deterministically produces one implementation. Many implementations may be valid. The objective is a specification that lets us determine, with as little interpretation as possible, whether an implementation is acceptable.

The product book must also define kill conditions. If an acceptance condition cannot be tested or two requirements contradict each other, the correct output is not heroic improvisation. It is STOP.

2. Lock your acceptance tests

The acceptance tests derived from the product book must be fixed before implementation and remain immutable throughout the development run.

The coding process may run them and may fail because of them. It may not change, weaken, delete or reinterpret them.

This is intentionally more rigid than ordinary iterative development. If we later discover that a test cannot be met or that the test itself is wrong, the process stops. The product book and acceptance set are updated through a separate decision, their version changes, and execution restarts from the specification and planning boundary. The finish line is not quietly moved by the participant currently failing to reach it.

Locked does not mean sacred. Tests can be wrong. OpenAI found difficult SWE-bench Verified tasks where tests rejected valid solutions or required behavior absent from the problem statement. That is why changing a test should be an explicit design event, not an opportunistic edit inside the coding loop. OpenAI’s audit shows how dangerous a bad oracle can be.

3. Break into steps before any code is written

The complete task must be broken into discrete execution steps before code is written. Each needs defined inputs, outputs, dependencies and review conditions, and must fit comfortably inside the model’s usable context, not merely its advertised context window.

This may be the single most important protection against an AI-development effort going sideways. Long contexts accumulate abandoned hypotheses, obsolete instructions, failed repairs and irrelevant tool output. Eventually the agent spends its reasoning budget navigating its own history.

Small should not mean arbitrary. Splitting one tightly coupled behavior across five agents simply moves the difficulty into integration. A good boundary ends in a coherent change another agent can verify without reconstructing the entire project.

4. Assign each step a new coder instance

Using one coding agent and looping it through every step looks compelling. The developer of step five already knows what happened during steps one through four, and fewer tokens are spent bringing it up to speed.

But the same accumulated context also contains every wrong assumption, abandoned path and local compromise made along the way. Anyone who has executed long-running agent tasks has seen performance deteriorate as the context shifts from one problem to another.

For non-trivial development, I want a fresh coding context for every step. Continuity should come from explicit artifacts: the product contract, architecture decisions, interfaces, git history, accepted outputs, open risks and a short task packet. It should not depend on a long conversation correctly remembering which parts are still true.

Anthropic’s recent experiments suggest that stronger models can sometimes sustain longer builds, allowing sprint decomposition to be removed and evaluation to happen less frequently. I understand the argument. My own experience leads me to a more conservative default: I would rather pay the visible cost of fresh contexts than accept the invisible risk of context corruption and false completion. Their current reasoning and trade-offs are described here.

5. The builder never grades his own work

The coder responsible for a step must never be the authority that approves it.

When the work is ready, the coder hands the diff, task contract and evidence to a fresh reviewer. The reviewer should not inherit the coder’s reasoning. It should inspect the result on its own terms, preferably with read-only access and the ability to run the system.

Where practical, I prefer a reviewer from a different model family. A different model is not automatically better, but it is less likely to reproduce exactly the same assumptions and blind spots. Deterministic tests remain a stronger oracle than either model; the second model exists to challenge what the tests do not express well.

The reviewer compares the change with the product intent, architecture, regressions and end-to-end behavior. It reports material gaps, not style objections manufactured merely to demonstrate activity.

Here current guidance and my experience agree: Anthropic recommends adversarial review in a fresh context so the evaluator sees the result rather than the story that produced it. See its current Claude Code guidance.

6. Final acceptance tests must be invisible to coder

This is probably my most controversial rule: the coder should not have read access to the final acceptance-test implementation.

It must know the required behavior. A hidden test may never demand a private method name, particular internal structure or functionality absent from the product contract. But the exact executable suite should belong to the control plane, not to the participant being evaluated.

The coder may write unit tests and receive visible examples or contract tests. What it cannot do is inspect the final oracle and shape the implementation around its precise assertions.

Agents are no better than us in this respect. Give the builder direct sight of the scoring mechanism and the target can quietly shift from “build the intended product” to “make these checks turn green.” The protected suite should therefore be largely black-box. Failures should identify the violated requirement without exposing or surrendering control of the examination.

7. Place hard cap on coder/reviewer cycles

It is legitimate for a skeptical reviewer, especially one using a different model, to reject a step and request repairs. That is the process doing its job.

But the loop must be strongly bounded. My red line is around four coder/reviewer attempts, and I stop earlier if the same defect returns twice. By then, the problem is probably not a missing if statement. Requirements may conflict. The step may be too large. The technology may be at its limit. The acceptance test may be wrong. Or the accumulated repair dialogue may already be corrupting both agents' judgment.

What I do not want is endless negotiation until the reviewer becomes tired, confused or sufficiently conditioned to approve bad code.

When the budget is exhausted, the reviewer produces a failure report: what remains wrong, the evidence, what was attempted, which assumptions are doubtful, and whether the likely fault is in implementation, decomposition, architecture, specification or the test oracle. Control returns to planning.

Failure is an acceptable result. Counterfeit success is not.

“Done” must be a system state

These rules lead to one architectural principle: the agent does not get to decide it is done.

Every feature begins in a failing state. A coder may propose completion, but only protected tests and an independent evaluator can accept it. Every step leaves code, evidence and a clean checkpoint. Integration checks confirm that individually valid steps still form one working product.

This is not the cheapest possible harness, and it is not my recommendation for changing a label or adding one obvious validation. It is my default for substantial autonomous product development, where a plausible but wrong result can consume days of review or reach production.

The current fashion is to assume that better models will let us remove more of this structure. They probably will remove some of it. Anthropic explicitly recommends retesting which pieces remain load-bearing as models improve. That is sensible engineering.

My own bias is to remove safeguards only after repeated evidence that they no longer catch meaningful failures. Model capability is improving quickly. The ability of a confident agent to make a wrong product look finished has not disappeared.

So let the model write the code. Let it plan, search, test and repair. Give it real autonomy inside clear boundaries.

But do not let it define the finish line, rewrite the examination, grade its own paper and then announce that it passed.

“Done” is not an agent opinion. It is a state the system must prove.

I am a principal software and protocol engineer. I have written production Solidity since 2017, led protocol-layer engineering on a live EVM L1, and built an AI-assisted audit workflow that I validated against a 219-contract corpus. It is now open source: github.com/giladHaimov/Smart-Contract-AI-Audit-Skill

The agent that writes the code cannot read the tests that grade it

Gilad Haimov — Sun, 23 Aug 2026 08:03:53 +0000

TL;DR:

Letting an AI coder review its own code and tests leads to compromised code quality, test-softening, and false positives. This article introduces a practical, multi-agent governance layer inspired by the Ralph loop but hardened with two core constraints:

The Locked Oracle: The coding agent is completely blocked from reading or editing the test suite, which is frozen before coding begins.
The Cross-Model Checker: Code review and testing are strictly delegated to a different model family to significantly reduce blind spots and self-verification bias.

Press enter or click to view image in full size

The Correcting Loop Architecture: The Director freezes the tests in advance (Locked Oracle). The Builder codes blindly without test access, and a cross-model Reviewer evaluates the diff against human Gates to prevent bias

Developers experimenting with auto AI coding, often hits the wall of self-verification bias, where the agent quietly softens your tests just and happily claims success. I have just shared a loop-based AI builder I’ve been using internally for some time. This builder gives me the best results from anything I have tested before in solving this problem.

The strongest impact that I see is in significantly reducing the ‘faux tests’ problem i.e. tests which are softened so that the app can be marked completed. An additional strong impact is on the PRD compliance, where I now get way fewer occurrences of the AI-dev process ‘forgetting’ parts of the requirements.

It relies heavily on loop concepts for automated AI coding, with some twists that I believe result in a more solid development process. You can basically look at it as a Locked-oracle Cross-model variant of the Ralph loop (an iterative AI-coding pattern).

It also borrows heavily from several other ideas floating around the AI dev sphere and combines them into something that has worked well for me in developing multiple smaller-scale projects.

These ideas, specified below, are not mine! I happily copied them from others, and you’re welcome to do the same with the model I’m sharing here.

Since the AI sphere has not yet reached the stage of naming consensus, I will use the names I believe are most common. I could be wrong.

The ideas that I borrowed

Ralph loop

From Ralph I’ve borrowed the most. Especially the part of looping with a refreshing session context in each unit of work in the loop, and the usage of git commits as both source-of-truth and checkpoints.

I did NOT take from Ralph its eternal loop, since I believe too many code/review iterations are a good indication of a broken process that needs to be stopped and reviewed.

I also did not take its single-agent self-review, which I think is inherently wrong for a valid code review. Instead I used:

Locked Oracle

Here I provide strong limitations to what the rolling dev loop is allowed to access.

The rule is simple: the builder may read all code, even code which appears unrelated to its current task. But it cannot read or modify the frozen acceptance suite used for system verification and later acceptance. Those are written once as part of system design, and are then locked before the first line of application code exists.

By doing this I solve two common problems facing test design:

Test-biased code, where the builder, knowing what the test checks for, creates a non-complete solution that still manages to pass the tests.
Test softening, where the process slowly relaxes the tested conditions to the point where the test no longer presents the hard demand set by the PRD for correctness.

With this addition I have way more assurance that a green test suite actually means the product is behaving as it should, rather than the agent having adjusted its own success criterion.

The lock is not an instruction in a rules file. It’s a deny rule plus a pre-tool hook, evaluated outside the model, on paths the model cannot influence.

# EXIT CODE SEMANTICS — the easiest way to disable this guard accidentally:
# exit 0  allow the tool call
# exit 2  BLOCK the tool call; stderr is returned to the model
# exit 1  does NOT block in Claude Code; the tool call proceeds.
# Using the conventional Unix failure code here silently disables the guard.

Cross-Model checker

Classic Ralph has one agent and no independent verifier. The same agent writes the code, writes the tests, and decides it is done. After using this approach for some time I dropped it. I simply could not rely on its correctness.

Instead, I found it more valid to replace it with a check model where each review/test session is done by a new agent instance belonging to a different model than that of the builder.

Now why is this important?

Because self-verification is the root of evil. The same model that generated the code always carries some ‘blind spot’ into the testing phase, where the considerations and context used to write the code find their way into scoring it. For this reason you would never assign the human coder to be the code reviewer.

Using a different model family (e.g., Claude building, GPT reviewing, or Gemini testing) brings a fresh set of reasoning and training biases, making it way more likely to intercept errors.

The fact that each reviewer agent is restarted, hence provided with clean context, also strengthens our review phase, by preventing ‘context rot’, i.e. a context so bogged with prior tasks that you can no longer rely on it to provide reasonable answers to the current task. Restarting the agents indeed solves this problem, admittedly at the cost of a higher learning curve when introducing each reviewer instance to the current code.

Inner/Outer loop model

The outer loop iterates execution steps to exhaustion, and an inner loop is deployed to verify by fix & trial the correctness of each step’s execution.

I have made multiple attempts not to use this model, which looked overly complex to me, but never managed to get it right using a single loop. I tend to think the reason is inherent: that you cannot really automate dev tasks without these two levels of looping.

Frozen judge model

The catalog is written before a single line of application code exists. That timing is what makes the lock enforceable rather than aspirational. A catalog written after the code exists is already shaped by that code, and by then there is nothing left to freeze.

Differential Agent Skills

This model relies on 3 different entities:

Director for overall orchestration and high-level review
Builder for coding
Reviewer for code review compared to design docs

Each of those is provided with different skills required for their role.

Not every role needs a strong model

The three roles do not need equal model quality, and treating them as if they do is the most expensive mistake available here.

Builder quality matters least. Every defect it produces gets caught by a frozen oracle it cannot influence. A weaker builder writes worse first drafts and burns more repair rounds, but it cannot ship something broken past the judge. That is what makes a cheap builder safe in this setup when it would not be safe in an ordinary one.

Director quality matters most. The Director writes the judge, and nothing downstream checks the judge except a human reading it once. A weak Director produces a shallow test catalog, the loop runs green all the way through, and the gap only surfaces during human testing. There is no automated recovery from that.

Reviewer quality matters in between, and its model family matters more than its raw capability. A reviewer from the builder’s family shares the builder’s blind spots and misses the same cases.

One practical note: don’t pick the builder model by benchmark scores. Measure repair rounds per step on a throwaway project. One or two is fine. Three or more consistently means the model is failing the spec rather than the gate, and a cheap model that needs three passes is not cheap.

The cost saving from this allocation is real, but it’s a side effect. Even if tokens were free, the family separation between builder and reviewer would still be required.

Why this works for me

This model has several benefits that make it great for my needs, which are rapid development of small-to-medium projects.

It audits the PRD before anything else

It runs extensive tests on the product book (PRD) and effectively blocks development until the PRD is corrected to be both self-contained and easily breakable into development steps. In general, ‘AI-dev friendly’.

It contains a double-loop mode

The entire project gets automatically broken into several execution steps so that each step is manageable in a single AI builder session with no significant context window decay.

Note that there’s an inherent plus in giving the AI the task of doing the breaking (I could have done it myself): AI should know best the task length for keeping the context window effective.

After each step completion, the process delegates to a newly created agent of a different model for ‘corrective review’. Corrective in the sense that the process expects the review to return with a changelist and handles the process of review, request-for-changes, new-code-review.

While the decision to force the reviewer to be of a different model than the builder is self-explanatory (you don’t want the job reviewer to think like the worker), the issue of restarting a new reviewer agent for each step is something I was not sure about. There are lots of benefits in keeping a reviewer context up and aware of all aspects of the entire project. At the end I went with restarting the reviewer so as to keep its context window clean, and in doing so got good enough results.

It never weakens a test to make it pass

It forces an aggressive policy of testing each stage AND of never weakening tests just so they pass. A failed test means an action item to be fixed in code.

As part of these testings, the automatic process will deploy a local realistic test env (DB in Docker container, node server, more components may be added) and test runs realistic scenarios locally.

It minimizes human intervention

It is very good at minimizing human intervention in the loop (it was built for it). The only human gates defined here are:

Gate A, where a human approves processed design/architecture docs, project steps, and directory structure.
Gate B, where a human reviews and approves the implementation of the 1st step.
Gate C, where development plus local testing is basically completed and the human reviews the complete system (and in real-world issues a change request…).

Even this level of human intervention can be reduced. Just add a short prompt stating that the agent:

“should work in full automation mode and circumvent any product book review and permission requests but must not reduce the product’s quality, and especially should retain working in steps and performing extensive per-step review by a different model”

Here the agent is basically on its own, so this should be limited to internal/throw projects.

It defaults to boring

The model defaults to a very basic manner of development where the ‘solid simple’ is almost always selected.

Backend will run on Node, DB layer will be an agnostic RDBMS, mobile apps will be written over Capacitor, but if app defs do not allow for it, will fall through to React Native, and only if impossible (e.g. complex animations, extensive mobile OS API needed) will work in native code.

All code, excluding scripts, mobile native if needed, etc., will be written in TypeScript, which adds extra simplicity to the model. It also adds resilience, since in my experience the no. 1 ‘code-level bug’ in AI-generated code is getting the types wrong.

It scales without a rewrite

While keeping it simple, the model was designed to accommodate an increase in app usage at a very low cost.

The backend is instructed to be stateless so as to make horizontal scaling trivial. DB layer is built with the most beneficial indexes already declared and is agnostic, hence should allow switching to a stronger DB with very few changes. And Node itself is very efficient and scale-friendly.

All this is to say: if and when success strikes, you will probably not need to rewrite even for way more users than you started with.

Same pattern, different domain: Solidity auditing

This same pattern is even stronger for smart contract security. Before any pragma solidity\ line, a Director (say Opus) drafts the security spec: reentrancy guards, access control, overflow checks. Now these become the locked test catalog.

Now a cheap builder (say Sonnet) writes the contract. A reviewer from a different model family (GPT-4o) audits with Slither + the frozen spec. The builder never sees the audit checklist, so it cannot soften its own guards.

Used on one helper contract (< 900 LoC), the Reviewer caught a spec mismatch at Gate A that would have cost a full rewrite later.

Two notes before you try it

1. This is a starting point, not a polished product. The core ideas are what I’m sharing — the implementation is yours to adapt. That said, the repo has a demo/\ folder with a working example. Just clone and run them and you should be good to go.

2. Know the ceiling. Works well up to ~20k LOC. Beyond that, context window limits kick in. Extensible — I just haven’t needed to yet.

Get the code: CorrectingLoopManagedFlow

Found this useful? Star the repo, drop a comment, or share how you manage your autonomous AI coding loops.

Originally published on Medium

We Stress-Tested an AI Solidity Auditor on 219 Production Contracts. Here’s the Honest Scorecard

Gilad Haimov — Thu, 20 Aug 2026 06:48:30 +0000

What a 293-entry vulnerability database, a weak model, a strong model, and one human reviewer found, and didn’t

All numbers below trace to the evaluation data in this repo

The setup

I built a thing: a 293-entry smart-contract vulnerability database, assembled from four sources — Cyfrin’s Solodit (the largest curated real-audit-finding database I know of), the SWC Registry plus ConsenSys/SCSFG best practices, SunWeb3Sec’s DeFiVulnLabs (PoC-backed, Foundry-reproducible), and the official Solidity docs’ known-compiler-bugs list. 411 raw items deduplicated down to 293, each with a stable ID, severity, detection guidance, and source aliases. Around it I built a skill with two modes: AUDIT (walk a codebase against all 293 entries, produce a findings report) and CODING (real-time guardrails while writing Solidity so you don’t reintroduce a cataloged bug).

The question I actually cared about: is this useful for real pre-audit work, or is it a toy? I don’t trust vibes on this. So I stress-tested it in three stages of increasing seriousness: seven well-known contracts as calibration, 219 production contracts as the corpus run, and twelve trap contracts with deliberately planted bugs as the adversarial check. Here’s everything, including the parts that don’t flatter the pipeline.

Stage 1: calibration on seven known contracts

Before pointing this at 219 contracts I’d never read, I pointed it at seven I (and everyone else) know well. If the auditor can’t find the planted reentrancy in a 43-line Ethernaut-style contract, nothing downstream matters. If it hallucinates criticals in WETH9, nothing downstream is trustworthy.

Three things worth noting. First, the planted bugs were all caught, at the right severity — the classic withdrawBalance reentrancy, the Ethernaut Reentrance.withdraw pattern, the Delegation ownership takeover via delegatecall. Second, on the battle-tested code (WETH9, OZ ERC20) it stayed quiet: no criticals, no highs, just the known warts — the ERC-20 approve front-running race (SWC-114, baked into the standard itself), pre-0.8 unchecked math that isn’t actually exploitable because ETH supply is bounded. Third, on real production code it found real things: the MasterChef run surfaced a Critical reentrancy pattern plus three Highs, including the inconsistent safeSushiTransfer (direct sushi.transfer() calls discarding the return bool while every other path in the file uses SafeERC20).

Calibration verdict: the auditor has a working nose and doesn’t scream fire in a crowded theater. Good enough to scale up.

Stage 2: the 219-contract corpus

The corpus: 219 production Solidity contracts. The pipeline had three stages, each with a distinct job.

The weak pass: candidate generation

A fast, cheap model (Composer 2.5) walked every contract against the database. 218 of 219 completed. Output: 327 findings: 4 Critical, 43 High, 147 Medium, 119 Low, 14 Info. 109 contracts came back completely clean.

The most instructive number in the whole run isn’t 327. It’s 66 : the number of contracts where entry V-187 fired — “Outdated Compiler / Floating Pragma / Known Compiler Bugs in Pinned Versions.” Floating pragmas and stale compiler pins are everywhere in production code, and a weak model dutifully reports every single one. This is what high-recall, high-noise looks like: the weak pass is a metal detector set to maximum sensitivity. It finds the coins. It also finds every bottle cap on the beach.

That’s not a failure, it’s the design. The weak pass’s job is to miss nothing, not to be right about everything. Being right is someone else’s job.

The strong pass: the filter

All 36 contracts with Critical or High candidates went to a stronger model (Opus 5 High) for adversarial re-examination: every candidate finding got challenged against the actual code. 36 of 36 completed. The strong model considered roughly 435 internal claims across those contracts and retained 23 findings. 21 of the 36 contracts were cleaned to zero — every candidate the weak model had flagged was rejected on review.

Read that again: 58% of the contracts the weak pass flagged as Critical/High had nothing actually wrong at that severity. The strong pass is where the pipeline earns its keep. A weak model alone would have handed you 47 Critical/High findings, most of them noise. The two-stage design exists because I measured what happens without the second stage.

Human review: the judge

Six Critical/High findings survived both models. I reviewed each one by hand against the code. The final tally: 2 confirmed, 2 qualified, 2 rejected.

The two confirmed were both in Reflexer’s MultiSAFEEngine: a missing-authorization issue on confiscateSAFECollateralAndDebt (Critical) and a state-update bug in updateAccumulatedRate where a memory copy was never stored (High). Real bugs, in a real protocol’s engine, found by the pipeline and verified by a human. That’s the headline result, and I’m not burying it.

The two qualified: a Vat add/sub issue under ⁰.8.x (real but conditional on the pragma range hence “qualified,” not “confirmed”) and a Solmate ERC-4626 first-deposit inflation issue (a genuine design-risk class, but whether it bites depends on deployment parameters). The two rejected didn’t survive contact with the actual code — the models had been too generous, and the human pass did its job.

The honest math on the full pipeline: out of 327 weak-pass findings, after strong filtering and human review, the confirmed-and-qualified residue is a handful. Anyone selling you “AI found 327 vulnerabilities in 219 contracts” is selling you the metal detector’s beep count. The pipeline’s actual output is the 4 findings that survived all three stages — and the 21 contracts it correctly cleared.

Stage 3: the trap set

Twelve production-style contracts, each with deliberately planted bugs from known classes: access control removed, reentrancy guards stripped, return values unchecked, allowlists opened. Filenames left unchanged so the model got no hint. The weak pass caught 11 of 12 planted issue groups.

The one miss is the most interesting data point in this section: a __acceptAdmin access gap in a GovernorAlpha-style contract. Why did it miss? The planted bug lived in the admin handover flow , not in a function that does something dangerous directly, but in the two-step transfer pattern where the danger is in what’s missing (a check that the pending admin was properly set). Pattern-matching against a vulnerability database is weakest exactly here: at bugs whose shape is “the absence of a step in a multi-transaction flow” rather than “this line is dangerous.” The database has the entry. The model didn’t connect it. That’s a recall gap in flow-level reasoning, and it’s the kind of thing I’d want fixed before trusting this pipeline on governance code.

11/12 is a strong signal for code-level bug classes. The miss tells you where the boundary is.

What the numbers actually mean

Three claims I think the data supports, and one it doesn’t:

AI is a good candidate generator. 218/219 completion, every planted bug in calibration caught, 11/12 trap groups caught. For known bug classes — the things in the database — recall is genuinely good.
AI is a good filter, with a stronger model. 435 claims → 23 retained, 58% of flagged contracts cleared. The two-stage design isn’t optional; it’s the whole game. A single weak-model pass is a noise machine.
AI is not the judge. Two of six surviving Critical/High findings were rejected by a human. The pipeline needs a human at the end, full stop. Anyone telling you otherwise hasn’t measured it.
What the data doesn’t support: that this replaces a formal audit. The strong pass only re-examined Critical/High candidates — the 147 Mediums and 119 Lows were never adversarially filtered. The trap set is twelve contracts and selected classes. Economic and incentive-layer risk (oracle manipulation games, governance attacks, MEV strategies) is out of scope for a pattern database by construction. The GovernorAlpha miss shows flow-level reasoning gaps. This is a pre-audit instrument, not an audit.

Limitations, stated plainly

The corpus was 219 contracts; the write-up’s original “~230” was loose rounding. The exact number is 219.
Only Weak-pass Critical/High candidates went through the strong pass. Medium/Low findings are unvalidated.
The trap set is small (12) and covers only classes I chose to plant.
Model names and versions are as logged during the runs; models change, and results will vary run to run. This is a measurement of a pipeline design, not a benchmark score for any model.
The evaluation summaries are committed in the repo; raw per-contract model outputs are not. Treat the numbers as experiment notes with published methodology, not as a reproducibility package.

Why I’m publishing this

I’m building a pre-audit review service on top of this pipeline: the same database, the same weak→strong design, with every finding cross-examined by independent models and a human sanity-check on every report before it ships. A formal audit typically runs into the tens of thousands of dollars — an order of magnitude or more above a scan like this — and most of what a scan catches is the low-hanging fruit neither you nor your auditor wants to spend senior hours on.

This article is the evidence for that service including the misses, the noise, and the parts where the human was load-bearing. If you want the pipeline run on your contracts before your audit: giladha@gmail.com.

The database, the skill, and the evaluation data are open: https://github.com/giladHaimov/Smart-Contract-AI-Audit-Skill.

AI-Assisted Solidity Development: Bringing Audit Knowledge into the Dev Loop

Gilad Haimov — Sun, 09 Aug 2026 19:10:06 +0000

How injecting an auditor’s mindset into the development loop catches vulnerabilities before the contract is frozen.

Smart-contract projects typically involve two distinct activities before a contract ships: implementation and independent security review. These may overlap in practice, but they are usually owned and approached by different teams with a significantly different knowledge set.

Phase one is implementation: the Solidity contracts, together with their test suite, deployment tooling, and off-chain ‘harness ’, i.e. integrations with backend and services that connect to the contract. The contract is almost always written in Solidity (Ethereum/EV-compliant network); the surrounding tooling is often written in Python or TypeScript.

Phase two is independent security review. An audit team examines the code, its design assumptions and relevant attack paths from a fresh adversarial perspective. The question is not merely whether the contract works as intended, but how it may fail when it meets hostile inputs, integrations or incentives.

Strong auditors know dozens of vulnerability classes by heart, and approach the code with a suspicious, independent mindset — constantly testing assumptions, privilege boundaries, invariants, and external dependencies.

Because implementation and independent review are usually separated in time, and in most cases require more than a single iteration, a significant part of contract development is spent on the round trip between them, rather than on writing the code itself. To a large extent, this is understandable, yet the cost in terms of time and team effort remains.

In many of the projects I was involved in, a large portion of total completion time went into audit/repair cycles: code that passed my own review, then came back flagged, then got patched, then went back for another pass.

The idea

After enough of these cycles, a clearer and more efficient approach became obvious. A development team may understand the system deeply, but it cannot keep every known failure mode and attack path actively in working memory while it is delivering features.

Auditors add an independent, adversarial perspective. My goal is not to replace that perspective, but rather, to bring this type of knowledge and critical thinking into the process way earlier. In fact: while the code is being written.

And, once AI is introduced into the process, the implementation becomes straightforward: first, obtain a curated vulnerability-database with is (sort of) well known in the community.

Once you have it, turn it into an AI skill that either a human developer or an AI coding agent can consult and activate during the ongoing development effort. This simple change significantly improves the effectiveness of the development process.

A 293-entry security knowledge base

I built this as an open-source tool called Smart-Contract-AI-Audit-Skill. It contains 293 known Solidity vulnerability and security-pattern entries, assembled from public audit findings, security registries, best-practice guides, proof-of-concept collections, and Solidity’s security documentation. The sources I mostly used were Solodit’s audit-finding catalog, the SWC Registry, ConsenSys and SCSFG best-practice material, DeFiVulnLabs’ proof-of-concept collection, and Solidity’s security-considerations documentation.

I began with 411 raw entries. Many described the same underlying issue from different sources or in different terms, so I merged duplicates while preserving a trail back to their original sources.

The resulting workflow supports two distinct modes.

Audit mode — code already written.

Give it a Solidity codebase, and it will check the relevant vulnerability entries against that code. For each possible finding, it reports the affected location, explains the failure mechanism, sets a severity score and suggests a concrete fix. Also, when the problem cannot be deduced from code structure but rather depends on runtime behaviour, it recommends which test should be added. These are candidate findings for review — not proof that the contract is secure.

CODING mode — while coding.

Same database, opposite direction. While you are writing or editing Solidity, the skill routes what you are working on into the relevant category checklists: a withdraw function pulls in Reentrancy plus External Calls, a price feed pulls in Oracle, touching an upgradeable contract pulls in Proxy & Upgradeability and Storage. Each checklist entry gives you a Don’t (the failure mode) and a Do / Detection (the concrete fix or check), with wording lifted straight from the database entry and a line pointer back to the full record if you need the aliases or the source citation. Most functions only touch one to three categories, so you never read all 21 files for a single function. There is also a short always-on list — 28 Critical and Critical/High items worth knowing by default no matter what you are touching. The rule is simple: consult the relevant checklist before you consider a function done, not after.

Note, that is not a deterministic static-analysis engine, but rather AI-guided review workflow. It uses the vulnerability entries to direct review and testing; its output is intended to guide human verification, not replace it.

As part of my work here, I tested the workflow against seven public Solidity targets, which I believe make up a decent representation of the types of vulnerabilities smart contracts may experience.

The test list is made of two clean baselines (OpenZeppelin’s current ERC20 implementation and WETH9), three deliberately vulnerable teaching examples, and two production contracts from GitHub: Uniswap V2’s pair and SushiSwap’s MasterChef. This is a small validation exercise, not a benchmark of detection accuracy, false-positive rate or audit coverage.

After compiling this list, I ran the AI-audit workflow and reviewed the resulting audit results. As expected, no critical/high findings were detected in the baselines contracts; rather, it surfaced the intended issues in the deliberately vulnerable examples. On Ethernaut’s Delegation, it identified the dangerous unrestricted delegatecall path that allows callee logic to write to the caller’s storage. On MasterChef, it raised a potential reentrancy concern in emergencyWithdraw because the token transfer occurs before the user state is cleared. That concern still requires a reproducible exploit path and independent review before it should be treated as a confirmed vulnerability.

A representative audit-mode result from Ethernaut’s Delegation challenge. The workflow identifies the unrestricted delegatecall path, traces its storage-impact mechanism, and links the finding to the supporting security entry.

Delegation is deliberately vulnerable. This example demonstrates the form and traceability of the output. It is not a benchmark of real-world detection accuracy.

Where this genuinely helps (and where not)

Some entries can be assessed primarily from code structure — for example, missing access control or a state update placed after an external interaction. For those cases, the workflow can identify relevant locations and suggest a remediation for review.

Other entries depend on behaviour rather than shape: whether a meaningful test exists, whether a particular state transition is reachable, or whether an external call can be exploited in context. The audit workflow can inspect the code and recommend the tests to add, but it cannot prove that those tests cover the relevant behaviour or were executed in a representative environment.

Of course, this is not intended to replace static analysis tools e.g., Slither. Static analysis is stronger at many mechanical static checks. Rather, it is aimed at review questions that require context , cases where a function may look reasonable in isolation but becomes dangerous because of its role in the surrounding protocol.

As is true to human contract auditing, code vulnerability detection is one layer of smart-contract security. The highest-impact failures are often protocol-specific: incorrect accounting, price and liquidation assumptions, privileged roles, upgrade paths, and cross-contract interactions. This workflow can bring those questions into development earlier, but it does not derive the protocol’s security specification or replace independent adversarial review.

Another problem: will you want to maintain your code in six months?

The vulnerability database solves one half of the problem: do not ship a known bug class. The other half is a different kind of debt: code that is technically fine today but that nobody, including future you, will want to maintain in six months.

For that, I built a second, separate skill: Solidity-Smart-Contract-Best-Practices-Skill. It is not an audit methodology; it is a handbook for engineering quality: naming, layout, invariants, how you write and structure tests, how you handle reviews and deploys. Seventy-four items, every one citing where it comes from (Solidity’s own docs, OpenZeppelin, SCSFG, Trail of Bits, Solcurity, the SWC Registry, the Foundry Book), organized into categories like access control, checks-effects-interactions, upgradeability, and oracle handling.

The interesting design choice here is the same one from the audit skill, applied to a different problem. Some of these 74 items are code-checkable: an AI can read the .sol file and just check whether the pattern is there. Others are test-checkable, meaning the item is not about the contract’s code at all; it is about whether a specific test exists for it, so the skill recognizes the situation that creates the need and tells you what the test should look like.

Try them

Feel free to use either skill on its own, or both together: the vulnerability database catching known attack patterns early, the best-practices skill keeping the code itself maintainable while you are at it.

Both repositories are intentionally curated rather than self-updating. Any extension should be sourced, reviewed and versioned before it is used on production code.

Both repos are also open source. If you try either one, I would genuinely like to know what it catches and what it misses.

Get the code: Smart-Contract-AI-Audit-Skill

Solidity-Smart-Contract-Best-Practices-Skill

If you are building Solidity systems and want to discuss adapting this workflow to your team, reach out on LinkedIn.