DEV Community: Gilad Haimov

What an ERC-4626 price test actually shows

Gilad Haimov — Mon, 14 Sep 2026 07:13:00 +0000

Vaults, shares, and price

ERC-4626 gives tokenized vaults a common interface for deposits, withdrawals, and share accounting. A user deposits an ERC-20 token and receives shares representing a claim on the vault's underlying holdings. The vault might lend those assets, stake them, or put them into another yield strategy. Users can later redeem their shares and, where the vault allows it, transfer them to someone else.

In a simple proportional model, a user's shares divided by total shares, multiplied by the vault's assets, gives their claim on those assets. Actual redemption amounts also depend on the implementation, including rounding, fees, and any virtual assets and shares used in the calculation.

I called this project "Vault Price Honesty" because I wanted to examine what those exchange rates mean for the people using a vault. The name needs a little care: an unexpected price change is not enough to call the accounting wrong. A direct donation, for example, can raise the value of existing shares without harming a later depositor.

I focused on five situations:

A tiny first deposit followed by a donation. Someone acquires a small number of shares, then transfers tokens directly into the vault. The resulting exchange rate can leave the next depositor with very few shares, or none. Whether the first depositor profits needs a separate calculation.
Assets remaining when no shares exist. This can happen after the last redemption or because someone donates to an empty vault. How the next deposit is priced depends on the implementation. The two situations are worth testing separately.
Direct donations counted as assets. Including donated tokens in the vault's accounting raises the value of existing shares. That can be intended behavior; the question is what follows from it.
Preview and execution disagreeing. Previews have specific requirements under ERC-4626. A difference needs to be checked against those requirements before it is called a violation.
Reported assets and assets held at the vault address differing. A strategy may hold assets elsewhere, so the difference can be legitimate. Stale or incorrect valuation is a separate concern.

Following the tokens

I started this project intending to build a reusable, hardened ERC-4626 base contract. Working through the cases made the scope less straightforward than I expected. Donation accounting, strategy valuation, and the treatment of assets remaining after the last redemption depend on the vault's design. Shared protections are useful, but they do not settle all of those choices.

I shifted the project toward a Foundry suite for examining that behavior. One local test made the reason particularly clear.

A user deposited 50 tokens and received zero shares. Before that deposit, an attacker had deposited one base unit of an 18-decimal token, then transferred 10,000 tokens directly into the vault. This fixture uses an OpenZeppelin-based vault with a decimals offset of zero.

The depositor's loss was clear. The attacker's position needed another calculation.

Participant	Contributed	Redeemable	Net
Attacker	10,000 + 1 wei	Approximately 5,025	Approximately −4,975
Depositor	50	0	−50

These are underlying-token amounts in a local fixture, excluding gas. The attacker's initial deposit was one base unit, not one whole token.

OpenZeppelin's virtual asset and share accounting explains the result: part of the donation cannot be recovered by the attacker. In this particular sequence, causing a depositor to lose 50 tokens costs the attacker roughly 4,975 tokens.

The depositor suffered a complete loss, but this sequence did not generate an attacker profit. It demonstrates griefing. It does not establish the economics of other donation amounts, multiple victims, or a different vault implementation.

The local inflation test also includes an OpenZeppelin-based vault with a decimals offset of three. With the same initial attacker deposit, donation, and victim deposit, the victim receives shares redeemable for approximately 45.02 tokens. That avoids the zero-share outcome, but still leaves the victim almost 10% short.

A test asserting only that the victim received shares would miss that loss. Comparing raw share counts between the two vaults would also be misleading because the offset changes share precision. The useful comparison is the amount of underlying assets those shares can recover.

Another local case starts with zero shares and 500 donated tokens already in the vault. The next user deposits 1,000 tokens and receives shares redeemable for 750. Here, too, a nonzero share balance conceals a substantial loss. This setup deliberately donates assets to an empty vault; it should not be confused with a normal redemption leaving a small rounding remainder.

Interpreting the other measurements

Direct donations require more care. In the recorded fork run, transferring underlying tokens into sUSDe increased its reported assets and share value. That observation alone does not establish depositor harm. A higher price per share can mean that each share represents more assets.

To make a stronger claim, I would need to follow a specific consequence: a depositor's round trip, a fee calculation, or an integration using that price as collateral value. The donation test does not demonstrate those outcomes.

The same caution applies to totalAssets(). Assets deployed through a strategy need not appear in the underlying token balance at the vault address. A difference between the two numbers is a starting point for examining the accounting. Establishing an error requires understanding what the vault owns and how it values those positions.

Preview checks have a more explicit reference: the ERC-4626 specification. When a preview is followed by the corresponding operation in the same transaction, under unchanged conditions, deposit and redeem must return at least their previewed amounts; mint and withdraw must require no more than theirs. The standard also requires previews to approximate the operation as closely as specified. Simple equality with convertToShares or convertToAssets is not a general substitute, since previews account for factors such as fees.

These checks need evidence from execution, including balance changes. A returned number is insufficient if the corresponding assets never arrive.

What I would use the report for

The local fixtures make particular sequences easy to inspect. Fork tests add real accounting structures and operational restrictions. Their conclusions remain limited to the paths actually exercised: the live tests cover deposit previews and redeem where possible, rather than all four operations. Empty-vault scenarios stay local for the populated vaults in the sample. An unsuccessful run leaves a coverage gap; it does not identify the cause of failure.

For a developer reviewing a result, I want the report to answer three questions: what operation ran, what changed, and which conclusion those measurements support. A standards violation, a demonstrated economic loss, and a design-dependent behavior each call for a different response.

The most useful result from this work was the small table above. It shows both the depositor's loss and the attacker's cost. Those two numbers say more about this particular attack than the zero-share result alone.

The Foundry project, tests, and reports are on GitHub. The suite is a focused investigation tool, not a comprehensive audit.

False Completion Is the Real Failure Mode of Coding Agents

Gilad Haimov — Thu, 03 Sep 2026 07:22:22 +0000

A deliberately strict process for preventing false completion in autonomous software development

In an earlier article, I argued that the model should have to earn every agent in an agentic system. One agent is a valid answer. Every additional boundary has to justify its cost.

This article starts one level lower. Assume we have selected the topology. Who gets to decide that the resulting product is actually the product we asked for?

After experiencing dozens of AI-development efforts and reviewing the results of many more, I no longer think the main danger is that the development procedure will manifestly fail. A crashed build, a red test or an explicit error report is relatively easy to deal with. The system has admitted that it has a problem.

The far more serious failure is that of FALSE COMPLETION: the AI system believes it has accomplished the development goal, reports success with complete confidence, and delivers a product that has little to do with what the system design had in mind.

The code may compile and the interface may look polished. Yet core flows are missing, persistence is half-wired, or the implementation has quietly solved a simpler problem than the one it was given.

And this is not merely anecdotal. An analysis of over 20,000 coding-agent sessions identified inaccurate self-reporting and misreading developer intent as recurring forms of developer-agent misalignment. Most visible resolutions still required explicit human correction. The agent’s confidence was not evidence that the job was done. The study is worth reading.

My response is a deliberately strict development process. It costs more tokens and wall-clock time than letting one agent run until it declares victory. I deliberately pay that cost. For substantial autonomous builds, this is now the only way I work. A failed process is visible. A successfully completed wrong product may survive until a user discovers it.

1. Make the product book executable

Before any code is written, the product book, call it a PRD, specification or product contract, must be reviewed by an AI agent whose task is not to improve the prose. Its task is to attack the design.

Can every important requirement be converted into observable behavior? Are the user flows, data rules, failure behavior and non-functional requirements clear? Is anything important hiding behind words such as “fast,” “intuitive” or “secure” without a measurable meaning?

The objective is not a specification that deterministically produces one implementation. Many implementations may be valid. The objective is a specification that lets us determine, with as little interpretation as possible, whether an implementation is acceptable.

The product book must also define kill conditions. If an acceptance condition cannot be tested or two requirements contradict each other, the correct output is not heroic improvisation. It is STOP.

2. Lock your acceptance tests

The acceptance tests derived from the product book must be fixed before implementation and remain immutable throughout the development run.

The coding process may run them and may fail because of them. It may not change, weaken, delete or reinterpret them.

This is intentionally more rigid than ordinary iterative development. If we later discover that a test cannot be met or that the test itself is wrong, the process stops. The product book and acceptance set are updated through a separate decision, their version changes, and execution restarts from the specification and planning boundary. The finish line is not quietly moved by the participant currently failing to reach it.

Locked does not mean sacred. Tests can be wrong. OpenAI found difficult SWE-bench Verified tasks where tests rejected valid solutions or required behavior absent from the problem statement. That is why changing a test should be an explicit design event, not an opportunistic edit inside the coding loop. OpenAI’s audit shows how dangerous a bad oracle can be.

3. Break into steps before any code is written

The complete task must be broken into discrete execution steps before code is written. Each needs defined inputs, outputs, dependencies and review conditions, and must fit comfortably inside the model’s usable context, not merely its advertised context window.

This may be the single most important protection against an AI-development effort going sideways. Long contexts accumulate abandoned hypotheses, obsolete instructions, failed repairs and irrelevant tool output. Eventually the agent spends its reasoning budget navigating its own history.

Small should not mean arbitrary. Splitting one tightly coupled behavior across five agents simply moves the difficulty into integration. A good boundary ends in a coherent change another agent can verify without reconstructing the entire project.

4. Assign each step a new coder instance

Using one coding agent and looping it through every step looks compelling. The developer of step five already knows what happened during steps one through four, and fewer tokens are spent bringing it up to speed.

But the same accumulated context also contains every wrong assumption, abandoned path and local compromise made along the way. Anyone who has executed long-running agent tasks has seen performance deteriorate as the context shifts from one problem to another.

For non-trivial development, I want a fresh coding context for every step. Continuity should come from explicit artifacts: the product contract, architecture decisions, interfaces, git history, accepted outputs, open risks and a short task packet. It should not depend on a long conversation correctly remembering which parts are still true.

Anthropic’s recent experiments suggest that stronger models can sometimes sustain longer builds, allowing sprint decomposition to be removed and evaluation to happen less frequently. I understand the argument. My own experience leads me to a more conservative default: I would rather pay the visible cost of fresh contexts than accept the invisible risk of context corruption and false completion. Their current reasoning and trade-offs are described here.

5. The builder never grades his own work

The coder responsible for a step must never be the authority that approves it.

When the work is ready, the coder hands the diff, task contract and evidence to a fresh reviewer. The reviewer should not inherit the coder’s reasoning. It should inspect the result on its own terms, preferably with read-only access and the ability to run the system.

Where practical, I prefer a reviewer from a different model family. A different model is not automatically better, but it is less likely to reproduce exactly the same assumptions and blind spots. Deterministic tests remain a stronger oracle than either model; the second model exists to challenge what the tests do not express well.

The reviewer compares the change with the product intent, architecture, regressions and end-to-end behavior. It reports material gaps, not style objections manufactured merely to demonstrate activity.

Here current guidance and my experience agree: Anthropic recommends adversarial review in a fresh context so the evaluator sees the result rather than the story that produced it. See its current Claude Code guidance.

6. Final acceptance tests must be invisible to coder

This is probably my most controversial rule: the coder should not have read access to the final acceptance-test implementation.

It must know the required behavior. A hidden test may never demand a private method name, particular internal structure or functionality absent from the product contract. But the exact executable suite should belong to the control plane, not to the participant being evaluated.

The coder may write unit tests and receive visible examples or contract tests. What it cannot do is inspect the final oracle and shape the implementation around its precise assertions.

Agents are no better than us in this respect. Give the builder direct sight of the scoring mechanism and the target can quietly shift from “build the intended product” to “make these checks turn green.” The protected suite should therefore be largely black-box. Failures should identify the violated requirement without exposing or surrendering control of the examination.

7. Place hard cap on coder/reviewer cycles

It is legitimate for a skeptical reviewer, especially one using a different model, to reject a step and request repairs. That is the process doing its job.

But the loop must be strongly bounded. My red line is around four coder/reviewer attempts, and I stop earlier if the same defect returns twice. By then, the problem is probably not a missing if statement. Requirements may conflict. The step may be too large. The technology may be at its limit. The acceptance test may be wrong. Or the accumulated repair dialogue may already be corrupting both agents' judgment.

What I do not want is endless negotiation until the reviewer becomes tired, confused or sufficiently conditioned to approve bad code.

When the budget is exhausted, the reviewer produces a failure report: what remains wrong, the evidence, what was attempted, which assumptions are doubtful, and whether the likely fault is in implementation, decomposition, architecture, specification or the test oracle. Control returns to planning.

Failure is an acceptable result. Counterfeit success is not.

“Done” must be a system state

These rules lead to one architectural principle: the agent does not get to decide it is done.

Every feature begins in a failing state. A coder may propose completion, but only protected tests and an independent evaluator can accept it. Every step leaves code, evidence and a clean checkpoint. Integration checks confirm that individually valid steps still form one working product.

This is not the cheapest possible harness, and it is not my recommendation for changing a label or adding one obvious validation. It is my default for substantial autonomous product development, where a plausible but wrong result can consume days of review or reach production.

The current fashion is to assume that better models will let us remove more of this structure. They probably will remove some of it. Anthropic explicitly recommends retesting which pieces remain load-bearing as models improve. That is sensible engineering.

My own bias is to remove safeguards only after repeated evidence that they no longer catch meaningful failures. Model capability is improving quickly. The ability of a confident agent to make a wrong product look finished has not disappeared.

So let the model write the code. Let it plan, search, test and repair. Give it real autonomy inside clear boundaries.

But do not let it define the finish line, rewrite the examination, grade its own paper and then announce that it passed.

“Done” is not an agent opinion. It is a state the system must prove.

I am a principal software and protocol engineer. I have written production Solidity since 2017, led protocol-layer engineering on a live EVM L1, and built an AI-assisted audit workflow that I validated against a 219-contract corpus. It is now open source: github.com/giladHaimov/Smart-Contract-AI-Audit-Skill

The agent that writes the code cannot read the tests that grade it

Gilad Haimov — Sun, 23 Aug 2026 08:03:53 +0000

TL;DR:

Letting an AI coder review its own code and tests leads to compromised code quality, test-softening, and false positives. This article introduces a practical, multi-agent governance layer inspired by the Ralph loop but hardened with two core constraints:

The Locked Oracle: The coding agent is completely blocked from reading or editing the test suite, which is frozen before coding begins.
The Cross-Model Checker: Code review and testing are strictly delegated to a different model family to significantly reduce blind spots and self-verification bias.

Press enter or click to view image in full size

The Correcting Loop Architecture: The Director freezes the tests in advance (Locked Oracle). The Builder codes blindly without test access, and a cross-model Reviewer evaluates the diff against human Gates to prevent bias

Developers experimenting with auto AI coding, often hits the wall of self-verification bias, where the agent quietly softens your tests just and happily claims success. I have just shared a loop-based AI builder I’ve been using internally for some time. This builder gives me the best results from anything I have tested before in solving this problem.

The strongest impact that I see is in significantly reducing the ‘faux tests’ problem i.e. tests which are softened so that the app can be marked completed. An additional strong impact is on the PRD compliance, where I now get way fewer occurrences of the AI-dev process ‘forgetting’ parts of the requirements.

It relies heavily on loop concepts for automated AI coding, with some twists that I believe result in a more solid development process. You can basically look at it as a Locked-oracle Cross-model variant of the Ralph loop (an iterative AI-coding pattern).

It also borrows heavily from several other ideas floating around the AI dev sphere and combines them into something that has worked well for me in developing multiple smaller-scale projects.

These ideas, specified below, are not mine! I happily copied them from others, and you’re welcome to do the same with the model I’m sharing here.

Since the AI sphere has not yet reached the stage of naming consensus, I will use the names I believe are most common. I could be wrong.

The ideas that I borrowed

Ralph loop

From Ralph I’ve borrowed the most. Especially the part of looping with a refreshing session context in each unit of work in the loop, and the usage of git commits as both source-of-truth and checkpoints.

I did NOT take from Ralph its eternal loop, since I believe too many code/review iterations are a good indication of a broken process that needs to be stopped and reviewed.

I also did not take its single-agent self-review, which I think is inherently wrong for a valid code review. Instead I used:

Locked Oracle

Here I provide strong limitations to what the rolling dev loop is allowed to access.

The rule is simple: the builder may read all code, even code which appears unrelated to its current task. But it cannot read or modify the frozen acceptance suite used for system verification and later acceptance. Those are written once as part of system design, and are then locked before the first line of application code exists.

By doing this I solve two common problems facing test design:

Test-biased code, where the builder, knowing what the test checks for, creates a non-complete solution that still manages to pass the tests.
Test softening, where the process slowly relaxes the tested conditions to the point where the test no longer presents the hard demand set by the PRD for correctness.

With this addition I have way more assurance that a green test suite actually means the product is behaving as it should, rather than the agent having adjusted its own success criterion.

The lock is not an instruction in a rules file. It’s a deny rule plus a pre-tool hook, evaluated outside the model, on paths the model cannot influence.

# EXIT CODE SEMANTICS — the easiest way to disable this guard accidentally:
# exit 0  allow the tool call
# exit 2  BLOCK the tool call; stderr is returned to the model
# exit 1  does NOT block in Claude Code; the tool call proceeds.
# Using the conventional Unix failure code here silently disables the guard.

Cross-Model checker

Classic Ralph has one agent and no independent verifier. The same agent writes the code, writes the tests, and decides it is done. After using this approach for some time I dropped it. I simply could not rely on its correctness.

Instead, I found it more valid to replace it with a check model where each review/test session is done by a new agent instance belonging to a different model than that of the builder.

Now why is this important?

Because self-verification is the root of evil. The same model that generated the code always carries some ‘blind spot’ into the testing phase, where the considerations and context used to write the code find their way into scoring it. For this reason you would never assign the human coder to be the code reviewer.

Using a different model family (e.g., Claude building, GPT reviewing, or Gemini testing) brings a fresh set of reasoning and training biases, making it way more likely to intercept errors.

The fact that each reviewer agent is restarted, hence provided with clean context, also strengthens our review phase, by preventing ‘context rot’, i.e. a context so bogged with prior tasks that you can no longer rely on it to provide reasonable answers to the current task. Restarting the agents indeed solves this problem, admittedly at the cost of a higher learning curve when introducing each reviewer instance to the current code.

Inner/Outer loop model

The outer loop iterates execution steps to exhaustion, and an inner loop is deployed to verify by fix & trial the correctness of each step’s execution.

I have made multiple attempts not to use this model, which looked overly complex to me, but never managed to get it right using a single loop. I tend to think the reason is inherent: that you cannot really automate dev tasks without these two levels of looping.

Frozen judge model

The catalog is written before a single line of application code exists. That timing is what makes the lock enforceable rather than aspirational. A catalog written after the code exists is already shaped by that code, and by then there is nothing left to freeze.

Differential Agent Skills

This model relies on 3 different entities:

Director for overall orchestration and high-level review
Builder for coding
Reviewer for code review compared to design docs

Each of those is provided with different skills required for their role.

Not every role needs a strong model

The three roles do not need equal model quality, and treating them as if they do is the most expensive mistake available here.

Builder quality matters least. Every defect it produces gets caught by a frozen oracle it cannot influence. A weaker builder writes worse first drafts and burns more repair rounds, but it cannot ship something broken past the judge. That is what makes a cheap builder safe in this setup when it would not be safe in an ordinary one.

Director quality matters most. The Director writes the judge, and nothing downstream checks the judge except a human reading it once. A weak Director produces a shallow test catalog, the loop runs green all the way through, and the gap only surfaces during human testing. There is no automated recovery from that.

Reviewer quality matters in between, and its model family matters more than its raw capability. A reviewer from the builder’s family shares the builder’s blind spots and misses the same cases.

One practical note: don’t pick the builder model by benchmark scores. Measure repair rounds per step on a throwaway project. One or two is fine. Three or more consistently means the model is failing the spec rather than the gate, and a cheap model that needs three passes is not cheap.

The cost saving from this allocation is real, but it’s a side effect. Even if tokens were free, the family separation between builder and reviewer would still be required.

Why this works for me

This model has several benefits that make it great for my needs, which are rapid development of small-to-medium projects.

It audits the PRD before anything else

It runs extensive tests on the product book (PRD) and effectively blocks development until the PRD is corrected to be both self-contained and easily breakable into development steps. In general, ‘AI-dev friendly’.

It contains a double-loop mode

The entire project gets automatically broken into several execution steps so that each step is manageable in a single AI builder session with no significant context window decay.

Note that there’s an inherent plus in giving the AI the task of doing the breaking (I could have done it myself): AI should know best the task length for keeping the context window effective.

After each step completion, the process delegates to a newly created agent of a different model for ‘corrective review’. Corrective in the sense that the process expects the review to return with a changelist and handles the process of review, request-for-changes, new-code-review.

While the decision to force the reviewer to be of a different model than the builder is self-explanatory (you don’t want the job reviewer to think like the worker), the issue of restarting a new reviewer agent for each step is something I was not sure about. There are lots of benefits in keeping a reviewer context up and aware of all aspects of the entire project. At the end I went with restarting the reviewer so as to keep its context window clean, and in doing so got good enough results.

It never weakens a test to make it pass

It forces an aggressive policy of testing each stage AND of never weakening tests just so they pass. A failed test means an action item to be fixed in code.

As part of these testings, the automatic process will deploy a local realistic test env (DB in Docker container, node server, more components may be added) and test runs realistic scenarios locally.

It minimizes human intervention

It is very good at minimizing human intervention in the loop (it was built for it). The only human gates defined here are:

Gate A, where a human approves processed design/architecture docs, project steps, and directory structure.
Gate B, where a human reviews and approves the implementation of the 1st step.
Gate C, where development plus local testing is basically completed and the human reviews the complete system (and in real-world issues a change request…).

Even this level of human intervention can be reduced. Just add a short prompt stating that the agent:

“should work in full automation mode and circumvent any product book review and permission requests but must not reduce the product’s quality, and especially should retain working in steps and performing extensive per-step review by a different model”

Here the agent is basically on its own, so this should be limited to internal/throw projects.

It defaults to boring

The model defaults to a very basic manner of development where the ‘solid simple’ is almost always selected.

Backend will run on Node, DB layer will be an agnostic RDBMS, mobile apps will be written over Capacitor, but if app defs do not allow for it, will fall through to React Native, and only if impossible (e.g. complex animations, extensive mobile OS API needed) will work in native code.

All code, excluding scripts, mobile native if needed, etc., will be written in TypeScript, which adds extra simplicity to the model. It also adds resilience, since in my experience the no. 1 ‘code-level bug’ in AI-generated code is getting the types wrong.

It scales without a rewrite

While keeping it simple, the model was designed to accommodate an increase in app usage at a very low cost.

The backend is instructed to be stateless so as to make horizontal scaling trivial. DB layer is built with the most beneficial indexes already declared and is agnostic, hence should allow switching to a stronger DB with very few changes. And Node itself is very efficient and scale-friendly.

All this is to say: if and when success strikes, you will probably not need to rewrite even for way more users than you started with.

Same pattern, different domain: Solidity auditing

This same pattern is even stronger for smart contract security. Before any pragma solidity\ line, a Director (say Opus) drafts the security spec: reentrancy guards, access control, overflow checks. Now these become the locked test catalog.

Now a cheap builder (say Sonnet) writes the contract. A reviewer from a different model family (GPT-4o) audits with Slither + the frozen spec. The builder never sees the audit checklist, so it cannot soften its own guards.

Used on one helper contract (< 900 LoC), the Reviewer caught a spec mismatch at Gate A that would have cost a full rewrite later.

Two notes before you try it

1. This is a starting point, not a polished product. The core ideas are what I’m sharing — the implementation is yours to adapt. That said, the repo has a demo/\ folder with a working example. Just clone and run them and you should be good to go.

2. Know the ceiling. Works well up to ~20k LOC. Beyond that, context window limits kick in. Extensible — I just haven’t needed to yet.

Get the code: CorrectingLoopManagedFlow

Found this useful? Star the repo, drop a comment, or share how you manage your autonomous AI coding loops.

Originally published on Medium