A smart contract can accept a new implementation and still have no safe route back. The upgrade transaction may succeed, the old bytecode may remain available, and an administrator may retain permission to install it. None of those facts proves that the old code can interpret the state users have created since the upgrade.
Rehearse recovery against those later states before describing a system as operationally upgradeable. The useful result is a manifest that identifies the exact deployment, the checkpoint tested, the recovery action permitted there and the evidence that user rights survive it. A successful pointer change is only one observation in that record.
This guide develops that manifest for a hypothetical EVM vault. Its scenarios are proposed tests, not results from a deployed protocol. Adapt the accounting, dependencies and authority model to the system under review.
1. Define the recovery promise before choosing a command
Use three separate terms in the runbook. An implementation rollback reinstalls an earlier implementation through the system's supported upgrade mechanism. A state repair transforms particular stored values under a reviewed procedure. A service recovery restores an acceptable user operation, possibly through a forward fix or a controlled migration. One incident may require all three, but each needs its own success criteria.
When upgrade acceptance stops at deployment success, the missing deliverable is evidence of recovery. Pharos Production documents a blockchain delivery process that includes contract testing, security review and staged deployment. A release review can attach the rehearsal described here to those activities and make the recovery assumptions explicit. That is a proposed acceptance artifact, not a claim that every contract has a reversible migration.
For the example vault, define the promise as follows: an existing user retains the same valid withdrawal entitlement after recovery, subject only to documented fees and rounding; a pending withdrawal remains identifiable and cannot be paid twice; operators can resume only the functions whose invariants have passed. Specify how each condition will be measured before executing any recovery transaction.
Also write down what the promise excludes. Restoring one contract's implementation cannot by itself reclaim a payment already received by another party, erase a message already executed on another chain or reverse a decision made by an external service. Those effects require separate authority and a separate reconciliation procedure.
Give the promise an explicit scope: one vault, its asset contract, the withdrawal queue and any settlement adapter. A statement about the vault alone should not silently become a claim about the entire protocol.
2. Freeze the deployed system you intend to rehearse
Start from the actual deployment inventory. Record the chain identity, fork block number and block hash, proxy addresses, active implementation addresses and runtime bytecode hashes. Include the source commit, compiler version and settings, dependency lockfile and storage-layout artifacts used to explain that bytecode. A repository branch name is insufficient because it can move.
Resolve the upgrade topology before selecting the recovery transaction. OpenZeppelin's proxy reference distinguishes transparent proxies, UUPS implementations and beacon-based deployments. Their upgrade logic and control points differ. In a beacon system, enumerate every proxy that follows the affected beacon; testing one instance does not establish compatibility for instances with different initialization histories.
For a UUPS deployment, establish that the currently installed implementation still exposes a usable authorized upgrade route. Do not assume that a route present in the previous release remains callable. Record any compatibility restriction that prevents reinstalling a particular historical version. The recovery target must be admissible through the deployed mechanism, not merely available in an artifact directory.
Build the local fork at a fixed block. Anvil's official documentation describes local forking, controlled mining, state management and account impersonation. These capabilities support a rehearsal, but each convenience changes what the exercise proves. Pin the tool version and relevant chain configuration, verify the starting block hash against the recorded source chain, and keep the transaction destination confined to the local test environment.
Document injected assumptions alongside the fixture: extra test balances, impersonated actors, mocked oracle responses and altered timestamps. Use dedicated test credentials. Production signing material is unnecessary for testing contract authorization rules, and its presence makes a local exercise harder to keep isolated.
Finally, choose representative existing positions. Include a long-lived depositor, an account with a pending withdrawal, an empty account and any privileged account with special accounting treatment. Record why each position matters. A fork containing real storage is still a weak fixture if every test touches only a newly created user.
3. Test the meaning of storage in both directions
A forward storage-layout check asks whether the new implementation can interpret the earlier layout. Recovery introduces another question: can the old implementation interpret every relevant state the new release is allowed to produce?
OpenZeppelin makes the persistence issue concrete in Writing Upgradeable Contracts:
And if you remove a variable from the end of the contract, note that the storage will not be cleared.
The quotation concerns removing a variable from a contract definition. Its relevance to recovery is that changing code does not erase historical storage. The same documentation warns against incompatible changes to variable ordering and types. A layout validation therefore belongs in the release evidence, while the rehearsal must also address the meaning of the values already written.
Consider a hypothetical vault whose first implementation stores withdrawal requests in asset units. A later version migrates those requests into shares while retaining a similarly shaped numeric field. The old implementation might read a perfectly ordinary integer after reinstallation and interpret it using the wrong unit. Successful reads and unchanged slot locations would not establish correct entitlements.
Make a state-meaning table for every changed field: previous interpretation, new interpretation, transition that writes the new form and behavior if old code reads it. Include enumerations, sentinel values, rounding conventions, timestamps and identifiers. Mark the first transition that makes a direct return invalid. That boundary can occur during initialization, the first deposit or a later maintenance transaction.
Use a concrete accounting fixture to expose the difference. Suppose the vault holds 1,000 asset units against 500 shares, with no fees or rounding in this example. A request for ten asset units becomes a request for five shares during migration. If old code later treats the stored five as asset units, the user receives only half the original entitlement. A test that merely confirms the request still exists would pass. A test that settles the request and compares the payment with its expected ten asset units would fail. Preserve both the stored representation and the economic expectation in the fixture so the assertion does not accidentally reuse the faulty conversion logic.
Treat initializers and migrations as state transitions with their own preconditions. Determine whether a recovery requires additional initialization, whether a version guard prevents it and whether replaying a migration could duplicate an allocation. The answer must come from the specific contract and reviewed payload. Avoid a generic instruction to call the initializer again.
If the reverse interpretation is undefined, record direct rollback as prohibited at that checkpoint. That finding is useful before release. Hiding it behind a green deployment test would turn a known architectural constraint into an incident-time surprise.
4. Branch the rehearsal at three checkpoints
Run independent branches from the pinned baseline. In each branch, apply the exact upgrade payload, advance to its named checkpoint and execute the recovery action against that checkpoint's state. Preserve transaction order and the arguments for every intervening operation.
The first checkpoint is immediately after the upgrade transaction. If installation and migration occur atomically, this checkpoint already includes that migration; there is no accessible production state between the two. Do not manufacture an intermediate recovery window that the actual transaction never exposes.
The second checkpoint is after any separate migration or initialization work. The third is after representative user activity and external interactions. These checkpoints describe progressively different conditions, not a guarantee that recovery becomes harder in a predictable numerical way.
Rehearsal branches for the hypothetical vault. A local snapshot resets the test fixture; recovery acts on the state produced within a branch.
Use a small scenario matrix tied to the release's actual changes:
| Branch | State reached | Recovery attempt | Required observation |
|---|---|---|---|
| Installation | Upgrade payload completed | Supported return to prior code | Existing positions still behave correctly |
| Migration | Changed records converted | Approved repair or forward fix | Record meaning and ownership reconcile |
| New activity | Deposit and withdrawal requested | Checkpoint-specific recovery | Claims remain payable exactly once |
| External effect | Settlement adapter acted | Containment and reconciliation | External obligations remain accounted for |
| Interrupted operation | One step pending or reverted | Resume the recorded procedure | No duplicated action or lost request |
Use local snapshots to create repeatable starting conditions, then distinguish their resets from the action being tested. A test that upgrades, restores the baseline snapshot and calls an old function demonstrates the old fixture. It says nothing about old code running against the post-upgrade state. Keep the checkpoint evidence before any fixture reset.
Order matters within a branch. A withdrawal requested before migration can exercise a different path from one requested afterward. A deposit followed by a withdrawal may expose a unit conversion that either operation alone misses. Select sequences from the changed behavior and known invariants, rather than expanding into a large arbitrary matrix.
Include at least one deliberate incompatibility in the fixture or expectations. The harness should reject a recovery action that violates the declared unit or ownership rule. A suite that passes both the intended case and a clearly invalid case cannot support the release decision.
5. Exercise the authority path and the waiting period
Rehearse through the actual control contracts. If the production route requires a multisig proposal followed by a timelock, the local sequence should exercise the corresponding proposal, authorization and execution checks. Calling an implementation directly as an impersonated administrator bypasses the part of the system that determines whether recovery is available.
Maintain a clear distinction between simulated authority and operational availability. Impersonation can test behavior for a particular caller. It does not prove that enough people can access their signing devices during an incident. Advancing the local clock can test a delay condition. It does not measure detection time, approval time, network congestion or the time needed to inspect the result.
Record those durations separately. Use measured operational evidence where it exists and label unmeasured estimates. The contract waiting period, signing process and detection path together determine what the system can do while affected operations remain available. Rehearse that exposure interval with realistic allowed actions, including a transaction already queued before a pause.
Recovery belongs in the same release conversation as audit remediation. The blockchain delivery process documented by Pharos Production includes testing, internal review, external audit coordination and staged deployment. Attach the checkpoint manifest to that process so a reviewer can identify which recovery payload was assessed and which subsequent writes invalidate it. This adds an inspectable condition to delivery without turning an audit into a guarantee of reversibility.
Test the pause boundary precisely. Identify the functions a guardian can stop, functions that remain callable and the authority required to resume them. A pause that prevents deposits but leaves an unsafe settlement path open does not provide the containment assumed by the runbook. A pause that blocks every exit may also change the recovery obligations to users.
Include a stale operation in the scenario. After choosing a recovery path, verify what happens to an already scheduled upgrade or maintenance transaction. Cancel it where the governance design permits cancellation, or demonstrate that its preconditions prevent execution against the recovered state. Leave no unexplained pending payload capable of undoing the repair.
6. Verify user outcomes after the recovery transaction
A successful receipt establishes that a transaction executed without reverting. It does not establish that balances, claims and permissions are correct. Capture observations before the upgrade, at the failed checkpoint and after recovery, then reconcile the expected changes between them.
For the hypothetical vault, start with ownership and accounting. Every sampled withdrawal request should retain its owner, status and entitlement under the declared accounting model. A completed payment must not remain claimable. A pending payment must not disappear merely because the previous implementation ignores a field introduced by the upgrade.
Define conservation using the system's actual assets and liabilities. Track deposits, withdrawals, fees and any permitted gain or loss in consistent units. Explain rounding tolerances and their maximum aggregate effect. A bare comparison between the vault's token balance and total shares is usually not a complete accounting rule because the quantities may have different meanings.
Check behavior as well as getters. Have a representative user finish a withdrawal, create a new permitted request and encounter the intended restriction on an invalid request. Verify allowances, role membership, pause settings and request identifiers where the release can affect them. A readable position is not necessarily a usable position.
Record the population covered by the checks. Sampling representative accounts is useful for scenario design, but it does not prove that every mapping entry survived a migration. If the migration touches a bounded set of records, reconcile that complete set. If the population is large, define the mechanism that establishes coverage, such as an enumerated migration input with totals and per-record checks. State the residual uncertainty rather than presenting a sample as exhaustive evidence.
Then examine consumers outside the upgraded contract. An indexer may have processed events emitted before recovery. A keeper may hold a pending job. An adapter may have accepted an identifier that the old code no longer understands. Record how these components rebuild, reject stale work or reconcile their records. The chain's current storage is only part of the service state.
Make every assertion report expected and actual values. When one fails, preserve the smallest reproducible sequence that reaches the discrepancy. This gives a reviewer an explanation of the boundary that broke instead of a screenshot containing a red test name.
7. Select recovery from the state that actually exists
Choose a direct implementation rollback only when the deployed mechanism permits it and the relevant post-upgrade states remain compatible with the earlier code. Rehearse the exact historical bytecode and the actual return payload. Recompiling an old source tag under different settings creates a different artifact to review.
If a reversible migration is part of the design, test its inverse as a separate operation. Identify the information needed to restore the earlier representation, the records it will touch and the transaction boundaries. If the forward migration discarded information, the inverse requires another trustworthy source for it; calling the procedure a rollback does not supply the missing data.
For a migration spread across transactions, rehearse a partially processed population. Record which entries use the old representation and which use the new one. Check that a retry cannot convert the same record twice and that recovery does not assume every batch completed. Run the largest supported batch under the intended gas conditions and preserve the boundary at which another transaction is required. Local success with an artificially generous block gas limit would not establish that the same batch is executable on the target chain.
A forward repair may preserve valid new state while correcting the faulty behavior. Its acceptance test must cover both the original defect and the recovery obligations. Reusing the upgrade route is convenient, but the repair is still new code with new assumptions. Do not substitute confidence in the previous release for review of the repair.
Some checkpoints should permit containment only. After an external payment or cross-chain execution, the immediate action may be to stop further affected operations and establish a reconciled claim set. Compensation, migration or administrator-assisted processing can require a separate decision. Record the unresolved obligation and its owner before allowing unrelated activity to obscure it.
Keep the incident decision narrow. The manifest should identify the permitted action for each observed checkpoint and the evidence needed to choose it. An operator should not need to invent a new storage transformation while deciding whether the system is safe to resume.
8. Preserve a manifest that another reviewer can reproduce
The manifest is the durable output of the rehearsal. Store it beside the release artifacts and bind its evidence to immutable hashes. Keep confidential endpoint credentials and signing material out of it; a provider label and the required access characteristics are enough to describe the environment.
The following is a template, not a completed run:
rehearsal_id: vault-release-candidate-01
status: NOT_RUN
baseline:
chain_id: REQUIRED
block_number: REQUIRED
block_hash: REQUIRED
deployment_inventory_hash: REQUIRED
release:
source_commit: REQUIRED
build_and_dependency_manifest_hash: REQUIRED
implementation_code_hashes: REQUIRED
upgrade_payload_hash: REQUIRED
scenario:
checkpoint: REQUIRED
ordered_transactions_hash: REQUIRED
simulated_privileges_and_time_changes: REQUIRED
recovery:
permitted_action: REQUIRED
payload_hash: REQUIRED
authority_and_delay_evidence: REQUIRED
verification:
invariant_definitions_hash: REQUIRED
expected_and_actual_results_hash: REQUIRED
covered_records_and_exclusions: REQUIRED
receipts_traces_and_state_diff_hash: REQUIRED
decision:
reviewer: REQUIRED
outcome: UNDECIDED
unresolved_obligations: REQUIRED
invalidation_conditions: REQUIRED
Give every scenario its own record or unambiguous entry. Do not overwrite a failed rehearsal with a successful rerun. Preserve the failed payload and observations, then link the revised run to the change that resolved them. The release decision should identify the exact accepted record rather than whichever file currently has the newest timestamp.
Retain enough environment information to reproduce the observation: tool version, chain configuration, dependency versions and any mock behavior. An archived result without a usable fixture can help incident analysis, but it is weaker evidence for a future release than a procedure another engineer can execute.
Keep local transaction receipts clearly labeled as rehearsal evidence. A local receipt is not a production receipt, and a local block height is not evidence that a live operation occurred. Store production approvals and eventual deployment observations in separately identified records linked to the same release.
9. Make the release claim expire when its assumptions change
Approve recovery per checkpoint. A release can legitimately support direct rollback before user activity while requiring forward repair afterward. Put that distinction in the operational runbook and the release decision so the team knows when the simpler route stops being valid.
Define invalidation conditions before deployment: different implementation bytecode, changed migration input, altered authority, a new dependency configuration or a newly enabled operation outside the rehearsed states. A recent rehearsal is not automatically current when the conditions it tested have changed.
Specify abort conditions as executable preflight checks wherever possible. An unexpected current implementation hash, an unknown migration version or an unresolved settlement record should stop the selected procedure before its first write. Test those rejection paths as well as the accepted path. Record who investigates an abort and how the incident remains contained while the assumption is unresolved. This prevents a prepared recovery payload from becoming an instruction to proceed regardless of the state operators actually find.
Before the real upgrade, compare the intended payload and deployment inventory with the accepted manifest. Review material state drift since the pinned block and rerun the affected scenarios when it changes their preconditions. After deployment, inspect the actual implementation and resulting state before resuming affected operations. Resolve an uncertain transaction outcome from on-chain evidence before submitting another potentially duplicative action.
The defensible claim is specific: this release has a demonstrated recovery procedure for these states, with these remaining limitations. That gives operators a usable decision under pressure and gives reviewers evidence they can challenge. Upgradeability becomes an operational property only when the system can reach an acceptable state through an authorized, understood and rehearsed procedure.
More insights to read
- Five Smart Contract Upgrade Patterns Compared
- Upgradeable Solidity Smart Contracts. Part 1 — Versioning
- Smart Contracts. Their Potential and Real Limitations. Part 2
- Smart Contracts. Their Potential and Real Limitations. Part 1
- Web3. Smart Contracts. Oracles. Part 1
About the author
Dmytro Nasyrov. Photo supplied by the author.
Written by Dmytro Nasyrov PhD, software architect with 24 years of production experience. Dmytro is the founder and CTO of Pharos Production. He works on production software architecture for FinTech, AI, Web3 and blockchain systems.
Top comments (0)