Edy Cu

Posted on Sep 10

400 rounds, and the bug never showed up. So I stopped rolling dice.

#showdev #testing #solidity #webdev

The most expensive bug in the game I just shipped could only fire on 1.01% of rounds.

Here is the shape of it. Replay is a betting game where the final score is public before you bet — the game ended 8–5, and what you buy is which of the 1,287 orderings of those 13 points actually happened. It runs on a casino platform whose host contract caps what a game may pay out:

maxAllowedPayout = escrowedStake + reservedProfit

Zero slack. The game quotes its own numbers up front — quoteCaps declares how much profit to reserve, quoteRiskParams declares the maximum payout — and then at settlement it returns the payout it actually owes. Three call sites, one number. If any of them re-derives that number independently and lands one wei high, the transaction reverts.

Not "sometimes reverts". It reverts every single time a player hits the top ticket. Which is the 96.03× ticket. Which wins on 13 orderings out of 1,287 — 1.0101% of rounds.

Read the failure mode again, because that is the nasty part: the rarer the outcome, the better the bug hides. It survives casual testing perfectly. It survives a demo. It fails on the one round somebody actually remembers.

Why I couldn't test it the obvious way

The obvious approach is to play until it happens.

So I played 400 rounds against the platform's local simulator — its own chain, a real ECVRF node rather than a mock, the full contract lifecycle. The tail never landed.

That is not surprising, and it is also not evidence. At p = 1.0101%, the odds of 400 rounds all missing are 0.98989^400 ≈ 1.7%. Unlucky, not informative.

And the flip side is worse. Suppose the tail had landed once, and paid correctly. What would I have learned? That one rank out of 1,287 works. Either way I would have shipped a game whose most valuable outcome was checked by luck.

Stop sampling, start enumerating

Two properties of the contract turned out to make the entire outcome space reachable.

First, settlement is a pure function.

function onRandomness(SessionContext calldata ctx, bytes32 randomness)
    external
    pure
    returns (StepResult memory)

It takes randomness as an argument and returns the result. Nothing is written, nothing is read from storage. So I can call it from a script, with randomness I choose, as often as I like, against the deployed contract — no VRF, no waiting, no state to reset between calls.

Second, the draw is a straight window read.

function _drawRank(bytes32 randomness, uint256 total) private pure returns (uint256) {
    uint256 limit = (65536 / total) * total;
    for (uint256 w = 0; w < 16; w++) {
        uint256 v = (uint256(uint8(randomness[2 * w])) << 8) | uint256(uint8(randomness[2 * w + 1]));
        if (v < limit) return v % total;
    }
    return 0;
}

That is rejection sampling over the sixteen 16-bit windows of the word, because plain byte % n is biased — and a biased draw is not a rounding error when the draw is the odds.

The useful consequence is the first line. For total = 1287, limit = 64350, so a bytes32 whose leading two bytes are R yields rank R. The randomness is addressable. Every outcome on the board has a preimage I can write down.

At which point the test stops being a test and becomes a sweep:

let wins = 0, losses = 0;

for (let r = 0; r < 1287; r++) {                    // every rank on the board
  const rnd = '0x' + r.toString(16).padStart(4, '0') + '00'.repeat(30);
  const res = await pub.readContract({
    address: game, abi, functionName: 'onRandomness', args: [ctx, rnd],
  });
  const [, , pathId, mask, maxDef, , won] = decodeAbiParameters(STATE, res.newGameState);

  if (pathId !== r) throw new Error(`rank ${r} decoded as pathId ${pathId}`);

  if (won) {
    wins++;
    if (res.payout !== maxPayout) throw new Error(`rank ${r}: payout != quoted maxPayout`);
    if (res.payout > cap)         throw new Error(`rank ${r}: PAYOUT EXCEEDS FACET CAP`);
    if (maxDef < 4)               throw new Error(`rank ${r}: won FOUR DOWN with maxDeficit ${maxDef}`);
  } else {
    losses++;
    if (maxDef >= 4) throw new Error(`rank ${r}: maxDeficit ${maxDef} but lost`);
  }
}

Real output:

8-5 FOUR DOWN, wager 1e18
  quoteCaps.maxReservedProfit  95030000000000000000
  facet cap = escrow + reserved 96030000000000000000
  quoteRiskParams.maxPayout     96030000000000000000
  cap == maxPayout ? YES — zero slack

swept all 1287 ranks: 13 wins, 1274 losses
closed form says C(13, 5-4) = C(13,1) = 13  ->  MATCH

first winning rank 0: mask 0000000011111, maxDeficit 5
  payout 96030000000000000000 = 96.03x
  equals the facet cap exactly: YES — Trap 1 holds at the extreme

Seconds, not 400 rounds. And the claim it produces is different in kind.

The two things sampling could never have given me

The winner count becomes checkable against the mathematics. Because winning ranks are counted rather than observed, I can compare the total against a closed form: this ticket should win on exactly C(13, 5-4) = C(13,1) = 13 orderings. It wins on 13.

That check catches a class of bug sampling structurally cannot see. If the payout arithmetic is perfect but three extra ranks are classified as winners, every sampled test still passes — you saw a win, it paid the right amount. The set of winners was wrong, and the set of winners is the thing that costs money.

One wei becomes visible. cap and maxPayout printed on adjacent lines, compared as integers, at the exact multiplier where the cap binds hardest.

There was a second trap in the same family, incidentally, and it is the reason I now distrust anything that only misbehaves at the top of a distribution. The host applies a game's reservedProfitDelta before finalising the session — so a game that helpfully releases its reserved profit at settlement collapses its own cap to the wager and reverts every win above 1×. The correct value is a slightly uncomfortable-looking zero:

// Do NOT release reserved profit here. _processStepResult applies this delta
// BEFORE _finalizeSession. Returning -maxReservedProfit would zero
// session.reservedProfit, collapse maxAllowedPayout to the wager, and revert
// every win above 1x. _finalizeSession zeroes it itself.
reservedProfitDelta: 0,

A 1.75× ticket wins often enough that you would catch this one by playing. A 96.03× ticket would have shipped broken.

What actually transfers

None of the above is really about blockchains. The property that made the sweep possible is this:

The randomness was an argument, not an ambient effect.

onRandomness does not call an RNG. It does not read a clock, a block hash, or a global. The variation is handed to it, which is precisely what makes the outcome space addressable — and enumeration is only available to code whose inputs you can name.

Most code forfeits this by accident. Math.random() inside the function. Date.now() inside the function. A fetch inside the function. Every one of those turns a countable space into a slot machine you have to pull.

The second half is a sizing rule: enumerate when the space is small, and it is small far more often than we assume. 1,287 orderings is small. All five boards together are 4,082, and the full sweep is a Block A line in npm run bench. Most genuinely rare paths in ordinary systems live in spaces of a few thousand too — status code × retry count, timezone × locale boundary, the dozen shapes a malformed header comes in. We sample those because sampling is the habit, not because the space is big.

Exhaustive beats lucky. It is usually also faster.

Honest limitations

The sweep proves rank → ordering → win/loss → payout. It does not prove the VRF is uniform — that is the network's property, not mine. What I can prove is that the mapping from a draw to an outcome is a bijection, so uniform in implies uniform out.
In standalone mode, the board deal itself is drawn client-side with crypto.getRandomValues. It is a bet parameter rather than an outcome, and expected return is flat at 0.97× on all 25 tickets, so there is nothing to grind — but it is not chain-derived, and I would rather say so than let you find it.
cancelStuckRandomness is not implemented. The local simulator rejects the call, so it is unbuilt rather than tested-and-working.
No mainnet deploy — there is no mainnet for this platform yet.

DEV Community