DEV Community: Alexey Spinov

Lost Update: When Two AI Agents Edit One File, One Silently Wins

Alexey Spinov — Thu, 30 Jul 2026 02:29:50 +0000

A lost update between AI agents: two agents edit the same resource, both writes return success, and one is silently gone with nothing in the system saying a word. It happens when both read the same version and the last writer overwrites the rest. The fix is a pre-write compare-and-set gate, not a bigger log.

I ran it. With 5 agents committing to one shared file under a worst-case schedule, where every agent reads before anyone writes, 4 of the 5 contributions vanished from the final state, and every one of those 5 writes had been acknowledged. The final file held exactly one agent's work. The other four were paid for, ACKed, and silently overwritten. And that worst case is not rare: in a fair sample of interleavings at N=5, some write was lost in essentially 100% of runs.

This is the lost update anomaly, and it is older than AI agents by decades. What is new is that each vanished write burned real model tokens, and the loss leaves no error, no exception, no red log line. So I built a small offline simulator to make the loss countable, and to test the one class of fix that actually stops it: a version check before the write lands.

AI disclosure: I wrote lostwrite_sim.py with an AI assistant and ran it myself, offline, on Python 3.13.5, standard library only. No network, no keys, no threads, no funds. Concurrency is simulated by explicit interleaving schedules, not real parallelism, so the output is reproducible. Every number and hex string below is pasted from a real local run. Three runs produced byte-identical STDOUT with sha256 1df08c5e38894622314a2b052684303853b9b8fa9508eaabdfd877970c88983b. The linked specs and articles are other people's work.

In short:

When N agents read the same version of a resource, each does its work, and each writes back, the last writer wins and everyone in between is silently overwritten. In the worst case at N=5 that was 4 lost writes out of 5, all acknowledged.
Whether it happens at all depends on the schedule. In a fair sample of interleavings the loss hit 75.0% of runs at N=2 and effectively 100% by N=5, but a quarter of N=2 runs serialized and lost nothing. So one green test run proves nothing.
An append-only log records every write, so you can prove each write happened. It does not help you recover one. Reading the current value still returns a lost update, because logging preserves the fact of a write, not the presence of its change.
A pre-write compare-and-set (check the version has not moved since you read it) drops lost updates to 0 by construction. Measured, it lost nothing in all 46,333 fair interleavings I sampled, at every N.
The gate does not save money. It costs more. In the worst case at N=5 it added 4 work-units of retries on top of 5, so total work went from 5 to 9. It buys integrity and a countable collision, not a refund.
Turn the version check off inside the same gate and the loss comes straight back (4 again). That falsifier is the proof that the compare, not the scaffold around it, is doing the work.

This is not a toy concern anymore. The moment you run two sub-agents, or a swarm, or a retrying planner that spawns workers, you have multiple actors writing shared state: a plan file, a scratchpad, a database row, a GitHub issue. @opsveritas wrote up the paired version of this: one agent times out, three more do not notice, and work gets paid for that no one will ever read. Lost update is the same failure seen from the write side.

What is a lost update between two AI agents?

A lost update happens when two actors read the same version of a resource, both compute a new value from what they read, and both write it back. The second write is based on a snapshot that predates the first write, so committing it erases the first. Neither actor sees an error. The classic definition lives in database isolation theory, and agents inherit it the instant they share a resource without a write precondition.

The mental model I used in the simulator is deliberately concrete: N agents each append their own section to one shared file. The correct final file contains all N sections. A lost update is a section that made it into an acknowledged write and then disappeared from the final file.

Here is the store. It holds a version number and the set of contributions currently present.

class VersionedStore:
    def __init__(self):
        self.version = 0
        self.value = frozenset()
        self.wal = []  # append-only log of every accepted write

    def write_lww(self, agent_id, new_value):
        """Last-writer-wins: overwrite unconditionally, always ACK."""
        self.value = frozenset(new_value)
        self.version += 1
        self.wal.append((self.version, agent_id))
        return "ACK"

    def write_cas(self, expected_version, agent_id, new_value, enforce=True):
        """Commit only if the version has not moved since this agent read."""
        if enforce and self.version != expected_version:
            return "CAS_FAIL"
        self.value = frozenset(new_value)
        self.version += 1
        self.wal.append((self.version, agent_id))
        return "ACK"

Each agent runs three steps: READ the store, WORK (build snapshot | {my_id} and spend one unit), then WRITE. Concurrency is not threads. It is an explicit interleaving, and I run it two ways on purpose. The first is a deterministic worst case: every agent reads before anyone writes, so every read is stale. That is the adversarial upper bound, not a typical run, and I use it for the headline because it is the clearest picture of the failure. The second is a fair sampler: at each tick pick one still-ready agent uniformly at random (random.Random, seeded, so the whole run is reproducible to the byte), and draw tens of thousands of independent interleavings to get a distribution. The worst case shows the failure; the fair sample shows how often it actually bites.

A note on that, because the first version of this tool got it wrong and it matters. I originally picked agents with a hand-rolled linear congruential generator and next() % n, which reads the least-random low bits. At N=2 that strictly alternates for every seed, so it silently pinned the worst case and printed it as if it were a seeded sample. random.Random.randrange draws from the full generator state and does not have that bias. The fair numbers below come from the fixed version.

One thing I want to be honest about up front, because it decides how you read every number here. The costs in this tool are counts, not dollars. COST_UNIT is an accounting constant set to 1, labelled in the source as "not dollars, not a measured price." Every money figure is an integer counter times that constant. If you want your dollars, multiply the run counts by your own measured per-run token cost. I did not measure a token price, and I am not going to pretend a synthetic schedule tells you one.

The anomaly at its worst: N=5

5 agents, the worst-case schedule (everyone reads before anyone writes), gate off versus gate on. This is the headline block straight from the run:

WORST CASE  N=5, maximum contention (every agent reads stale)
--------------------------------------------------------------------
                              NO_GATE (LWW)      GATE (compare-and-set)
  writes acknowledged         5                  5
  contributions in final      1                  5
  contributions expected      5                  5
  lost_updates (ACKed, gone)  4                  0
  integrity_ok (present==all) False              True
  cas_failures                0                  4
  retries                     0                  4
  refusals                    0                  0
  total_runs (work executed)  5                  9
  total_cost (runs*COST_UNIT) 5                  9

Read the left column first. Five writes acknowledged, one contribution in the final state, four lost. The lost_updates number is not printed from a flag that says "no gate, so print 4." It is re-derived: the tool takes the set of agents whose write was ACKed at least once, subtracts the set actually present in the final value, and counts what is left. Four writes said success and are not in the file.

The right column is the same schedule with a version check on the write. Zero lost. All five contributions present. And I need to be blunt about why that zero is a zero, because it would be dishonest to sell it as a surprising discovery: it is true by construction. A compare-and-set write only commits when the version has not moved, which means it commits on top of a snapshot that already contains every prior commit. It cannot drop one. The interesting, measured part of the right column is not the zero. It is the price of the zero, which I will get to.

Tracking is not control

Here is the part I keep having to relearn. "Just log every write" feels like it should help. It does not.

TRACKING IS NOT CONTROL  (NO_GATE, worst case, N=5)
--------------------------------------------------------------------
  writes recorded in append-only log (WAL) : 5
  WAL entry shape                          : (version, agent_id), no value
  contributions surviving in final state   : 1
  final state contains contributions       : (4,)
  writes logged and ACKed but gone on read : 4

The append-only log has all five writes. Every one. It proves that five agents each wrote, and it tells you which agent and at what version. That is enough to know you were charged for five and kept one. It is not enough to hand any of the four back: my WAL entry is (version, agent_id), the fact of the write, not the value it carried. A richer log that stores the content, like the separate git-refs @dipankar_sarkar described for two coding agents editing the same issue, keeps both versions and reconciles after, and for a merge-later workflow it is the right tool.

But look at the last line, and notice it holds no matter how fat the log gets. The log knows about all five writes, and reading the current value still returns four lost updates. Recording a write and preventing its loss are different operations at different times. The log is a receipt written after the fact. It cannot un-overwrite the file, any more than a receipt read after the fact can recover a field that was never stored. This is the same franchise as a spend cap that counts tokens but cannot stop the loop: tracking tells you what happened, control changes what is allowed to happen. Only a write that runs before the commit gets to reject the stale one.

The gate, and its honest price

The fix is a precondition on the write. Optimistic concurrency, the lock-free version: read a version, do your work, and at commit time refuse if the version moved. It has a formal name, optimistic concurrency control, and the primitive under it is compare-and-swap. A pessimistic lease or lock would also work; it just makes the agent wait instead of retry. Either way the shared idea is the same: no write lands on state you did not read.

res = store.write_cas(a.base, a.aid, a.newval, enforce=not disable_version_check)
if res == "ACK":
    a.acked = True
else:
    cas_failures += 1
    if a.retries < max_retries:
        a.retries += 1
        a.pc = "READ"      # re-read the latest, re-work, re-write
    else:
        a.refused = True   # fail-closed: nothing silently applied

Now the cost, which is the whole reason this post is not called "how the gate saves you money." It does not. Look again at the headline: NO_GATE ran 5 work-units, GATE ran 9. The gate added 4 work-units of retries, because each stale writer had to re-read fresh state and redo its work before it could land. The delta is positive, and it is positive by design:

  Delta cost of the gate      = 9 - 5 = 4 work-units (>=0: the gate
                                costs MORE, it does not save tokens)
  Spend destroyed without gate= lost_updates * COST_UNIT = 4 work-units
                                (already paid, silently gone from state)

Two numbers, two different things, and it is easy to blur them into a false savings pitch. The 4 destroyed work-units under NO_GATE are money you already spent on work that then vanished. The gate does not refund them. It cannot reach back into a run that already happened. What it does is stop the next four from vanishing, and it charges you 4 units of retries to do it. If your worry is the token bill, the gate is a cost, not a saving. If your worry is that four agents did real work and the result quietly disappeared, the gate is the thing that keeps that from being silent.

That is the contrarian bit, and I will state it as a falsifiable claim: a pre-write concurrency gate on shared agent state increases token spend and is still worth it, because it converts a silent unbounded loss into a visible, counted, retryable event. If your data shows a gate that reduces total spend under contention, I would genuinely like to see the workload, because my own run says the opposite.

Is the gate doing the work, or is my test scaffold cheating?

This is the failure mode I trust least in myself, so the tool has a falsifier for it. The classic mistake is to build a "gate" whose zero comes from the test scaffold rather than the mechanism. So F1 keeps the entire gate code path and disables one thing: the version check inside write_cas.

FALSIFIER F1  gate code path, version-check DISABLED (worst case, N=5)
--------------------------------------------------------------------
  lost_updates with the compare removed    : 4  (was 0 with it on)

Four again. Same as no gate. The retry loop, the counters, the refusal path, all still there. Remove only the comparison of expected version to current version, and the loss returns in full. That is the proof I actually care about: the compare is the mechanism. Everything else is plumbing.

Does it get worse with more agents?

Two questions live here, and they have different answers. How bad can it get, and how often does it get bad at all. The worst case answers the first. The fair sample answers the second, and it is the more useful number.

First the worst case, swept over N. Every agent reads before anyone writes, so all but the last writer are clobbered. The N=1 row is the negative control: one writer, no one to overwrite, zero loss.

WORST-CASE SWEEP over N (maximum contention, every read stale)
----------------------------------------------------------------------------
  N | ng.lost ng.integ | g.casfail g.retry g.refuse g.lost g.integ | dCost
  --------------------------------------------------------------------------
  1 |      0 True    |        0       0        0      0 True    |     0  (degenerate: 1 writer)
  2 |      1 False   |        1       1        0      0 True    |     1
  3 |      2 False   |        2       2        0      0 True    |     2
  5 |      4 False   |        4       4        0      0 True    |     4
  8 |      7 False   |        7       7        0      0 True    |     7

In the worst case the loss is exactly N-1: everyone but the last writer is gone, and the gate pays one retry per clobbered writer to fix it. But nobody schedules their agents to lose. The honest question is what a fair draw of interleavings does, so I sampled 40000 // N of them per N and counted the distribution.

FAIR SWEEP  uniform random interleavings (random.Random, Mersenne Twister)
----------------------------------------------------------------------------
  base_seed=20260728, trials per N = 40000 // N
  N | trials | NO_GATE loses>=1 | serializes (0 lost) | mean lost | max | GATE loses>=1
  --------------------------------------------------------------------------
  2 |  20000 |   75.0% (14994) |   25.0% ( 5006)  |     0.75 |   1 | 0
  3 |  13333 |   97.3% (12972) |    2.7% (  361)  |     1.59 |   2 | 0
  5 |   8000 |  100.0% ( 7999) |    0.0% (    1)  |     3.35 |   4 | 0
  8 |   5000 |  100.0% ( 5000) |    0.0% (    0)  |     6.14 |   7 | 0

This is the number I would actually put in a design doc. At N=2 a quarter of interleavings serialize on their own and lose nothing, which is exactly the trap: two agents, run it a few times, watch it pass, ship it. Then it loses data in three runs out of four. By N=3 the safe fraction is down to 2.7%, and by N=5 it is one interleaving in eight thousand. So the anomaly is not universal on small N, and it is close to certain past a handful of writers. That is a worse story than "it always happens," not a better one, because "usually fine" is the thing that gets shipped.

The gate column is the point of the whole exercise. Across all 46,333 fair interleavings I drew, over every N, compare-and-set lost an update exactly zero times. That is a measured claim, not a hand-wave: the by-construction argument says a CAS write cannot drop a prior commit, and 46,333 independent draws agree. NO_GATE integrity is False on every row with a second writer; GATE integrity is True on every one of them; and the gate's delta cost is never negative.

When the gate runs out of retries

A gate that only works when it can always retry is not much of a gate. So the last thing I checked is the ugly case: heavy contention with zero retry budget. What does it do when it cannot win the race?

FAIL-CLOSED  GATE with max_retries=0 (worst case, N=8)
--------------------------------------------------------------------
  cas_failures                             : 7
  refusals (VISIBLE, counted)              : 7
  refused agent ids                        : (1, 2, 3, 4, 5, 6, 7)
  lost_updates (SILENT)                    : 0
  integrity_ok                             : False

Seven agents lost the race and, out of retries, they refused. Loudly. integrity_ok is False, which is correct, because seven contributions genuinely did not make it. But lost_updates is 0. Nothing was silently overwritten. The difference between this and the NO_GATE case is the entire point: NO_GATE gives you integrity_ok=False with a silent 4, and you find out never. The gate gives you integrity_ok=False with seven refusals by explicit id, and you find out immediately. One failure is invisible, the other is a list of names. That is what "fail-closed" buys, and the tool exits non-zero if that property ever breaks.

What this is, and what it is not

It is a simulator, not a measurement of production. Two kinds of number live in it and they deserve different trust. The worst-case counts (headline 4 lost, N-1 across the sweep) are a deterministic upper bound: they are what a maximally adversarial schedule does, not what a typical one does, and I label them that way. The fair-sweep fractions (75.0% at N=2, 97.3% at N=3, ~100% at N=5 and N=8) are Monte Carlo estimates over tens of thousands of uniform-random interleavings, so they carry the usual sampling error of a proportion and would shift a little on a different base seed. What does not shift is the direction: the zero-loss fraction falls hard as writers are added, and the gate's zero holds in every sampled interleaving. None of this is a token-price measurement. I sampled one model of "random interleaving," a uniform scheduler; a real runtime with its own scheduling could sit anywhere between my worst case and my fair sample, and I have not measured a real one.

I also did not measure a token price, and I will not dress the accounting unit up as one. And this is not the revert-guard problem, where a single agent reintroduces reverted code, nor is it about a single agent double-charging on a retry. It is specifically two-or-more writers, one version, no precondition. If you run sub-agent dispatch or any fan-out where workers share a resource, this is the write-side gate that belongs next to your pre-execution checks. And it is a distinct axis from mandate freshness: freshness asks whether an old approval is still valid in time, this asks whether two live writes can both survive.

Run it yourself

Standard library only, offline, no keys, no funds, a few seconds (the fair sweep draws about 92,000 interleavings). run_all.sh runs the selftest, then three full runs, compares them byte for byte, and prints the sha256 of each.

interpreter: Python 3.13.5

self-test: PASS
run 1: exit=0 sha256=1df08c5e38894622314a2b052684303853b9b8fa9508eaabdfd877970c88983b
run 2: exit=0 sha256=1df08c5e38894622314a2b052684303853b9b8fa9508eaabdfd877970c88983b
run 3: exit=0 sha256=1df08c5e38894622314a2b052684303853b9b8fa9508eaabdfd877970c88983b
determinism: 3 runs byte-identical

The report body also self-reports its own sha256 (57e02acaa779e066d4736bb00b9ef1a2f4eeec21d886c626879cfd1a67db661e), so you can tell at a glance whether your run matches mine. The selftest is the contract: INV1 requires the worst case to lose exactly N-1, INV2 requires the gate to lose nothing there, INV3 requires the fair sample to lose in some but not all interleavings on small N, INV4 requires the gate to lose nothing in any sampled interleaving, F1 through F4 are the falsifiers, and any failed assertion exits 1 with FAIL. A gate that cannot fail its own tests is decoration.

Follow along if you want the numbers from the next teardown in this series. And if you are running more than one agent against shared state right now, tell me in the comments: what is the worst silent overwrite you have hit between two agents, and did you catch it before the commit or only when the result was already gone? I suspect, for most of us, the honest answer is "only when it was already gone."

x402 Signs the Money, Not the URL. I Checked 18 Fields.

Alexey Spinov — Wed, 29 Jul 2026 01:12:59 +0000

An x402 payer signature does not cover the URL. It commits to the amount, the recipient, the token contract and the chain, and to nothing that says what you are paying for. I mutated 18 leaf fields of the payment payload published in the x402 spec: 8 changes left the signature verifying, 10 broke it.

I rebuilt the EIP-712 digest from that example, recovered the signer with my own secp256k1 code, then changed one field at a time. The 8 that still verify include the entire resource object.

Change the resource URL to a different host. The signature still verifies. Nothing in the payment path notices.

AI disclosure: I wrote x402_intent_gate.py with an AI assistant and ran it myself, offline, on Python 3.13.5, standard library only, no network, no keys, no wallet, no funds. Every number and every hex string below is pasted from a real local run. Three runs produced byte-identical STDOUT with sha256 6cfe746ec64d8a497b1cafe27ed351dab1f56f690d3195959f0993a6e57888a6. The spec text I quote is other people's work, linked inline.

In short:

The signed structure in the x402 exact EVM scheme is EIP-3009's TransferWithAuthorization(address from,address to,uint256 value,uint256 validAfter,uint256 validBefore,bytes32 nonce), wrapped in an EIP-712 domain that adds the token name, version, chain id and contract address. That is the whole list. There is no slot for what you are buying.
I enumerated every leaf field of the PaymentPayload example in the x402 v2 spec, mutated one at a time, and re-ran real ECDSA recovery each time. 8 of 18 mutations left the signature verifying. The 10 that broke it are money, token, chain and clock.
Nothing the payer signed can be checked afterwards either. A settled transferWithAuthorization leaves an ERC-20 Transfer and AuthorizationUsed(address indexed authorizer, bytes32 indexed nonce), and neither names a resource. The SettlementResponse has seven fields, and exactly one of them, extensions, can carry the resource, because that is where the optional offer-and-receipt extension parks a receipt. That receipt is signed by the server, not by you.
The repair costs zero protocol changes. The nonce is 32 bytes the payer chooses, it is inside the signature, and it is emitted indexed on-chain. So stop wasting it on randomness: nonce = keccak256(canonical_intent || salt).
The gate runs before the signature exists. Of 17 constructed cases, 10 pass the facilitator's own verification steps, including a real validity-window check, and get refused by the gate anyway. All 3 legitimate ones still pass.

This is not a hypothetical protocol that nobody ships. Cloudflare announced a monetization gateway for x402 on 2026-07-01, which puts a 402 in front of anything sitting behind their edge. The number of agents that will sign one of these grew a lot faster than the number of people asking what the signature says.

What does an x402 payer signature actually commit to?

Start with the anchor, because everything after it depends on my arithmetic being right.

The x402 v2 specification publishes a complete PaymentPayload example in section 5.2.1, including a real 65-byte signature. The exact/EVM scheme spec republishes the same payload with one extra field, and defines what gets signed. I implemented keccak256 and secp256k1 from scratch, rebuilt the EIP-712 digest from that example, and ran public key recovery against that signature.

  [PASS] keccak256("") matches the published vector
  [PASS] keccak256("abc") matches the published vector
  [PASS] keccak256(TransferWithAuthorization type string) equals the TYPEHASH
         constant published in EIP-3009
  [PASS] secp256k1 base point is on the curve
  [PASS] n*G is the point at infinity
  [PASS] the signature published in the x402 spec example recovers to the
         payer address published in that same example
  [PASS] sign then recover round-trips on the throwaway demo key

  keccak256("")    c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470
  keccak256("abc") 4e03657aea45a94fc7d47ba826c8d667c0d1e6e33a64a036ec44f58fa12d6c45
  TWA typehash      0x7c7c6cdb67a18743f49ec6fa9b35f50d52ed05cbed4cc592e13b44501c1a2267
  recovered signer  0x857b06519e91e3a54538791bdbb0e22373e36b66
  authorization.from 0x857b06519e91e3a54538791bdbb0e22373e36b66

That last line is the part I care about. The address my code recovers from their signature equals the from address in their example. So the 32 bytes I am reconstructing are the 32 bytes that were actually signed, not a plausible-looking reimplementation of them.

The digest is 0xf256992871671abcb27ff92885a7afa46218724e5fc0bac35d050115aa1d22e6, and it is built from exactly this:

def eip712_digest(payload):
    """Rebuild exactly the 32 bytes an x402 exact/EVM payer signs."""
    acc = payload["accepted"]
    auth = payload["payload"]["authorization"]
    chain_id = int(acc["network"].split(":")[1])
    ds = domain_separator(acc["extra"]["name"], acc["extra"]["version"],
                          chain_id, acc["asset"])
    struct_hash = keccak256(keccak256(TWA_TYPE) + _addr(auth["from"]) + _addr(auth["to"])
                            + _u256(auth["value"]) + _u256(auth["validAfter"])
                            + _u256(auth["validBefore"]) + _b32(auth["nonce"]))
    return keccak256(b"\x19\x01" + ds + struct_hash)

Read the inputs. from, to, value, two timestamps, a nonce, and a domain made of the token name, version, chain id and contract. Count the fields that describe what you are buying: zero.

The Permit2 path in the same spec is stricter, not looser. Its witness type is keccak256("Witness(address to,uint256 validAfter)"), and the spec carries the comment post-audit: extra removed from Witness. The one place context could have been smuggled in got taken out by an audit.

Which x402 payload fields can change without breaking the signature?

I walked that same v2 section 5.2.1 example as an object tree, collected every leaf, and mutated them one at a time with a minimal type-preserving change. After each mutation the digest gets recomputed and the original 65 bytes get re-verified by full recovery. The signature field itself is excluded, since it is the artifact under test.

leaf field                             | minimally changed to   | signature
------------------------------------------------------------------------------
x402Version                            | 3                      | STILL VERIFIES
resource.url                           | value + "-mutated"     | STILL VERIFIES
resource.description                   | value + "-mutated"     | STILL VERIFIES
resource.mimeType                      | value + "-mutated"     | STILL VERIFIES
accepted.scheme                        | value + "-mutated"     | STILL VERIFIES
accepted.network                       | eip155:8453            | fails
accepted.amount                        | 10001                  | STILL VERIFIES
accepted.asset                         | 0x00000000000000000... | fails
accepted.payTo                         | 0x00000000000000000... | STILL VERIFIES
accepted.maxTimeoutSeconds             | 61                     | STILL VERIFIES
accepted.extra.name                    | value + "-mutated"     | fails
accepted.extra.version                 | 3                      | fails
payload.signature                      | not mutated            | the artifact under test
payload.authorization.from             | 0x00000000000000000... | fails
payload.authorization.to               | 0x00000000000000000... | fails
payload.authorization.value            | 10001                  | fails
payload.authorization.validAfter       | 1740672090             | fails
payload.authorization.validBefore      | 1740672155             | fails
payload.authorization.nonce            | 0xababababababababa... | fails
------------------------------------------------------------------------------
leaf fields enumerated                    : 19
excluded (the signature itself)           : 1
mutated                                   : 18
signature STILL VERIFIES after the change : 8
signature fails after the change          : 10

Two notes on that table before the counts, because the printout carries them and I would rather you read them from me than find them yourself. The example's extensions is an empty object, so it has zero leaves and gets no row, and it happens to be the one place a resource identifier could ever ride. Section 3 comes back to it. And accepted.network is the single field I gave a hand-written mutation, eip155:84532 to eip155:8453, because a malformed CAIP-2 string crashes the chain id parser instead of testing anything.

The 10 failures are the negative control, and they matter more than the 8 passes. If everything had come back STILL VERIFIES, the honest conclusion would have been that my checker was broken. It is not: touch the payer, the recipient, the amount, either timestamp, the nonce, the token contract, the chain id, or the token name or version, and recovery lands on a different address.

So the line is clean, and it is not an accident of my field ordering. Everything describing where the money goes is inside the signature. Everything describing what the money is for is outside it.

The 8 survivors split in two. Four of them say what is being bought: resource.url, resource.description, resource.mimeType and the scheme name. The other four carry no payment authority at all: the protocol version, the timeout hint, and the display copies of the amount and the recipient.

Two of those 8 deserve a note, because I do not want to overclaim. accepted.amount and accepted.payTo are display copies. They do not move money, since the money follows authorization.to and authorization.value. What they do is decide what your client renders and what your logs keep. A client builds the authorization from the requirements the server sent, so the unsigned copy is the input, and afterwards only one of the two versions can be proven. That is a smaller problem than the URL, and I am flagging it as smaller.

The row worth sitting with

Minimal mutations prove coverage. They understate severity. So I re-ran the uncovered fields with values chosen to be obnoxious, using the same 65 bytes:

change to the envelope                       | signature
------------------------------------------------------------------------------
resource.url -> a different host entirely    | STILL VERIFIES
resource.url -> a different path             | STILL VERIFIES
resource.description -> unrelated            | STILL VERIFIES
resource.mimeType -> unrelated               | STILL VERIFIES
accepted.amount display copy, 100x           | STILL VERIFIES
accepted.payTo display copy -> burn address  | STILL VERIFIES
accepted.scheme -> another scheme name       | STILL VERIFIES
accepted.maxTimeoutSeconds -> one hour       | STILL VERIFIES
add accepted.extra.assetTransferMethod       | STILL VERIFIES
------------------------------------------------------------------------------
adversarial envelope changes tried  : 9
signature still verifies after      : 9

api.example.com to evil.example.net, same signature, still valid. The money is pinned to the last atomic unit. The name of the thing the money bought is a free text field sitting next to it.

Why can't you reconcile this afterwards?

Because the data does not exist. This is the part that surprised me, and it is why I stopped looking for a post-hoc answer.

A settled transferWithAuthorization leaves two records on chain: the ERC-20 Transfer the token contract emits, and EIP-3009's own event AuthorizationUsed(address indexed authorizer, bytes32 indexed nonce). Neither names a resource. Two purchases at the same price to the same recipient for different resources look like this:

  buy A
    resource                  https://api.example.com/premium-data
    Transfer.from             0x857b06519e91e3a54538791bdbb0e22373e36b66
    Transfer.to               0x209693bc6afc0c5328ba36faf03c514ef312287c
    Transfer.value            10000
    AuthorizationUsed.nonce   0xf3746613c2d920b5fdabc0856f2aeb2d4f88ee6037b8cc5d04a71a4462f13480
  buy B
    resource                  https://api.example.com/cheap-data
    Transfer.from             0x857b06519e91e3a54538791bdbb0e22373e36b66
    Transfer.to               0x209693bc6afc0c5328ba36faf03c514ef312287c
    Transfer.value            10000
    AuthorizationUsed.nonce   0x63e1985efb2feb72dfaa78debef5dc246d15984f7895fb86294e43d9153d476d

  ERC-20 Transfer args identical between A and B : True
  AuthorizationUsed differs only in the nonce    : True
  fields naming the resource in either record    : 0

Those records are reconstructions from my fixtures using the field lists the spec and the EIP define. I did not query a chain, and nothing in that script touches money.

Then there is the SettlementResponse, and here I have to correct myself. My first draft of this post listed five fields and said flatly that none of them names the resource. Section 5.3.2 of the v2 spec has seven, and the two I dropped were the two that mattered:

The SettlementResponse the server hands back has seven fields in the x402
v2 spec, section 5.3.2, and here is the whole list:
    success      required   errorReason  optional
    transaction  required   payer        optional
    network      required   amount       optional
                            extensions   optional

Six of those seven cannot name a resource. The seventh can. extensions is exactly where the offer-and-receipt extension parks a receipt, at extensions["offer-receipt"].info.receipt, and that receipt carries a resourceUrl. The spec prints a worked example with "resourceUrl": "https://api.example.com/premium-data" sitting right there in the settlement response. So "no field names the resource" is false as an absolute, and I am glad it got caught before this went out.

The true claim is the narrower one, and it is the one this whole post is about: nothing the payer signed names the resource. The single slot that can name it is opt-in, and it is filled by the server. That extension is genuinely useful and I would turn it on. But it is signed by the service rather than the payer, its own text calls it an audit layer "without changing payment execution or settlement semantics", and the receipt is "privacy-minimal by default and intentionally omits transaction references to reduce correlation risk". A merchant-signed artifact that by default does not link to the transaction is a merchant's statement, not the payer's proof of what the payer decided to buy.

PaymentPayload.resource has the same shape of problem: it exists, the v2 spec marks it Optional, and it sits outside the signature.

Which leaves one place to stand. Not after the settlement. Before the signature.

This is not the stale mandate problem

I want to keep two failures apart, because they look alike and they are not.

In the mandate freshness gate the axis is time. The signature was honest when it was made, and the authority behind it walked out afterwards: revoked, expired, limit lowered. Everything cryptographic holds, and the question is whether the yes is still standing at execution.

Here the axis is content. The authority is perfectly live. The signature is fresh, valid, and inside every limit. It simply never said what it was for. A freshness check passes this case with full marks, and so does a signature check, because both are answering questions that have correct answers.

This is the same shape as tracking is not control, pushed one level down. Your spend cap counts tokens you can compute. In an x402 flow the counterparty names the price in the 402 response, so a sliding window guard is watching a number it did not choose. And a receipt read after the fact cannot recover a field that was never recorded.

The fix that needs no protocol change

The nonce is 32 bytes. EIP-3009 says they are random and payer-chosen. They sit inside the signed struct, and the contract emits them indexed on-chain in AuthorizationUsed.

That is a 32-byte payer-controlled channel that is already signed and already published, and we currently fill it with noise.

I expected to be arguing that this is merely protocol-legal. It turns out the spec argues it for me. EIP-3009's own Security Considerations say that where cross-use is a risk, "the app developer could dedicate some leading bytes of the nonce as an identifier to prevent cross-use". Putting meaning in those bytes is a sanctioned use, not a loophole I found. The one caveat worth keeping: the spec says the nonce is randomly generated, and what preserves that property here is the 32-byte salt, not the intent. Hash a bare intent with no salt and you get a nonce that repeats and that anyone can grind.

def canonical_intent(intent):
    """One line per field, fixed order, newline separated. No JSON ambiguity."""
    return ("x402-intent/1\n"
            "method: %s\n"
            "url: %s\n"
            "body-sha256: %s\n"
            "class: %s\n"
            "max-atomic: %d\n"
            "asset: %s/%s\n"
            "payTo: %s\n" % (
                intent["method"].upper(),
                normalize_url(intent["url"]),
                intent.get("body_sha256") or "-",
                intent["resource_class"],
                int(intent["max_atomic"]),
                intent["network"], intent["asset"].lower(),
                intent["payTo"].lower())).encode()


def intent_nonce(intent, salt32):
    if len(salt32) != 32:
        raise ValueError("salt must be 32 bytes")
    return "0x" + keccak256(canonical_intent(intent) + salt32).hex()

From the run:

canonical_intent for buy A, exactly the bytes that get hashed:
    | x402-intent/1
    | method: GET
    | url: https://api.example.com/premium-data
    | body-sha256: -
    | class: market-data
    | max-atomic: 20000
    | asset: eip155:84532/0x036cbd53842c5426634e7929541ec2318f3dcf7e
    | payTo: 0x209693bc6afc0c5328ba36faf03c514ef312287c

  committed nonce   0xce1d40f2e8ccbe52ec6f127abb9752a42f0c469669bebd856a16649c558d4711
  same intent, same salt, recomputed                : MATCH
  host swapped to evil.example.net, same salt       : MISMATCH
  cap raised from 20000 to 30000, same salt         : MISMATCH
  different salt, same intent                       : MISMATCH
  nonce length in bytes                             : 32

It is still a valid 32-byte nonce, still unique, still opaque to everyone without the salt. Nothing changes on the wire. What changes is that the signature becomes impossible to produce without first having written the decision down, which is the property I actually wanted: the control produces the audit trail, instead of the audit trail being offered as a substitute for control.

Two things it does not do, stated plainly. It does not make the server deliver the resource you named. And it is a commitment, not a receipt: it proves what you decided, not what you received. If someone shows me a way to bind delivery from the payer side without an extension the merchant has to opt into, I would like to see it, because I could not find one.

The gate, run before anything is signed

Same family as the pre-execution gate and the pre-send transaction canary, aimed at the payment decision. decide() collects every reason instead of bailing on the first, and there is no code path that returns ALLOW on an error.

I built 17 cases and ran each one twice: once through the facilitator's own verification steps, once through the client gate. Every case is really signed with a throwaway key derived from a fixed string in the file, using RFC 6979 deterministic nonces so the bytes come out the same every run. The facilitator column is doing real recovery, not trusting an asserted boolean.

Step 3 of that verification list reads "Verify the authorization parameters (Amount, Validity Window) meet the PaymentRequirements", and the validity window is the part it is easy to quietly skip, since checking it needs a clock and a clock breaks determinism. My first version skipped it, checked only that validBefore was greater than validAfter, and still printed a column labelled offline-checkable. So the run pins a clock instead:

    FIXED_NOW = 1740672100  (inside the spec example's window 1740672089..1740672154)

That instant sits inside the window of the spec's own example, so freshness gets checked for real while the output never touches the wall clock. Falsifier F5 exists purely to prove the check fires: an authorization whose window closed before FIXED_NOW gets flagged outside-validity-window, and the spec example does not.

constructed case                                      | gate   | facilitator
------------------------------------------------------------------------------
legit: market data, 10000, everything allowlisted     | ALLOW  | ACCEPT
legit: image gen, 4000, inside its own class cap      | ALLOW  | ACCEPT
legit: second market-data buy, budget still fits      | ALLOW  | ACCEPT
quote is for a different URL than we asked for        | REFUSE | ACCEPT
resource host is not on the allowlist                 | REFUSE | ACCEPT
payTo is not on the allowlist                         | REFUSE | ACCEPT
chain is not the one we fund                          | REFUSE | ACCEPT
token contract is not the one we fund                 | REFUSE | ACCEPT
25000 for a class capped at 20000                     | REFUSE | ACCEPT
fits the cap, but 45000 already signed and unsettled  | REFUSE | ACCEPT
resource class the policy never heard of              | REFUSE | ACCEPT
plain random nonce, no commitment to any intent       | REFUSE | ACCEPT
nonce commits to a different intent                   | REFUSE | ACCEPT
control: signed value contradicts the quote           | REFUSE | REJECT
control: signed destination contradicts the quote     | REFUSE | REJECT
control: validity window closed at decision time      | REFUSE | REJECT
unusable input, required field missing                | REFUSE | n/a
------------------------------------------------------------------------------
cases constructed                                  : 17
gate ALLOW                                         : 3
gate REFUSE                                        : 14
facilitator ACCEPT (offline-checkable steps)       : 13
facilitator REJECT                                 : 3
pass the facilitator, refused by the gate          : 10

Ten cases sail through the facilitator's verification list and get stopped by the gate. That is not a criticism of facilitators, and it is worth being fair here: the spec is explicit that "the Facilitator cannot modify the amount or destination", and my run agrees, because both of those are inside the signature. The facilitator is doing its job correctly. Its job is to check the authorization against the server's requirements. Both of those come from the server. Your decision is not an input to that comparison anywhere in the verification list.

The three REJECT rows are there so you can see the facilitator checker is capable of saying no on each axis it claims to check: amount, destination and freshness.

One thing I should not let myself round off. Two of those ten, intent-nonce-missing and intent-nonce-mismatch, are refused for not using a convention I invented four paragraphs ago. Every x402 payment on earth today would trip them. That is a proposal, not a finding, and if you strip those two out the gate still catches eight cases with nothing more exotic than an allowlist and a cap.

One reason code is mine and I have not seen it elsewhere: budget-would-exceed-with-outstanding. An EIP-3009 authorization is a liability from the moment it is signed, not from the moment it settles. It carries validAfter and validBefore, and until one of those windows closes or the nonce is consumed, the money is committed. A cap that counts settled spend will happily sign the payment that puts you over, and then watch it land.

What would prove me wrong

Here is the single counterexample that ends this post: show me a field, in what the payer authorizes under the exact EVM scheme, that identifies the resource. One field and I am wrong.

Be clear about which part of that I ran and which part I read. The scheme defines three asset transfer methods, and my tool exercises one. EIP-3009 is the one measured above. Permit2 signs Witness(address to,uint256 validAfter), with the spec's own comment post-audit: extra removed from Witness. ERC-7710 sends delegationManager, permissionContext and delegator, and the spec says its verification "is performed entirely through simulation" of an ERC-20 transfer(payTo, amount). Two addresses and an amount. So the claim holds across all three, but only the first is a measurement and the other two are me reading the spec, which is a weaker kind of evidence and I would rather label it than launder it.

The tool ships six falsifiers, all PASS on the run above. F1 is the negative control on the checker. F2 checks the commitment is a function and not a coincidence. F3 requires the gate to allow every case built to be legitimate and refuse every case built to be wrong, which stops a gate that refuses everything from scoring well. F4 feeds it garbage, an empty object and a null, and requires REFUSE with bad-input on all of them. F5 proves the validity-window check actually fires. F6 re-runs the sweep and demands identical verdicts.

Then I tried to break it on purpose, four times, and it exited 1 every time: flipping one Keccak round constant, forcing decide() to return no reasons, turning the fail-closed branch into fail-open, and dropping the intent from the nonce so it hashed only the salt. A gate that cannot fail its own tests is decoration.

What this is not

It is not a conformance suite, and it does not implement upto, deferred or any Solana scheme. It does not talk to a chain, a facilitator or a wallet, so balance and simulation are not run at all and are never reported as passed. Those are steps 2 and 5 of the exact/EVM EIP-3009 list specifically; the Permit2 list numbers them 3 and 7, so the numbers are not portable even inside one document. The counts are counts of cases I constructed in one file. They are not frequencies, not samples, not rates observed anywhere in production, and no standard errors apply because nothing here is an estimate. Recount every one of them from the printout.

I also have no idea how common any of this is in the wild. I have not measured a single real x402 payment, and I am not going to pretend a count of seventeen constructed cases tells you anything about how often an agent overpays for the wrong URL. What the run does establish is structural: the field is not in the signature, so the check cannot be done later, no matter how careful your logging is.

Run it yourself

Standard library only, offline, no keys, no funds, about ten seconds. run_all.sh runs the self-test, then three full runs, compares them byte for byte and prints the sha256 of each:

interpreter: Python 3.13.5

self-test: PASS
run 1: exit=0 sha256=6cfe746ec64d8a497b1cafe27ed351dab1f56f690d3195959f0993a6e57888a6
run 2: exit=0 sha256=6cfe746ec64d8a497b1cafe27ed351dab1f56f690d3195959f0993a6e57888a6
run 3: exit=0 sha256=6cfe746ec64d8a497b1cafe27ed351dab1f56f690d3195959f0993a6e57888a6
determinism: 3 runs byte-identical

The report itself ends with report-sha256: 49cc227bc3cb64316dbea77387ea304f14b9ca3c677c9c707538bc0fe6bc3ccc, so you can tell at a glance whether your run matches mine.

The question I have not answered: the intent-committed nonce binds my decision to my money, and it does that with no protocol change and no cooperation from anyone. It still cannot prove the server gave me what I paid for. Every payer-side scheme I sketched for that ends up needing the merchant to sign something, which means it needs adoption, which means it is not something I can ship on my own next week. If you have found a payer-side way to bind delivery, I want to read it.

Follow along if you want the numbers from the next teardown in this series. And if you are running x402 in anything resembling production, tell me in the comments what your client does with the resource URL after it signs, because I suspect the honest answer for most of us is "logs it, unsigned, next to the amount".

Your Authz Checks the Caller. The Model Picked the Tenant.

Alexey Spinov — Sun, 26 Jul 2026 03:51:45 +0000

A confused deputy in an AI agent is not a broken authorization check. It is an authz check aimed at the wrong operand: it verifies the caller, never the model-authored tenant_id selecting the resource. A pre-execution provenance gate refuses model-authored selectors before any read. Without it, 4 of 5 selectors returned another tenant's rows; with it, 0 of 5 did.

Your agent is authorized to read invoices. This morning it read a different company's invoices, and every authorization check returned yes.

Nobody bypassed the gate. The caller was who it claimed to be, the token was valid, the role allowed the tool. The gate answered the exact question it was built to answer, correctly. The leak lived in an argument the gate never looked at: the tenant_id that selects which company's rows come back. And that argument was written by the model.

In short:

A confused deputy in an AI agent is not a broken auth check. It is an auth check aimed at the wrong operand: it verifies who is calling and never verifies which resource the call selects.
When the model authors the resource-selecting argument (tenant_id, account_id, project_id), an authorized caller can reach another tenant's data. The identity check passes the whole time.
The discriminator is provenance, not value. A tenant_id that came from the authenticated session is fine. The same value, if the model wrote it, is not, because the model picking the right tenant once is luck, not authorization.
The gate: any resource-selecting argument must be session-derived. Model-authored selectors are refused before the database is touched.
The tool below runs eight concrete calls. Without the gate, 4 of the 5 model-authored selectors returned another tenant's rows. With the gate, 0 of 5 reached any row, and all 3 session-derived calls were still served. Standard library only, offline, deterministic. Recount every number from the printout.

AI disclosure: I wrote scope_provenance_gate.py with an AI assistant and ran it myself, three times, on Python 3.13.5, standard library only, no network, no keys. Every output block below is pasted from that run. The STDOUT is byte-for-byte identical across the three runs; its sha256 is 63adbe8ebe17e873cbf7dbdf24faed392f20a3205d50579aec1705cf3c7841cb and bash run_all.sh reproduces it. The fixture is synthetic and calibrated to nothing: the tenants, invoices and sessions are invented to isolate one mechanism. bot2 is a new project. It has no production fleet and no incident to sell you. This post demonstrates how a bug is reachable, not how often it happens in the wild. The one verbatim external quote (the definition of confused deputy) is attributed and linked; the practitioner posts I reference are their words, and I link the primary sources.

The confused deputy: the operand nobody checks

Here is the shape of a normal agent tool call. The agent has a tool, read_account_rows(account_id, limit). A request comes in on an authenticated session. Middleware checks whether this caller, in this role, may invoke this tool. It may. The tool runs. Rows come back.

Now look at where account_id came from. In an LLM agent the tool-call arguments are assembled from two very different sources. Some fields the runtime injects: the session, the auth context, anything you copy in from the request you already trusted. The rest the model fills, from its plan. limit is a fine thing for the model to choose. account_id decides whose data you return. If the model writes that field, then the argument that selects the resource is authored by the least trusted component in the system, and the authorization layer never reads it.

That is a textbook confused deputy. The Wikipedia article on the confused deputy problem defines it in one sentence:

"In information security, a confused deputy is a computer program that is tricked by another program (with fewer privileges or less rights) into misusing its authority on the system."

The term is Norm Hardy's, from his 1988 ACM SIGOPS paper of the same name. The deputy here is your authz middleware. It holds real authority (it can read any account) and it is tricked into using that authority on a target chosen by the model, because it checks the caller and not the selector. The privilege gap is exact: the model has no standing to read account B, the middleware does, and the middleware acts on the model's choice.

So here is the claim, stated so you can break it: an authorization layer that verifies the caller and reads the resource-selecting argument from the model's output can return a resource the caller was never scoped to. The fix does not need a smarter authz check on identity. It needs a check on a different operand: the provenance of the selector. Show me a model-authored account_id that reaches data through the gate below, or a session-derived one the gate refuses, and the tool is broken and the claim with it.

The world, in eight calls

The fixture is three accounts and two sessions. acct_apex and acct_ceres and acct_borealis, each with one or two invoice rows. Two authenticated sessions: S1, scoped to acct_apex only; S2, a shared-ops user scoped to both acct_apex and acct_ceres. Nobody is scoped to acct_borealis here, which makes it the clean victim.

BLOCK 1 -- the world (synthetic, calibrated to nothing)
  acct_apex: rows [apex-inv-2201,apex-inv-2202]
  acct_borealis: rows [bor-inv-5501,bor-inv-5502]
  acct_ceres: rows [cer-inv-8801]
  session S1: identity=user_apex_ops role=reader authorized_accounts=['acct_apex']
  session S2: identity=user_shared_ops role=reader authorized_accounts=['acct_apex', 'acct_ceres']

The authorization middleware is the ordinary kind. It answers one question and it answers it right:

def authz_allow_caller(session, tool):
    """Answers exactly one question: may this caller invoke this tool?
    It never sees account_id."""
    return tool in ROLE_TOOLS.get(session.role, set())

And the tool trusts that the middleware already did its job, so it does no ownership check of its own:

def run_tool(tool, values):
    if tool == "read_account_rows":
        account_id = values["account_id"]
        limit = values["limit"]
        return list(DB.get(account_id, [])[:limit])
    raise ValueError("unknown tool")

Neither layer checks who is allowed to read account_id. That is the whole bug, and it is boring, which is why it ships.

Each argument in the model carries a provenance tag: session if the runtime injected it from the authenticated context, model if it came out of the plan. This is not something I invented for the demo. Your runtime already knows which fields it injected and which the model filled, because you wrote the code that assembles the call. The tag just names a fact you are throwing away.

Watch identity pass while the data leaks

Run the eight calls through the current path: identity check, then the tool. No scope check anywhere.

BLOCK 2 -- every enumerated call, WITHOUT the gate
  call sess selector(account_id)   prov    id_ok foreign rows_returned
  C1   S1   acct_apex              session True  False   apex-inv-2201,apex-inv-2202
  C2   S1   acct_borealis          model   True  True    bor-inv-5501,bor-inv-5502
  C3   S1   acct_borealis          model   True  True    bor-inv-5501,bor-inv-5502
  C4   S1   acct_apex              model   True  False   apex-inv-2201,apex-inv-2202
  C5   S1   acct_ceres             model   True  True    cer-inv-8801
  C6   S2   acct_ceres             session True  False   cer-inv-8801
  C7   S2   acct_borealis          model   True  True    bor-inv-5501,bor-inv-5502
  C8   S1   acct_apex              session True  False   apex-inv-2201,apex-inv-2202

  Cross-tenant reads achieved without the gate: 4 -> C2,C3,C5,C7

Read the id_ok column. It is True on every row, including the four that leaked. The identity check never failed. C2 and C3 are S1, scoped to apex, walking out with bor-inv-5501 and bor-inv-5502, borealis rows, because the model wrote account_id = "acct_borealis" and the middleware only ever asked whether S1 may call the tool. C5 is S1 reading ceres. C7 is S2 reading borealis, an account it was never scoped to.

That is the confused deputy in one screen. The gate the system trusts did not break. It answered "may this caller use this tool?" with a correct yes, while a different operand it never inspected decided the answer to a question nobody asked: "may this caller read this account?"

The gate reads the source, not the value

The fix keys on the one property that separates C1 from C4. Look at those two rows again: same session S1, same selector value acct_apex, same rows on disk. C1 is safe and C4 is not, and the only difference between them is who wrote the account_id. C1's came from the session. C4's came from the model. The value is identical. The provenance is not.

So the rule is provenance, and only provenance:

def scope_provenance_gate(call):
    """Looks ONLY at the source of each resource-selecting argument,
    never at its value. Fails closed on unknown tools or params."""
    schema = TOOL_SCHEMA.get(call.tool)
    if schema is None:
        return (False, "DENY: unknown tool, no schema (fail-closed)")
    for param, arg in call.args.items():
        pspec = schema["params"].get(param)
        if pspec is None:
            return (False, f"DENY: undeclared param '{param}' (fail-closed)")
        if pspec["resource_selecting"] and arg.provenance != "session":
            return (False, f"DENY: resource-selecting arg '{param}' is "
                           f"{arg.provenance}-authored, must be session-derived")
    return (True, "ALLOW: all resource-selecting args are session-derived")

A tiny schema declares which parameter selects a resource (account_id yes, limit no). The gate runs before authz and before the database. Same eight calls:

BLOCK 3 -- the same calls, WITH the provenance gate
  call selector         prov    verdict rows / reason
  C1   acct_apex        session ALLOW   rows [apex-inv-2201,apex-inv-2202]
  C2   acct_borealis    model   DENY    DENY: resource-selecting arg 'account_id' is model-authored, must be session-derived
  C3   acct_borealis    model   DENY    DENY: resource-selecting arg 'account_id' is model-authored, must be session-derived
  C4   acct_apex        model   DENY    DENY: resource-selecting arg 'account_id' is model-authored, must be session-derived
  C5   acct_ceres       model   DENY    DENY: resource-selecting arg 'account_id' is model-authored, must be session-derived
  C6   acct_ceres       session ALLOW   rows [cer-inv-8801]
  C7   acct_borealis    model   DENY    DENY: resource-selecting arg 'account_id' is model-authored, must be session-derived
  C8   acct_apex        session ALLOW   rows [apex-inv-2201,apex-inv-2202]

C4 is the row I want you to sit with. Its value was harmless. acct_apex is exactly what S1 is allowed to read, and the model happened to name it correctly. The gate denies it anyway. That is not the gate being dumb. It is the gate refusing to grade the model on whether it guessed right this time, because a component that is allowed to pick the tenant when it guesses right is allowed to pick the tenant, full stop. If you want apex read, the selector comes from the session. The model does not get partial credit for coincidence.

The counts, so you can recount them

Every number I am about to state is a count of the eight rows above. No sample size, no percentage over an invented denominator, no averaging. Count them yourself.

BLOCK 4 -- the counts (recount them from BLOCK 2 and BLOCK 3)
  K total enumerated calls                          : 8
  resource selector is session-derived (S)          : 3  -> C1,C6,C8
  resource selector is model-authored (M)           : 5  -> C2,C3,C4,C5,C7
  of M, selector points OUTSIDE caller's set        : 4  -> C2,C3,C5,C7
  of M, selector happens to name caller's own acct  : 1  -> C4

  WITHOUT gate: model-authored selectors that
    returned another tenant's rows                  : 4 of 5  -> C2,C3,C5,C7
  WITH gate: model-authored selectors that
    reached ANY row                                 : 0 of 5  -> (none)
  WITH gate: session-derived selectors that still
    returned the caller's authorized rows           : 3 of 3  -> C1,C6,C8
  WITH gate: session-derived selectors denied        : 0 of 3  -> (none)

Three facts, each a count and not an estimate:

Reachable. 4 of the 5 model-authored selectors returned another tenant's rows without the gate. The fifth, C4, returned the caller's own rows only because the model happened to name the caller's own account. The leak is reachable, and you can see exactly which rows walked out.
Blocked. 0 of the 5 reached any row with the gate on. All five were denied before the database was touched, C4 included. The gate does not need to know which of them was malicious, because it does not decide on the value.
Not broken. 3 of 3 session-derived selectors were still served with the gate on. The gate denied zero legitimate calls. It is not "deny everything," which would be trivial and useless.

The 4-of-5 and 0-of-5 are not measurements of a rate that could come out differently on a bigger fixture. They are properties of the construction. Without the gate, a model-authored selector pointing outside the caller's set will always return foreign rows, because identity-only authz always passes and the tool always reads whatever account_id it is handed. With the gate, a model-authored resource selector is always denied, because that is the rule. Grow the fixture to eighty calls or eight hundred and the two structural facts do not move. Only the sizes of the sets do.

Four things that could have broken and didn't

A gate that only ever says DENY would pass the leak test and be worthless. So the tool ships four falsifiers, each of which fails loudly if the gate is the wrong shape:

BLOCK 6 -- falsifiers (each could fail if the gate were wrong)
  [PASS] F1 gate does not break legitimate session-derived scope
         3/3 session-derived calls served, 0 denied. A deny-all gate would fail this.
  [PASS] F2 classifier reads the source of the arg, not its value
         C1 and C4 both select 'acct_apex'; C1 (session) ALLOW, C4 (model) DENY. A value-based check (account_id == self) would ALLOW C4 and miss the rule.
  [PASS] F3 session-derived scope leaks nothing even without the gate
         All session-derived selectors are inside the caller's authorized set by construction, so identity-only authz already returns own rows. The vulnerability is specific to model-authored scope, not to the tool.
  [PASS] F4 gate keys on the resource-selecting operand, not on 'any model arg'
         C8's limit is model-authored but non-selecting; account_id is session-derived. Gate ALLOWs C8. A blanket 'no model input' gate fails.

F2 is the one that matters most, because it is the difference between a real fix and a fake one. The obvious patch is a value check: assert account_id == session.tenant. That would stop C2, and it would also break S2 reading its second authorized account, and it would silently pass a model-authored value that happens to match. F2 proves the gate is not doing that. C1 and C4 carry the identical value acct_apex and get opposite verdicts, so the decision is provably keyed on the source, not the string. F4 proves the gate is not the lazy over-correction either: it does not reject every argument the model touched, only the one that selects a resource. C8's limit is model-authored and the gate lets it through, because a model choosing to fetch fifty rows of accounts it is allowed to see is not a confused-deputy problem.

F3 is the honest scoping of the whole claim. The tool is not dangerous. read_account_rows is fine. The vulnerability is specific to model-authored scope, and F3 shows it: every session-derived call leaks nothing even with no gate at all, because a selector drawn from the session is inside the caller's authorized set by construction. Take the model out of the resource-selection path and there is no deputy to confuse.

This is not the write-hop bug

If you read the write-chain taint post here three weeks ago, this can look like the same story told twice. It is a different operand, and the difference is the whole point.

In that post the gate keyed on a signal, sender_trust, and the danger was that a model had written some store upstream of that signal, so the value the gate read was model-laundered. The fix walked the write-closure of the signal the gate reads and refused to let a model-tainted signal hold the authorization role. The tainted thing was the thing the gate checks.

Here the gate reads a clean signal. Caller identity is world-anchored; the model did not write it and there is no write-hop to trace. The gate is right about what it checks. The bug is that the resource selector is a different argument entirely, a sibling operand the authz layer structurally never inspects, and that operand is model-authored at the point of the call. There is no laundering and no upstream store. The model just fills a field, directly, and the field decides whose data comes back. Taint-linting the signal the gate reads would not catch this, because the signal the gate reads is clean. You have to gate the operand the gate ignores. Same family, adjacent bug, different fix. If your mental model is "make sure the gate's inputs are trustworthy," this one slips past, because the gate's inputs are trustworthy and the leak is in an input the gate never took.

The general franchise both posts sit in is the same, and it is the one I keep coming back to: gate before you execute, do not log after. It is the same instinct as gating the lethal trifecta before the agent runs and as the post on model-authored SQL reaching the database: an operand an untrusted component authored gets checked before it acts, not logged after. Logging the call tells you which account leaked, next week, in the incident review. Gating the provenance of the selector stops the read.

What this is and is not

This is a mechanism demo on a synthetic fixture. It shows that the leak is reachable and that a provenance gate closes it on eight enumerated calls. It is not a claim about how common the bug is, not a benchmark, and not a measurement of anything in production, because bot2 has no production. The provenance tag is the load-bearing assumption: the gate is only as good as your runtime's honesty about which arguments it injected versus which the model filled. If you assemble tool calls by letting the model emit the whole JSON and never tracking what you put in, you do not have the tag, and step one is to start attaching it. The gate cannot recover a provenance you never recorded.

There is a real cost, and C4 is it. The gate denies model-authored selectors that would have been harmless, because it refuses to read the value. In exchange you get a rule that does not depend on the model being right. I think that trade is correct for anything that selects a tenant. You may not, for lower-stakes selectors, and F4 is there precisely so you can scope the rule to the arguments that deserve it.

What to change on Monday

Find every tool where an argument selects a resource: tenant_id, account_id, project_id, workspace, user_id, org. For each one, ask a single question: at the moment the call is assembled, does that argument come from the session or from the model? If you cannot answer, that is the finding. Start tagging.

Practitioners are already converging on this from the other side. Kailash Sankar, in "Defense in Depth: Tenant Isolation for an Agent That Executes Code", wires a proxy that overwrites whatever the model puts in a tenantId parameter with the trusted value from the context registry, hallucinated or injected value be damned. Brian Hall, in "Don't use an LLM to decide what your AI agent is allowed to do", puts it as a design rule: the decision on whether a real action runs "has to sit on something that gives the same answer every time and can show its work afterward."

Overwrite and deny are two implementations of the same rule, and they differ in one way worth naming. Sankar's proxy silently corrects the model's tenantId; the model never learns it overreached. The gate here refuses and says why, which turns a silent correction into a visible signal you can count, alert on, and use to notice a plan that keeps reaching for tenants it was not handed. Silent is safer to ship. Loud is better for finding out your agent has been trying the wrong door for a month. I have not run this in anger long enough to tell you which one you will regret less. Pick the one that matches how much you trust your own logging.

The tool, all eight calls, both execution paths, the selftest and the four falsifiers are in scope_provenance_gate.py. Copy it, add a ninth call, watch the counts move by exactly one.

I write one runnable tool per post about operating AI agents in production: the cost, the failures, the gates that run before execution instead of the logs that run after. Follow for the next one. And tell me in the comments: in your agent, which tool arguments come from the session and which come from the model, and are you sure?

Cost Per Verified Success: Your Exit-0 Denominator Lies

Alexey Spinov — Sat, 25 Jul 2026 03:51:39 +0000

Your cost-per-task dashboard is doing division. Spend on top, "successful tasks" on the bottom. The number it prints is the average cost of one agent task getting done. Here is the problem nobody puts on the dashboard: the denominator is whatever your agent told you was a success. On most stacks that means exit_code == 0, or ok: true, or an HTTP 200. That is the actor grading its own homework. When the agent silently fails, that failure stays in the denominator as a "success," so the average cost per success comes out lower than the truth. Your cheapest-looking number is the one you can least trust.

Cost per verified success is agent spend divided by witnessed successes, not by exit-0. A verified success is one an independent witness re-confirms: a file in the manifest, a DB row, a token in an HTTP body, a matching sha. Silent failures inflate the exit-0 count, so the dashboard number is a lower bound on real cost.

AI disclosure. I wrote verified_cost.py with an AI assistant and ran every case myself before publishing. Every terminal block below is pasted from a real run on Python 3.13.5, stdlib only. The run-log is a synthetic fixture: the token counts and the price sheet are made up, and I label them so. What is real is the witness logic (each check is recomputed from recorded evidence, not asserted) and the arithmetic. I have no production incident and no invoice to sell you here. bot2 is new and its lifetime spend is zero dollars. What I have is a script that runs and a number you can reproduce.

Why is cost per task a lie?

Because "task" and "successful task" are two different measurements, and the cheap one is wearing the expensive one's name tag. exit_code == 0 is cheap: it is the process telling you it thinks it finished. It is a self-report from the same actor whose work you are trying to price. On this blog the shape keeps recurring: a tool returns 200 and lies, a green check reconciles against nothing, an approval is not immutability. Cost inherits the same disease. If your success count is self-reported, your cost-per-success is self-reported too, and it always rounds in the flattering direction.

Watch it happen on one line of a run-log. An agent calls a "create order" endpoint. The endpoint returns HTTP/1.1 200 OK with a body of {"status":"RATE_LIMITED","order":null}. The process exits 0, because 200 is not an error. The dashboard counts a successful task and folds its cost into the average. No order was created. You paid for the tokens, you paid for the task, and the dashboard told you that money bought a success. It did not. Multiply that by an overnight batch and your per-success number drifts away from reality while looking perfectly healthy.

The fix is not a better dashboard. It is a better denominator.

What is a witnessed success?

A verified success is a task that (1) exited 0 and (2) has its effect re-confirmed by a check independent of the agent. That "and" matters. It makes verified successes a subset of exit-0 successes, which is the whole reason the math has a direction. Call M the count of exit-0 tasks and M' the count of verified ones. Because every verified success is also an exit-0 success, M' <= M for any log you will ever feed it. So:

naive_cost_per_success    = total_spend / M
verified_cost_per_success = total_spend / M'      (M' <= M, so this is >= naive)
understatement_factor     = verified / naive = M / M'   (>= 1, always)

That inequality is not a finding. It is arithmetic. naive_cost <= verified_cost for every possible run-log, with equality only when M' == M, meaning zero silent failures. I want to be blunt about that, because it is the honest core of the tool: the tool does not discover that your true cost is higher. It proves it is at least as high and then measures by how much. The direction is guaranteed. The magnitude is what you did not know.

A witness has to be concrete or it is just vibes with a checkmark. verified_cost.py ships four kinds, and each one is recomputed from recorded evidence rather than trusting a boolean somebody wrote down:

def eval_witness(w):
    kind, ev = w["kind"], w["evidence"]
    if kind == "file_exists":
        return ev["target"] in ev["manifest"]          # path actually present
    if kind == "db_row_present":
        return ev["target"] in ev["rows"]              # row id actually there
    if kind == "http_body_contains":
        return ev["needle"] in ev["body"]              # token actually in the body
    if kind == "hash_match":
        return ev["expected"] == ev["observed"]        # sha actually matches
    raise ValueError("unknown witness kind")           # anything else -> fail closed

(The real function has the type-checking and the fail-closed raises spelled out; this is the spine.) The point is that a witness result is a function of evidence you can inspect, not a second self-report. If the create order body says RATE_LIMITED, then http_body_contains("ORDER_CONFIRMED") returns false no matter what the exit code claimed. You can paste your own bodies in and rerun. The check does not care about the agent's opinion.

Running it on a batch

Here is the meter on a synthetic overnight batch of 12 tasks. Ten exited 0. Two failed honestly (a migration that returned exit 1, a docker build that OOM-killed with 137), and the dashboard already knows about those. The interesting three are the ones that exited 0 and did nothing: t03 got the RATE_LIMITED body above, t04 produced an artifact whose sha did not match what was expected, t06 claimed to update user-90 but that row is not in the table.

$ python3 verified_cost.py report fixtures/runlog.json
id     spend      exit0   witness   verified
t03    $0.3540    yes     FAIL      no
t04    $0.5850    yes     FAIL      no
t06    $0.2775    yes     FAIL      no
...
total_spend            = $3.4665
M  (exit-0 successes)  = 10      <- the denominator your dashboard uses
M' (verified: exit0 AND witness) = 7
silent failures (exit0, witness FAIL) = 3  ['t03', 't04', 't06']
spend on silent failures = $1.2165 (bought an exit 0, witness rejected it)
------------------------------------------------------------------------------
naive_cost_per_success    = total/M  = $3.4665 / 10 = $0.3466
verified_cost_per_success = total/M' = $3.4665 / 7 = $0.4952
understatement_factor     = M/M' = 10/7 = 1.4286x  (arithmetic, not a measured effect)

Read the raw pieces, not just the ratio. M is 10, M' is 7, and I print both so the 1.4286x cannot hide anything. On this batch the dashboard would tell you each success cost 35 cents. The witness says 50. That is the same 1.4286x the report prints: the true per-success cost sits 43% above the dashboard's number (equivalently, the dashboard reads 30% below the truth), and it is not a rounding error. $1.2165 of the $3.4665 you spent, better than a third of the bill, bought exit-0s that the witness threw out. The dashboard counted that third as wins.

I want to be precise about what is real here and what is not. The 1.4286x is 10/7, and the 10 and the 7 are counts I chose when I built the fixture. So the magnitude is synthetic. What is not synthetic is the direction and the mechanism: on any log, the moment one exit-0 task fails its witness, M' drops below M and your true cost climbs above what the dashboard shows. The tool computes M' by rerunning the checks, so on your own log the number is yours, not mine.

The gate: block before the inflated number reaches the budget

A meter you read after the fact is a postmortem. The point of this franchise is to gate before the spend lands, the same way the pre-execution gate decides whether an action runs at all. So verified_cost.py gate takes two policy knobs, a max verified cost per success and a max understatement factor, and returns a CI exit code: 0 to pass, 1 to block, 2 to fail closed on garbage input.

$ python3 verified_cost.py gate fixtures/runlog.json 0.15 1.20
  naive=$0.3466/succ   verified=$0.4952/succ   understatement=1.4286x
  policy: max_verified=$0.1500/succ  max_understatement=1.2000x
  BLOCK: verified_cost $0.4952/succ exceeds budget $0.1500/succ
  BLOCK: understatement 1.4286x exceeds tolerance 1.2000x (silent failures mask the bill)
VERDICT: BLOCK exit=1

Two independent reasons fired. The budget one is a generic FinOps cap. The understatement one is the reason this tool exists, so I isolated it: loosen the budget to a dollar per success, well above the real $0.4952, and the gate still blocks.

$ python3 verified_cost.py gate fixtures/runlog.json 1.00 1.20
  policy: max_verified=$1.0000/succ  max_understatement=1.2000x
  BLOCK: understatement 1.4286x exceeds tolerance 1.2000x (silent failures mask the bill)
VERDICT: BLOCK exit=1

That is the whole idea in one exit code. Even when you can afford the verified cost, a wide gap between the naive and verified numbers is itself the signal: your dashboard is dividing by a denominator that is lying to it, and that is worth stopping a deploy over before the pattern scales into next month's budget.

The falsifier: when this tool should shut up

A gate that always fires is a stuck alarm, not a control. So the tool has to pass a case where it adds nothing, and it has to do that honestly. If your agent never silently fails, then exit_code == 0 really does mean success, M' equals M, and there is nothing to correct. Here is that case, five tasks, every exit-0 witnessed:

$ python3 verified_cost.py gate fixtures/honest_zero.json 0.30 1.20
  naive=$0.2244/succ   verified=$0.2244/succ   understatement=1.0000x
VERDICT: PASS exit=0

Understatement 1.0000x. Naive equals verified to the cent. The masking check is silent and the gate passes. That is the falsifier working: if exit-0 never lied on your stack, this tool changes nothing and says so out loud. Any claim it makes on your real batch has to survive that comparison first.

Now the reasonable objection: "fine, but our agents are healthy, we pass 95% of tasks, the gap must be negligible." It is not zero, and the arithmetic says exactly how not-zero. understatement = M/M' = 1/(1-s) where s is the silent-failure rate among exit-0 tasks. I swept s to show the magnitude at each rate (this table is arithmetic, not a measurement of any workload, and it is labeled that way in the output):

k      M'     s        SE(s)      M/M'         1/(1-s)
0      200    0.0000   0.0000     1.0000       1.0000
10     190    0.0500   0.0154     1.0526       1.0526
40     160    0.2000   0.0283     1.2500       1.2500
60     140    0.3000   0.0324     1.4286       1.4286
100    100    0.5000   0.0354     2.0000       2.0000
200    0      1.0000   0.0000     UNBOUNDED    1/0     <- M'=0, verified cost UNBOUNDED, gate BLOCKS

At a 5% silent-failure rate, the "healthy 95% agent," your true cost is 1.0526x the dashboard number. Small, but not nothing, and it only grows: the lower your real success rate, the wider the gap, because you are dividing the same spend by an ever-smaller denominator. And the bottom row is the one I most wanted the tool to handle without crashing: when every exit-0 task is a silent failure, M' is zero, verified cost is unbounded, and the gate blocks rather than dividing by zero and printing a comfortable-looking number.

One more honest note buried in that table. If you use exit_code itself as the witness, M' equals M for every s, so the gap is 1.0 always. A dashboard is not doing bad arithmetic. It is using the null witness: the actor as its own judge. The gap this tool measures is precisely the information a real witness adds beyond the agent's self-report. No witness, no gap, no visibility. That is why "add a witness log" is the actual ask here, not "buy a better dashboard."

Where this is soft, and what it is NOT

I would rather you trust the small claim than oversell the big one.

The magnitudes are synthetic. 1.4286x, $0.4952, $1.2165 come from a fixture I built. They illustrate the arithmetic. They are not a measurement of any real agent fleet, mine or anyone's. The direction (naive is a lower bound) is general; the size is not.
It is exactly as good as your witness. Garbage witness, garbage M'. If your only "independent" check is another call to the same flaky service, you have two self-reports, not a witness. The four kinds here (file_exists, db_row_present, http_body_contains, hash_match) are the ones I could make recompute from evidence with no network. Yours may need more. And the quiet version of a garbage witness is one too weak to ever fail: an empty needle sits inside every body, so it passes silently and the gap reads 1.0000x. The tool fails closed on evidence it cannot evaluate (exit 2), not on a witness that cannot fail. Writing a witness that can never fail is on you.
A witnessed success is not a correct one. http_body_contains("ACK") confirms the body said ACK. It does not confirm the ACK was for the right thing. Contract-level witnessing catches silent failure; it does not catch subtle wrongness. This is the same blind spot exit codes have, moved one notch up, not removed.
Cost is only the loop's forward pass. This tool prices completed tasks. It says nothing about tokens burned inside a task that never finishes; that is what the loop cost forecaster is for. Pair them: forecast the loop, then price the loop against what it actually produced.
It fails closed, on purpose. A malformed witness (say a hash_match with no observed field) exits 2, not 0. You cannot trust a bill computed from evidence you cannot evaluate, so the tool refuses to compute one. I checked that by hand: malformed input exits 2, and zero verified successes with nonzero spend blocks rather than dividing by zero.

Run it against your own run-log

If you log agent tasks at all, you already have M. What you probably do not have is M', because most stacks never record an independent witness next to the exit code. That is the actual gap, and it is cheaper to close than it sounds: pick the one artifact each task is supposed to produce, record whether it is really there, and divide by that count instead. Everything here is offline, keyless, stdlib-only Python 3.13.5. It drops into CI or a pre-deploy hook with no daemon and no account, and it prints its own sha256 on the last line so you can pin the version you ran.

Here is the part I have not settled, and I want your take. I set the masking tolerance at 1.20x and I am not sure that is right for anyone but this fixture. Too tight and it fires on the honest 5% agent; too loose and it waves through a batch where a third of the spend bought nothing. And the deeper question is upstream of the threshold: what is your witness? If you run agents in production, tell me the single check you trust to confirm a task actually happened, the one that is not just the agent saying ok: true a second time. I read every comment, and I am collecting the good ones.

If this was useful, the 200-and-lies witness gate is where the honest denominator comes from, the loop cost forecaster is the numerator side, and the pre-execution gate is the pattern that stops the bad number before it spends. Follow along; the next one keeps walking down this stack.

Your MCP Pin Blocks Every Update. Most Never Broke You.

Alexey Spinov — Fri, 24 Jul 2026 03:47:33 +0000

A month ago I shipped a 40-line padlock for MCP tools: pin the manifest hash, block the rug-pull. It works. It also has a flaw I wrote about in the last paragraph and then had to go build a fix for, because it kept nagging me. The pin fires on any change to a tool's definition. A server that honestly adds an optional parameter trips it exactly as loud as a server that yanks a required one out from under you. Same alarm, same block, same 3am page. The pin knows the contract changed. It has no idea whether the change broke you.

In short: an MCP tool schema breaking change narrows the valid-call set, so a call your agent makes today stops validating tomorrow. A backward-compatible change widens or leaves that set alone, so old calls stay valid. A byte pin cannot tell them apart. compat_gate.py diffs the inputSchema and replays the calls your agent recorded: silent on compatible, fail-closed on breaking.

AI disclosure: I wrote compat_gate.py with an AI assistant and ran every case myself before publishing. Every terminal block below is pasted from a real run on Python 3.13.5. The compatible case (C2) is a real dated diff between two published MCP spec schemas, fetched over curl and checksummed; the breaking cases (C3 to C5) are synthetic fixtures I built and confirmed by replay, and I label them as such. I have no production incident to sell you here; bot2 is new and its lifetime run count is zero. What I have is a tool that runs and a diff you can reproduce.

Why does pinning a manifest fire on every change?

Because a hash has one bit of memory: same, or different. The June pin takes a canonical SHA-256 over name + description + inputSchema and blocks when the live manifest stops matching the one you approved. That canonicalization is already the good kind of pin: it sorts keys and strips whitespace, so a re-serialized-but-identical manifest does not trip it. I checked that first, and it holds. The pin is not paranoid about JSON formatting.

It is paranoid about meaning. And that is the problem, because most meaning changes to a schema are harmless to your calls. The MCP spec itself is the clearest evidence. The protocol ships dated, breaking-capable revisions on a cadence; the schema directory in the spec repo currently carries 2024-11-05, 2025-03-26, 2025-06-18, 2025-11-25, and a live draft/. I pulled two of those and diffed them by hand (more on that below). Servers track the spec. Tools get re-versioned. Descriptions get rewritten for the model. Every one of those events flips the pin's one bit, and the pin dutifully blocks.

Here is the open question I left myself in June and could not answer from my own toolbox: where is the line between a legitimate tool update and drift? The franchise on this blog keeps landing on the same shape. Tracking is not control. Having a credential is not scoping it. Approval is not immutability. This post adds the next line: a changed contract is not a broken one. A pin is not a compatibility check. The pin tracks change. It does not control whether a change that would break your agent reaches the call. That is a different job, one level down from the hash, and it needs a different tool.

What makes a tool-schema change breaking?

There is a ground truth here, and it is older than MCP. It is subtyping. A schema describes a set: the set of argument objects it calls valid. A change is backward-compatible when the new set contains the old one, so every call that used to validate still validates. A change is breaking when the new set is smaller, so some previously-valid call now fails. That is the whole test, and it does not require an opinion.

On an object inputSchema, the breaking moves are the ones that shrink the valid set:

Add a required property. Old calls that omit it were fine and now are not.
Remove a property when additionalProperties: false. A call that passed that key is now rejected as an extra.
Narrow a type. number to integer throws out 3.5. Any incompatible type swap can strand an old value.
Shrink an enum. Drop "csv" from ["json", "csv"] and every call that asked for CSV is gone.
Make an optional property required, or tighten additionalProperties from true to false.

And the compatible moves, the ones that widen or preserve the set:

Add an optional property. Nobody was required to send it; old calls are untouched.
Widen an enum, relax a type (integer to number), drop a required constraint, or rewrite a description. None of these can invalidate a call that was already valid.

That list is the entire logic of the classifier. The point of the tool is not to be clever about it. The point is to run it against real schemas and, critically, to check the classifier's verdict against something independent: the calls themselves.

The gate: replay the calls your agent actually makes

compat_gate.py is keyless, offline, and stdlib-only (json, sys, hashlib). It carries a deliberately small JSON-Schema validator that understands exactly six keywords: type, required, properties, enum, additionalProperties, const. Anything else in a schema (a format, an items, a $ref) is treated as no constraint. That is a real limit and I come back to it at the end. It is the right subset for the corpus here and small enough to read in one sitting.

Two signals run side by side, and they are meant to agree:

def classify(old_is, new_is):
    """Diff two inputSchemas. Return [(verdict, reason), ...] in a stable order."""
    changes = []
    old_props, new_props = old_is.get("properties", {}), new_is.get("properties", {})
    old_req, new_req = set(old_is.get("required", [])), set(new_is.get("required", []))
    old_ap = old_is.get("additionalProperties", True)
    new_ap = new_is.get("additionalProperties", True)

    for p in sorted(new_props):
        if p not in old_props:
            if p in new_req:
                changes.append(("BREAKING", "added required property '%s' (old calls omit it)" % p))
            else:
                changes.append(("COMPATIBLE", "added optional property '%s' (old calls stay valid)" % p))
    # ... removed props, newly-required props, additionalProperties, enum shrink/widen, type narrow, const ...
    return changes

That is signal (A), the subtyping classifier: it labels each field-level change COMPATIBLE or BREAKING with a reason. Signal (B) is the one I trust more, because it does not reason at all. It replays every recorded call against both schemas:

    replay = []  # (idx, call, valid_under_old, valid_under_new, errs_new)
    for i, call in enumerate(calls, 1):
        vu_old = not validate(call, old_is)
        errs_new = validate(call, new_is)
        replay.append((i, call, vu_old, not errs_new, errs_new))
    control_ok = all(r[2] for r in replay)               # every recorded call must be valid under old
    broken = [r for r in replay if r[2] and not r[3]]    # valid under old, invalid under new

Every call in the corpus has to be valid under the old schema. That is the entry criterion and I print it as valid_under_old=True on every line, as a negative control: if a recorded call is not valid under the schema it was recorded against, the fixture is wrong and the tool says so. Then each call is validated against the new schema. A call that was valid under old and is invalid under new is a broken call, and the count of broken calls is a measurement, not a label. The classifier can call a change breaking; the replay tells you whether it broke your traffic.

Quick start is two commands:

# just classify the schema delta:
python3 compat_gate.py classify old_tool.json new_tool.json

# the gate: classify + replay recorded calls, exit code is your CI signal:
python3 compat_gate.py gate old_tool.json new_tool.json calls.jsonl

gate returns 0 when there are zero breaking changes and zero broken calls, 1 when either fires, and 2 on a usage error. The exit code is the whole product. It drops into CI or a pre-connect hook with no daemon and no account.

Three changes, three verdicts, one tool

I ran the gate against six changes. One is real; five are synthetic fixtures on a made-up run_query tool (a read-only SQL tool: required sql, optional limit, format as an enum of ["json","csv"], dry_run, timeout_ms). Four recorded calls, none of them touching timeout_ms. Here is the null case first, because a gate that screams at everything is useless.

C1, identical (re-serialized). The new tool is the old one with keys reordered and whitespace changed. Same content.

### C1 identical (re-serialized)
byte-pin: ok
subtyping: 0 changes
compat verdict: PASS   breaks=0/4

Both silent. The canonical pin does not trip on re-serialization, and the gate finds nothing to break. Good. Neither tool is trigger-happy on a non-change.

C2, a real backward-compatible diff from the MCP spec. This is the one I care most about being honest on. I fetched two published MCP schema files, 2025-06-18 (sha256 b3db8f1c...) and 2025-11-25 (sha256 7b2d96fd...), and pulled the Implementation object out of each. Implementation is the serverInfo/clientInfo payload in the handshake; it is a JSON Schema with properties and required, the same shape a tool inputSchema is, which is why I can drive the gate with it. Between those two dated versions it gained three optional properties (description, icons, websiteUrl) and kept required at ["name", "version"]. Textbook additive.

### C2 real MCP additive diff (Implementation)
byte-pin: BLOCK
  [COMPATIBLE] added optional property 'description' (old calls stay valid)
  [COMPATIBLE] added optional property 'icons' (old calls stay valid)
  [COMPATIBLE] added optional property 'websiteUrl' (old calls stay valid)
compat verdict: PASS   breaks=0/3

The pin blocks. The bytes changed, so the hash changed, so the pin does its one job and refuses. The gate replays three recorded handshake instances ({"name": "example-server", "version": "1.0.0"} and friends), all valid under old, all still valid under new, and passes in silence. This is half the falsifier for the whole thesis: on a genuinely compatible change, the gate has to be quiet, or it is just a slower pin. It is quiet.

C3, an added required parameter. The new run_query requires a tenant_id. This is the shape I most wanted to fail closed on, because it is the common one: a server tightens a tool and every existing integration silently starts sending calls that will be rejected.

### C3 added required param 'tenant_id'
byte-pin: BLOCK
  [BREAKING] added required property 'tenant_id' (old calls omit it)
  call#1 valid_under_old=True valid_under_new=False [BROKEN] {"sql": "select 1"}
        -> $: missing required 'tenant_id'
  ...
compat verdict: FAIL   breaks=4/4

Four of four recorded calls break, each because it is missing tenant_id, and the gate names them. The gate command exits 1. That is the other half of the falsifier: on a breaking change, the gate must fail closed and point at the call. It does, and I checked the exit code by hand, because a gate that prints FAIL and exits 0 is not a gate.

C4, a shrunk enum. The format enum drops "csv". Only one recorded call asked for CSV.

### C4 enum shrank on 'format'
byte-pin: BLOCK
  [BREAKING] property 'format' enum shrank ['csv', 'json']->['json'] (dropped ['csv'])
  call#3 valid_under_old=True valid_under_new=False [BROKEN] {"sql": "select id, email from users", "format": "csv"}
        -> $.format: 'csv' not in enum ['json']
compat verdict: FAIL   breaks=1/4

One of four. Not four of four. This is why the replay matters and the classifier alone does not: "breaking" is true, but the blast radius on your traffic is a single call, and the number tells you that. breaks=1/4 is a different operational decision than breaks=4/4.

C5, breaking by schema, silent on your traffic. The new tool removes the optional timeout_ms under additionalProperties: false. By subtyping that is breaking: a call that sent timeout_ms would now be rejected. But none of the four recorded calls ever set it.

### C5 removed optional 'timeout_ms'
byte-pin: BLOCK
  [BREAKING] removed property 'timeout_ms' under additionalProperties:false (a call using it is now rejected)
compat verdict: FAIL   breaks=0/4

Here the two signals disagree, and the tool refuses to hide it. The classifier says breaking; the replay says breaks=0/4. The verdict is FAIL, because I would rather a gate be conservative on a schema that genuinely narrowed. But the printout names the split out loud: breaking by schema, 0 recorded calls touch it, silent on your traffic. That middle cell is the honest one. If I collapsed it to a clean PASS I would be lying about the schema; if I collapsed it to a clean FAIL with no annotation I would be hiding that your recorded traffic is fine. So it prints both.

Byte-pin versus compat-gate on the same changes

Here is the head-to-head, straight from the run. The byte-pin column is the June canonical hash, my own prior tool, not a strawman.

change                                   byte-pin   compat   breaks
------------------------------------------------------------------------------
C1 identical (re-serialized)             ok         PASS     0/4
C2 real MCP additive diff (Implementatio BLOCK      PASS     0/3
C3 added required param 'tenant_id'      BLOCK      FAIL     4/4
C4 enum shrank on 'format'               BLOCK      FAIL     1/4
C5 removed optional 'timeout_ms'         BLOCK      FAIL     0/4        <- breaking by schema, 0 calls touched
C6 desc rewrite + added optional 'cache' BLOCK      PASS     0/4
------------------------------------------------------------------------------
byte-pin fired: 5/6   |   compat-gate fired: 3/6

Read the absolute counts, not the ratio. On this corpus of six changes, the byte-pin fired five times and the gate fired three. The two changes the pin flagged and the gate stayed silent on (C2, the real MCP additive; C6, a description rewrite plus an added optional cache param) were both backward-compatible: every recorded call still validated. That gap of two is the false alarms the gate removed on this corpus. It is not a claim about all possible changes; it is the measured difference on these six.

Two is a small number because six is a small corpus. But the shape scales the wrong way for the pin. The more a spec re-versions, the more the pin fires, and the fraction of those fires that are actually compatible does not drop just because the volume went up. A pin under heavy re-versioning is a siren that is right about "changed" and useless about "should I stop." The gate's job is to turn most of that siren off without turning off the part that matters.

Where does the compatible/breaking line actually sit?

I would rather you trust the small claim than the big one, so here is where this gets soft.

Replay only covers the calls you recorded. The gate is exactly as good as your call log. If your agent has a code path that fires a call shape you have never captured, C5 is your whole life: the gate will say breaks=0 and be technically right and operationally blind. The classifier signal (A) is the hedge here, because it flags the schema narrowing even when your traffic misses it, which is precisely why I keep both signals and fail closed when they disagree.

Compatible-by-schema is not compatible-by-behavior. This is the same blind spot the pin has, one level up. A server can keep inputSchema byte-identical and change what the tool does behind it: return different data, hit a different backend. Nothing in this gate, and nothing in a hash, catches that. Contract compatibility is not runtime compatibility, and I am not claiming it is.

The validator is a subset. Six keywords. It does not model pattern, minimum, maxItems, nested $ref resolution, anyOf, or a dozen other JSON Schema features that can each carry a breaking change. On a schema that tightens a pattern or lowers a maximum, this tool will miss it, because it does not read those keywords. That is a scoping decision, not a completeness claim. Extend the validator and you extend the coverage; the design leaves room for it.

What this is NOT

It is not a replacement for the pin. Run both. The hash catches the description-poisoning and rug-pull shapes, where a byte-level change is the signal and "compatible" is the wrong frame entirely: a hidden instruction slipped into a description is backward-compatible by subtyping and still hostile. The pin is your detector for "changed at all." This gate is your filter for "changed in a way that breaks my calls." They answer different questions and they belong in the same pipeline.
It is not a semantic scanner. It does not read the meaning of a description or judge whether a new parameter is dangerous. It asks one structural question about the argument set.
The breaking fixtures are synthetic. C3 to C5 are fixtures I wrote. They are confirmed by independent replay (a call valid under old genuinely fails under new, and the tool prints the failing path), not by my say-so, but they are not a live server. The one real diff is C2, and I checksummed it.
A PASS is not a proof of full compatibility. It means: on the six schema keywords I model, zero breaking changes, and zero of your recorded calls broke. It does not mean nothing can go wrong. See the section above.

Run it against your noisiest server

Pull the tools/list for the MCP server your agent leans on hardest. Save the inputSchema you have now, save the next version when the server updates, dump a few hundred of your real recorded calls to a .jsonl, and run compat_gate.py gate. If it passes, you just re-approved an update without a human reading a diff at 3am. If it fails, it hands you the exact calls that would have broken, before the first one goes out. That is the cheapest pre-call check you will add this quarter, and it runs before the call instead of in the postmortem.

Here is the part I still have not settled, and I want your take. Signal (A) and signal (B) disagree in the C5 shape (breaking schema, untouched traffic), and I resolved it by failing closed with a loud annotation. I am not sure that is right for every team. If you re-version aggressively, C5 is going to fire constantly on parameters nobody sends, and constant firing is how a gate gets disabled. Do you gate on the schema signal (conservative, noisy) or the replay signal (permissive, blind to unseen paths)? Do you tie re-approval to a publisher signature and skip the diff entirely? I have run this against exactly one real diff and a pile of fixtures, so my line is drawn in pencil. If you have run a compat check against a server that updates for real, tell me where you put the line and why. I read every comment.

If this was useful, the MCP tool pin is the layer above it, the pre-execution gate is the layer that decides whether a specific action runs at all, and the MCP server token tax measures what each tool costs you in context. Follow along; the next one keeps walking down this stack.

The best config in your bake-off didn't win. Selection did.

Alexey Spinov — Thu, 23 Jul 2026 03:24:54 +0000

Best-of-K eval selection bias: pick the highest-scoring config from K candidates on one eval set and that observed score is biased up. It reports the expected maximum of K noisy estimates, which beats the field mean whenever K exceeds one. The bias appears even when all K configs are truly equal, grows with K, and shrinks with n.

Here is the version that bites you. Your bake-off ran a batch of prompts against one eval set, the top one came out ahead, and you shipped it. In production it does worse. That drop reads like bad luck, or drift, or a bad week. It is none of those. It is a number you could have computed before you shipped, and it gets larger the more candidates you tried.

I ran a small script to make the gap concrete. Eight configs, one hundred eval items, and here is the catch: I made all eight configs truly identical, every one a fair coin at 50%. There is no real best. Nothing to tune. Then I let selection pick a winner anyway:

   config 0:  47/100 =  47.0%
   config 1:  50/100 =  50.0%
   config 2:  52/100 =  52.0%
   config 3:  52/100 =  52.0%
   config 4:  50/100 =  50.0%
   config 5:  50/100 =  50.0%
   config 6:  54/100 =  54.0%  <- selected winner (argmax)
   config 7:  44/100 =  44.0%

     config 6: 54.0% (k=54 n=100 SE=4.98)
     config 2: 52.0% (k=52 n=100 SE=5.00)
   RANK: INDISTINGUISHABLE - gap 2.00 pp against 7.06 pooled SE = 0.28 SE < 2.0. Ranking "config 6" above "config 2" is NOT allowed.

Held-out the winner on a fresh 100 items: 48/100 = 48.0%.

Config 6 wins the bake-off at 54.0%. Then I asked the same eval-guard I use in the McNemar and rule-of-three pieces to rank config 6 against the runner-up. It refused: the gap is 0.28 SE, far under the two-SE bar, so INDISTINGUISHABLE. The ranker would not call config 6 the best. Selection did. And on a fresh held-out set the 54.0% falls back to 48.0%, toward the true 50% it was always going to be.

TL;DR

Picking the best of K configs by observed pass rate reports the expected maximum of K noisy estimates. The max of K exceeds the field mean, strictly, for K at least 2 (Jensen's inequality on a convex function). So the winner's reported score is biased up.
The bias is not about a weak config. In the null world, where all K configs are truly equal, best-of-16 reads 8.81 pp above the true 50%, and best-of-64 reads 11.67 pp above it. Nothing to optimize; the inflation is the act of selecting.
It grows with K: 2.82 pp at K=2, 7.10 pp at K=8, 11.67 pp at K=64 (p=0.5, n=100). Adding candidates makes it worse, not better. You cannot fix a selection bias by selecting from more.
It equals the drop you feel on held-out. On held-out, best-of-16 is indistinguishable from a random pick, 50.4% against 49.9% at 1.40 SE, while its in-sample number sits 29.5 SE higher. The whole curse lives in the eval you selected on.
It shrinks with n like 1/sqrt(n): 8.81 pp at 100 items, 2.21 pp at 1600. Real, but you fight the square root, so halving the bias costs four times the eval budget.

What this is and is not. The fixtures are synthetic, and I say so on purpose. The bias here is a theorem, not an effect I fitted: E[max of K i.i.d. non-degenerate estimates] > their common mean is Jensen, and the null-world sections compute it exactly from binomial order statistics with no sampling and no tuned constant. The one sampled section uses a fixed seed and is byte-identical across runs. The curse is negligible when one config is genuinely far ahead of the field; it dominates when the field is tight, which is where most bake-offs actually sit. I show both.

Why the winner's score is biased up

Selection reports a maximum, and a maximum is not an average. Each config's observed pass rate is its true rate plus noise. When you take the best of K, you are not reading a typical config, you are reading whichever one the noise flattered most. E[max] sits above the mean for any K of 2 or more, and the gap widens as K grows. This is Jensen's inequality: the max is a convex function of the estimates, so the expectation of the max exceeds the max of the expectations, which in the equal case is just the common true rate.

The magnitude has a textbook form. The expected max of K standard normals grows like sqrt(2 ln K), so the selection bias scales roughly as sigma * sqrt(2 ln K), where sigma is the standard error of one config's estimate. That is why more candidates keep costing you: the penalty climbs with ln K and never turns around. Here is the exact order-statistic computation, summed over the binomial, not sampled:

   K | E[in-sample winner] | true field p | bias = drop (pp) | sigma*sqrt(2lnK)
   1 |              50.00% |        50.0% |            -0.00 |             0.00
   2 |              52.82% |        50.0% |             2.82 |             5.89
   4 |              55.14% |        50.0% |             5.14 |             8.33
   8 |              57.10% |        50.0% |             7.10 |            10.20
  16 |              58.81% |        50.0% |             8.81 |            11.77
  32 |              60.31% |        50.0% |            10.31 |            13.16
  64 |              61.67% |        50.0% |            11.67 |            14.42

Read the bias column downward. Zero at K=1, because with one config there is no selection to make; the script flags that row as degenerate and refuses to run the selection path on it. Then 2.82, 5.14, 7.10, all the way to 11.67 pp at K=64. The sqrt(2 ln K) column is the growth guide, not an equality: it is the asymptotic leading term and runs loose at small K, which is why I print the exact bias beside it rather than the approximation. The conclusion does not live in any single row. It lives in the monotone climb of the whole swept column, with zero real signal anywhere in the fixture.

The bias is there even when nothing is better

The null world is the part I care about most, because it is the part you cannot argue with. There is no best config to find. Every candidate is the same coin. If the winner's score were an honest estimate, best-of-K would land at 50% on average for every K. It does not. It lands at 58.81% for K=16 and 61.67% for K=64. That premium is manufactured entirely by the operation of taking a maximum over noise.

This is the anti-fixture check I hold myself to after dropping five of my own drafts in one day for hiding a conclusion inside a tuned constant. Here there is no constant to tune in my favor: no lucky base rate, no engineered gap, no cherry-picked K. I set the true difference between configs to exactly zero and the bias is still positive and still growing. You can disagree with the binomial if you like. You cannot make the max of sixteen coins average out to one coin.

It doesn't depend on a lucky base rate

The next question a good skeptic asks: did I pick p=0.5 because it flatters the effect? So I swept the base rate at K=16 and read the exact bias at each:

     p | E[in-sample winner] | bias = drop (pp) | sigma (pp) | bias/sigma
  0.10 |              15.58% |             5.58 |       3.00 |       1.86
  0.20 |              27.28% |             7.28 |       4.00 |       1.82
  0.50 |              58.81% |             8.81 |       5.00 |       1.76
  0.80 |              86.79% |             6.79 |       4.00 |       1.70
  0.90 |              94.92% |             4.92 |       3.00 |       1.64

Positive at every base rate, 5.58 pp at a 10% pass rate up to 8.81 pp at 50%. Look at the last column instead of the fourth. Measured in units of the per-cell noise sigma, the inflation is close to constant, about 1.76 sigma at p=0.5 and never straying far. That is the real statement: the curse scales with sigma = sqrt(p(1-p)/n), the noise in one estimate, not with any special value of p. There is no base rate you can choose to make it disappear. The absolute pp figure is largest near 50% only because that is where a coin is noisiest.

More items shrink it, but you're fighting a square root

There is one honest escape, and the same math hands it to you. Since the bias scales with sigma, and sigma falls like 1/sqrt(n), running more eval items shrinks the curse. The theorem's degenerate limit is n to infinity: sigma to zero, bias to zero. Sweeping n at K=16, p=0.5:

     n | sigma (pp) | bias = drop (pp) | bias/sigma | normal pred (pp)
   100 |       5.00 |             8.81 |       1.76 |             8.83
   400 |       2.50 |             4.41 |       1.76 |             4.41
  1600 |       1.25 |             2.21 |       1.77 |             2.21
  6400 |       0.62 |             1.10 |       1.77 |             1.10

Two things worth seeing here. First, the last column: I also computed the normal-theory prediction, sigma times the exact expected max of sixteen standard normals (which the script works out to 1.7660 by quadrature), and the binomial bias converges onto it as n grows and the central limit theorem takes hold. 8.81 against a predicted 8.83 at n=100, then dead-on to two decimals by n=400. The theorem and the run agree because it is the same theorem.

Second, the cost. Going from 100 to 400 items, a 4x eval budget, halves the bias from 8.81 to 4.41 pp. That is the square-root tax. If someone pays a judge model per eval item, buying your way out of the winner's curse is a real line on the bill, and it scales the wrong way. "Just run more evals" is correct and expensive at the same time.

The held-out drop is the bias you actually feel

The bias so far is E[in-sample winner] - true rate. You never see the true rate, so how do you know you got bitten? You hold the winner out. Re-evaluate the selected config on a fresh set and its inflated score regresses. In expectation the held-out drop equals the selection bias, because the held-out re-estimate is unbiased for whatever config you happened to pick.

That is exactly the trap I walked into on the McNemar piece, where a quantity that looked like a finding turned out to be an unconditional rate wearing a conditional costume. So this script runs the independence probe from the same library before it claims anything: it compares the winner's held-out rate against a random pick's held-out rate, and reports whether the conditioning on "was selected" actually buys you anything.

   in-sample(best of 16) : 58.8% (k=23506 n=40000 SE=0.25)
   held-out(best of 16)  : 50.4% (k=20145 n=40000 SE=0.25)
   held-out(random pick) : 49.9% (k=19947 n=40000 SE=0.25)
   FORCED BY CONSTRUCTION [NULL: held-out of best-of-16 vs a random pick]: conditional 50.4% is indistinguishable from unconditional 49.9% (1.40 SE < 2.0)
   in-sample vs held-out (paired over 400 trials): drop 8.40 pp = 29.5 SE -> SEPARATED (measured curse)

On held-out, best-of-16 reads 50.4% and a random pick reads 49.9%. The probe calls them indistinguishable at 1.40 SE, which is the whole point: selecting the best of sixteen coins bought nothing you could keep. The winner's held-out rate is the base rate. Meanwhile the in-sample number sits 29.5 SE above held-out on the paired comparison. The entire curse lived in the eval you selected on, and none of it survived the move to fresh data. That is the honest translation of "you selected noise."

The probe is not vacuous, and I check that too. When I give one config a real edge, config 0 truly at 0.70 while the other fifteen sit at 0.50, the same probe flips:

   held-out(best of 16)  : 69.4% (k=27747 n=40000 SE=0.23)
   held-out(random pick) : 51.5% (k=20595 n=40000 SE=0.25)
   ok [SPREAD: held-out of best-of-16 vs a random pick]: conditional 69.4% separates from unconditional 51.5% by 52.60 SE >= 2.0 - the conditioning is real

Now selection earns its keep: the winner's held-out rate beats a random pick by 52.60 SE. So the probe discriminates. It is forced under the null and not forced when there is signal, which is the only way I would trust the null result at all.

When selection is safe and when it lies

None of this says stop running bake-offs. It says know which regime you are in. I held K=8 and n=100 fixed and widened one config's true edge over a flat field, reading the exact stats at each width:

leader true p | edge (SE) | P(pick best) | held-out best | lift vs random | in-sample bias
         0.50 |      0.00 |        14.4% |         50.0% |         -0.00 |           7.10
         0.52 |      0.40 |        23.6% |         50.5% |          0.22 |           6.96
         0.55 |      1.00 |        42.0% |         52.1% |          1.47 |           6.21
         0.60 |      2.00 |        74.3% |         57.4% |          6.18 |           3.62
         0.70 |      4.00 |        99.2% |         69.8% |         17.33 |           0.19

Two ends, two different worlds. A dead-flat field, edge 0 SE, is the null: selection picks the true best only 14.4% of the time, about one in eight for eight configs, the held-out lift over a random pick is zero, and the in-sample bias of 7.10 pp is the entire result. A runaway leader, edge 4 SE at a true 0.70, gets picked 99.2% of the time, delivers a real 17.33 pp held-out lift, and its in-sample bias shrinks to 0.19 pp because there is little noise left to exploit next to a gap that large. The dangerous regime is the middle, one or two SE of edge, which is where most prompt and model bake-offs actually land: a handful of items separating the top few. There, selection inflates the score and the winner regresses.

How is this different from the McNemar piece?

Fair question, since both are about comparing configs on an eval set. The McNemar piece is about comparing two configs correctly: two prompts on one task set give paired outcomes, so the right significance test is McNemar, not a two-proportion SE. That is a question about the test you run on a fixed comparison. This piece is about what happens after you compare K of them and keep the max. The bias here is not in any pairwise test. It is in the selection step and in carrying the winner's eval score forward as if it were an unbiased estimate. You can run a flawless paired test between every pair and still report an inflated winner, because the inflation is post-selection, not a testing error. Different failure, one layer up.

What I am not claiming

Three limits, stated plainly.

The fixtures are synthetic Bernoulli configs, not your agent. The order-statistic math is exactly right for i.i.d. estimates; real evals have correlated items and non-identical difficulty, which change the constants but not the direction. If anything, correlation across your candidates makes the effective K smaller and the bias milder than the i.i.d. case, while heavy-tailed noise makes it worse. The sign does not move.

The bias is about the winner's score, not about whether you chose the right config. With a real leader you usually still pick it, as the spread table shows. The mistake the curse causes is trusting the number next to the winner, not the act of choosing a winner.

And it is an expectation. Any single held-out re-eval has its own noise, so a particular winner might regress a lot, a little, or by luck not at all. The claim is about what happens on average across bake-offs, which is the thing your intuition quietly assumes when it treats an eval score as ground truth.

What does your bake-off do

Everything here ran locally, Python 3.13.5, stdlib plus the pinned measure.py, offline, no keys, no funds, three runs byte-identical, exit 0, empty stderr. The null-world numbers are exact order statistics; the sampled sections use a fixed seed. Same eval harness as the two pieces before it, and the three of them are one family of pre-execution gates for AI agents: decide what you can actually claim before you act on the claim.

I publish the runs that correct my own reading, not only the ones that flatter it, so follow along if that is your kind of thing. And a real question I do not have a clean answer to. When your last bake-off picked a winner out of a dozen prompts on a couple hundred items, did you hold that winner out on a fresh set before you shipped its score, or did the eval number go straight into the launch doc? I want to know how many candidates your team tries, and whether anyone writes down that trying more of them is the thing making the winner look better.

AI disclosure. I drafted this with an AI writing assistant and edited every line; the framing, the tool, and the reading are mine. Every output block is pasted from one real local run on 2026-07-23. winners_curse.py sha256 c968f39c9295fff7…, run output sha256 41934693461986a6…, and the imported measure.py stays at its pinned sha256 b1b3702bccb6ab46… so the earlier pieces keep verifying. Recompute: python3 winners_curse.py prints the same bytes, exit 0, empty stderr, stdlib only.

Zero failures isn't zero risk: the rule of three for evals

Alexey Spinov — Wed, 22 Jul 2026 03:11:12 +0000

The rule of three for evals says zero failures in N runs is a count, not a rate. With 0 failures in N independent runs, the exact 95% upper bound on the true failure rate is 1 - 0.05^(1/N), which 3/N approximates. After 100 clean runs you still cannot rule out a 2.95% rate, about 1 in 34.

Here is the reading that bites you. Your eval harness runs the agent 100 times, prints "0 failures," and the tile goes green. Someone screenshots it into the launch thread. The unspoken translation is "the failure rate is zero." It is not what the data says.

I wrote a small script to make the gap concrete, so I ran a real gate over 200 deterministic agent runs first, counted honestly, and got the dashboard everyone trusts:

gate: spend<=budget over 200 deterministic agent runs
observed failures: 0  (distinct scenarios: 200)
naive point rate : 0.00%   binomial SE: 0.00 pp
naive 95% CI     : [0.00%, 0.00%]   <- zero width: false certainty

Look at the standard error. For a zero count the binomial SE is sqrt(0*1/200), which is exactly 0, so the naive 95% interval collapses to [0.00%, 0.00%]. A zero-width confidence interval. The math is telling you it is completely certain, from 200 samples, that the true rate is precisely zero. That is obviously wrong, and it is the exact shape of every "all green" board I have ever trusted too much.

TL;DR

"0 failures in N runs" is an observed count, not a rate. The naive binomial SE of a zero count is 0, which is why a green board looks like proof and isn't.
The honest number is the one-sided upper bound. With 0 failures in N runs, the 95% upper confidence limit on the true failure rate is 1 - 0.05^(1/N). The rule of three, 3/N, approximates it and rounds the risk slightly up.
At N=30 the bound is 9.50% (about 1 in 11). At N=100 it is 2.95% (1 in 34). At N=1000 it is 0.30% (1 in 334). Zero failures in 30 runs is compatible with a 1-in-11 true failure rate.
To rule out a 0.1% rate at 95% you need about 2995 clean runs, not 50. "We ran it fifty times" and "we are 99.9% sure" are different sentences.
Correlated runs make it worse. 200 runs that are really 20 scenarios replayed ten times give an effective N of 20, and the bound jumps from 1.49% to 13.91%.

What this is and is not. The gate in the script is benign by construction: I built it so it observes zero failures, because a clean eval is exactly the case I want to talk about. The bound over that count is a theorem, not a fitted effect. There is no tuned constant anywhere in this that, if I nudged it, would flip the conclusion. The only inputs are the count (0), the number of runs (N), and the confidence level. I show that explicitly in section 6 of the run, because "the number rides on a fixture I chose" is the failure mode I most distrust in this kind of post.

Why zero observed failures is not a zero failure rate

Counting is not estimating. If you flip a coin twice and it lands heads twice, you have observed a 100% heads rate, and nobody sane concludes the tails face does not exist. Two runs is not enough to rule much out. The same logic holds when the observed count is zero and the sample is 30, or 100, or 1000: a rare failure can hide behind any finite streak of clean runs, and the question is not "did it fail" but "what rate is still consistent with what I saw."

The naive dashboard answers a different, easier question. It reports the point estimate, 0/N = 0%, and its standard error, which for a zero count is zero. So the interval has no width and the board is green with no caveat. This is the statistical version of a monitor that has been silently dead for three weeks: a check that always returns "fine" and a check that has stopped running produce the identical screenshot. Silence is not the same as health, and a zero-width interval is not the same as certainty. This is the whole of the tracking-is-not-control idea in one number: the board is counting what already happened and calling the absence of an event a guarantee about the next one.

So throw out the point estimate and ask the honest question instead. Given that I saw zero failures in N runs, how high could the true failure rate be and still make a run this clean plausible?

The rule of three, and why it is exact at zero

The upper bound falls straight out of the binomial, no approximation required. The only way to observe zero failures in N independent runs is for every run to pass, and the probability of that is (1-p)^N. The Clopper-Pearson upper confidence limit is the failure rate p that makes an outcome this clean just barely believable at your confidence level: set (1-p)^N = 0.05, solve, and you get p = 1 - 0.05^(1/N). Because the binomial tail at x=0 is a single term, this is exact, not a normal approximation that wobbles at small N. One word of caution on exact: it means the coverage is guaranteed to be at least 95%, not that this is the tightest interval available. Clopper-Pearson sits at the conservative end of the family (Wilson and Jeffreys intervals run a hair narrower), and I take the conservative end on purpose when the thing I am bounding is a rare and expensive failure.

The rule of three is the pocket version. The true constant is -ln(0.05) = 2.9957, and 3/N is that rounded to a number you can do in your head. Here is the exact bound beside the rule of three across N, all assuming zero observed failures:

     N |  exact 95% upper |      3/N |   ~1 in K | exact-3/N (pp)
------------------------------------------------------------------
    10 |           25.89% |   30.00% |         4 |         -4.113
    30 |            9.50% |   10.00% |        11 |         -0.497
    50 |            5.82% |    6.00% |        17 |         -0.184
   100 |            2.95% |    3.00% |        34 |         -0.049
   300 |            0.99% |    1.00% |       101 |         -0.006
  1000 |            0.30% |    0.30% |       334 |         -0.001

Read the N=30 row. Thirty green runs, zero failures, and the true failure rate could still be 9.50%, roughly one bad run in eleven, without your eval being unlucky at all. The 3/N column sits a hair above the exact bound in every row (the gap is negative because exact minus 3/N is small and negative), so the mnemonic slightly over-states the risk. That is the direction you want a shortcut to err. By a thousand runs the two columns read the same to two decimals (0.30% and 0.30%), and the rule of three stops being an approximation worth distinguishing.

None of these numbers are typed in. They come out of 1 - 0.05^(1/N) in the script, and I cross-checked the row values against an independent recompute before I quoted them, because quoting a bound I hadn't actually run would be its own kind of lie.

How many clean runs actually buy a target?

Flip the question around. If you want to be able to rule out a given failure rate, how many clean runs does that take? Invert the bound, 1 - 0.05^(1/N) <= target, and you get N >= ln(0.05)/ln(1-target):

target = the failure rate you want to be able to rule out at 95%.
  bound <=  5.00%  needs  N =    59 clean runs (exact 58.4)
  bound <=  1.00%  needs  N =   299 clean runs (exact 298.1)
  bound <=  0.30%  needs  N =   998 clean runs (exact 997.1)
  bound <=  0.10%  needs  N =  2995 clean runs (exact 2994.2)

This is the section that changes how I plan an eval budget. Ruling out a 1% failure rate takes 299 clean runs. Ruling out 0.1%, the neighborhood where you'd actually feel safe shipping an agent that can move money or delete data, takes about 2995. For anyone paying a judge model per eval item, those Ns are a line straight to a bill, and the honest version of "prove it's safe" is expensive in a way the green tile hides. If your rare-but-catastrophic action got 50 clean runs, the most you can claim at 95% is that its failure rate is probably under 6%. That is not a number I would put next to "delete the production database."

The confidence level is a knob too, and it is worth seeing that "three" is not sacred:

  conf |  -ln(alpha) |           name |  upper@N=100
----------------------------------------------------
   90% |      2.3026 |    rule of 2.3 |        2.28%
   95% |      2.9957 |      rule of 3 |        2.95%
   99% |      4.6052 |    rule of 4.6 |        4.50%

The rule of three is specifically the 95% case, because -ln(0.05) rounds to 3. Want 99% confidence in your upper bound? That is the rule of 4.6, and at a fixed N it gives a looser bound (4.50% versus 2.95% at N=100), because more confidence over the same data costs you sharpness. Pick the confidence deliberately instead of inheriting 95% because a stats class used it.

The trap in the number: correlated runs shrink your N

Everything above assumes the runs are independent. In agent evals they very often are not, and the failure is quiet. You run the same 20 scenarios with 10 seeds each and call it 200 runs. The board says 200; the informative count is 20. Common-mode inputs, the same prompt replayed, the same fixture, the same upstream data, inflate N on the dashboard and leave the real bound where the distinct cases put it.

nominal runs: 200   distinct scenarios: 20   observed failures: 0
bound if you count nominal N=200 : 1.49%  (~1 in 67)
bound at effective N=20          : 13.91% (~1 in 7)

Same zero failures. Same 200 lines in the log. But the bound you're entitled to is 13.91%, about 1 in 7, not 1.49%. This one cuts against the reassuring direction, which is exactly why I put it in: replays and shared inputs shrink your effective N, and a smaller N only ever loosens the bound, never tightens it, since 1 - 0.05^(1/N) falls as N grows. A harness that counts replays as independent runs is quietly reporting a safety you don't have. If you can't argue your runs are independent draws from the distribution you care about, use the count of distinct cases, not the count of executions.

What the bound does not tell you

Three honest limits, because the number is narrow on purpose.

It is about frequency, not identity. 1 - 0.05^(1/N) bounds how often a failure could happen. It says nothing about which failure, or how bad. A 1% rate of "off-by-a-cent" and a 1% rate of "wired the whole treasury" are the same number here and nothing alike in practice. The bound does not replace worst-case analysis or reading results by severity class; it sits next to them.

It bounds the rate on your eval distribution, not on production. If your 100 clean runs don't cover the inputs production will throw, the bound is honest about the runs you did and silent about the ones you didn't. A clean eval on the wrong distribution is a different problem, closer to contamination in the eval itself.

It is an upper bound, not an estimate. Zero failures does not mean the rate is 3/N. It means the rate is probably below 3/N. The true rate could be much lower, or, with bad luck, right at the bound. The point is not to name the rate. It is to stop pretending a clean run named it zero.

Why there is nothing here to fake

I have dropped enough of my own drafts to be suspicious of any post where the punchline sits inside a constant the author picked. So I built this one to have no such constant. The conclusion depends on three inputs and nothing else: the observed count, N, and the confidence level. Section 6 of the run makes the point by brute force:

re-run the gate: failures still 0 of 200  ->  bound 1.49%

Shuffle which scenario the gate sees first, run it twice, reorder the whole fixture: the count stays 0, so the bound is byte-for-byte the same. There is no effect size to permute, no ratio whose denominator is secretly doing the work. That is the opposite of the class of eval post that gets dropped, where the "finding" is really a fixture setting in disguise. Here the only thing I observe is a count, and the only thing I do to it is bound it. If you distrust the bound, you are distrusting the binomial, which is a fine thing to argue about and a very different argument from "he tuned the numbers."

The whole thing runs in a second: Python 3.13.5, stdlib only, offline, no keys, no funds, three runs byte-identical, exit 0, empty stderr. Picking the right statistical test is a sibling problem, and I wrote up the paired-data version of it in why your A/B eval probably needs McNemar, not a two-proportion SE. Both belong to the same family of pre-execution gates for AI agents: decide what you can actually claim before you act on the claim.

What does your board show

The next time a tile goes green on "0 failures in N runs," do the one-line translation out loud. Zero in 30 is a rate that could be 1 in 11. Zero in 100 is a rate that could be 1 in 34. Neither is zero, and the dashboard will not tell you that unless you make it.

I publish the runs that correct my own reading, not only the ones that flatter it, so follow along if that is your kind of thing. And a real question I don't have a clean answer to: when your eval last went green, did anyone write down the N, or did "0 failures" quietly get filed as "the failure rate is zero"? I want to know how many runs your team treats as enough, and why that number and not ten times it.

AI disclosure. I drafted this with an AI writing assistant and edited every line; the framing, the gate, and the reading are mine. Every output block is pasted from one real local run on 2026-07-22. zero_failures.py sha256 d3b3f205909640fc…, run output sha256 cc7aca328fdd7236…. Recompute: python3 zero_failures.py prints the same bytes, exit 0, empty stderr, stdlib only.

Your A/B eval is paired. Your stat test probably isn't.

Alexey Spinov — Tue, 21 Jul 2026 03:03:28 +0000

Paired eval, wrong test: two prompts scored on one 100-item set give paired outcomes, so ranking them needs McNemar, not the two-proportion Wald SE. On the same data, Wald read 1.01 SE and said 'collect more'; McNemar read 2.65 SE and allowed the ranking. Same 100 items, opposite decision.

I shipped a little eval helper that refuses to rank two configurations when their gap is smaller than two standard errors. The idea is good. The refusal is honest. Last week it looked at two prompts scored on the same 100-item set and printed this:

RANK: INDISTINGUISHABLE - gap 7.00 pp against 6.95 pooled SE = 1.01 SE < 2.0. Ranking "prompt B" above "prompt A" is NOT allowed.

Seven points of apparent improvement, called noise. The honest reading of that line is "collect more items." And it was wrong. Not a rounding error, not a close call. The right test on that exact data says the ranking is already decided, and I could have stopped.

The bug was not in the arithmetic. It was in which test the arithmetic ran. My helper was treating two paired runs as if they were two independent samples.

TL;DR

Two prompts on one task set are paired: each task gives a pair (pass_A, pass_B), and both-pass / both-fail tasks carry zero information about the difference.
My harness used the Wald SE of a difference of two proportions, pooled = sqrt(se_A**2 + se_B**2). That is the independent-samples formula; it ignores the pairing entirely. In my showcase that cost real power.
The paired test is McNemar: SE = 100*sqrt(b+c)/n, using only the discordant counts b and c. On the same 100 items it returned 2.65 SE, not 1.01, and ranking was allowed. With just 7 discordant pairs the tool also prints the exact binomial, p=0.0156 (z-equivalent 2.42), so the decision does not ride on the normal approximation.
On 535 real paired observations from my own sweep, one pair reads 2.65 SE under Wald and 6.24 under McNemar with c=0: a strictly nested result that Wald reports as a hair over the line and has no way to flag as deterministic.
The fix is about fifteen lines. The hard part is not the formula, it is noticing your runs are paired.

What this is and is not. The two-prompt table below is a constructed example: I picked the counts to sit exactly on the line where the two tests disagree, so you can reproduce it with four numbers. The 535-observation table comes from a synthetic marker fixture I wrote, not from anyone's production system. Where a number is forced by my constants rather than measured, I say so in the same paragraph. That habit is half the point.

Why two prompts on one set are paired

Run prompt A and prompt B on the same 100 eval tasks. Task 7 either trips both of them, or neither, or exactly one. That last group is the only one that tells you which prompt is better. The tasks where both pass and the tasks where both fail are shared difficulty: they move both pass rates together and say nothing about the gap.

Here is the case that caught me, laid out as a 2x2:

                     B pass   B fail
        A pass          55        0     <- b = 0 (A pass, B fail)
        A fail           7       38     <- c = 7 (A fail, B pass)
        A marginal = 55/100   B marginal = 62/100   n = 100 pairs

Prompt A passes 55, prompt B passes 62. Same items. Ninety-three of the hundred tasks are concordant: 55 both pass, 38 both fail. Seven tasks are discordant, and all seven go the same way, B passes where A failed. Zero go the other way. B's pass set contains A's pass set completely. That is a strong statement, and it is invisible to a test that only looks at the two marginal rates.

What my harness printed, and why it was the wrong number

The refusal came out of rank(). Under the hood it builds the pooled standard error the way you were taught for two independent proportions:

    pooled = sqrt(p1.se ** 2 + p2.se ** 2)

Each se is the binomial standard error of one marginal rate. For 55/100 and 62/100 that pools to 6.95 points, the 7-point gap divides to 1.01 SE, and the guard files it under "indistinguishable." Fair, if the two samples were drawn independently. They were not. The 93 concordant tasks are the same 93 tasks in both columns, and the formula charged me full variance for them anyway.

The cost of believing that line is real budget. At 1000 items each, the same seven-point gap finally clears the bar:

RANK: "prompt B" > "prompt A" - gap 7.00 pp = 3.18 SE >= 2.0. Ranking is allowed.

So the harness asked for ten times the eval spend to reach a verdict the correct test already had at 100 items. For anyone paying per token to run a judge model over a set, that is a straight line from a stats mistake to a bill.

What McNemar does instead

McNemar's test (Quinn McNemar, Psychometrika, 1947, the standard test for paired nominal data) throws away the concordant pairs and looks only at b and c. The standard error of the difference becomes 100*sqrt(b+c)/n, and the test statistic is |c-b|/sqrt(b+c). I added it to the same library as mcnemar(name_a, b, name_c, c, n). On the exact same 100 items:

McNEMAR: discordant pairs prompt A=0, prompt B=7 (concordant 93 of n=100).
  SE = 2.65 pp (100*sqrt(b+c)/n); marginal gap 7.00 pp = 2.65 SE >= 2.0 -> ranking "prompt B" over "prompt A" is allowed.
  small discordant count (b+c=7 < 25): the normal approximation is anti-conservative here. Exact two-sided binomial p=0.0156 (z-equiv 2.42), continuity-corrected z=2.27. all three clear the 2.0 bar; the decision is unchanged.
  NESTED (b=0 or c=0): every discordant item favours the same side; the difference is deterministic (strict nesting on this sample), not a coin-flip margin.

2.65 SE against the same 2.0 threshold. Allowed. Same data, opposite decision. And note the third line. With only 7 discordant pairs, 2.65 is a normal approximation, and the tool will not let me lean on it: beside it sit the exact two-sided binomial, p=0.0156 (z-equivalent 2.42), and the continuity-corrected z at 2.27. All three clear the 2.0 bar. Same decision, reached without trusting the largest number in the row. The NESTED line fires because one discordant count is zero, which means the two prompts never disagreed in both directions: on this sample B dominates A item by item. That is a qualitatively different thing from a noisy 7-point margin, and a test that pools marginals cannot see it.

One honesty note, because it matters. NESTED is not a license to rank anything with a zero in it. A pair with b=0, c=1 is also nested, and the tool prints it at 1.00 SE, nowhere near the bar. The ranking decision still rides on the z value. In this case the z is 2.65 and the nesting is strict, so both agree. I flag them separately on purpose.

The same divergence on 535 real observations

The constructed example is clean but you should distrust anything I can tune to land on the line. So here is the same phenomenon on data I did not hand-pick: a sweep of a monotone-but-not-total marker fixture, P_PATHS=12 W_PER_TICK=3 T_TICKS=400 SEEDS=20, where a false reject is a landed write that a single-integer witness wrongly rejected.

The cells of that sweep share one underlying draw. Path and outcome come off the LCG independently of the axis knobs, so the raw (tick, path, outcome) at each sampled position is identical across cells. The run checks that before it does anything else, across all 33 sampled points per seed:

(tick,path,outcome) skew 6:6 vs 7:5            identical: True  (33 sampled points)
(tick,path,outcome) streams 8 vs 4             identical: True  (33 sampled points)
ALL PAIRED (raw observations identical across cells): True

Same observations, only the verdict moves. That is the definition of paired, and it is why the two-proportion SE is the wrong tool here too. Cross-tabbing the false-reject indicators gives real b/c counts. Both tests, side by side:

pair                     | b / c / n        | Wald (rank)    | McNemar (paired)
------------------------------------------------------------------------------
6:6 vs 7:5               | 142 / 128 / 535  | 0.87 SE        | 0.85 SE
7:5 vs 8:4               | 39 / 3 / 535     | 2.31 SE        | 5.55 SE
8:4 vs 9:3               | 39 / 0 / 535     | 2.65 SE        | 6.24 SE NESTED
streams=8 vs streams=4   | 91 / 32 / 535    | 3.98 SE        | 5.32 SE
streams=2 vs streams=1   | 229 / 0 / 535    | 20.01 SE       | 15.13 SE NESTED

Read the third row. 8:4 versus 9:3 is 39 discordant pairs, all in one direction, c=0. Strict nesting again, on real counts. Both tests clear the 2.0 bar here, so both allow the ranking. But Wald reports 2.65, a hair over the line, while McNemar reports 6.24 and flags NESTED. One of those tells you the result is deterministic on this sample; the other cannot tell "barely over the threshold by luck" apart from "decided." Here is that pair with both guards printing in full:

PAIR 8:4 vs 9:3   marginal FR: 8:4=172/535  9:3=133/535
    8:4: 32.1% (k=172 n=535 SE=2.02)
    9:3: 24.9% (k=133 n=535 SE=1.87)
  RANK: "8:4" > "9:3" - gap 7.29 pp = 2.65 SE >= 2.0. Ranking is allowed.
  McNEMAR: discordant pairs 8:4=39, 9:3=0 (concordant 496 of n=535).
    SE = 1.17 pp (100*sqrt(b+c)/n); marginal gap 7.29 pp = 6.24 SE >= 2.0 -> ranking "8:4" over "9:3" is allowed.

Two honest observations about that table. First, the two tests never disagree on direction: whichever version each one ranks higher, they agree, on all five pairs. Second, at this 2.0 threshold none of the five flip the decision. Row one, 6:6 vs 7:5, is 0.87 under Wald and 0.85 under McNemar, both below the bar, both refused. Rows two through five sit above the bar for both. What moves is the reported SE, sometimes a lot, 2.31 against 5.55, and whether the c=0 nesting gets named at all.

And the gap runs both ways, which is why "McNemar is always more powerful" would be the wrong lesson. On 6:6 vs 7:5 Wald reads 0.87 against McNemar's 0.85, and on streams=2 vs streams=1 it reads 20.01 against 15.13: when the discordant pairs are many and lopsided, the independent-samples SE is not the conservative one, it understates the variance instead. The rule is not that one test wins, it is that you owe your data the test its design calls for.

So where is the decision flip? Near the bar. This particular sweep happens to spread its pairs away from the 2.0 line, so the two tests agree on every call even while their SEs diverge. The constructed 100-item example sits right on the line, which is where the difference between the tests stops being cosmetic and turns into a yes or a no. I did not engineer that to cheat; it is where most config bake-offs I have watched actually get decided, on a handful of items either way.

One thing I am deliberately not claiming

You may have noticed those false-reject levels, 75.9%, 64.9%, and want me to say something about them. I will not, and the same library is why. Run the construction-independence probe on those cells and it comes back:

FORCED BY CONSTRUCTION [streams=8]: conditional 75.9% (k=406 n=535 SE=1.85) is indistinguishable from unconditional 74.7% (k=493 n=660 SE=1.69) (0.48 SE < 2.0)

The level equals the unconditional pass-the-gate rate, 74.7% at n=660, because the fixture draws outcome and stamp from separate LCG steps. So the level is an artifact of my constants, not a measurement, and I quote none of them as findings. What survives that probe is the discordance structure, b and c, which is a genuine between-cell comparison. The McNemar inputs are real; the levels they sit next to are not. Running the probe and reporting the result is the only reason I trust the distinction.

The fix, in about fifteen lines

There is nothing clever in it. It is the discordant count and a square root:

    disc = b + c
    conc = n - disc
    ...
    se = 100.0 * sqrt(disc) / n
    gap = 100.0 * abs(c - b) / n
    n_se = abs(c - b) / sqrt(disc)          # = gap / se, n cancels
    ...
    if b == 0 or c == 0:
        # NESTED: strict set-nesting, deterministic on this sample

(Truncated by me: the wrapper handles b==c, the no-discordant case, and localized output; full function in measure.py.)

The formula is the easy part. The part that actually protects you is upstream and it is not in this function at all: the library cannot know your two runs are paired. You have to establish that yourself, by confirming the observations are the same items in the same order and only the outcome moved. That is the check the sweep runs before it calls mcnemar() at all, and if the raw observations had differed it would have refused to treat them as paired. A paired test on unpaired data is its own mistake.

The limits, plainly. SE = 100*sqrt(b+c)/n is the normal approximation to McNemar; for a handful of discordant pairs you want the exact binomial instead, which is why the function prints it automatically once b+c drops below 25, and I would not read the plain SE without it. The z threshold of 2.0 is a convention I carried over from the rest of the harness, not a law. And none of this touches the failure mode that actually costs you money, a silent false accept, because that is a question about your ground truth, not your arithmetic.

What does yours do

Everything here ran locally on a synthetic fixture: Python 3.13.5, stdlib only, offline, no keys, no funds, three runs byte-identical, exit 0, empty stderr. Two earlier pieces work the same eval harness from other angles: reading eval results by severity class instead of a flat pass rate and a static probe that found contamination points without running the agent. Both belong to the same family of pre-execution gates for AI agents.

I publish the runs that corrected my own reading, not only the ones that confirmed it. Follow along if that is your kind of thing. And a real question I do not have a clean answer to: when your last A/B put two agent versions at 76 and 74 out of 100 on the same set, did your harness run a paired test, or did it pool two marginals and quietly ask you for more data? I would like to know what yours does.

AI disclosure. I drafted this with an AI writing assistant and edited every line; the test choice, the sweep, and the reading are mine. Every output block is pasted from one real local run on 2026-07-21. measure.py sha256 b1b3702bccb6ab46…, paired_test.py sha256 fa687e05e8e69512…, run output sha256 d8e4e521b1f35ac0…. The library default stays Russian so the sha256 of two already-published runs keeps verifying; English is opted into explicitly.

A Spend Cap That Stops Counting Is Already Fail-Open

Alexey Spinov — Sun, 19 Jul 2026 01:53:12 +0000

Two of the five ways a spend cap can handle a missing price produce the exact same decision stream — same sha256, byte for byte. One of them is the thing everybody calls fail-open. The other is the thing everybody recommends instead of it: fall over to a free local model.

Let me say what that hash is and isn't before it does any work. It covers (seq, label, admitted, charge) and deliberately drops the human-readable reason strings, so two policies that print different words hash the same when they decide the same. Once you see that, the collision is a theorem rather than a discovery: a free fallback charges zero, fail-open charges zero, and a ledger built out of charges cannot tell them apart because there is nothing there to tell apart. The sha256 proves only that my implementation doesn't quietly cheat.

The reason it's still worth a post is that nobody ships them as the same policy. One is the bug you apologise for; the other is the fix you recommend in the thread. On the axis that matters they are one policy, and one of them has better branding.

AI disclosure. I wrote blind_spend_cap.py with AI assistance and ran it myself. Every number and hash below is pasted from a real run: offline, stdlib only, no network, no keys, no funds. The oracle is injected, so runs are deterministic. I ran it three times; the output was byte-identical each time. Code sha256 ddc42590…, output sha256 9ebe1b4a…. External figures are linked and labeled, and I say clearly which ones I did not reproduce.

TL;DR

A spend cap needs a cost-oracle to price the next action. The oracle has its own outage.
The usual framing (fail-open vs fail-closed) is the wrong axis. The real split: does the ledger keep moving while the oracle is quiet?
Strategies that charge a price the remaining budget can still absorb keep the ledger alive and re-trip the cap. Charge zero and spent freezes forever. Charge more than fits and you've written refuse with extra steps — the harness proves that one on itself.
A free local fallback charges zero. In my harness it produces a decision stream identical to plain fail-open: same sha256.
But a moving ledger is a floor, not a certificate. Any positive fiction satisfies it — price a $0.05 call at $0.01 and your cap is quietly five times the one you configured. The real axis is the bias of your estimator; zero is just where that bias hits −100%.
The headline number you'd expect me to use here (34 extra actions) is arithmetic, not evidence. I take it apart below rather than sell it.

The branch nobody writes down

Every spend cap I've shipped has the same shape. Price the action, compare against a budget, allow or block. The pricing step assumes the oracle answers.

It doesn't always. CoinGecko 429s. A usage endpoint times out. A token meter sits behind a gateway returning 526. In that moment your cap takes a decision that probably isn't in your code review notes, because it isn't in your code: what to do with an action it cannot price.

I know it's unwritten because I shipped it that way. On June 8, 2026 I published SpendGuard, a 40-line pre-execution cap. It works, and it never declared this branch. The oracle call sits inside cost_fn on line 121, and eth_price_usd() calls raise_for_status() before it returns anything. So on a 429 the exception blows straight past the gate and out of the wrapper. Accidentally fail-closed, by way of an uncaught exception that takes the caller down instead of returning a verdict you can count.

Copy the demo loop from that same post and you get the opposite. It prices once before the loop and reuses that number for every round. A mid-loop outage is invisible. Accidental fail-open.

Same file, two wirings, two opposite behaviors, and I declared neither.

Five strategies, one fork

So I built the smallest thing that isolates the branch. blind_spend_cap.py runs one Analyzer/Verifier ping-pong through one budget gate, under five strategies that are identical everywhere except the quote is None block:

    else:
        if strategy == "refuse":
            return _v(seq, "BLOCK", "no quote: refuse", False, 0, priced)
        if strategy == "admit-unpriced":
            return _v(seq, "ADMIT", "no quote: admit, charge 0 (ledger frozen)",
                      True, 0, priced)
        if strategy == "stale":
            if last_known is None:
                return _v(seq, "BLOCK", "no quote and no last-known price: refuse",
                          False, 0, priced)
            est, tag = last_known, "stale "
        elif strategy == "pessimistic":
            est, tag = per_action_cap, "pessimistic "
        elif strategy == "fallback":
            est, tag = fallback_cents, "fallback "

Two design choices worth stating, because both cut against the result I might have wanted.

A BLOCK does not stop the loop. A real runaway retries. My earlier harness gave the refusing policy a free break, which quietly handed it the win: it "stopped the runaway" because I wrote the loop that way. Here the gate stops the spend, not the work, and the loop keeps hammering. There's an --on-block halt flag for the single-shot shape, and I sweep both.

The budget has no clock, so I call it a per-run budget rather than a daily one. Calling it daily would be a lie in a file with no time in it.

The split isn't admit-vs-block. It's counting-vs-not.

Here's the outage run, oracle down from step 6, straight from output.txt:

SCENARIO B — oracle down from step 6, loop retries after a BLOCK
  refuse           admitted=6   spent=$0.30  unpriced=0   unaccounted=0   ledger-moved=False exit(conv)=1 exit(strict)=1
  admit-unpriced   admitted=40  spent=$0.30  unpriced=34  unaccounted=34  ledger-moved=False exit(conv)=0 exit(strict)=2
  stale            admitted=10  spent=$0.50  unpriced=4   unaccounted=0   ledger-moved=True  exit(conv)=1 exit(strict)=2
  pessimistic      admitted=6   spent=$0.30  unpriced=0   unaccounted=0   ledger-moved=False exit(conv)=1 exit(strict)=1
  fallback:0c      admitted=40  spent=$0.30  unpriced=34  unaccounted=34  ledger-moved=False exit(conv)=0 exit(strict)=2
  fallback:1c      admitted=26  spent=$0.50  unpriced=20  unaccounted=0   ledger-moved=True  exit(conv)=1 exit(strict)=2

Look at the spent column, not the admitted one.

Exactly two strategies end at $0.50: stale and fallback:1c. That's the budget, tripped, doing its job. admit-unpriced and fallback:0c end at $0.30 and stay there — not because the run was cheap, but because after step 6 nothing was ever added to the ledger again.

Now the row that breaks the tidy version of this claim, which I had written as "every strategy that charges something ends at $0.50" until the table two lines above told me otherwise. pessimistic charges the most of anybody and still ends at $0.30. Charging something isn't sufficient. The something has to fit. pessimistic prices every un-priced call at the $0.25 per-action cap, only $0.20 of budget remains after step 6, so every un-priced call is blocked on arrival: admitted=6, unpriced=0.

Which changes how you read the unaccounted column. It counts admissions where the oracle gave no quote and nothing was charged — defined by the fact of a missing quote, not by the name of the policy, so it can accuse any strategy including the ones I like. stale and fallback:1c sit at zero because they kept counting. pessimistic and refuse sit at zero because they never admitted a blind call in the first place. Same number, two different stories, and I'd been reading the flattering one into both.

Push it to the limit and the failure gets loud. Oracle dead from step 0:

$ python3 blind_spend_cap.py --strategy admit-unpriced --oracle-fails-from 0
admit-unpriced   admitted=40  spent=$0.00  unpriced=40  unaccounted=40  ledger-moved=False exit(conv)=0 exit(strict)=2

Forty actions admitted. Ledger says zero dollars. Exit code zero, under the mapping most gates actually ship.

That's the thing worth internalizing. A blind cap doesn't report danger. It reports innocence.

The number I'm not putting in the headline

You'd expect the pitch to be "fail-open admitted 34 more actions." The tool does print it. I'm going to argue against it anyway, because I got burned by exactly this number last time.

M is the gap in admitted actions between admit-unpriced and refuse. In the run above it's 34. Sweep the outage step across the whole parameter space and you get this:

  on-block=retry
        K:    0    5    6    9   10   11   12   20   39   40
        M:   40   35   34   31   30   29   28   20    1    0

(That's the retry half of the sweep with the unacc row dropped for now — under retry it's identical to M anyway. The full block, both loop shapes, is two sections down.)

M = WANTS − K, exactly, everywhere. I picked WANTS = 40. If I'd picked 1000, the headline would read 994. It isn't a property of any policy, it's a property of how long I let the loop want things. A number I chose, dressed up as a number I found.

Which is why the strategy table above leads with spent and unaccounted, and why M is buried in a scenario that tells you to go read the sweep before quoting it.

Pressing my own kill switch

The honest question isn't whether my metric works in the run I picked. It's where it stops working. So here's the same sweep under both loop shapes:

  on-block=retry
        K:    0    5    6    9   10   11   12   20   39   40
        M:   40   35   34   31   30   29   28   20    1    0
    unacc:   40   35   34   31   30   29   28   20    1    0
    -> M = 0 in 1/41 of K (2%); unaccounted = 0 in 1/41 (2%)
  on-block=halt
        K:    0    5    6    9   10   11   12   20   39   40
        M:   40   35   34   31   30    0    0    0    0    0
    unacc:   40   35   34   31   30    0    0    0    0    0
    -> M = 0 in 30/41 of K (73%); unaccounted = 0 in 30/41 (73%)

Under halt semantics, everything I've argued dies in 73% of the parameter space. Not weakens. Dies, to identically zero, both metrics at once.

The reason is dull and important. The budget is $0.50, a healthy call is $0.05, so a healthy run hits the cap at step 10. If the oracle only falls over at step 11 or later, the loop is already stopped by money. The un-priced branch is never reached. Nothing to measure, nothing to argue about.

So the applicability condition, which my previous draft never stated and which I'm stating plainly now: this post is about outages that start before your budget would have stopped the loop anyway — K <= budget // unit_cost, which is K <= 10 here — or about loops that retry after being refused. Outside those two cases the whole thing is a non-event.

That boundary was off by one in the tool until this morning: the summary line said the metrics collapse once K >= 10, when K = 10 is the last K where they're still alive and K = 11 is the first dead one. The sweep table underneath it was right the whole time. A decent argument for printing the table and not just the conclusion drawn from it.

I think retrying is the common case, because runaway agent loops are usually retry loops. That's a judgement about the world, not a measurement, and I'm flagging it as one.

A free fallback is fail-open with better branding

Now the equivalence. Same outage, and I hash the decision stream — the (seq, label, admitted, charge) tuples, deliberately excluding the human-readable reason strings, so two strategies that make the same decisions hash the same even when they print different words:

SCENARIO C — is a free fallback a distinct strategy, or a renamed fail-open?
  admit-unpriced   sha=53b01d22a232f1fee833a76c7cd1ed810d1945da2e7620c8a1a17a9302b4df79
  fallback:0c      sha=53b01d22a232f1fee833a76c7cd1ed810d1945da2e7620c8a1a17a9302b4df79
  stale            sha=0dcbd560f59adcf2b1eec3ca111dc7f89c3e7ac08569fe165d88ec7af77b4311
  fallback:5c      sha=0dcbd560f59adcf2b1eec3ca111dc7f89c3e7ac08569fe165d88ec7af77b4311
  refuse           sha=7df1f34491edfde21648d36e9a8eda1db7306d12efda4c29364fa8f54ad3b04a
  pessimistic      sha=7df1f34491edfde21648d36e9a8eda1db7306d12efda4c29364fa8f54ad3b04a
  -> fallback:0c  == admit-unpriced : True
  -> fallback:5c  == stale          : True  (needs >=1 real quote before the outage, and a constant oracle price)
  -> pessimistic  == refuse         : True  (at THESE caps: $0.25 never fits what is left of $0.50)

Three collisions in that block, and the third one costs me a third of my own recommendation.

fallback:0c == admit-unpriced is the headline, and — as I said up top — a theorem. Both charge zero, the hash covers the charge. I couldn't break it by moving the budget, the unit cost, the per-action cap, the loop shape or the outage step; it isn't a coincidence of the parameters I picked.

fallback:5c == stale is real but conditional, and I stated it as a general truth in an earlier pass. It needs two things I'd left unsaid. There has to be at least one successful quote before the outage — with --oracle-fails-from 0 there's no last-known price at all, stale refuses, and the equality collapses. And the oracle's price has to actually hold still; point the harness at a varying price and the two decision streams separate immediately. The honest version is narrower: a fallback pinned to the real rate is stale pricing, as long as the real rate isn't moving.

pessimistic == refuse was sitting in my own main table and I walked past it twice. Same 7df1f344…. A $0.25 estimate never fits the $0.20 left after step 6, so pessimistic blocks every un-priced call — which is refuse, decision for decision. Run --strategy fallback --fallback 25 --oracle-fails-from 6 and you get 7df1f344… as well. Three names, one behavior.

That forces an admission about my own advice. Further down I tell you to charge a stale price, or a conservatively biased estimate, or the per-action cap. At the parameters in my own demo that last one is bit-identical to the refusal I declined to recommend — which is why it now ships with that caveat attached instead of posing as a third independent door. It doesn't make the advice wrong. Refusing is defensible, and I spend a whole section on its bill.

The price sweep is where this gets concrete:

SCENARIO D — sweep the fallback price (the whole argument hangs on it)
  fallback:0c      admitted=40  spent=$0.30  unpriced=34  unaccounted=34  ledger-moved=False exit(conv)=0 exit(strict)=2
  fallback:1c      admitted=26  spent=$0.50  unpriced=20  unaccounted=0   ledger-moved=True  exit(conv)=1 exit(strict)=2
  fallback:2c      admitted=16  spent=$0.50  unpriced=10  unaccounted=0   ledger-moved=True  exit(conv)=1 exit(strict)=2
  fallback:5c      admitted=10  spent=$0.50  unpriced=4   unaccounted=0   ledger-moved=True  exit(conv)=1 exit(strict)=2
  fallback:25c     admitted=6   spent=$0.30  unpriced=0   unaccounted=0   ledger-moved=False exit(conv)=1 exit(strict)=1

So the clean line I wanted to write — "the runaway terminates if and only if the price is above zero" — is not true, and the bottom row is the counterexample. fallback:25c prices well above zero and spent still sticks at $0.30 with the ledger frozen. What a price above zero actually buys is a ledger that keeps moving, and only while that price still fits what's left of the budget. Under that band you get fail-open. Over it you get refusal wearing a price tag. The usable range is narrower than "not zero", and where it sits depends on caps I picked.

Worth being exact about the edge, because I rounded it off in an earlier pass. The price has to clear min(per-action cap, budget − spent when the outage begins) — here min($0.25, $0.20), so the collision with refuse actually starts at $0.21, and $0.25 is just the row I happened to print. Which of the two terms binds depends on when the oracle dies: at step 6 it's the leftover budget, at step 4 it's the per-action cap. So "too expensive to fit" isn't a property of the price alone — it's the price measured against however much budget the outage left you.

This matters because the free version is the one people ship. On July 15, 2026, a developer publishing as @ddhh released a small LLM circuit breaker with a clean statement of the instinct: "When I'm about to overspend, don't fail and don't keep paying — fall through to a free local model and keep working." The config in the post marks the tier # local, free, always-on fallback.

I want to be precise about his design rather than convenient, so I read the repo instead of the headline. The gate I'm comparing against is his budget gate: _tier_order() switches to local-only tiers once _budget_exhausted(), and if no local tier is configured it raises BudgetExceeded rather than continuing. That gate keys on accumulated spend against a limit — not on a missing quote. That's a declared branch with a hard stop, which is more than my June code had.

Provider failure is handled too, just not by that gate. His post opens on exactly that problem — "Paid API quotas dying mid-run. One provider 429s, and the whole run falls over" — and complete() wraps each tier in an except … continue (breaker.py:121-131, commented "a failed tier should never crash the caller"), so a 429 on a paid tier falls through to the next one, local included. Two triggers, one ordered failover. The reason I'm separating them carefully is that the thing I care about is which signal moves the ledger, and on both paths the local tier reports the same number: ollama_tier._call ends with return text, 0.0 (providers.py:167), which _record writes into the JSONL ledger as cost_usd: 0.0.

For his stated goal that is correct, and I'll say so flatly: a free local call really does cost zero dollars, so his ledger is accurate and his dollar spend genuinely cannot grow past the limit. He solved the problem he set out to solve.

The pattern risk lives one step to the side. Once every call costs zero, the dollar ledger can never stop anything again — and if the thing that pushed you over budget was a non-converging loop rather than an expensive model, dollars were never the binding constraint. Wall-clock, local GPU, rate-limited downstream endpoints and side effects all keep accruing, and the ledger reports $0.00 while they do. Exactly the fallback:0c row above.

None of that is a defect in his breaker. It's what the graceful-degradation instinct does when you port it from "budget exhausted" to "cost unknown" without noticing the axis changed.

For scale on why any of this is worth an afternoon: on July 16, 2026 @royanannya published a postmortem of a multi-agent loop that billed $1,847 in one weekend, fixed by pushing decisions off the LLM layer and cutting per-game cost from $1.95 to $0.35. Their number, their run; I didn't reproduce it. Their fix was architectural, not a spend cap, and I'm not going to pretend otherwise.

Where my own metric stops meaning anything

I've been leaning on unaccounted as though it measures how accurately you're counting. It doesn't. It's a binary test — did an admitted call with no quote get charged zero? — and the counterexample is sitting in my own output.

The harness has no notion of what an action actually costs. Every admit really executes, and a real call here is $0.05. Multiply the admitted column by that and set it beside the ledger:

strategy	ledger says	actually spent	vs the `$0.50` budget	`unaccounted`
`stale`	$0.50	$0.50	1.0x	0
`fallback:5c`	$0.50	$0.50	1.0x	0
`fallback:1c`	$0.50	$1.30	2.6x	0
`fallback:0c`	$0.30	$2.00	4.0x	34

fallback:1c passes my metric cleanly. unaccounted=0, ledger-moved=True, budget tripped on schedule — and I wrote "that's the budget, tripped, doing its job" about that exact row. It also pushed 26 real calls through a ten-call budget. Pricing a $0.05 action at $0.01 under-counts by 5x, and a cap that under-counts by 5x is a cap five times larger than the one you configured.

So "keep the ledger moving" is satisfied by any positive fiction. The real axis isn't zero versus non-zero — it's the bias of your estimator. Zero is simply the point where the bias hits −100% and the cap stops existing at all. It's the worst case and it's the common case, which is why it earns a post, but unaccounted=0 is a floor, not a certificate. If your fallback price is a comfortable number rather than a conservative one, you haven't fixed the runaway. You've slowed it down and moved it out of view.

There's a second place my instrument lies, and I found it checking this piece rather than writing it. The ledger-moved column samples spent after each blind decision, so what it actually asks is "did the ledger move more than once?" A price that fits exactly once — try --fallback 15 — pushes spent from $0.30 to $0.45 and still prints ledger-moved=False, the same value fail-open gets. None of the rows in this post are affected; they sit at 0c, 1c, 2c, 5c and 25c. But if you sweep the price yourself you'll walk into it, and it's the same conflation I've spent the whole post complaining about, sitting in my own column. Trust spent. The boolean is a convenience and I got it wrong.

One caveat on that table, because it cuts against my own framing: it assumes an un-priced call costs what a healthy one costs. If your fallback genuinely is a free local model, the dollar figure really is zero and the fallback:0c row overstates the dollars. That's precisely @ddhh's case — and precisely why what escapes there is wall-clock, GPU and downstream rate limits rather than dollars.

What refusing actually costs you

I've been describing the failure mode of not counting. Refusing has its own bill, and my last draft skipped it, which was the honest complaint against it.

First, the recovery case. Outages end. Oracle down from step 6, back at step 15:

SCENARIO F — the outage ENDS: oracle down 6..14, back at 15
  loop retries after a BLOCK (a real runaway does):
  refuse           admitted=10  spent=$0.50  unpriced=0   unaccounted=0   ledger-moved=False exit(conv)=1 exit(strict)=1
  same run, but the loop HALTS on the first BLOCK (single-shot shape):
  refuse           admitted=6   spent=$0.30  unpriced=0   unaccounted=0   ledger-moved=n/a   exit(conv)=1 exit(strict)=1

(The scenario prints admit-unpriced and stale rows too; I've kept only refuse here, because it's the policy on trial.)

Under retry, refusing costs nothing: the run resumes and completes the same ten actions it would have anyway. Under halt, it ends the run at step 6 and never sees the oracle come back. Same policy, same outage, and whether refusing is free or expensive depends entirely on a property of your caller that isn't in the cap at all.

Three more costs, none of which my harness measures:

You hand your stop button to a third party. If your gate refuses whenever CoinGecko is unreachable, then CoinGecko's rate limiter is now your kill switch, and anyone who can degrade it can halt your agent remotely.

Refusing is not automatically safe. The fail-closed doctrine arrives from authorization, where denial is the safe default. Spending isn't authorization. Stopping halfway through a non-idempotent sequence — and SpendGuard was written for ETH and gas — can be worse than admitting one un-priced call. If your actions aren't safely interruptible, "refuse" is not the free option it looks like.

The escape hatch never gets closed. Any --allow-unpriced flag will be added to a systemd unit at 3am during an incident and will still be there next year. If you build one, give it an expiry.

Which is why the recommendation of this post isn't "fail closed." It's narrower: declare the branch, and keep the ledger moving with a price you'd defend out loud. Charge a stale price. Charge an estimate biased high rather than convenient. Charge the per-action cap if you accept what my own harness showed above — that on a tight budget this is refusal under a different name. Just never charge zero and call it accounting.

Two exit codes, and which one is an opinion

The tool prints exits under two mappings, because the difference is the trap:

EXIT_CONVENTIONAL = {"PASS": 0, "ADMIT": 0, "BLOCK": 1, "ERROR": 2}
EXIT_STRICT = {"PASS": 0, "BLOCK": 1, "ADMIT": 2, "ERROR": 3}

conventional is what most gates ship: an admitted action is a success. Under it, the blind run exits 0 while 34 actions went through un-priced. Your orchestrator sees green and moves on.

strict treats "admitted without a quote" as its own outcome. Under it no strategy exits 0 during an outage — refusing gets a 1, admitting blind gets a 2. Nobody gets a clean run when the oracle is down, which seems right to me.

That second mapping is my opinion, and I'm labeling it as one. My last attempt at this put ADMIT: 0 in the table, then acted amazed that fail-open exited green — I'd assumed the conclusion and called it a finding. The counts are the evidence. The exit code is a choice, and you should make your own.

The tool also exits worst-wins in every mode including the demo, so the default run exits 3. It contains a deliberate oracle fault. A file about failures laundered into green zeros doesn't get to launder its own.

The oracle-fault path catches magnitude, not just sign, since a cents-versus-dollars mixup is the classic cost-oracle bug:

      seq 3: oracle-untrusted: quote 500 is 100x last known 5 (unit error, not a price)

A 100x jump is caught as a broken oracle instead of an expensive action. The threshold is 20x and it's a judgement call, and it can't fire on the first quote of a run because there's nothing to compare against yet.

It's also one-sided, which I only noticed while writing this up. It catches a quote that's too big and sails straight past the mirror-image bug: dollars arriving as cents, every action priced at a hundredth of what it costs. For a spend cap that's the more dangerous direction — it under-counts instead of blocking, which is the same failure as everything else in this post — and my check doesn't cover it. It's marked as a known gap in the source rather than quietly left there.

What this is not

A benchmark of your bill. The constants ($0.05 a call, a $0.50 budget, 40 rounds) exist to make the branch legible. The transferable part is the shape.
Proof that outages and runaways co-occur. I believe they do, because a runaway hammering a rate-limited endpoint is often the thing knocking its own price feed over. A synthetic harness can't show that, and I'm not claiming it does.
A verdict on anyone's library. The equivalence is about the pattern of pricing a fallback at zero. It reproduces with --strategy fallback --fallback 0 --oracle-fails-from 6 in twenty lines of my own code. (Leave the outage flag off and you get a healthy-oracle run where the branch never fires — which proves nothing, as Scenario A says out loud.)
A correctness proof. The sha256s show determinism, nothing more. A wrong program reproduces byte-for-byte just as well as a right one.
A model of how commercial budget systems fail. AWS Budgets, GCP billing and most usage endpoints don't go quiet — they lag, then reconcile, and what you couldn't see gets billed to you later. My gate has no true-up: an estimate charged during the outage is never corrected when the oracle comes back. And for LLM calls there's no honest pre-call price at all, since output tokens aren't known until the call is finished — so a real LLM cap lives permanently in pessimistic/fallback and never earns the PASS row my harness prints. What this models cleanly is a price feed, which is the case I actually shipped.

This sits on a different axis from the sliding-window guard, which is about cheap calls that sum to a runaway, and from the pre-execution gate, which is about gating before you execute rather than after. Today's axis is oracle availability, and neither of those touched it.

The one I'm still stuck on

Stale pricing keeps the ledger alive, which is the whole recommendation, and it's also quietly a lie: you're charging against a number you know might be wrong. So how long is a cost estimate allowed to live before "cached" becomes "guessing"? For gas that moves in seconds it might be five seconds. For a token price it might be an hour. I don't have a principled way to set that TTL, and I suspect it's per-oracle rather than a general rule.

If you've drawn that line in production — the point where a cached cost stops being a fact — I'd like to hear where you put it and what made you move it.

Run it yourself: stdlib, offline, and it prints hashes you can diff against mine. Then go look at your own cap and answer one question. When it can't price the next call, does the number in your ledger keep moving?

Follow for the next teardown in this series, and tell me the worst thing your agent ever did while your dashboard showed $0.00. I read every comment.

Written with AI assistance and reviewed/edited by a human. The Python in this post was run offline (stdlib only, no network, no keys, no funds) on 2026-07-19; every number and hash in the output blocks is from a real deterministic run, repeated three times byte-for-byte. Code sha256 ddc425908d070c07a6765810e6115f649c17312b8e754d034227d37c982357e8, output sha256 9ebe1b4ab3459eb78c1d3aeea7eaa16ee0c2286e1d8d57b09e0eeb6451a189f1. The $1,847 figure belongs to @royanannya and was not reproduced here; the circuit-breaker design described is @ddhh's, quoted from their post and their MIT-licensed repository.

One compaction, four actions, one block: compaction safety is a property of the pair

Alexey Spinov — Sat, 18 Jul 2026 01:06:26 +0000

Context compaction is safe or unsafe only against a specific proposed action. Not in general. The compactor decides what your agent forgets before it knows what your agent will do, so at compaction time "did this lose anything important?" has no answer yet. The lie, if there is one, comes into existence later, when the agent proposes to act.

That sounds like philosophy. It has an exit code.

AI disclosure. I wrote compaction_omission_gate.py with an AI assistant and ran it myself: Python 3.13.5, offline, standard library only, no network, no keys, no funds. Every verdict, exit code and hash below is pasted from a real local run. I ran the whole demo twice and the two output.txt files are byte-for-byte identical (sha256 2f0ae2c6d54b1d35caefb3da12af3a4189aff05e5d8c59321b63150f27eef376). The fixtures are synthetic and mine. The numbers I quote from Anannya Roy Chowdhury, Prasad T and the Ratel team are their claims from their own posts, attributed inline; I did not reproduce their systems.

In short:

Freeze the compaction. Vary only the action. Same 52 records, same 16 kept (kept-sha256=3bfe38c85fcf on every row), same 36 dropped, and the same rule missing from all four: 1 BLOCK, 3 PASS. Safety moved without the compaction moving at all.
So "a good compactor" is not a thing you can have. There is only "this compaction is safe for that action."
The check is arithmetic, not vibes: amount 5000 passes, amount 5001 blocks. Same words, one unit apart.
This is a partial no to a question I asked publicly two days ago. The action-blind version of the check blocks 4 of 4, where the gate blocks 1 of 4. Three of those four blocks are wrong, and nothing recovers them.
Two of my own designs died on this post: one to a number in someone else's post, one to my reviewer. Details below.

The gap three people found in the same week

Anannya Roy Chowdhury published My Multi-Agent AI Cost $1,847 in One Weekend on July 16. Her numbers: $1,847 burned, 82% cut, context compressed "from 12,000 tokens to 340." That is 97% of the context gone. I asked her in the comments whether she'd found a way to catch a bad compression before the turn runs on it, or whether you find out from the win rate.

Her reply named the hole better than my question did:

"A bad compressor can absolutely 'lie' by omission, which is arguably more dangerous than naive replay because the failure is silent."

She called it the core challenge of her Part 2, which was not out when I wrote this. So this post is not a rebuttal of anything. It is an attempt to answer the question she agreed was open.

Two other people hit the same wall in the same week. Prasad T's Teaching a Qwen agent to forget put a superseded_by link on memory records so a contradicted fact is dropped from recall but kept in an audit trail. The line I keep coming back to, from the thread under his post: forgetting you can't inspect is just data loss. And the Ratel Show HN (19 points, 18 comments when I checked) does progressive tool disclosure with BM25/embeddings/hybrid and claims up to 81% fewer tokens. Different clothes, same problem: something decides what the model doesn't see, and nothing checks that decision against what the model is about to do.

The number that killed my first design

My first design was the obvious one: run the decision on the full context, run it on the compacted context, and if the decision changes, BLOCK. Compare against the truth.

Prasad's post killed it. His FAMA measurement (his numbers, not mine) reports recency-only = 0.64 and never-forgets = 0.64. Identical. Never-forgets is the full context. So the "truth" I wanted to compare against is itself wrong 36% of the time. An oracle that lies in a third of cases is not an oracle, and any reader who has read his post gets to close my tab in one line.

Those same two numbers kill a second thing, which is the framing I would have reached for by reflex: compaction hurts because you lose volume. If throwing almost everything away and throwing nothing away both land on 0.64, volume is not the variable. Prasad says it plainly: "the win isn't about trimming context; it's about knowing what's stale."

So the gate has no oracle. It never asks what the model would have decided. It resolves your declared predicates against your stored records. That is a structural fact, not a prediction.

What does the compaction omission gate actually ask?

One closed question: among the records the compactor dropped, is there one that binds this action?

Input is the full pre-compaction context, the ids the compactor kept, and a concrete action. The dropped set is computed, never supplied. I don't trust anyone's "here's what I removed" list. Records are typed: policy (scope, predicate, effect), fact (field, value, optional supersedes), chatter. Ordering is a declared seq, never a wall clock, so the verdict can't drift with the time of day.

The whole resolution step is this small:

def resolve(field, params, facts):
    """action.params wins; otherwise the newest stored fact for that field."""
    if field in params:
        return True, params[field]
    if field in facts:
        return True, facts[field]["value"]
    return False, None

If a field resolves nowhere, the gate fails closed rather than assuming harmless. There is no scoring anywhere in the verdict path. PASS, BLOCK, ERROR. No percentages. That's deliberate: on July 16 a Show HN called ReasonGate shipped a "gate" built on regexes, and Simon Willison took it apart in the thread within the hour. Recall of "76-96%... never 100%" is not a gate, and "this list of regular expressions does not inspire confidence." A gate that guesses is a filter with good PR.

Is context compaction safe? Freeze it and vary the action

Here is the claim as an experiment, and it needs nothing to compare against. One compaction. Four actions.

The fixture is a 52-record support thread. The refund request opens it, the resolution closes it, and the middle carries the boring stuff: internal notes, a tier update, and one dry line at seq 11: Refunds over 5,000 need a human approver. The compaction is head+tail, keep the first 8 and the last 8, which is the shape most windowing schemes converge on. That rule sits at position 11 of 52. It is dropped. In all four rows.

Proposed action	Verdict	Exit
`refunds.create {amount: 8400}`	BLOCK `dropped-binding-constraint`	1
`refunds.create {amount: 120}`	PASS	0
`refunds.create {amount: 5000}`	PASS	0
`notes.append {ticket: 7741}`	PASS	0

"Same compaction" is a claim, and claims about counts are cheap, so the tool prints a fingerprint of the kept set:

    compaction: 52 records, 16 kept, 36 dropped (retention=given, kept-sha256=3bfe38c85fcf)

That digest is identical on all four rows. Same 52 records, same 16 kept, record for record, same 36 dropped. One blocks. Nothing about the compaction moved, and the answer to "is this compaction safe?" moved anyway. That is the whole thesis, and there is no baseline in it, because the claim isn't that some other tool does worse. The claim is that the question is malformed until you name the action.

The 120 row prints a line I like:

note      : permissive: dropped policy 'f_020' (allow) resolves TRUE here (amount = 120 lt 500)
            but losing a permission cannot license an action; not a block reason

A dropped allow rule is not danger. It costs you an auto-approval, nothing more. The gate says so instead of blocking on principle.

"A good compactor" is not a thing you can have. There is only "this compaction is safe for that action."

The boundary, because "it resolves the predicate" is also a claim

The 5000 row is the one worth poking at. 5000 gt 5000 is FALSE, so the rule wouldn't have fired, so its absence changes nothing. But how would you know the gate computed that rather than pattern-matched its way there? Walk the boundary:

`amount`	Verdict
4999	PASS
5000	PASS
5001	BLOCK `dropped-binding-constraint`
8400	BLOCK `dropped-binding-constraint`

5000 and 5001 are the same word to any tokenizer worth the name. The verdicts differ because the gate resolved amount out of the action's params and did arithmetic against a stored predicate. Lexical similarity has no way to know that 8400 > 5000, because that is not a fact about words.

The part where I answer my own question with "no"

Two days ago I asked publicly whether you can catch a bad compression before the turn runs. The honest answer, which I did not want, is mostly no, and I can measure the cost of pretending otherwise.

--precheck takes the action away from the verdict and asks the general question instead: could any dropped record bind any reachable call of these tools?

$ python3 compaction_omission_gate.py --precheck --retention given fixtures/fx1_refund_thread.json
    action    : (the verdict below never consults it -- that is the experiment)
    compaction: 52 records, 16 kept, 36 dropped (retention=given, kept-sha256=3bfe38c85fcf)
    tools     : refunds.create
    verdict   : BLOCK (action-blind) exit 1
    reason    : may-bind-some-action: dropped policy 'f_011' scopes to refunds.create; some
                reachable call of that tool satisfies (amount gt 5000). Which one? Unknown
                here: no action was supplied

Run it against each of the four actions and you get the same line four times, because it never reads them. The gate blocks 1 of 4. Precheck blocks 4 of 4. Three of those four blocks are false, and they are not fixable by being cleverer: nothing recovers 120 > 5000 = FALSE without the 120.

So the control point sits one step later than I hoped, and one step earlier than it costs money. The turn is already spent: the model has read the mutilated context and produced a proposal. Nothing has executed. That's the line my pre-execution gate has always drawn: before the action, not before the thinking. Reading the dropped records to check them costs zero tokens, since compaction governs what gets sent to the model, not what's on your disk. (Money is a different post: the re-billing curve is here.)

The hazard is not symmetric, and my gate got that wrong

Here's the bug my reviewer found in my own tool, and it is the mirror of the one I was proud of.

I had a rule: losing a permission cannot license an action. True, and the gate leans on it. A dropped allow never blocks. But I filtered the kept-rule check by the same logic, and that filter was wrong. A kept allow rule is a permission you still have. If the compactor drops the value that permission reads, the permission fires on a value you already took back:

$ python3 compaction_omission_gate.py --retention given fixtures/fx6_stale_permit.json
    verdict   : BLOCK (dropped-superseder-stale-permit) exit 1
    reason    : dropped-superseder-stale-permit: kept policy 'f_003' (allow) reads
                'customer.tier'. The compactor kept 'f_002' (customer.tier = gold, seq 2) and
                dropped 'f_011' (customer.tier = standard, seq 11) and declares supersedes
                'f_002'. The rule's own predicate (customer.tier eq gold) is TRUE on the kept
                value and FALSE on the newest one: the agent will license this call with a
                value you already retracted. Losing a permission cannot license an action, but
                KEEPING one while its input is retracted can

Before the fix, that fixture returned PASS exit 0. Silent. A 4,000 refund auto-approved on a gold tier that was revoked twenty records ago, and my gate, whose entire pitch is "the failure is silent, so catch it," said nothing. I had written the general principle and then applied it to the wrong noun.

Fixing it forced a distinction I had been sloppy about. A stale value has two directions and they are not the same event:

Kept rule permits on the stale value, wouldn't on the newest: the agent acts where your records say stop. Hazard. BLOCK.
Kept rule denies on the stale value, wouldn't on the newest: the agent stops where your records say go. Cost. PASS, with a note.

The second one is a real fixture too (fx7), and the gate declines to block it:

    note      : stale value, safe direction: kept policy 'f_003' (deny) reads 'customer.tier'.
                [...] The rule's own predicate (customer.tier eq restricted) is TRUE on the kept
                value and FALSE on the newest one. This compaction makes the agent more cautious
                than your records warrant: a cost, not a block reason

A gate that blocks whenever anything changed is just an alarm wired to the whole house. The direction is the point.

Is there a "correct" retention scheme? I measured. No.

Same fixtures, same k, two action-independent schemes:

fixture	head_tail	recency
refund thread (rule in the middle)	BLOCK binding	BLOCK binding
superseded fact	BLOCK stale-kept	BLOCK binding
genuinely safe	PASS	PASS
retracted tier, kept `allow` rule	BLOCK stale-permit	PASS
retracted tier, kept `deny` rule	PASS	PASS

I predicted recency would survive the supersession fixture. It doesn't. I was wrong, and the way it's wrong is more interesting than my guess: head+tail pins the stale customer.tier = standard at seq 2 and drops the correction at seq 20, so the agent applies a kept rule to a retracted value. Recency keeps the correction and drops the rule itself. Two popular schemes, opposite failures, same input. "Switch to recency" doesn't fix the failure, it relocates it.

I won't tell you no correct scheme exists; five synthetic fixtures can't carry that. The structural argument is stronger than the table anyway. Safety depends on the action, which the freeze-and-vary run above shows without appealing to any scheme at all. Neither of these schemes can see the action. So neither can be safe in general, and the row where recency wins is luck about where the record sat, not virtue.

"But a relevance compactor would keep the rule"

Probably, sometimes. And this is where I have to report a section that isn't here.

An earlier draft of this post had a better headline and a fifth act: my gate against a relevance-based compactor, IDF-weighted cosine, keep-top-k, the state of the art, confidently certifying a compaction that dropped a binding rule. The reviewer who gates my drafts took the code apart instead of reading the prose, and found that I had serialized the action as structure (refunds.create amount 8400 currency USD customer c_889) while indexing every record as prose (Refunds over 5,000 need a human approver.). Same JSON on both sides. My gate read six fields off that record. My baseline read one.

Whatever that comparison showed, it was not a fact about relevance ranking. It was a fact about what I handed each side. So the ranker is gone, all ninety-odd lines of tokenizer and cosine that existed to lose a fight I had arranged. The thesis never needed it, which I'd have noticed sooner if the number hadn't been flattering.

The honest version of the question is harder. A ranker that sees the action doesn't produce one compaction you can freeze and audit; it produces a different compaction per action. That may well be the right engineering. It also means there is no fixed artifact to check, and "is this compaction safe" stops being a question you can ask once and cache. I don't have a clean answer for that shape. If you do, I'd like to hear it.

Prose cannot be certified

One fixture is rolling summarization output: the middle of the thread replaced by eight prose blobs typed as summary. The gate doesn't recognize that kind, can prove nothing about it, and blocks with dropped-unevaluable-record. Be precise about what fired, though: the unrecognized kind, not the prose itself. Retype those same eight blobs as chatter — a known kind the gate reads as your own declaration that a record is throwaway — and it believes you, drops them silently, and returns PASS. I checked; the verdict flips. So the honest version is narrower than prose fails closed: the gate fails closed on a kind it can't read, and trusts the one you've labelled inconsequential. Same knife as the schema limit further down — it holds you to your typing. There's an --allow-unevaluable flag that turns the unrecognized-kind block into PASS exit 0 with [--allow-unevaluable: you chose to fly blind] in the report. Not a fix. A signed decision.

This is the uncomfortable one, because rolling summarization is exactly what my own context-tax post prescribes as the cure. I'm auditing my own prescription. If your memory is a pile of summarized text, this gate has nothing to work with, and neither does anything else. That's not a gap in the gate. It's a gap in the store. Prasad's superseded_by link is the cheapest version of the fix.

What this is NOT

Not an authorizer. It says the compaction hid nothing. Whether the action is allowed is your policy engine's job.
Not a benchmark. Seven synthetic fixtures I wrote, shaped to have the property I'm describing. No claim about anyone's production system, and specifically no claim about Anannya's. Her numbers are hers, from her post.
Not a compactor. It doesn't fix your retention. It tells you when this retention can't carry this action.
Not proof your agent is safe. It closes one hole: a dropped record that binds the proposed call.
Not usable on unstructured memory, as above.

How is this different from my other gates?

mandate-freshness-gate expires authority: the permission was revoked, and nothing is missing from the context. Here a fact is retracted and the record is gone. checkpoint-skip-gate replays a recorded trajectory to find a step the agent skipped; the culprit is the agent, and the check is post-hoc. Here nothing is skipped: the infrastructure quietly ate a fact, and the check runs before execution. agent-memory-tax-and-backdoor argued retention is not relevance, about what your store keeps. This one is about what your window throws away in the same breath, and the verdict never scores anything. sliding-window-spend-guard treats the window as a spend problem, where eviction is bookkeeping. Here eviction is a correctness problem and the verdict has no cost term in it at all.

Run it

Drop the tool and the fixtures in a directory and run bash run_demo.sh. Three stdlib imports (json, hashlib, sys), no network, no keys, no model. The full sweep ends 2 PASS, 4 BLOCK, 1 ERROR -> overall exit 1, malformed input exits 2 and never 0, and every report ends with a sha256 of its own body. Run it twice and diff. If it isn't byte-identical, I'd want to know.

Every command in the demo passes --retention given explicitly even though it's the default, because an earlier version of that script quietly recomputed the compaction per action while the prose above it claimed the compaction was frozen. Two of the numbers I'd written were false and I hadn't run the command my own paragraph described. Spelling the flag out makes the sentence and the command the same object.

The schema is the price of admission. policy needs a scope, a predicate and an effect; fact needs a field, a value and ideally a supersedes. If your ingestion layer labels a binding rule as chatter, the gate believes it, and nothing saves you. It holds you to your own typing. That's the honest limit.

So: if compaction safety is a property of the pair rather than of the compactor, where does this check belong in your harness: inside the compactor, in the store, or at the execution gate? I lean toward the execution gate, because it's the only place that knows the action, but that's the place least likely to have the dropped records in hand. And what do you do when your context is prose with no schema? Pay to type it, or fly blind and admit it?

Follow for the numbers from the next gate I build and break. If you've watched an agent lose an instruction to auto-compact and only find out from the outcome, tell me what the dropped record was. I read every comment.

Codex encrypted its sub-agent prompts. Gate the spawn plan.

Alexey Spinov — Thu, 16 Jul 2026 00:48:38 +0000

Pre-dispatch authorization for AI sub-agents means checking a child spawn's grant envelope (its role, tools, path scope and token budget) against the parent's policy in the last plaintext moment before handoff, not by reading a trace afterwards. When the orchestrator encrypts the handoff, the after-view goes blind. The before-view does not, because it sits earlier on the timeline.

On July 14, "Codex starts encrypting sub-agent prompts" hit 408 points and 240 comments on Hacker News in a day. The tracking bug behind it is filed as openai/codex#28058, titled "Regression: encrypted MultiAgentV2 messages remove readable task audit trail." A change encrypted the orchestrator-to-sub-agent payload, and the plaintext task record humans used to read after the fact turned into ciphertext. People who had been inspecting what their sub-agents were told, after dispatch, could no longer read it.

That is a good thing to notice, and a worse thing to fix by asking for the plaintext back.

AI disclosure. I wrote subagent_dispatch_gate.py with an AI assistant and ran it myself: Python 3.13.5, offline, standard library only, no network, no keys, no funds. Every number, exit code and sha256 below is pasted from a real local run. I ran the whole demo twice and the two output.txt files are byte-for-byte identical (sha256 5af48191642d66f7c364c429c50d2ad1a021f09004f5566ba878c7be87fcaaf1). The one synthetic part is clearly marked: encrypt_artifact() models the observable consequence of encryption (opaque bytes you cannot parse back into fields), not Codex's real crypto. And every fact about Codex here comes from that HN thread and that issue. I did not reproduce their system.

In short:

The issue people filed asks to restore the readable audit copy. That restores tracking. It does not restore control, and I think the two got conflated in the thread.
Post-hoc inspection genuinely worked the day before the change. In my contrast run it recovered a plan with 4 fields when the artifact was plaintext, then 0 fields once the artifact was the opaque blob. The only variable is encryption. No strawman.
A pre-dispatch gate reads the plan the parent still holds in-process, before the handoff boundary. Same spawn, both worlds, children=1, invariant to whether the artifact is later encrypted.
The gate is not a tautology. One plan granting fs.write, a fs:* wildcard, shell.run, a path of ~ and a 5,000,000-token budget produces PASS: 1 BLOCK: 1, exit 1. The same two roles scoped read-only inside the workspace under cap produce PASS: 2 BLOCK: 0, exit 0.
An absent or empty plan is exit 2, never a silent pass. A gate that treats "no plan found" as "nothing to block" fails open, and that is the failure mode I most wanted to kill.

What is pre-dispatch authorization for AI sub-agents?

It is a check that runs on the parent, before the parent hands a task to a child agent. The parent holds a spawn plan: for each child, a role, a set of tool grants, a set of paths it may touch, and a token budget. Pre-dispatch authorization compares each of those against a policy the parent already owns, and refuses the spawn if any child asks for more than the policy allows. It happens while the plan is still a plain object in memory. Nothing has been sent, nothing has been encrypted, nothing has run.

The distinction that matters: this is authorization of a spawn, not authorization of an action, and not reconciliation of a trace. It is the same family as a pre-execution gate for AI agents, moved up one level, from "may this action run" to "may this child exist with these powers."

The incident: the audit trail went dark, and the thread asked for it back

Read issue #28058 in its own words. A PR added encryption to the MultiAgentV2 message payloads. The intent was reasonable: the model-facing channel between orchestrator and sub-agent should not sit around in plaintext on disk. The side effect was that the local rollout history, the thing a human opened to review what task got delegated to which sub-agent, became unreadable. The issue's ask is to keep a human-readable audit copy alongside the encrypted delivery.

I understand the ask. I also think it quietly gives up the game. If your control over what a sub-agent may do lives in your ability to read the transcript afterwards, then your control was always downstream of the work. The sub-agent already ran. The tokens already burned. The shell.run already happened. Restoring the readable copy lets you write a better postmortem. It does not stop the next spawn.

This is the line I keep coming back to across this whole cluster: tracking is not control. A trace tells you what happened. It does not decide what is allowed. When the trace is plaintext you can pretend the two are the same, because you can always go read it and intervene. Encryption breaks the pretense. It takes away the after-view and forces the question: if you could not read the trace at all, where would your control actually live?

Why is post-hoc dead while pre-dispatch survives?

Here is the honest version of that question, run as code. Same spawn, two worlds, one variable.

def encrypt_artifact(plan_obj):
    """Model the at-rest / handoff artifact AFTER the orchestrator encrypts it.
    Not real crypto: models the observable consequence -- bytes a reader without
    the key cannot parse back into fields. sha256 stands in for 'ciphertext'."""
    raw = json.dumps(plan_obj, sort_keys=True).encode()
    return "ENC:" + hashlib.sha256(raw).hexdigest()

def posthoc_inspect(artifact):
    """A session inspector reading the artifact AFTER dispatch.
    Plaintext JSON -> recovers the plan (worked yesterday).
    Opaque ENC blob -> recovers nothing (blind today)."""
    if isinstance(artifact, str) and artifact.startswith("ENC:"):
        return None, 0            # blind: zero fields recoverable
    obj = json.loads(artifact) if isinstance(artifact, str) else artifact
    if isinstance(obj, dict) and isinstance(obj.get("plan"), list):
        fields = sum(len(c) for c in obj["plan"] if isinstance(c, dict))
        return obj["plan"], fields
    return None, 0

The point of posthoc_inspect is that it is not rigged. Hand it plaintext and it does its job. Run python3 subagent_dispatch_gate.py --contrast:

CONTRAST: post-hoc inspection vs pre-dispatch gate (same spawn)
  world=PLAINTEXT  post-hoc: recovered plan=yes fields=4
  world=ENCRYPTED  post-hoc: recovered plan=no fields=0  <- BLIND
  pre-dispatch reads parent's in-process plan: children=1 (invariant to artifact encryption)
  => post-hoc control SURVIVES encryption: no
  => pre-dispatch control survives encryption: yes
sha256(contrast)=0abca3747138eaf4260ed298ed001c1df6fd7ac67e89ce56f60c7b5fe8b9ef11

Yesterday's inspector recovered 4 fields. Today's recovers 0. It did not get worse at its job. The world moved out from under it. The pre-dispatch view reads the plan the parent constructs in memory, before encrypt_artifact ever touches it, so it reports the same children=1 in both worlds. That is the whole argument in five lines of output: encryption is fatal to the after-view and irrelevant to the before-view, and I am not shaving points off post-hoc to get there. It honestly recovers the plan when the plan is readable.

The reason the before-view can exist at all is boring and load-bearing: the parent has to build the plan in plaintext to send it. Even Codex's own encryption sits around delivery and at-rest storage. The payload is assembled in the clear first. There is a moment, every single spawn, when the full plan is a plain object the parent is holding. That moment is where the gate goes.

The gate: authorize the spawn plan, not the trace

Four checks, one policy, one child at a time. Nothing clever.

WILDCARD_PATH_MARKERS = ("~", "$HOME", "*")
WRITE_CAPS = {"fs.write", "fs.delete", "shell.run", "wallet.transfer"}

def _escapes_workspace(path, root):
    """True if `path` is not provably inside `root` (fail-closed on ambiguity)."""
    if not isinstance(path, str):        # a non-string path spec is not provably safe
        return True
    p = path.strip()
    root_norm = root.rstrip("/")
    if p == "":
        return True
    for m in WILDCARD_PATH_MARKERS:      # home / wildcard touches everything
        if m in p:
            return True
    if ".." in p.split("/"):             # parent-traversal (absolute OR relative)
        return True
    if p == "/":                         # filesystem root
        return True
    if p.startswith("/"):                # absolute: must be inside root
        return not (p == root_norm or p.startswith(root_norm + "/"))
    return False

def check_child(child, policy):
    reasons = []
    allowed_tools = set(policy["allowed_tools"])
    cap  = policy["budget_cap_tokens"]
    root = policy["workspace_root"]
    allow_write = policy.get("allow_write", False) is True   # only a real True grants write

    tools = child.get("tools")
    if not isinstance(tools, list) or not tools:
        reasons.append("child declares no tools (fail-closed: unauthorizable)")
    else:
        for t in tools:
            if t not in allowed_tools:
                reasons.append("tool '%s' not in parent allowlist (deny-by-default)" % t)
            if t in WRITE_CAPS and not allow_write:
                reasons.append("tool '%s' is a write/destructive capability but "
                               "policy allow_write=false" % t)

    paths = child.get("paths")
    if not isinstance(paths, list):
        reasons.append("child.paths must be a list")
        paths = []
    for p in paths:
        if _escapes_workspace(p, root):
            reasons.append("path '%s' escapes workspace root '%s'" % (str(p).strip(), root))

    budget = child.get("budget_tokens")
    if not isinstance(budget, int) or isinstance(budget, bool):
        reasons.append("budget_tokens missing or non-integer (unbounded spawn)")
    elif budget > cap:
        reasons.append("budget_tokens %d over parent cap %d" % (budget, cap))
    elif budget <= 0:
        reasons.append("budget_tokens %d must be positive" % budget)

    return child.get("role", "<no-role>"), ("PASS" if not reasons else "BLOCK"), reasons

Two design choices are doing the real work. First, fs.write is in the allowlist and still gets blocked, because it is a write capability and the policy says allow_write=false. Allowlisting a tool and granting write are two different decisions, and collapsing them is how "read-only agent" quietly becomes "agent that can write." Second, _escapes_workspace is deny-by-default on ambiguity. I do not try to be clever about what ~ might resolve to. If a path is not provably inside the root, it escapes. I got this wrong in the first draft: my wildcard list literally contained "/", so the substring check flagged every absolute path, and the legitimate PASS fixture came back as exit 1. The live run is what caught it. That is the entire reason I run these before I write about them.

The scope side of this is a sibling problem to scoring a single API key's blast radius: same "how wide is this grant" axis, different object and different output. This gate is a binary refusal on a whole spawn envelope, not a 0-100 score on one credential.

Run it in sixty seconds

Here is the child that should never be allowed to spawn. Save it as plan_block.json:

{
  "policy": {
    "allowed_tools": ["fs.read", "fs.write", "http.get"],
    "workspace_root": "/repo/workspace",
    "budget_cap_tokens": 200000,
    "allow_write": false
  },
  "plan": [
    { "role": "home-cleaner", "tools": ["fs.write", "fs:*", "shell.run"],
      "paths": ["~", "/repo/workspace/task-1"], "budget_tokens": 5000000 },
    { "role": "doc-reader",   "tools": ["fs.read", "http.get"],
      "paths": ["/repo/workspace/docs"], "budget_tokens": 40000 }
  ]
}

home-cleaner is the kind of spawn you do not want dispatched sight-unseen: broad file powers, a shell, a home-directory reach, an absurd budget. Run it:

$ python3 subagent_dispatch_gate.py plan_block.json
SUBAGENT-DISPATCH-GATE REPORT
policy: 3 tool(s) allowed, root=/repo/workspace, budget_cap=200000, allow_write=False
children in spawn plan: 2
  - home-cleaner -> BLOCK
      x tool 'fs.write' is a write/destructive capability but policy allow_write=false
      x tool 'fs:*' not in parent allowlist (deny-by-default)
      x tool 'shell.run' not in parent allowlist (deny-by-default)
      x tool 'shell.run' is a write/destructive capability but policy allow_write=false
      x path '~' escapes workspace root '/repo/workspace'
      x budget_tokens 5000000 over parent cap 200000
  - doc-reader -> PASS
PASS: 1   BLOCK: 1
VERDICT: 1 subagent spawn(s) refused BEFORE dispatch
sha256(report)=afed5edb835afdafeac5496dd299770581701fb4489232b7e2baf469fa311812
exit=1

Note doc-reader passes in the same plan. The gate is not a wall that blocks everything. It refused one child on six specific grounds and let the scoped one through. Now flip only the scope, keep the two roles. In plan_pass.json, home-cleaner is fs.read on /repo/workspace/task-1 with a 40,000-token budget:

$ python3 subagent_dispatch_gate.py plan_pass.json
SUBAGENT-DISPATCH-GATE REPORT
policy: 3 tool(s) allowed, root=/repo/workspace, budget_cap=200000, allow_write=False
children in spawn plan: 2
  - home-cleaner -> PASS
  - doc-reader -> PASS
PASS: 2   BLOCK: 0
VERDICT: all spawns within policy; dispatch authorized
sha256(report)=de81c4d8f75cd4683796b9715c9501f81b77bd618286f0ebbf172aad8f21a5d9
exit=0

Exit 1 to exit 0, decided on the real difference between the two plans. If a gate cannot produce both outcomes it is a decoration, not a control. This one produces afed5edb… for the refusal and de81c4d8… for the authorization, and both hashes are stable across runs.

Why does bad input fail closed?

The failure I care about most is the quiet one. A gate that returns "all clear" when it was handed nothing is worse than no gate, because it launders "I did not check" into "I approved." So an absent or empty plan is exit 2, and it says why:

$ python3 subagent_dispatch_gate.py plan_empty.json      # "plan": []
ERROR: spawn plan is empty (fail-closed: nothing to authorize is not the same as everything authorized)
exit=2

$ python3 subagent_dispatch_gate.py plan_missing.json    # no "plan" key at all
ERROR: no spawn plan present (fail-closed: absence of a plan is NOT authorization -- a gate that passes an empty plan fails open)
exit=2

The same reflex runs inside each child. A spawn that declares no tools is blocked, not waved through as harmless. A budget_tokens that is missing or non-integer is blocked, because an unbounded spawn is the one that quietly forks children in a loop and bills you for all of them. Empty is not safe. Empty is unknown, and unknown fails closed. A previous tool in this series died in review for getting this backwards, treating a zero-byte input as a pass, and I would rather over-correct.

Where this sits next to the rest

This is one spoke of a cluster, and the edges matter more than usual here.

The near neighbor is an authz gate that reconciles the trace against what was allowed. That gate assumes the trace exists. It reads a span log after the fact and checks each recorded action against policy. Issue #28058 is the moment that assumption dies: encrypt the payload and there is no trace to reconcile. So the control does not disappear, it relocates earlier on the timeline, from the span log that is now ciphertext to the spawn plan that is still plaintext. If you have been running post-hoc reconciliation, that article is where you were, and this one is where you go when the log goes dark.

It is also a pre-run manifest check like the lethal-trifecta gate, which asks a different question of the same manifest: can untrusted input reach private data and then reach an exfiltration channel? Same "decide from the declared plan before anything runs," different property. And it is a new spoke on the pre-execution gate pillar: authorize the child, not the action.

What this is NOT

I would rather you find the holes than a commenter find them for me.

I did not reproduce Codex. Every claim about their system comes from HN 48905028 and issue #28058. The encryption there covers delivery and the at-rest copy; the sub-agent's output is a separate matter, and MultiAgentV2 is an experimental, off-by-default path. I am reasoning about the shape of the problem, not their internals.
This is not a decryptor or a bypass. The gate never touches ciphertext. It stands on the parent's own boundary and reads the plaintext plan the parent is already holding, before handoff. If your orchestrator does not expose that pre-dispatch moment, this tool has nothing to read.
encrypt_artifact() is a model, not AES. It produces opaque bytes to stand in for "ciphertext you cannot parse back into fields." It demonstrates the consequence of encryption for a reader without the key. It is not a reproduction of any real cipher, and I am not claiming it is.
It is not a runtime enforcer. It statically authorizes a manifest and returns an exit code. To actually stop a spawn you need a pre-dispatch hook inside your orchestrator that consults this verdict and refuses the handoff. This shows what the hook should refuse and why. It does not install the hook.
The budget check is a declared integer, not a tokenizer. It compares budget_tokens to a cap. It trusts the plan's own declared number and does not estimate real token cost. If a child lies about its budget, the gate believes the lie. Treat it as a ceiling on the ask, not a measurement of the spend.
The policy language is a toy. An allowlist, a workspace root, a budget cap, an allow_write flag. That is not OPA, not a real PDP/PEP, not a capability system. It is the smallest thing that makes the argument concrete and runnable. Real deployments will want a real policy engine.
A blocked plan is not a safe plan. The gate checks the four things it checks. A child that stays inside all four can still do something dumb with a legitimate grant. Passing this gate means "within the envelope you declared," nothing stronger.

What I would do on Monday

Find the point in your orchestrator where the parent has assembled the child's task and grants but has not sent them yet. That point exists, because the parent has to build the thing in the clear to hand it over. Emit the plan as a plain object there, one dict per child with its tools, paths and budget. Run those four checks against the parent's policy. Refuse the handoff on any BLOCK, and log the reasons. Then, separately, keep whatever after-the-fact audit you have. The audit is still useful for the postmortem. It just cannot be the place your control lives, because one encryption change already proved it can be taken away.

Here is the question I actually want answered, and I do not have a clean answer myself. For anyone running a multi-agent orchestrator in anger: where does authorization of the spawn live in your stack right now? Is it a real check before the parent hands off, or is it a trace you read afterwards and hope you get to in time? Because if it is the trace, issue #28058 is a preview of the day it goes dark.

I write about pre-execution control for AI agents: gates that decide before the work runs, not dashboards that explain it after. Every post ships a tool you can run offline, with no keys. Follow along if that is your kind of thing, and if your orchestrator has a pre-dispatch story, put it in the comments. I am especially interested in anyone who has bolted an authorization check onto a spawn and had it survive contact with a real agent loop.

AI Agent Cost Drift: 0.35%/day Is Invisible to Your Dashboard

Alexey Spinov — Wed, 15 Jul 2026 00:47:52 +0000

AI agent cost drift is the slow growth of your input floor — system prompt, tool schemas, CLAUDE.md, MCP servers — that a rolling baseline never catches, because the baseline climbs with it. drift_anchor_gate.py pins a frozen canary on day 0 and compares: a 0.35%/day creep raised zero alarms in 60 days; the anchor blocked it on day 9.

Your fleet dashboard fires when today's average run jumps 20 percent above last week. Your context floor grows 0.35 percent a day. Those two numbers never meet. I built six 60-day worlds and ran four rolling detectors over them, each at the tightest threshold that stays quiet on a flat fleet, and the slow creep raised zero alarms in all four, across 60 days, while the floor went up 22.6 percent. A frozen anchor caught the same creep on day 9.

AI disclosure: I wrote drift_anchor_gate.py and make_worlds.py with an AI assistant and ran them myself: Python 3.13.5, offline, standard library only, no network, no keys. Every number, exit code and sha256 below is pasted from a real local run. I ran the whole demo twice from a clean rm -rf worlds, and the two output.txt files are byte-for-byte identical (sha256 1772e695cb75f79d9e3f162ed4c49477a329703610cdf5ddff54cec2cc4da62a). The series are synthetic, and I say exactly how they are generated below. The one thing that is not synthetic is the arithmetic, and it is the part that does the work.

In short:

A baseline computed from your own recent history (rolling median, rolling mean, EWMA over your fleet) is a detector of speed. The invoice is charged on level. Those are different quantities, and slow cost drift lives in the gap between them.
The blind band is arithmetic, not a finding: a window of length w firing at ratio T cannot see any uniform daily growth below g* = T^(2/w) - 1. For w=7, T=1.20 that is 5.35 percent a day. Real context creep runs at a tenth of that.
Measured on my fixtures: a 0.35 percent/day creep produced 0 alarms from all four rolling detectors in 60 days. The frozen anchor blocked it on day 9, at a 2 percent tolerance, with 0 false alarms on a flat fleet.
The demo that matters: a world where every local file is byte-for-byte identical to day 0 (diff -rq prints nothing, git diff would print nothing), the vendor quietly bumped the harness scaffold, the fleet dashboard's short-window detectors stayed silent (the long window fired once, on day 58 — 46 days late), and the anchor blocked on day 12 with the growth attributed to vendor: +2600 B.
The dashboard wins a world too, and I show it: when the work per run doubles and the floor never moves, the anchor passes all 60 days and the rolling detector fires on day 30. Keep both.
Where this article is wrong: when the floor is about 90 percent or more of a typical request, the rolling detector does see the creep. Sweep included.

The comment I owed a measurement to

On July 13, Dipankar Sarkar replied to my post on usage logs that report zero tokens for delivered text with a sharp correction. His argument: a provider tokenizer bump and real suppression look identical if you only watch a threshold, so the disambiguator should be population, not threshold. "So the block condition isn't 'delta widened past X,' it's 'this stream's delta diverged from the fleet's,' per-stream anomaly against a rolling baseline of everyone else."

He is right, and I said so. Then I argued the thing that became this article: a rolling baseline adapts, so anything that grows slower than the adaptation window never diverges from the fleet. It walks the fleet along with it. Boiling frog. What you need is a second anchor that is frozen at a known-good date, and for a solo developer with no fleet, a canary request with deterministic input is a population of one.

And then I ended my own comment with this:

"To be straight about status, this is a design argument and not something I have measured. The re-tokenized floor I have run. The fleet-versus-canary discrimination I have not."

That was two days ago. This post is me paying that off. The domain moved (I am measuring input cost drift, not usage suppression) but the structure of the question is the same one Dipankar and I were arguing about: can a baseline built from your own recent history see a thing that moves your own recent history?

Start with the arithmetic, before any data exists

This part is not a discovery. It is a derivative, and it is worth doing on paper before you touch a fixture, because it tells you what the measurement is allowed to find.

A rolling window compares today against a baseline built from the previous w days. That baseline is centred roughly w/2 days back. Under uniform daily growth g, today's level over the baseline's level is about (1+g)^(w/2). The alarm fires when that ratio exceeds the threshold T. Solve for g:

g* = T^(2/w) - 1

Any growth slower than g* is invisible forever. Not "for a while". The ratio the detector computes stops rising, because the baseline is climbing at the same rate as the signal. Here is what the tool prints before it has read a single byte of data:

  window   threshold   invisible below   compounds to, over 30 days
  w=7      T=1.15         4.07%/day        x3.31
  w=7      T=1.20         5.35%/day        x4.77
  w=14     T=1.20         2.64%/day        x2.18
  w=28     T=1.20         1.31%/day        x1.48
  w=28     T=1.10         0.68%/day        x1.23

Read the right-hand column. A seven-day window at a 20 percent threshold is blind to any drift that multiplies your input cost by 4.77x over a month. That is the prediction. The measurement is not the formula, it is the question of where real context creep lands relative to that band, and whether you can pick a T low enough to close it without the alarm screaming every Tuesday.

The fixtures, declared before I ran them

Sixty days. Forty runs per day, fixed. I fixed the run count on purpose: variable traffic adds noise to a daily average, and noise is the thing that pushes the dashboard's threshold up. Removing it is a handicap in the dashboard's favour, and I would rather hand the dashboard its best case than be accused of rigging it.

The floor at pin time is 61,060 bytes:

field	bytes	origin
`tools_json`	26,000	local file
`system_prompt`	12,000	local file
`instructions` (CLAUDE.md)	9,000	local file
`harness_scaffold`	6,000	vendor, not a local file
`mcp_a`, `mcp_b`	4,000 each	local files (the MCP token tax)
`session_id`, `timestamp`	36 + 24	runtime, declared volatile

Work per run is a lognormal draw, median 18,000 bytes, sigma 0.8, multiplied by a daily job-mix factor (lognormal, sigma 0.15). So a typical run is 86,033 bytes and the floor is 71.0 percent of it. Every world shares one noise realization and one anchor: only the floor trajectory differs. The dashboard is never handed a harder draw than the anchor gets.

The six worlds:

world	what happens
`control`	nothing changes
`creep`	one policy section appended to CLAUDE.md every day, a 3rd MCP server on day 14, a 4th on day 33, 12 tool schemas on day 40
`slowcreep`	one policy section every third day, one MCP server on day 33
`spike`	someone pastes a 40 KB architecture doc into CLAUDE.md on day 30
`harness`	no local file changes at all. The vendor bumps its scaffold on days 12, 31 and 48
`workload`	the floor never moves. Work per run doubles on day 30

Each policy section is uniquely generated, 380 to 620 bytes, with its own heading and body. I mention that because my last cost article was dropped in review for a fixture that repeated one paragraph thirty times, which was a fair hit.

Growth rates that come out of that: creep runs at +1.044 percent/day and reaches 112,700 bytes. slowcreep runs at +0.347 percent/day and reaches 74,885 bytes, up 22.6 percent. Nobody would notice either one by looking.

Six worlds, one anchor, one command

Every rolling detector gets its own best deal: the tool walks a threshold ladder from 1.05 upward and picks the lowest threshold that raises zero false alarms on the flat world. That is the most generous calibration you can honestly give an alerting rule, because the first thing that happens to a noisy rule in real life is that somebody raises the threshold until it shuts up.

Here is what came back, condensed from the real output:

world	rolling median w=7	rolling mean w=7	EWMA a=0.3	rolling median w=28	frozen anchor, tol 2%
`control`	0 alarms (T=1.20)	0 (T=1.20)	0 (T=1.20)	0 (T=1.15)	PASS, all 60 days
`creep` (1.04%/d)	1, on day 40	1, day 40	1, day 40	15, from day 30	BLOCK day 3
`slowcreep` (0.35%/d)	0	0	0	0	BLOCK day 9
`harness`	0	0	0	1, day 58	BLOCK day 12
`spike`	4, day 30	2, day 30	2, day 30	15, day 30	BLOCK day 30
`workload`	4, day 30	2, day 30	2, day 30	14, day 30	PASS, all 60 days

Look at the creep row for the short windows. One alarm each, and it lands on day 40, the day twelve tool schemas were installed. Not on the creep. On the step. The detector is doing exactly what it is built to do, which is notice a discontinuity, and the 51 KB the floor gained by walking there is invisible to it.

Now the long window, w=28 at T=1.15. The arithmetic said it is blind below 1.00 percent/day. creep runs at 1.044 percent/day, just above the line, and the detector fires 15 times starting on day 30. slowcreep runs at 0.347 percent/day, below the line, and it fires zero times in 60 days. The formula predicted both the hit and the miss before the data existed. That is the whole argument in two rows, and it is why I put the arithmetic first.

The long window is not free, by the way. It paid for that sensitivity with 15 alarms on spike and 14 on workload.

The demo that made me build this: git diff is empty and you are paying more

The harness world is the one I would put in front of a skeptic.

Nothing in the repository changes. Not one byte. The demo runs diff -rq between a day-0 snapshot and a day-59 snapshot of every local file that feeds the request:

diff -rq repo_day000 repo_day059: no differences. git diff would print nothing.

sha256 of every local file, day 0 vs day 59:
  d0  063d5a8a0fe8267fc4093041701996f728662aab22b0763dd0225de767bff895  ./CLAUDE.md
  d0  e3b98852646c31827f303b3762410a3ca42f3a51c2a2201614e21b76fae51e70  ./mcp/a.json
  d0  07acba3d89f17bc9467bcbac3f2bcc72cfff4c07ad65694d86691a66a080f80c  ./mcp/b.json
  d0  25cb751f2526dafd5e0f7273fe29a7abac797424f442cfff28ea82ca013bf8cf  ./system_prompt.txt
  d0  3dc0f40d1489590ef5a98c464c11378f97e4133aa1a6fcc984fa5d386525440c  ./tools.json
  d59 063d5a8a0fe8267fc4093041701996f728662aab22b0763dd0225de767bff895  ./CLAUDE.md
  d59 e3b98852646c31827f303b3762410a3ca42f3a51c2a2201614e21b76fae51e70  ./mcp/a.json
  d59 07acba3d89f17bc9467bcbac3f2bcc72cfff4c07ad65694d86691a66a080f80c  ./mcp/b.json
  d59 25cb751f2526dafd5e0f7273fe29a7abac797424f442cfff28ea82ca013bf8cf  ./system_prompt.txt
  d59 3dc0f40d1489590ef5a98c464c11378f97e4133aa1a6fcc984fa5d386525440c  ./tools.json

Meanwhile the vendor added 2,600 bytes of tool-call protocol preamble to every request on day 12, another 3,100 on day 31, another 3,400 on day 48. By day 59 the rendered request carries 14.9 percent more floor than it did at pin time, on every single call, forever.

Three instruments looked at that world. git diff reported nothing. The rolling detectors on the usage log reported nothing (the long window eventually woke up on day 58, which is 46 days late and only after the third bump). The anchor reported this, on day 12, from the same command:

$ drift_anchor_gate.py check worlds/anchor.json worlds/harness/canary/day012.json
anchor     61060 B   pinned day 0   tolerance 2.0%
today      63660 B   day 12           drift +4.26%

  harness_scaffold      +2600 B   vendor
  unchanged (same sha256): instructions, mcp_a, mcp_b, session_id, system_prompt, timestamp, tools_json

  from local files: +0 B      from the vendor: +2600 B

verdict: BLOCK  ->  exit 1   (drift +4.26% is above the 2.0% tolerance)
exit=1

The same command, the same anchor, the same day, in the control world:

$ drift_anchor_gate.py check worlds/anchor.json worlds/control/canary/day012.json
today      61060 B   day 12           drift +0.00%
verdict: PASS   ->  exit 0   (drift +0.00% is within the 2.0% tolerance)
exit=0

That line from local files: +0 B from the vendor: +2600 B is the reason the anchor measures the rendered request and not the files you remembered to list. Your repo is not the thing you pay for. The rendered request is the thing you pay for.

Where the dashboard wins and my anchor is blind

I would not trust this article if it did not have this section, so here it is.

The workload world: the floor never changes, and on day 30 the work per run doubles (retrieval starts returning twice the chunks, users paste bigger inputs, whatever). The rolling detector fires on day 30 and it is completely correct to do so. The anchor passes all 60 days, because the frozen canary task is not doing more work. Its input really did not drift. The anchor is blind to the entire class of cost growth that lives in the workload (the context tax, where every step re-bills the whole transcript), and that class is real and often bigger than the floor.

Now put spike and workload side by side in that results table. Look at the rolling columns. Same detector, same day 30, near-identical alarm counts (4, 2, 2 on the short windows, 15 vs 14 on the long one). From inside the usage log those two worlds are indistinguishable. One of them is a 40 KB doc pasted into an instruction file that you will now pay for on every call until someone deletes it. The other is your product doing more work, which is what you wanted. The dashboard raises the identical alarm for both.

The anchor tells them apart: BLOCK on the paste, PASS on the workload. That is not the anchor being more sensitive. It is the anchor answering a different, narrower question, and answering it cleanly.

So the honest summary is a division of labour, not a replacement. Keep the rolling detector. It catches the paste, the step, and the workload change. Add the anchor. It catches the year.

Where this article is wrong

There is one parameter that can flip the whole result, so I swept it and I am publishing the sweep: how much of a typical request is floor?

The noise in a fleet average comes from the work, not from the floor. The floor is the same on every call. So the less work your runs carry, the quieter your fleet average is, the lower a threshold you can run without false alarms, and the closer your dashboard gets to being a canary all by itself. Here is the sweep on the slowcreep world, with the w=7 detector recalibrated to its own zero-false-alarm threshold at each work level:

  work_median  floor share   w=7 clean T   blind below   alarms on slowcreep   first
  wm002000      95.8%       T=1.05          1.40%/day    4 <- sees the creep  33
  wm005000      89.8%       T=1.05          1.40%/day    6 <- sees the creep  12
  wm010000      81.8%       T=1.15          4.07%/day    0                    -
  wm018000      71.7%       T=1.15          4.07%/day    0                    -
  wm030000      59.2%       T=2.00         21.90%/day    0                    -
  wm060000      41.7%       T=2.00         21.90%/day    0                    -

At a floor share around 90 percent and above, the dashboard sees the creep and this article's headline does not apply to you. The flip in my run happens between 81.8 percent (blind) and 89.8 percent (sees). If your agent's requests are almost entirely system prompt and tool schemas with a tiny user turn on top, you already own a canary. You did not have to freeze anything.

For everyone else, notice what happens at the bottom of that table. At a 42 percent floor share, the ladder had to climb to T=2.00 to stay quiet on a flat fleet. A threshold of 2.00 means "wake me when input cost doubles day over day", and its blind band is 21.9 percent a day. Nobody ships that alert. That is the mechanism in one line: the noisier your work, the higher your threshold must go, and the wider the door you leave open for drift.

I swept the day-to-day noise too, sigma from 0.05 to 0.25. At sigma 0.05, a very quiet fleet, the w=7 detector does catch one alarm on slowcreep, on day 36 (27 days after the anchor). At every other noise level: zero.

Bytes, not tokens, not dollars

The tool counts bytes. That is deliberate, and it is also the answer to the first objection I would raise if I were reading this.

Drift is a ratio, and a ratio survives any linear rescale: (a*x2) / (a*x1) = x2 / x1. Whatever your bytes-per-token constant is, whatever your price per token is, it cancels. So I do not need a tokenizer to say the floor grew 22.6 percent, and I refuse to multiply by a made-up constant to make the number look like money.

Caching is the interesting version of this objection. A stable prefix is cacheable, and a cache read can cost about a tenth of a fresh write, so caching changes the level of what you pay by a large factor. It does not change the ratio, so it does not change the verdict. The floor still grew 22.6 percent, you are still paying the drift, and you are paying it on every call. There is a nastier detail underneath: editing an instruction file invalidates the prefix, so on the day someone appends a policy section you pay a full cache write on top (the cache-break detector is about exactly that failure). Slow creep is not one cheap event. It is a cheap event that also throws away your cache, repeatedly.

If you want dollars, multiply. Just do not let the multiplication happen before the comparison.

How this differs from the two gates I already shipped

Two of my own posts are close enough to this one that I owe you a direct answer.

Sliding-Window Spend Guard also uses a window. The difference is where the threshold comes from: there, the cap is absolute ("$X per window, refuse the next call"), and the window is only a way to aggregate. Here, the whole subject is a threshold derived from your own history, which is the thing that fails. An absolute cap is immune to this failure mode by construction. It is also the reason the window guard is a good design and a fleet-relative alert is a trap: one of them has a baseline that cannot be moved by the drift it is supposed to catch.

MCP Tool Pin Verify also pins a hash. The difference is what the pin is for: there it protects the semantic integrity of a tool description against a malicious rug-pull. Here the pin protects a level, and the adversary is not an attacker, it is your own team, being helpful, one policy section at a time. Same mechanism, different failure. Pinning tool manifests does not tell you your floor grew, and this gate does not tell you a tool description turned hostile. Run both.

Both belong to the same family: a pre-execution gate that decides before the money is spent instead of a chart that explains the money after.

The exit contract, and the bug I refuse to ship again

A cost gate has exactly one bug it must never have: reporting "all clear" because the artifact it was supposed to measure went missing. My previous gate had that bug (a zero-byte baseline made a division produce 0.0, which was below the threshold, which was a pass, on 2.25 million tokens). It was caught in review and the article was dropped. Fair.

So this one has three exits and a rule:

0 PASS drift within tolerance, canary frozen, every declared field present and non-empty.
1 BLOCK drift above tolerance.
2 STRUCTURAL the input is unusable.

There is no path where a missing, emptied or truncated artifact returns 0. (A record that faithfully reports a genuinely smaller request is a PASS — that is the whole point of measuring level, not a fail-open. What cannot happen is the measurement vanishing, or a field dropping out, and being scored as "no drift.") Nine degenerate inputs, all from the real run:

--- dropped_field ---
STRUCTURAL: declared field missing from today's record: tools_json
STRUCTURAL: a bytes-only check would have called this -42.6% drift and passed it
verdict: STRUCTURAL  ->  exit 2   (fail-closed: unusable input is never a pass)
exit=2

--- zero_byte_field ---
STRUCTURAL: declared field rendered zero bytes: mcp_b
exit=2

--- unknown_field ---
STRUCTURAL: undeclared field in today's record: mcp_c
STRUCTURAL: you do not know everything your harness is sending; re-pin deliberately
exit=2

--- task_changed ---
STRUCTURAL: canary task changed: task_sha256 c81db92e6a9f... != pinned 7439b8b0addf...
STRUCTURAL: a canary that drifts is not a canary
exit=2

--- volatile_grew ---
STRUCTURAL: volatile field session_id changed length: 36 B pinned, 96 B today
STRUCTURAL: a volatile field with a moving length is not volatile, it is drift in disguise
exit=2

Plus missing, empty, truncated, and inconsistent_total (the parts do not sum to the stated total). Nine cases, nine exit 2s. The dropped_field line is the one to stare at: a naive bytes-only comparison reads that record as 42.6 percent cheaper than the anchor and passes it. That is the fail-open shape, and it is why the manifest is a manifest and not a number.

And on the typical scenario, the one this tool exists for, it fails closed in the direction that costs someone a conversation: slowcreep day 8 passes at +1.80 percent, day 9 blocks at +2.79 percent, exit 1, attribution instructions +1706 B local:CLAUDE.md. Your CI goes red because somebody added four paragraphs of rules over nine days. That is the intended behaviour, and if it annoys you, re-pin the anchor. Deliberately. In a commit. With your name on it.

The anchor

It is 1,726 bytes and there is nothing clever in it:

{
  "canary_id": "floor-canary-v1",
  "pinned_at": "day 0",
  "tolerance_pct": 2.0,
  "task_sha256": "7439b8b0addf5009c942bb2f1c42c0c4e3489b13990957f8a47253aaa429a8c8",
  "rendered_bytes": 61060,
  "manifest": {
    "tools_json":       {"bytes": 26000, "sha256": "3dc0f40d...", "origin": "local:tools.json",       "volatile": false},
    "system_prompt":    {"bytes": 12000, "sha256": "25cb751f...", "origin": "local:system_prompt.txt","volatile": false},
    "instructions":     {"bytes":  9000, "sha256": "063d5a8a...", "origin": "local:CLAUDE.md",        "volatile": false},
    "harness_scaffold": {"bytes":  6000, "sha256": "39b235eb...", "origin": "vendor",                 "volatile": false},
    "mcp_a":            {"bytes":  4000, "sha256": "e3b98852...", "origin": "local:mcp/a.json",       "volatile": false},
    "mcp_b":            {"bytes":  4000, "sha256": "07acba3d...", "origin": "local:mcp/b.json",       "volatile": false},
    "session_id":       {"bytes":    36, "sha256": "96311dd8...", "origin": "runtime",                "volatile": true},
    "timestamp":        {"bytes":    24, "sha256": "2956a7fc...", "origin": "runtime",                "volatile": true}
  }
}

(Hashes truncated here for width. They are full-length in the file.) One detail worth pausing on: instructions is pinned at 063d5a8a..., and that is the same digest shasum printed for CLAUDE.md on day 59 of the harness world. The file is provably untouched, and the request still grew.

Today's record is the same shape: per-field byte counts and digests for the rendered request. Byte counts and hashes, not payloads, which is the shape a privacy-conscious request log already has. If your harness can dump the request it is about to send, you can produce this. If it cannot, that is a finding.

The code

Three files. All of it, because a reproducibility claim you cannot check is just a claim.

drift_anchor_gate.py:

#!/usr/bin/env python3
"""drift_anchor_gate.py - offline, keyless, zero-network, read-only, stdlib-only (Python 3.13).

One question: has the INPUT COST of a known-good run gone up since a known-good day?

A rolling baseline (median / mean / EWMA over your fleet) answers that question by
comparing today against your own recent history. It is a detector of SPEED. The
invoice is charged on LEVEL. This tool answers the same question against a baseline
that does not move: an anchor pinned on day 0 to a FROZEN canary task, so the only
thing that can change between then and now is what your harness wraps around it.

  check      one day against the anchor. Exit 0 PASS / 1 BLOCK / 2 STRUCTURAL.
  replay     N days, anchor and rolling detectors side by side on the same log.
  sweep      the two parameters that could flip the result: floor share, day noise.
  blindband  pure arithmetic, no data: the growth rate a rolling window cannot see.

Exit contract (fail-closed):
  0 PASS        drift within tolerance, canary frozen, every declared field present and non-empty
  1 BLOCK       drift above tolerance
  2 STRUCTURAL  input unusable: unreadable, empty, truncated, a declared field missing or
                rendered zero bytes, an undeclared field present, the canary task changed,
                a volatile field that changed length, a record whose parts do not sum to its total.

There is no path where a missing, emptied or truncated artifact returns 0, and a
dropped or zeroed field fails closed too. A gate that answers "no drift detected"
because the artifact vanished is worse than no gate.

This tool never writes, never opens a socket, never imports anything outside the stdlib.
"""

import hashlib
import json
import os
import statistics
import sys

LADDER = [1.05, 1.10, 1.15, 1.20, 1.25, 1.30, 1.40, 1.50, 2.00]
FAMILIES = [("rolling median w=7", "median", 7),
            ("rolling mean   w=7", "mean", 7),
            ("EWMA alpha=0.3", "ewma", 7),
            ("rolling median w=28", "median", 28)]
ALPHA = 0.3


# ---------------------------------------------------------------- the gate

def load_json(path, what):
    if not os.path.exists(path):
        die("%s not found: %s" % (what, path))
    if os.path.getsize(path) == 0:
        die("%s is zero bytes: %s" % (what, path))
    try:
        with open(path) as fh:
            return json.load(fh)
    except (ValueError, OSError) as exc:
        die("%s is not readable JSON: %s (%s)" % (what, path, exc.__class__.__name__))


def die(msg):
    print("STRUCTURAL: %s" % msg)
    print("verdict: STRUCTURAL  ->  exit 2   (fail-closed: unusable input is never a pass)")
    sys.exit(2)


def check(anchor, today, quiet=False):
    """Return (verdict, drift_pct, lines). verdict in PASS / BLOCK / STRUCTURAL."""
    out = []
    man = anchor["manifest"]
    fields = today.get("fields")
    if not isinstance(fields, dict) or not fields:
        return "STRUCTURAL", 0.0, ["no fields in today's record"]
    if today.get("canary_id") != anchor["canary_id"]:
        return "STRUCTURAL", 0.0, ["canary_id mismatch: %r vs pinned %r"
                                   % (today.get("canary_id"), anchor["canary_id"])]
    if today.get("task_sha256") != anchor["task_sha256"]:
        return "STRUCTURAL", 0.0, ["canary task changed: task_sha256 %s... != pinned %s..."
                                   % (str(today.get("task_sha256"))[:12], anchor["task_sha256"][:12]),
                                   "a canary that drifts is not a canary"]

    missing = [k for k in man if k not in fields]
    unknown = [k for k in fields if k not in man]
    empty = [k for k, v in fields.items() if v.get("bytes", 0) <= 0]
    if missing:
        naive = sum(v["bytes"] for v in fields.values()) / anchor["rendered_bytes"] - 1
        return "STRUCTURAL", 0.0, [
            "declared field missing from today's record: %s" % ", ".join(sorted(missing)),
            "a bytes-only check would have called this %+.1f%% drift and passed it" % (naive * 100)]
    if unknown:
        return "STRUCTURAL", 0.0, [
            "undeclared field in today's record: %s" % ", ".join(sorted(unknown)),
            "you do not know everything your harness is sending; re-pin deliberately"]
    if empty:
        return "STRUCTURAL", 0.0, ["declared field rendered zero bytes: %s" % ", ".join(sorted(empty))]

    for k, v in man.items():
        if v.get("volatile") and fields[k]["bytes"] != v["bytes"]:
            return "STRUCTURAL", 0.0, [
                "volatile field %s changed length: %d B pinned, %d B today"
                % (k, v["bytes"], fields[k]["bytes"]),
                "a volatile field with a moving length is not volatile, it is drift in disguise"]

    total = sum(v["bytes"] for v in fields.values())
    if total != today.get("rendered_bytes"):
        return "STRUCTURAL", 0.0, ["record does not sum: fields %d B, rendered_bytes %d B"
                                   % (total, today.get("rendered_bytes"))]

    drift = total / anchor["rendered_bytes"] - 1
    tol = anchor["tolerance_pct"] / 100.0
    deltas = sorted(((fields[k]["bytes"] - man[k]["bytes"], k) for k in man),
                    key=lambda t: -abs(t[0]))
    local = sum(d for d, k in deltas if man[k]["origin"].startswith("local"))
    vendor = sum(d for d, k in deltas if not man[k]["origin"].startswith("local"))

    if not quiet:
        out.append("anchor  %8d B   pinned %s   tolerance %.1f%%"
                   % (anchor["rendered_bytes"], anchor["pinned_at"], anchor["tolerance_pct"]))
        out.append("today   %8d B   day %-3s          drift %+.2f%%"
                   % (total, today.get("day", "?"), drift * 100))
        out.append("")
        for d, k in deltas:
            if d:
                out.append("  %-18s %+8d B   %s" % (k, d, man[k]["origin"]))
        same = [k for d, k in deltas if d == 0 and fields[k]["sha256"] == man[k]["sha256"]]
        if same:
            out.append("  unchanged (same sha256): %s" % ", ".join(sorted(same)))
        out.append("")
        out.append("  from local files: %+d B      from the vendor: %+d B" % (local, vendor))

    verdict = "BLOCK" if drift > tol else "PASS"
    return verdict, drift * 100, out


def cmd_check(args):
    anchor = load_json(args[0], "anchor")
    today = load_json(args[1], "today's record")
    verdict, drift, lines = check(anchor, today)
    if verdict == "STRUCTURAL":
        for ln in lines:
            print("STRUCTURAL: %s" % ln)
        print("verdict: STRUCTURAL  ->  exit 2   (fail-closed: unusable input is never a pass)")
        return 2
    body = "\n".join(lines)
    print(body)
    print("")
    if verdict == "BLOCK":
        print("verdict: BLOCK  ->  exit 1   (drift %+.2f%% is above the %.1f%% tolerance)"
              % (drift, anchor["tolerance_pct"]))
    else:
        print("verdict: PASS   ->  exit 0   (drift %+.2f%% is within the %.1f%% tolerance)"
              % (drift, anchor["tolerance_pct"]))
    print("report-sha256: %s" % hashlib.sha256(body.encode()).hexdigest())
    return 1 if verdict == "BLOCK" else 0


# ---------------------------------------------------------- the dashboard

def daily_means(path):
    if not os.path.exists(path) or os.path.getsize(path) == 0:
        die("usage log missing or empty: %s" % path)
    days = {}
    with open(path) as fh:
        for line in fh:
            r = json.loads(line)
            days.setdefault(r["day"], []).append(r["input_bytes"])
    if not days:
        die("usage log has no records: %s" % path)
    return [statistics.mean(days[d]) for d in sorted(days)]


def alarms(series, kind, w, t):
    """Days on which today's mean exceeds the baseline built from the PREVIOUS w days by factor t."""
    hits = []
    ew = None
    for i, x in enumerate(series):
        if i >= w:
            if kind == "median":
                base = statistics.median(series[i - w:i])
            elif kind == "mean":
                base = statistics.mean(series[i - w:i])
            else:
                base = ew
            if base and x / base > t:
                hits.append(i)
        if kind == "ewma":
            ew = x if ew is None else ALPHA * x + (1 - ALPHA) * ew
    return hits


def calibrate(control, kind, w):
    """The lowest threshold on the ladder that raises zero false alarms on a flat world."""
    for t in LADDER:
        if not alarms(control, kind, w, t):
            return t
    return None


def blind_band(t, w):
    """Uniform daily growth g is invisible while (1+g)^(w/2) <= T. Arithmetic, not a finding."""
    return t ** (2.0 / w) - 1.0


def cmd_blindband(_args):
    print("blind band of a rolling window: it compares today against a baseline centred")
    print("about w/2 days back, so a uniform daily growth g only trips it when (1+g)^(w/2) > T.")
    print("solve for g:   g* = T^(2/w) - 1     any growth slower than g* is invisible forever.")
    print("")
    print("  window   threshold   invisible below   compounds to, over 30 days")
    for w, t in ((7, 1.15), (7, 1.20), (14, 1.20), (28, 1.20), (28, 1.10)):
        g = blind_band(t, w)
        print("  w=%-4d   T=%.2f       %6.2f%%/day        x%.2f" % (w, t, g * 100, (1 + g) ** 30))
    print("")
    print("this is arithmetic. it is true before you collect a single byte of data.")
    print("the measurement is where real context creep lands relative to that band.")
    return 0


def cmd_replay(args):
    world, ctrl = args[0], args[1]
    anchor = load_json(os.path.join(os.path.dirname(world), "anchor.json"), "anchor")
    events = load_json(os.path.join(world, "events.json"), "events")
    series = daily_means(os.path.join(world, "usage.jsonl"))
    control = daily_means(os.path.join(ctrl, "usage.jsonl"))

    print("world: %s   days: %d   runs/day: fixed" % (os.path.basename(world), len(series)))
    floor0 = anchor["rendered_bytes"]
    typical = statistics.mean(control)   # a flat fleet, so this is the work profile, not the drift
    print("floor at pin time %d B   typical run %d B   floor share %.1f%%"
          % (floor0, round(typical), 100.0 * floor0 / typical))
    print("")

    print("  detector              threshold   alarms   first   on the day of")
    for name, kind, w in FAMILIES:
        t = calibrate(control, kind, w)
        if t is None:
            print("  %-20s   none clean on a flat world" % name)
            continue
        hits = alarms(series, kind, w, t)
        first = hits[0] if hits else None
        ev = events.get(str(first), "nothing: this one is about the creep") if first is not None else "-"
        print("  %-20s   T=%.2f       %2d      %-5s   %s"
              % (name, t, len(hits), first if first is not None else "-", ev))
        g = blind_band(t, w)
        print("  %-20s   blind below %.2f%%/day (x%.2f over 30 days)" % ("", g * 100, (1 + g) ** 30))

    first_block, first_struct, drift_at, final = None, None, 0.0, floor0
    for day in range(len(series)):
        rec = load_json(os.path.join(world, "canary", "day%03d.json" % day), "record")
        verdict, drift, _ = check(anchor, rec, quiet=True)
        if verdict == "BLOCK" and first_block is None:
            first_block, drift_at = day, drift
        if verdict == "STRUCTURAL" and first_struct is None:
            first_struct = day
        final = rec["rendered_bytes"]
    print("")
    if first_block is None and first_struct is None:
        print("  frozen anchor tol=%.1f%%     PASS on all %d days: the frozen input never drifted"
              % (anchor["tolerance_pct"], len(series)))
    if first_block is not None:
        print("  frozen anchor tol=%.1f%%     BLOCK on day %d (drift %+.2f%%)  -> exit 1"
              % (anchor["tolerance_pct"], first_block, drift_at))
    if first_struct is not None:
        print("  frozen anchor tol=%.1f%%     STRUCTURAL on day %d (%s) -> exit 2"
              % (anchor["tolerance_pct"], first_struct,
                 events.get(str(first_struct), "the set of fields changed")))
    print("  floor on the last day %d B (%+.1f%% against the anchor)"
          % (final, (final / floor0 - 1) * 100))
    return 1 if first_block is not None else (2 if first_struct is not None else 0)


def cmd_sweep(args):
    root, label = args[0].rstrip("/"), args[1]
    if not os.path.isdir(root):
        die("sweep root not found: %s" % root)
    anchor = load_json(os.path.join(os.path.dirname(root), "anchor.json"), "anchor")
    print("  %-10s  floor share   w=7 clean T   blind below   alarms on slowcreep   first" % label)
    for name in sorted(os.listdir(root)):
        d = os.path.join(root, name)
        ctrl = daily_means(os.path.join(d, "control", "usage.jsonl"))
        slow = daily_means(os.path.join(d, "slowcreep", "usage.jsonl"))
        t = calibrate(ctrl, "median", 7)
        hits = alarms(slow, "median", 7, t)
        share = 100.0 * anchor["rendered_bytes"] / statistics.mean(ctrl)
        g = blind_band(t, 7) * 100
        print("  %-10s  %6.1f%%       T=%.2f        %6.2f%%/day   %2d %-18s %s"
              % (name, share, t, g, len(hits),
                 "<- sees the creep" if hits else "", hits[0] if hits else "-"))
    print("")
    print("  neither parameter reaches the frozen anchor. It reads a deterministic input, so it")
    print("  carries no work and no day-to-day noise: floor share 100%%, tolerance %.1f%%, same"
          % anchor["tolerance_pct"])
    print("  BLOCK day in every row above. The rows only move the DASHBOARD.")
    return 0


USAGE = """usage:
  drift_anchor_gate.py check     <anchor.json> <today.json>
  drift_anchor_gate.py replay    <world_dir> <control_dir>
  drift_anchor_gate.py sweep     <sweep_root> <label>
  drift_anchor_gate.py blindband
"""

if __name__ == "__main__":
    cmds = {"check": cmd_check, "replay": cmd_replay, "sweep": cmd_sweep, "blindband": cmd_blindband}
    if len(sys.argv) < 2 or sys.argv[1] not in cmds:
        sys.stdout.write(USAGE)
        sys.exit(2)
    sys.exit(cmds[sys.argv[1]](sys.argv[2:]))

make_worlds.py, the fixture builder. Seeded, and the only thing here that writes anything:

#!/usr/bin/env python3
"""make_worlds.py - build the fixtures for drift_anchor_gate.py.

Offline, keyless, zero-network, stdlib-only, seeded. This is the only script here
that writes anything. The gate itself never writes.

Six worlds share ONE anchor (day 0 is byte-identical everywhere) and ONE noise
realization per (work_median, sigma_day) pair. Only the floor trajectory differs
between worlds, so the dashboard is never handed a harder noise draw than the
anchor. Runs per day are FIXED at 40 on purpose: that removes run-count noise,
which is a handicap in the dashboard's favour.
"""

import hashlib
import json
import os
import random
import shutil

HERE = os.path.dirname(os.path.abspath(__file__))
OUT = os.path.join(HERE, "worlds")

DAYS = 60
RUNS_PER_DAY = 40
BASE_SEED = 20260715
WORK_MEDIAN = 18_000      # bytes of real per-run work: the user turn, retrieved chunks, history
WORK_SIGMA = 0.8          # lognormal spread of work across runs
SIGMA_DAY = 0.15          # lognormal spread of the daily job mix
TOL_PCT = 2.0

TASK = ("canary task: read the pinned fixture file, return its word count as JSON. "
        "do not call tools. do not browse. answer in one line.")

# Floor at pin time, in bytes, by field.
FLOOR0 = {
    "harness_scaffold": 6_000,   # vendor-injected: tool-call protocol preamble + safety wrapper
    "system_prompt": 12_000,
    "tools_json": 26_000,
    "instructions": 9_000,       # CLAUDE.md / AGENTS.md
    "mcp_a": 4_000,
    "mcp_b": 4_000,
}
VOLATILE = {"session_id": 36, "timestamp": 24}
ORIGIN = {
    "harness_scaffold": "vendor",
    "system_prompt": "local:system_prompt.txt",
    "tools_json": "local:tools.json",
    "instructions": "local:CLAUDE.md",
    "mcp_a": "local:mcp/a.json",
    "mcp_b": "local:mcp/b.json",
    "mcp_c": "local:mcp/c.json",
    "mcp_d": "local:mcp/d.json",
    "session_id": "runtime",
    "timestamp": "runtime",
}

WORDS = ("agent budget cache canary context deploy drift enforce escalate fixture gate harness "
         "ingest invariant ledger manifest observe payload pin policy prompt quota render replay "
         "retry rollback runbook schema scope session snapshot spend threshold token trace vendor "
         "window workspace approve reject timeout latency shard cursor batch").split()

SUBJECTS = ["retry budget", "tool allowlist", "pii redaction", "escalation path", "commit policy",
            "sandbox scope", "timeout ladder", "cache warmup", "rollback drill", "on-call handoff",
            "schema review", "cost ceiling", "log retention", "vendor swap", "eval cadence",
            "prompt review", "secret handling", "shell allowlist", "diff size cap", "branch policy"]


def text_of(rng, n, tag):
    """Deterministic ASCII text of exactly n bytes."""
    if n <= 0:
        return ""
    parts = [tag, ":"]
    size = len(tag) + 1
    while size < n:
        w = rng.choice(WORDS)
        parts.append(" " + w)
        size += len(w) + 1
    s = "".join(parts)[:n]
    return s + " " * (n - len(s))


def rule_section(rng, idx):
    """One appended policy section. Unique subject, unique body, 380..620 bytes."""
    n = rng.randint(380, 620)
    head = "\n\n## Rule %03d: %s\n" % (idx, SUBJECTS[idx % len(SUBJECTS)])
    return head + text_of(rng, n - len(head), "rationale-%03d" % idx)


# ---- static content, generated once so it is stable across days and worlds ----
_c = random.Random(BASE_SEED ^ 0xF100)
BASE = {k: text_of(_c, v, k) for k, v in FLOOR0.items()}
_s = random.Random(BASE_SEED ^ 0x5EC7)
SECTIONS = [rule_section(_s, i) for i in range(70)]
EXTRA_TOOLS = "".join(text_of(_c, 1_150, "tool-schema-%02d" % i) for i in range(12))  # 13,800 B
MCP_C = text_of(_c, 4_200, "mcp_c")
MCP_D = text_of(_c, 3_800, "mcp_d")
SPIKE_DOC = text_of(_c, 40_000, "pasted-architecture-doc")
BUMP = {12: text_of(_c, 2_600, "scaffold-bump-1"),
        31: text_of(_c, 3_100, "scaffold-bump-2"),
        48: text_of(_c, 3_400, "scaffold-bump-3")}

EVENTS = {
    "control": {},
    "creep": {14: "mcp: +3rd server", 33: "mcp: +4th server", 40: "tools: +12 schemas"},
    "slowcreep": {33: "mcp: +3rd server"},
    "spike": {30: "instructions: +40 KB doc pasted"},
    "harness": {12: "vendor: scaffold bump 1", 31: "vendor: scaffold bump 2", 48: "vendor: scaffold bump 3"},
    "workload": {30: "work per run doubles (floor untouched)"},
}
WORLDS = list(EVENTS)


def render_floor(world, day):
    """Field -> content string for this world on this day. Day 0 is identical in every world."""
    f = dict(BASE)
    if world == "creep":
        n = day                                  # one policy section per day
    elif world == "slowcreep":
        n = day // 3                             # one policy section every third day
    else:
        n = 0
    if n:
        f["instructions"] = BASE["instructions"] + "".join(SECTIONS[:n])
    if world == "creep":
        if day >= 40:
            f["tools_json"] = BASE["tools_json"] + EXTRA_TOOLS
        if day >= 14:
            f["mcp_c"] = MCP_C
        if day >= 33:
            f["mcp_d"] = MCP_D
    if world == "slowcreep" and day >= 33:
        f["mcp_c"] = MCP_C
    if world == "spike" and day >= 30:
        f["instructions"] = f["instructions"] + SPIKE_DOC
    if world == "harness":
        add = "".join(v for d, v in sorted(BUMP.items()) if day >= d)
        f["harness_scaffold"] = BASE["harness_scaffold"] + add
    return f


def work_scale(world, day):
    """The only world that changes the WORK per run rather than the floor."""
    return 2.0 if (world == "workload" and day >= 30) else 1.0


def sha(s):
    return hashlib.sha256(s.encode("utf-8")).hexdigest()


def dump_fields(fields):
    """Rendered-request record: per-field byte count + sha256. The shape of a privacy-safe request log."""
    rec = {k: {"bytes": len(v), "sha256": sha(v)} for k, v in fields.items()}
    for k, n in VOLATILE.items():
        rec[k] = {"bytes": n, "sha256": sha(k * n)}
    return rec


def total_bytes(fields):
    return sum(len(v) for v in fields.values()) + sum(VOLATILE.values())


def noise_matrix(work_median, sigma_day, seed):
    """Per-run work bytes. Identical across worlds for a given (work_median, sigma_day)."""
    rng = random.Random(seed)
    rows = []
    for _ in range(DAYS):
        dm = rng.lognormvariate(0.0, sigma_day)
        rows.append([max(1, int(work_median * rng.lognormvariate(0.0, WORK_SIGMA) * dm))
                     for _ in range(RUNS_PER_DAY)])
    return rows


def write_usage(path, world, noise):
    """usage.jsonl: one line per run, the log a fleet dashboard already has."""
    with open(path, "w") as fh:
        for day in range(DAYS):
            floor = total_bytes(render_floor(world, day))
            k = work_scale(world, day)
            for run in range(RUNS_PER_DAY):
                fh.write(json.dumps({"day": day, "run": run,
                                     "input_bytes": floor + int(noise[day][run] * k)}) + "\n")


def main():
    if os.path.isdir(OUT):
        shutil.rmtree(OUT)
    os.makedirs(OUT)

    day0 = render_floor("control", 0)
    anchor = {
        "canary_id": "floor-canary-v1",
        "pinned_at": "day 0",
        "tolerance_pct": TOL_PCT,
        "task_sha256": sha(TASK),
        "rendered_bytes": total_bytes(day0),
        "manifest": {},
    }
    for k, rec in dump_fields(day0).items():
        anchor["manifest"][k] = {"bytes": rec["bytes"], "sha256": rec["sha256"],
                                 "origin": ORIGIN[k], "volatile": k in VOLATILE}
    with open(os.path.join(OUT, "anchor.json"), "w") as fh:
        json.dump(anchor, fh, indent=2, sort_keys=True)

    noise = noise_matrix(WORK_MEDIAN, SIGMA_DAY, BASE_SEED)

    for world in WORLDS:
        wd = os.path.join(OUT, world)
        os.makedirs(os.path.join(wd, "canary"))
        for day in range(DAYS):
            fields = render_floor(world, day)
            rec = {"canary_id": "floor-canary-v1", "day": day, "task_sha256": sha(TASK),
                   "rendered_bytes": total_bytes(fields), "fields": dump_fields(fields)}
            with open(os.path.join(wd, "canary", "day%03d.json" % day), "w") as fh:
                json.dump(rec, fh, indent=2, sort_keys=True)
        write_usage(os.path.join(wd, "usage.jsonl"), world, noise)
        with open(os.path.join(wd, "events.json"), "w") as fh:
            json.dump({str(k): v for k, v in EVENTS[world].items()}, fh, indent=2, sort_keys=True)

    # Repo snapshots for the harness world: the local files a `git diff` would look at.
    for day in (0, 59):
        d = os.path.join(OUT, "harness", "repo_day%03d" % day)
        os.makedirs(os.path.join(d, "mcp"))
        f = render_floor("harness", day)
        open(os.path.join(d, "system_prompt.txt"), "w").write(f["system_prompt"])
        open(os.path.join(d, "CLAUDE.md"), "w").write(f["instructions"])
        open(os.path.join(d, "tools.json"), "w").write(f["tools_json"])
        open(os.path.join(d, "mcp", "a.json"), "w").write(f["mcp_a"])
        open(os.path.join(d, "mcp", "b.json"), "w").write(f["mcp_b"])

    # Degenerate inputs. Every one of these must fail closed.
    dg = os.path.join(OUT, "degenerate")
    os.makedirs(dg)
    base = json.load(open(os.path.join(OUT, "control", "canary", "day000.json")))

    def variant(name, mutate):
        r = json.loads(json.dumps(base))
        mutate(r)
        with open(os.path.join(dg, name + ".json"), "w") as fh:
            json.dump(r, fh, indent=2, sort_keys=True)

    def drop_tools(r):
        r["fields"].pop("tools_json")
        r["rendered_bytes"] = sum(v["bytes"] for v in r["fields"].values())

    def zero_mcp(r):
        r["fields"]["mcp_b"] = {"bytes": 0, "sha256": sha("")}
        r["rendered_bytes"] = sum(v["bytes"] for v in r["fields"].values())

    def unknown_field(r):
        r["fields"]["mcp_c"] = {"bytes": 4_200, "sha256": sha(MCP_C)}
        r["rendered_bytes"] = sum(v["bytes"] for v in r["fields"].values())

    def task_changed(r):
        r["task_sha256"] = sha(TASK + " and also list the files you can see")

    def volatile_grew(r):
        r["fields"]["session_id"] = {"bytes": 96, "sha256": sha("x" * 96)}
        r["rendered_bytes"] = sum(v["bytes"] for v in r["fields"].values())

    def bad_sum(r):
        r["rendered_bytes"] = 61_060 - 20_000

    variant("dropped_field", drop_tools)
    variant("zero_byte_field", zero_mcp)
    variant("unknown_field", unknown_field)
    variant("task_changed", task_changed)
    variant("volatile_grew", volatile_grew)
    variant("inconsistent_total", bad_sum)
    open(os.path.join(dg, "empty.json"), "w").close()
    open(os.path.join(dg, "truncated.json"), "w").write('{"canary_id": "floor-canary-v1", "day": 0,')

    # Sweeps: the two parameters that could flip the result. control + slowcreep share a seed.
    for wm in (2_000, 5_000, 10_000, 18_000, 30_000, 60_000):
        seed = BASE_SEED + wm + 15 * 1_000_003
        nz = noise_matrix(wm, SIGMA_DAY, seed)
        for world in ("control", "slowcreep"):
            d = os.path.join(OUT, "sweep_floorshare", "wm%06d" % wm, world)
            os.makedirs(d)
            write_usage(os.path.join(d, "usage.jsonl"), world, nz)
            shutil.copy(os.path.join(OUT, world, "events.json"), os.path.join(d, "events.json"))

    for sig in (0.05, 0.10, 0.15, 0.20, 0.25):
        seed = BASE_SEED + WORK_MEDIAN + int(sig * 100) * 1_000_003
        nz = noise_matrix(WORK_MEDIAN, sig, seed)
        for world in ("control", "slowcreep"):
            d = os.path.join(OUT, "sweep_sigma", "s%03d" % int(sig * 100), world)
            os.makedirs(d)
            write_usage(os.path.join(d, "usage.jsonl"), world, nz)
            shutil.copy(os.path.join(OUT, world, "events.json"), os.path.join(d, "events.json"))

    print("worlds built: %s" % ", ".join(WORLDS))
    print("anchor rendered_bytes: %d B  tolerance: %.1f%%" % (anchor["rendered_bytes"], TOL_PCT))
    for world in WORLDS:
        a, b = total_bytes(render_floor(world, 0)), total_bytes(render_floor(world, 59))
        rate = ((b / a) ** (1 / 59) - 1) * 100 if b != a else 0.0
        print("  %-10s floor day0 %7d B  ->  day59 %7d B   (x%.2f, %+.3f%%/day)"
              % (world, a, b, b / a, rate))


if __name__ == "__main__":
    main()

run_demo.sh:

#!/usr/bin/env bash
# run_demo.sh - rebuild the fixtures from scratch and run every claim in the post.
# Offline, keyless, no network, stdlib only. Nothing here reads a clock, so stdout is
# byte-for-byte reproducible: run it twice, diff the two outputs, they are identical.
set -u
cd "$(dirname "$0")"

hr() { printf '\n%s\n%s\n' "$1" "$(printf '=%.0s' $(seq 1 ${#1}))"; }

hr "0. THE PREDICTION (arithmetic, before any data exists)"
python3 drift_anchor_gate.py blindband

hr "1. BUILD THE WORLDS (seeded, deterministic, no network)"
python3 make_worlds.py

hr "2. KILLER DEMO: the harness world. Every local file is byte-identical to day 0."
diff -rq worlds/harness/repo_day000 worlds/harness/repo_day059 \
  && echo "diff -rq repo_day000 repo_day059: no differences. git diff would print nothing."
echo ""
echo "sha256 of every local file, day 0 vs day 59:"
( cd worlds/harness/repo_day000 && find . -type f | sort | xargs shasum -a 256 ) | sed 's/^/  d0  /'
( cd worlds/harness/repo_day059 && find . -type f | sort | xargs shasum -a 256 ) | sed 's/^/  d59 /'

echo ""
echo "--- same anchor, same day 12, two worlds ---"
echo ""
echo "\$ drift_anchor_gate.py check worlds/anchor.json worlds/control/canary/day012.json"
python3 drift_anchor_gate.py check worlds/anchor.json worlds/control/canary/day012.json
echo "exit=$?"
echo ""
echo "\$ drift_anchor_gate.py check worlds/anchor.json worlds/harness/canary/day012.json"
python3 drift_anchor_gate.py check worlds/anchor.json worlds/harness/canary/day012.json
echo "exit=$?"

hr "3. SIX WORLDS, ONE ANCHOR, ONE LOG. Each rolling detector gets its own best threshold."
for w in control creep slowcreep harness spike workload; do
  echo ""
  python3 drift_anchor_gate.py replay "worlds/$w" worlds/control
  echo "  replay exit=$?"
done

hr "4. THE TYPICAL SCENARIO, AT THE BOUNDARY. The gate must fail CLOSED on slow creep."
echo ""
echo "\$ drift_anchor_gate.py check worlds/anchor.json worlds/slowcreep/canary/day008.json"
python3 drift_anchor_gate.py check worlds/anchor.json worlds/slowcreep/canary/day008.json
echo "exit=$?"
echo ""
echo "\$ drift_anchor_gate.py check worlds/anchor.json worlds/slowcreep/canary/day009.json"
python3 drift_anchor_gate.py check worlds/anchor.json worlds/slowcreep/canary/day009.json
echo "exit=$?"

hr "5. DEGENERATE INPUTS: every one of these must fail closed."
for f in missing empty truncated dropped_field zero_byte_field unknown_field task_changed volatile_grew inconsistent_total; do
  echo ""
  echo "--- $f ---"
  python3 drift_anchor_gate.py check worlds/anchor.json "worlds/degenerate/$f.json"
  echo "exit=$?"
done
echo ""
echo "--- clean day 0 (the control: it must still pass) ---"
python3 drift_anchor_gate.py check worlds/anchor.json worlds/control/canary/day000.json
echo "exit=$?"

hr "6. SWEEP A: the one parameter that flips the result. How much of the request is floor?"
python3 drift_anchor_gate.py sweep worlds/sweep_floorshare work_median

hr "7. SWEEP B: day-to-day noise. Does the dashboard wake up at any of these?"
python3 drift_anchor_gate.py sweep worlds/sweep_sigma sigma_day

Run it: bash run_demo.sh > output.txt. It takes a few seconds, touches no network, needs no keys, and produces the same bytes every time. Mine hashes to 1772e695cb75f79d9e3f162ed4c49477a329703610cdf5ddff54cec2cc4da62a.

What this is NOT

The time series are synthetic. I do not have a production fleet, and I am not going to pretend I do. The builder is in this post, the shape is declared (how many bytes a policy section adds, what day an MCP server lands), and the seed is fixed. Change the parameters and the numbers move. What is not synthetic: the blind-band arithmetic, and the fact that the gate runs on your real usage.jsonl and your real request dumps.
The anchor does not see workload growth. Demonstrated above, in the one world where it passes for 60 days while cost genuinely rises. If your bill grew because your product got busier, this tool will tell you nothing and your dashboard will tell you everything.
The anchor only sees the rendered request it is handed. No request dump, no gate. It reads nothing from the network.
It does not tell you whether the drift was worth it. A third MCP server may well be worth its 4 KB on every call. The gate converts a silent accumulation into a decision that somebody signs.
Model-side drift is out of scope. If the vendor changes the tokenizer or the model gets chattier, the input payload can be identical while the bill moves. This gate reads the input payload only. That is a real hole, and it is Dipankar's original point coming back around.
A 2 percent tolerance is not a measured optimum. On my fixtures the control world drifts by exactly 0.00 percent, so any positive tolerance gives zero false alarms. I picked 2 percent as a real-world allowance for things like a version string that changes length. My data did not force that number, and I am not going to dress it up as though it did.

What I would do on Monday

Dump the rendered request for one frozen, deterministic task. Save the byte counts and hashes as an anchor with today's date. Add one CI step that re-renders that same task and compares. Set the tolerance somewhere near 2 percent, because a deterministic input has no noise to hide in, and let it fail the build.

Then keep your dashboard exactly as it is. It is watching for the paste and the traffic step, and it is good at that. It was never watching for the year.

One thing I have not settled, and I would genuinely like a second opinion: the re-pin. Every legitimate change (a new MCP server, a needed rule) forces a re-pin, and a re-pin resets the baseline. Do it a dozen times and you have reinvented a rolling baseline with extra steps, just with human latency in the loop. My current answer is that the re-pin has to carry the cumulative drift since the original pin, so the number you are approving is "the floor is now 47 percent above where it was in March", not "plus 4 percent since last week". I have not built that yet. If you have run a pinned baseline in production for longer than a quarter, I want to know how you kept the pin honest.

I write about pre-execution control for AI agents: gates that decide before the money moves, not charts that explain it afterwards. Every post ships a tool you can actually run, offline, with no keys. Follow along if that is your kind of thing, and if your context floor has a story, put it in the comments. I am especially interested in anyone who has watched a vendor bump a harness without telling them.