DEV Community: Alexey Spinov

Your Guardrail Test Still Passes With the Guardrail Deleted

Alexey Spinov — Wed, 09 Sep 2026 07:51:28 +0000

To test whether an AI agent guardrail is still working, run your labelled corpus twice: once as built, once with the gate replaced by a pass-through. ablation_probe.py sorts each bad input into LOAD_BEARING, SHADOWED or MISSED, and exits 1 when no verdict moved. A rejection shows that something said no. It does not show that the gate did.

Here is the shape of the problem. You have a policy gate. You wrote a fixture that it is supposed to reject. CI runs the fixture, the pipeline says DENY, the build is green. Now delete the gate. On my corpus, 9 of 26 bad inputs are still rejected, by other stages that were in the pipeline the whole time.

Those 9 are decoration. Denied with the gate, denied without it. Whatever they test, it is not the gate.

AI disclosure. I wrote ablation_probe.py and make_corpus.py with AI assistance and ran them myself before publishing: offline, standard library only, no network, no keys, no funds. Every output block below is pasted from a real run on Python 3.13.5. run_all.sh executes 31 scenarios, each three times, and compares the copies byte for byte: it reported 31 of 31 byte-identical across three runs. Code sha256: ablation_probe.py 52035306…d6e4, make_corpus.py e5d86c35…39c4, run_all.sh 96ff9499…5fb8, corpus 9ac269d7…06ef9, second corpus 3eb86f67…c315. Output blocks are excerpts from one run of the named scenario: [...] marks rows dropped from inside a table, and where a block is not the whole report I have cut headers, the pre-flight or the good-input table around it. Nothing inside a block is reworded. The pipeline and both corpora are synthetic, written by me, and I say so again where they appear. Five defects of my own die further down the page. Two I found; the other three were found by a pre-publication review of this draft, after I had already written the numbers that turned out to be describing something else. One of those three is the exact failure this article is about, sitting inside the tool that is supposed to detect it.

In short:

With the gate deleted, 9 of 26 bad inputs on my corpus are still denied. Grouped by class: 3 of my 6 declared bad classes survive the gate's removal intact. A known-bad twin drawn only from those classes stays green with no gate at all.
SHADOWED does not mean "caught earlier". In order A all 9 are masked by a stage before the gate; in order B, 6 are masked by a stage after it and only 2 before. My tool printed the word "earlier" as a hardcoded constant for a day, in the very order whose numbers this post publishes.
Moving the gate two positions earlier changes nothing: order C has a class map identical to order A, 17/9, exit 0. What flips a class is crossing the path canonicaliser, which is an undeclared precondition of the gate, not a position in a list.
Crossing it turns 1 of 6 bad classes from LOAD_BEARING to MISSED. Six inputs that were rejected now reach execute. Both orders exit 0; the reports differ, the exit codes do not.
Counting attribution by the first stage that says no undercounts the gate by 6 of 26 inputs in order A and by 0 in order B.
The "measured trade-off" I published in the first draft was a property of how I wrote six paths. Rewrite the good rows in absolute form and the strict gate goes from 4 false positives and exit 1 to 0 and exit 0. The MISSED result survives that rewrite; the trade-off does not.
--record and --expect in one command was green by construction, and a failing run could freeze itself as the baseline. Both are now refusals.
Null control: ablate a declared no-op stage and the probe reports 26 SHADOWED, exit 1, no crash, no claim that the pipeline is broken.

Nobody checks whether the guardrail is running

The framing is not mine. arun rajkumar (@mickyarun) published Nobody Checks Whether the Guardrail Is Running on Dev.to on 2026-09-07. His own line for it: "A guardrail that has never fired and a guardrail that silently stopped running produce identical output. Green." In the same post he relays a point from Heinrich Neb, that every grader needs a known-bad twin, "an input it is supposed to reject, plus a recorded date of when it last actually rejected something", and calls Neb's framing the sharpest version he has seen.

We said the statistical half of that ourselves in July, in zero failures is not zero risk: "a check that always returns 'fine' and a check that has stopped running produce the identical screenshot." That post answered with a bound on the failure rate you have not yet observed. It did not build anything that could tell the two checks apart. This one does, and finds that the twin has a hole of its own.

A known-bad twin proves something said no

Write the assert and look at it: assert pipeline(bad_input) == DENY.

The subject of that sentence is the pipeline. Not the gate. And the gate is not first: arguments have to be parsed before a policy gate has anything to read. Whether schema validation and the path allowlist sit in front of it or behind it is a design choice, and I measure three arrangements below. Either way, a known-bad input that is also malformed gets rejected by the schema, whether or not the gate is alive.

That is not hypothetical, it is the boring case. Fixtures get written by hand, in a hurry. My guess is that the first bad inputs most people reach for are a path traversal and a malformed payload, and it is only a guess: I have not surveyed anyone.

The observable quantity is not the verdict. It is whether the verdict moves when the gate goes.

How do you test whether a guardrail is still working? Delete it and rerun

One mutant, pointed at a guardrail. The mutant is the deleted stage. If your suite does not notice, your suite is not testing that stage. Three outcomes per bad input:

class	with gate	gate removed	what it means
`LOAD_BEARING`	`DENY`	`ALLOW`	this input can detect a dead gate
`SHADOWED`	`DENY`	`DENY`	some other stage catches it; useless as a canary
`MISSED`	`ALLOW`	`ALLOW`	a hole; the gate did not cover it in either run

Good inputs are a separate control. A good input denied is a FALSE_POSITIVE, because a gate that rejects too much scores well on bad inputs alone. Two of the runs below do exactly that.

Run it, no keys, no network:

python3 make_corpus.py corpus.jsonl
python3 ablation_probe.py corpus.jsonl --order A --gate v2 --ablate policy_gate

The pipeline under test is in the same file: normalize (trims and lowercases), schema_check (types and required fields), resolve_path (collapses .. against the workspace root, string arithmetic only, no filesystem), path_allowlist, audit_log (a declared no-op), policy_gate, execute (a sink; nothing is executed). The gate refuses destructive SQL, tools outside the mandate, absolute paths outside /work/ws, and writes to .git, .env and .ssh inside it.

About the corpus, precisely, because the first version of this post overstated it: two of the six bad classes are the product of declared axes, destructive_sql at four verbs by two tables and protected_inside_workspace at three targets by two spellings, which is 14 of the 26 bad rows. The other four classes are lists of three that I wrote. The nine good rows are five column names crossed with one table, plus four tool-and-path pairs I wrote. So: partly enumerated, partly picked, and the split is 14 generated against 21 hand-written across the whole 35-row file.

The column axis in the good rows is doing real work, and it is the reason it exists. Four of those five column names contain a destructive verb as a substring: deleted_at, altered_at, dropped_by, truncated_at.

Three classes on the same corpus

corpus          corpus.jsonl  sha256 9ac269d75eb19cbd
                35 rows: 26 bad in 6 classes, 9 good
pipeline order  A
stages          normalize -> schema_check -> resolve_path -> path_allowlist -> audit_log -> policy_gate -> execute
gate variant    v2
ablated stage   policy_gate

note: the counts below are exhaustive over a declared corpus, not a sample
      drawn from one. No standard error is printed because nothing here is
      estimated; every row is a decision the pipeline actually made.

pre-flight
  determinism     two full ON passes byte-identical: yes
                  policy_gate invoked 61 times per pass. A flaw whose period divides
                  that number survives THIS check: both passes start in phase.
  order-independence  same rows visited back to front, same verdicts: yes
                  this is the one that catches a phase-aligned flaw, and it is
                  here because the check above let one through.
  transforms      resolve_path   (cannot be ablated: removing one changes the input downstream)
                  detected on THIS corpus, not declared: a transform with nothing
                  to transform here reads as pure. Scope of that claim is 35 rows.
  target purity   policy_gate left all 35 corpus rows byte-identical  OK

The note: block is printed by the tool, not added here. These counts are exhaustive over a declared corpus rather than a sample from one, so no standard error appears anywhere in the output. Nothing is estimated. Every row is a decision the pipeline made.

by bad class   (n = inputs in that class; survives = still DENY with policy_gate deleted)
bad_class                         n  classes        survives gate deletion
--------------------------------------------------------------------------------------------
destructive_sql                   8  LOAD_BEARING   0 of 8
escape_workspace_absolute         3  SHADOWED       3 of 3
escape_workspace_traversal        3  SHADOWED       3 of 3
protected_inside_workspace        6  LOAD_BEARING   0 of 6
tool_outside_mandate              3  LOAD_BEARING   0 of 3
schema_broken                     3  SHADOWED       3 of 3

summary
  LOAD_BEARING    17
  SHADOWED        9
  ok              9
verdict  GATE-LOAD-BEARING
exit 0
report-sha256 a09db437922eb50d1447e3e0b7db32d151c42df973b6a0d78d5755f2a726bade

Which of your bad fixtures can actually detect a dead gate?

This is the part I would run first on someone else's repo.

known-bad twin blindness
  with policy_gate deleted, 9 of 26 bad inputs are still DENIED by another stage.
  where that stage sits relative to policy_gate:  before it 9  ·  after it 0  ·  both 0
  'shadowed' does not mean 'caught earlier'. In this order it means caught at all.
  a twin built only from these classes stays 100% green with no gate at all:
    escape_workspace_absolute
    escape_workspace_traversal
    schema_broken
  that is 3 of 6 classes, 9 of 26 bad inputs.

Three of six. Path traversal, absolute-path escape, malformed payload.

The denominator here is mine: I declared six classes and 26 inputs. The split is not mine. Which class lands on which side falls out of where the stages sit.

A twin covering the whole corpus does catch deletion in this order: 9 of 26 still deny, so 17 flip to allow and the suite goes red. The blindness is a subset problem, and it bites whoever writes fixtures by hand rather than enumerating classes. I have never watched anyone enumerate them, which is not the same as saying nobody does.

Note the second line of that block, and read it in order B before you trust the word "shadowed":

  where that stage sits relative to policy_gate:  before it 2  ·  after it 6  ·  both 1

In order B the gate runs before path_allowlist, so 6 of the 9 masked inputs are masked by a stage that comes after the gate. Shadowing is not a fact about being earlier. It is a fact about being covered twice, anywhere in the chain.

Why does the first stage that says no lie about coverage?

Short-circuit the pipeline and you learn which stage rejected the input first. That is not the same as which stages would have. So the probe asks every stage in place, without short-circuiting, and builds a set.

stage census   (bad inputs each stage would deny, asked stage by stage, no short circuit)
  normalize       deny_set  0 of 26      first-denier  0 of 26
  schema_check    deny_set  3 of 26      first-denier  3 of 26
  resolve_path    deny_set  0 of 26      first-denier  0 of 26
  path_allowlist  deny_set  7 of 26      first-denier  6 of 26
  audit_log       deny_set  0 of 26      first-denier  0 of 26
  policy_gate     deny_set 23 of 26      first-denier 17 of 26   <- the stage under test
  execute         deny_set  0 of 26      first-denier  0 of 26
  coverage of policy_gate as a SET: 23 of 26 bad inputs. By first denier only: 17.
  Short-circuit attribution undercounts policy_gate by 6 input(s) in this order.

Six inputs where the gate would have said no and never got asked. In order B the same numbers are 14 and 14, undercount 0. That 6 is a fact about two things at once: about the ordering, and about how much of my corpus two stages cover jointly. It is the size of the two classes that path_allowlist and policy_gate both reject, 3 + 3. Change either and the number changes.

I take this one personally. My last tool, a static gate for unbounded LLM call sites, shipped with a bug of this shape: it took the largest caller instead of the sum over callers, printed a ceiling of $0.2496 for 8 runs, and executed 14. Same mistake wearing different clothes. Here the set is the primitive and denied_by is printed beside it, never instead of it.

What happens to the same gate in three different stage orders?

Order A canonicalises the path, then applies policy. Order B applies policy early, then canonicalises. Order C moves the gate exactly two positions earlier than A but keeps it after the canonicaliser. All three are legal arrangements of the same seven stages, and I am not claiming a distribution over which one teams pick. The gate source is byte-identical across them.

Order C is the one that matters, because it is the control I did not have in the first draft:

C_v2                               exit 0  deterministic

Its class map is identical to order A, line for line: 17 LOAD_BEARING, 9 SHADOWED, 0 MISSED. A genuine two-position move changes nothing. Checked against the recorded baseline from order A:

baseline comparison against baseline_A.json
  recorded under order A gate v2; this run order C gate v2
  no class changed against the recorded baseline

So position is not the variable. Crossing resolve_path is. Order B is the only one of the three where the gate is handed a path that has not been canonicalised yet, and there the map moves:

summary
  LOAD_BEARING    11
  SHADOWED        9
  MISSED          6
  ok              9
  MISSED classes are holes, not credit for the gate: protected_inside_workspace
verdict  GATE-LOAD-BEARING
exit 0

The mechanism is dull and old. The gate reasons about absolute paths, because after resolve_path there are no relative ones left. Placed before it, the gate is handed notes/../.git/hooks/pre-commit, sees a relative path, has no opinion, and passes it on. Then resolve_path turns it into /work/ws/.git/hooks/pre-commit, the allowlist sees a path inside the workspace and approves, and the write to .git/hooks/pre-commit goes through.

That precondition is written in the gate's docstring and printed in no report. Which is the actual lesson: a guardrail has preconditions, they are usually undeclared, and moving it across one is not a refactor.

Look at the verdict line. exit 0. Both orders.

My own exit code cannot tell A from B, because in order B there are still 11 load-bearing inputs and no false positive, which is all exit 0 asks for. What separates them is the recorded map:

  recorded under order A gate v2; this run order B gate v2
  CHANGED  protected_inside_workspace: LOAD_BEARING -> MISSED
  1 of 6 classes changed against the recorded baseline
verdict  GATE-NOT-PROVEN (class map moved against baseline)
exit 1

So the CI shape is --record once on the order you shipped and --expect on every pull request. To be exact about what that catches: it goes red when a change moves the class map, which order B does and order C does not. A reorder that changes nothing stays green, correctly.

Two ways to be wrong about a relative path

If the gate's blind spot is relative paths, tighten it. That is one line: refuse relative paths outright. I ran that as a third gate implementation, strict, across all three orders. Three orders by three gate variants, one corpus.

gate	order	LOAD_BEARING	MISSED	FALSE_POSITIVE	exit
`v2`	A	17	0	0	0
`v2`	B	11	6	0	0
`v2`	C	17	0	0	0
`strict`	A	17	0	0	0
`strict`	B	17	0	4	1
`strict`	C	17	0	0	0
`v1`	A	17	0	4	1
`v1`	B	11	6	4	1
`v1`	C	17	0	4	1

The first draft of this post read that strict/B row as a measured trade-off: the one variant I wrote that survives reordering pays for it on the good control. A reviewer took the four good rows whose paths I had written relatively, rewrote them in absolute form, changed nothing else, and the trade-off evaporated:

abs_B_strict                       exit 0  deterministic

LOAD_BEARING 17, SHADOWED 9, ok 9, FALSE_POSITIVE 0. Under an absolute calling convention strict is not a trade at all, it is strictly better than v2. So the honest statement is narrow: under the relative-path calling convention I chose for four good rows, strict rejects 4 of 9 legitimate requests in order B. That is a fact about my corpus, not about the gate.

What does survive the rewrite is the thing this post is actually about. On the absolute corpus, v2 in order B still loses the whole protected_inside_workspace class to MISSED and still exits 0. The MISSED result holds under both conventions; the trade-off held under one.

v1 and strict are also not two points on one scale of strictness. v1 is too broad about SQL substrings, strict is too broad about relative paths. Different axes, and I have swept one point on each, not a range.

v1 is the version I wrote first, matching destructive verbs as substrings rather than leading tokens. It scores the same 17 load-bearing inputs as the correct gate. What separates them is 3 hunks and 19 lines of a 105-line report: the variant label, four good rows, and the four summary lines those rows move.

good-02   FALSE_POSITIVE -                           DENY   ALLOW  policy_gate     {policy_gate}                   -   <- gate rejects legitimate traffic
good-03   FALSE_POSITIVE -                           DENY   ALLOW  policy_gate     {policy_gate}                   -   <- gate rejects legitimate traffic
good-04   FALSE_POSITIVE -                           DENY   ALLOW  policy_gate     {policy_gate}                   -   <- gate rejects legitimate traffic
good-05   FALSE_POSITIVE -                           DENY   ALLOW  policy_gate     {policy_gate}                   -   <- gate rejects legitimate traffic

SELECT deleted_at, SELECT altered_at, SELECT dropped_by, SELECT truncated_at. In the first draft this rested on a single row I had written by hand, already knowing about the substring delete inside deleted_at. Now it is a column axis, and four of the five columns trip it. A load-bearing count is not a quality score: without the good control, a broken gate and a correct gate were indistinguishable on this corpus.

Where the probe refuses to answer

Fourteen scenarios, run by run_all.sh on every invocation. Twelve exit 2, two exit 1 for reasons given below.

ref_transform                      exit 2  deterministic
ref_normalize_noop                 exit 1  deterministic
ref_normalize_real                 exit 2  deterministic
ref_flaky                          exit 2  deterministic
ref_flaky_aligned                  exit 2  deterministic
ref_nogood                         exit 2  deterministic
ref_nobad                          exit 2  deterministic
ref_brokenjson                     exit 2  deterministic
ref_unknownstage                   exit 2  deterministic
ref_othercorpus                    exit 2  deterministic
ref_badutf8                        exit 2  deterministic
ref_recordexpect                   exit 2  deterministic
ref_otherablate                    exit 2  deterministic
ref_recordonfail                   exit 1  deterministic

The one that matters most:

CANNOT ATTRIBUTE: removing resolve_path changed the input downstream
  this is not 'the stage is not needed', it is 'the experiment is impossible'

Ablate a stage that rewrites the request and the two runs are no longer comparable: the second one is running on different data. The probe detects that itself by comparing the canonical JSON before and after each stage, then refuses. A tool that answered "not load-bearing" here would be telling you to delete your path canonicaliser.

And the good-control refusal:

CANNOT ATTRIBUTE: corpus has no good inputs
  without a good control a gate that denies everything looks perfect

Exit 2 beats exit 1 beats exit 0. Across the fourteen refusal scenarios the runner exercises, none returned 0.

Five things my own tool got wrong

Two I found. Three came out of a pre-publication review that reran everything and went looking for the seams.

1. The determinism check is blind at one period, and it shipped a false green. First line of the pre-flight, and the guard I trusted most. The flaky gate variant returns ALLOW on every Nth invocation, standing in for a gate that reads a clock. policy_gate is invoked 61 times per pass on this corpus. Set the period to 61:

  determinism     two full ON passes byte-identical: yes

Yes. It said yes, because the second pass started at exactly the phase the first one did. For a while that run went all the way to verdict GATE-LOAD-BEARING, exit 0 on a gate that is not deterministic, which is precisely the failure this post is about, in the tool that detects it. The fix is not a different modulus, it is a different question: run the same rows back to front and compare verdicts keyed by input.

  order-independence  same rows visited back to front, same verdicts: NO
CANNOT ATTRIBUTE: the pipeline depends on the order its inputs are visited in

I left the blind check in and printed the invocation count next to it, so you can see when it is going to be useless.

2. The purity check is empirical, and I over-read it. --ablate normalize returns exit 1 on my main corpus, not the refusal I expected. normalize trims whitespace and lowercases the tool name, and the generated corpus is already trimmed and lowercase, so it changes nothing and really is an identity here. Rewrite the good rows with " FS.WRITE " and a padded path and the same command refuses with exit 2. Two corpora, two answers, both correct, and the claim "this stage is pure" is scoped to the rows you fed it.

3. "Shadowed" did not mean what the tool printed. The report said masked inputs were "still DENIED by an earlier stage". That word was a constant in a format string, not a measurement. In order B, six of the nine masked inputs are caught by path_allowlist, which runs after the gate. The classification was right and the explanation was wrong, in the order whose numbers this post publishes. The tool now computes the masking stages and prints each one's position.

4. --record and --expect in one command was green by construction. The baseline was written before the comparison, so a CI job written as a single record-and-check command agreed with itself by construction. Worse, a failing run wrote its baseline anyway, so a broken gate could be frozen as the reference and the pull request that fixed it would go red. Both are refusals now, and the baseline is written only after a clean verdict. A baseline recorded for a different corpus or a different ablation target is also refused; a different order or gate is not, because that is the signal.

5. A corpus that is not valid UTF-8 crashed instead of refusing. The decode sat outside the try, so invalid bytes produced a traceback and exit 1, which in CI reads as "gate not proven" rather than "I cannot answer". One line moved.

What this is not

It is not a replacement for the known-bad twin. The twin checks that the verdict is right; this checks who produced it. Run both.

It does not catch a gate turned off by config in production while it is on in CI. Different problem, and the probe cannot see your production config from a test process.

It is not mutation testing in general. One mutant, chosen by you, on one stage.

It is not a drop-in for your repo, and the first draft of this post promised otherwise. The stage list and the two-run pattern are hardcoded; what ports is the mechanism, and the mechanism is 21 lines: run the chain short-circuited, run it again asking every stage in place, do both with the target replaced by an identity, and compare. Rewriting those against your own stage list is an afternoon, not a download.

It says nothing about inputs outside the corpus you hand it. Every classification count above describes 35 declared rows, and the MISSED class exists precisely because holes get found by inputs nobody added.

It is not a static check. Neighbouring pieces cover the neighbouring questions: the green checkmark auditor reads test files with ast and never runs them; checkpoint skip gate is about a required check that never ran at all in a recorded trajectory. Here the gate ran, said DENY, and the verdict did not depend on it. The root of the series is still tracking is not control: gate before the action, not a dashboard after it. This adds one line to that: fired is not load-bearing.

The pipeline and both corpora are synthetic. I wrote them, and three of the five defects above were found by someone rerunning them and disagreeing with my prose. What is not synthetic is the mechanism.

If you run this on your own guardrail suite, I want one number from you: with the gate cut out, how many of your known-bad fixtures still fail, and which stage catches them instead. Say whether that stage runs before your gate or after it, because I got that wrong for a day and the answer surprised me. Follow along for the next runner and its numbers.

Unbounded LLM Call Sites: A Budget Is Not a Ceiling

Alexey Spinov — Tue, 08 Sep 2026 07:48:15 +0000

Unbounded LLM call sites are model calls whose repeat count is not in the source, so a per-call budget does not multiply into a ceiling. loop_bound_gate.py reads a Python tree with ast, offline and keyless, and answers one question per call site: BOUNDED with a dollar ceiling, UNBOUNDED with a named structural reason, or CANNOT ANALYZE.

A per-call budget is a multiplicand. The bill is that number times how many times the line runs. On the fixture below the budget stays fixed at $0.0312 in every column, and one call site spends $0.0936 on one input and $1.2480 on another. Same code. Same budget. Thirteen times the money, because I handed it thirteen times the rows.

So the question a budget never asks is the one that decides the bill: how many times can this line run? For a lot of call sites, that number is written in the code. For a lot of others it does not exist there at all, and the honest answer is a refusal rather than a comforting figure.

loop_bound_gate.py never imports or runs the code it reads.

AI disclosure. I wrote loop_bound_gate.py, loop_bound_counter.py and loop_bound_check.py with AI assistance and ran them myself before publishing: offline, standard library only, no network, no keys, no funds. Every output block below is pasted from a real run on Python 3.13.5. The runner executes 29 scenarios, each three times, and byte-compares the copies: it reported 29 deterministic, 0 not. Code sha256: loop_bound_gate.py b593cc34…69c9, loop_bound_counter.py b7d84407…3af8, loop_bound_check.py 4755d232…1263, run_all.sh 4b5f58ce…30d3. Where an output block is trimmed, [...] marks the lines I dropped from that same run; nothing inside a block is reworded. The fixtures under fixtures/ are synthetic, written by me for this post, and I say so again where they appear. The one piece of real code here is urllib3 2.7.0, and I checked my local copy against the upstream tag byte for byte before quoting line numbers from it. The per-call budget $0.0312 in the fixture comments is an input, not a price I am claiming about any vendor; put your own number there. I went looking for primary sources on the widely-quoted runaway-agent bills before writing this and could not confirm a single one, so not one of those numbers is in this post. Four bugs of my own die further down the page. Two of them I found; the third and fourth were found by a pre-publication review of this very draft, after I had already written that the tool was correct, and the fixture that caught the worst of them is that reviewer's, not mine.

In short:

Two printed ceilings of mine turned out to be exceedable, and neither miss was a fixed multiple. On the first, the overshoot is exactly the number of input rows: 3.00x at 3 rows, 40.00x at 40, 1000.00x at 1000, against the same printed $0.1560. The size of the error is the size of your input, which is another way of saying there was no ceiling.
That failure is not exotic. The call site is for attempt in range(5), five calls, obviously bounded. The function containing it is called from for row in rows in a different file.
The second one survived the first fix. A function reached from two bounded loops runs the sum of them; my code took the larger. Printed $0.2496 for 8 runs, executed 14 for $0.4368, and returned exit 0. A false green in a pre-merge gate is the worst output this tool can produce, and it was mine for a day.
The fix is a refusal, not a smarter number. The gate now adds up every call edge it can resolve, inherits an unbounded caller's reason, and where callers do not resolve at all prints UNBOUNDED (unresolved caller). After both fixes: 8 of 8 ceiling checks held, against 8 of 12 for the one-function baseline.
The price of that refusal is measured. The one-function baseline priced 6 of 10 files, the whole-tree gate prices 4 of 10. Two files out of ten trade a dollar figure for a reason.
The flag is not free signal. Of 6 files marked UNBOUNDED, 3 ran an identical number of calls on both datasets. It cries wolf on a while n < 5 loop you can bound by eye, and I left that fixture in the corpus rather than tuning it away.
Every positive result here rests on one config line. Delete "entrypoints": ["*:main"] and the same corpus prices 0 call sites instead of 4 files: fixtures/repo goes BOUNDED 2 to BOUNDED 0, fixtures/clean goes BOUNDED 2 to BOUNDED 0.
On real code, urllib3 2.7.0, watching self.urlopen: 6 call sites, 0 bounded, 4 recursion, 2 unresolved caller. Retry-by-recursion, bounded at runtime by a retries object, not by anything in the structure. Zero dollars printed, because nobody declared a budget and the tool refuses to invent one.
My first alias rule flagged 12 places in requests 2.34.2 as unanalysable. After two bug fixes: 1. Both broken rules are still reachable behind flags, so 12 and 8-against-14 are numbers you can re-measure rather than take from me.

What does "bounded" mean for an LLM call site?

A repeat count is a fact about structure. for i in range(3) runs three times whatever the input is. for row in rows runs len(rows) times, and len(rows) is not in the file. Nest them and the counts multiply.

So there is a small set of shapes the gate can turn into an integer: a for over a literal list or tuple, over range() with literal integer arguments, over a module-level constant sequence that is assigned once and never mutated, and the product of nested such loops. Everything else gets a name instead of a number: a data-driven iterable, a while True, a while whose condition is assigned from the model's own response, recursion, an unresolved caller.

The ceiling is then the declared budget times that integer. The budget comes from a # budget: 0.0312 comment on or above the call site. No comment, no dollars: the verdict is NO BUDGET and the exit code is 2. A default price would be the worst possible feature here, because a made-up number reads exactly like a measured one.

Run it in 60 seconds, no keys

python3 loop_bound_gate.py fixtures/repo --config config.json --max-spend 1.00

config.json is two fields: which dotted names count as a model call, and which functions are entry points that run once per program.

{
  "model_calls": ["client.messages.create"],
  "entrypoints": ["*:main"]
}

The entry point list looks like boilerplate. It is not. A function nobody calls inside the tree you handed over is a function whose repeat count you did not give the tool. It says so rather than assuming one.

Where are the unbounded LLM call sites in a synthetic repo?

fixtures/repo is six synthetic files I wrote for this post. Nothing in it is production code, and the dollar figures below are the declared budget times a repeat count, not anyone's bill.

$ python3 loop_bound_gate.py fixtures/repo --config config.json --max-spend 1.00
loop_bound_gate 1.0.0  root=fixtures/repo  python=3.13.5
model calls watched: client.messages.create
entry points declared: *:main
mode: whole-tree (callers resolved)

CALL SITE                          VERDICT         CEILING / REASON
------------------------------------------------------------------------------
digest.py:13                       BOUNDED         $0.0936  = $0.0312 x 3 run(s)
digest.py:17                       BOUNDED         $0.1872  = $0.0312 x 6 run(s)
planner.py:13                      UNBOUNDED       while condition set by the model's own output
refiner.py:9                       UNBOUNDED       recursion (refine calls itself at refiner.py:11)
triage.py:12                       UNBOUNDED       for over a data-driven iterable
worker.py:10                       UNBOUNDED       for over a data-driven iterable at batch.py:12 (inside main)
------------------------------------------------------------------------------

call sites found: 6   BOUNDED 2   UNBOUNDED 4   CANNOT ANALYZE 0   NO BUDGET 0
  no ceiling because: for over a data-driven iterable                            2
  no ceiling because: recursion                                                  1
  no ceiling because: while condition set by the model's own output              1
worst case allowed by the code: NOT FINITE (4 call site(s) have no ceiling, so there is no sum to report)
exit 1

digest.py:17 sits in a for tone in ["short", "long"] nested inside a for section in SECTIONS, where SECTIONS is a three-element tuple at module level. Three times two is six, and $0.0312 x 6 is $0.1872.

planner.py:13 is the shape this whole post is about:

def main(client, data):
    done = False
    steps = 0
    while not done:
        # budget: 0.0312
        resp = client.messages.create(model="m", prompt="step %d" % steps)
        done = resp.stop
        steps += 1
    return steps

The gate does a small taint walk here: done is assigned from a call it is watching, done appears in the loop test, so the reason is not the generic "condition not statically bounded" but the specific one. That distinction is the whole shape of an agent loop. The exit condition is an output of the thing you are paying for.

Then the counting replay, which is a separate program. It imports each fixture with a stub client that increments a counter and returns canned text, runs it under two datasets, and holds the per-call budget fixed:

$ python3 loop_bound_counter.py
dataset A = 3 rows / 1 item x 1 pass      dataset B = 40 rows / 2 items x 3 passes
per-call budget held constant at $0.0312 in both columns

CALL SITE                                   obs A    spent A    obs B    spent B      B/A
--------------------------------------------------------------------------------------------
digest.py:13+17   fixed passes                  9    $0.2808        9    $0.2808    1.00x
triage.py:12      one call per row              3    $0.0936       40    $1.2480   13.33x
planner.py:13     model says when to stop        4    $0.1248        4    $0.1248    1.00x
refiner.py:9      retry by recursion            2    $0.0624        8    $0.2496    4.00x
worker.py:10      ATTACK-1                     15    $0.4680      200    $6.2400   13.33x
multi.py:12      ATTACK-2                      14    $0.4368       14    $0.4368    1.00x
[...]
--------------------------------------------------------------------------------------------
call sites whose observed count changed with the data: 4 of 12
total observed calls: A 186 ($5.8032)   B 451 ($14.0712)

triage.py:12 is the sentence at the top of this post, with its absolutes next to the ratio: 3 calls and $0.0936 on dataset A, 40 calls and $1.2480 on dataset B, and $0.0312 per call in both columns. The 13.33x in that row is 40 over 3, the two dataset sizes I picked, so read the column as a demonstration that the spend tracks the input rather than as a measurement of anything. The budget was never wrong. It was never a ceiling either.

The control matters as much. digest.py is 9 on both datasets, and on a size sweep of 1, 3, 10, 40 and 100 rows it is 9 five times out of five while triage.py tracks the row count exactly. A gate that flagged everything would be useless, and this is the run that says it does not.

ATTACK-1: how far off was my ceiling?

Here is worker.py, minus its docstring. Read it and price it.

def summarize(client, row):
    for attempt in range(5):
        # budget: 0.0312
        client.messages.create(model="m", prompt="%s try %d" % (row, attempt))

Five iterations, $0.0312 each, ceiling $0.1560. That is what my first version printed, and it is what --naive still prints on demand. The problem is in another file that contains no model call at all, which is exactly why a gate that prices a call site from its own function does not look there:

from worker import summarize

def main(client, data):
    for row in data["rows"]:
        summarize(client, row)

loop_bound_check.py puts the printed ceiling against the dollars the replay actually spent, one row per file, one question per row: observed <= ceiling.

$ python3 loop_bound_check.py --naive
gate mode: NAIVE (call site's own function only)
per-call budget: $0.0312   datasets: A = 3 rows, B = 40 rows

FILE              CEILING      spent A      spent B     HOLDS?
--------------------------------------------------------------
digest.py         $0.2808      $0.2808      $0.2808        yes
report.py         $0.0936      $0.0936      $0.0936        yes
sweep.py          $3.7440      $3.7440      $3.7440        yes
worker.py         $0.1560      $0.4680      $6.2400         NO
               overshoot: $6.2400 against a printed ceiling of $0.1560 = 40.00x
multi.py          $0.0624      $0.4368      $0.4368         NO
               overshoot: $0.4368 against a printed ceiling of $0.0624 = 7.00x
refiner.py      UNBOUNDED      $0.0624      $0.2496        n/a
exported.py       $0.1248      $0.1248      $0.1248        yes
[...]
--------------------------------------------------------------
files the gate priced: 6 of 10
ceiling checks: 8 of 12 held
VERDICT: 4 printed ceiling(s) were exceeded. A ceiling that can be exceeded is not a ceiling.

One line in that block is there because the same review caught me stacking the deck. refiner.py recurses into itself inside the very function the naive mode reads, and my naive mode was not checking for that at all: the recursion test lived only in the whole-tree path. So the baseline I was comparing against was doing less work than a one-function gate honestly can, and two of its four failures were mine, not the method's. A gate reading one function can see that function call itself in three lines of code, so now it does. refiner.py reads UNBOUNDED in both modes above, and the failures that remain are worker.py and multi.py, the two files where the count genuinely lives somewhere else.

Now the part I got wrong when I first wrote this page. I led with 40.00x as though it were a finding. It is not. worker.py runs five calls per row, the naive ceiling is those five calls, so the overshoot works out to (rows x 5) / 5, which is rows and nothing else. I picked 40. Here is the same run with the row count as the only moving part:

$ python3 loop_bound_check.py --overshoot
worker.py, naive ceiling held constant at $0.1560, only the row count moves
    rows    obs calls        spent  naive ceiling    overshoot
--------------------------------------------------------------
       3           15      $0.4680        $0.1560        3.00x
      10           50      $1.5600        $0.1560       10.00x
      40          200      $6.2400        $0.1560       40.00x
     100          500     $15.6000        $0.1560      100.00x
    1000         5000    $156.0000        $0.1560     1000.00x
--------------------------------------------------------------
overshoot equals the row count, exactly, at every size tried.
The miss is not a constant. It is whatever the input is.

That is a better result than the one I was claiming. A fixed 40x would be a bounded error you could pad around. An error equal to the input means no multiple saves you, which is the argument of this whole post arriving through the back door. A number in the headline was hiding it.

The repair to the gate is not a cleverer estimate. It is a refusal:

$ python3 loop_bound_check.py
gate mode: whole-tree (callers resolved)
per-call budget: $0.0312   datasets: A = 3 rows, B = 40 rows

FILE              CEILING      spent A      spent B     HOLDS?
--------------------------------------------------------------
digest.py         $0.2808      $0.2808      $0.2808        yes
report.py         $0.0936      $0.0936      $0.0936        yes
sweep.py          $3.7440      $3.7440      $3.7440        yes
worker.py       UNBOUNDED      $0.4680      $6.2400        n/a
multi.py          $0.4368      $0.4368      $0.4368        yes
[...]
--------------------------------------------------------------
files the gate priced: 4 of 10
ceiling checks: 8 of 8 held
VERDICT: every printed ceiling held on both datasets.

8 of 8 held. Read that denominator carefully, because it is not the denominator of the naive 8 of 12: the checker only counts a file it was handed a price for. So 8 of 8 is the 4 files this mode still prices, twice each, and 8 of 12 is the 6 files the naive mode prices, twice each. Fewer files, not a better hit rate on the same ones. And this is a sweep, not a theorem: this corpus, these inputs, these verdicts.

Correctness cost two files out of ten their dollar figure, 6 of 10 priced down to 4 of 10. A tool that prints fewer numbers is a harder sell and a better answer.

ATTACK-2: the same bug again, in the fixed version

I would have published the section above as the end of the story. A pre-publication review of this draft went looking for a second instance of the same class and found one, in the code I had just called correct.

def helper(client, tag):
    for i in range(2):
        # budget: 0.0312
        client.messages.create(model="m", prompt="TAG %d" % i)

def main(client, data):
    for _a in range(3):
        helper(client, "first")
    for _b in range(4):
        helper(client, "second")

Nothing data-driven anywhere. Two loops, both literal, (3 x 2) + (4 x 2) = 14 calls. My program_count() walked both call edges and kept the larger instead of adding them:

$ python3 loop_bound_gate.py fixtures/multi --config config.json --max-spend 1.00 --v1-max-callers
[...]

CALL SITE                          VERDICT         CEILING / REASON
------------------------------------------------------------------------------
multi.py:12                        BOUNDED         $0.2496  = $0.0312 x 8 run(s)
                                                   via helper <- main
------------------------------------------------------------------------------

call sites found: 1   BOUNDED 1   UNBOUNDED 0   CANNOT ANALYZE 0   NO BUDGET 0
worst case allowed by the code: $0.2496
exit 0

$0.2496, 8 runs, exit 0. Green in CI, while the replay spends $0.4368. Same class as ATTACK-1, one layer up, and this time in the version I had already declared fixed after being burned once.

The correction is two lines, sum instead of max, and the flag above stays in so the wrong answer can be re-measured:

$ python3 loop_bound_gate.py fixtures/multi --config config.json --max-spend 1.00
[...]

CALL SITE                          VERDICT         CEILING / REASON
------------------------------------------------------------------------------
multi.py:12                        BOUNDED         $0.4368  = $0.0312 x 14 run(s)
                                                   via helper <- main
------------------------------------------------------------------------------

call sites found: 1   BOUNDED 1   UNBOUNDED 0   CANNOT ANALYZE 0   NO BUDGET 0
worst case allowed by the code: $0.4368
exit 0

Exactly the 14 the counter ran. What bothers me is not the bug. It is that my ceiling check reported a clean sweep before this fixture existed, because no file in my corpus had a function with two callers. The check was honest and the corpus was thin. Those two look identical from the outside, and the only thing that told them apart was somebody else writing a file I had not thought of.

Does the flag separate anything?

A gate that marks everything unbounded would post the same 8 of 8 and be worthless, for the same reason exit 0 makes a bad success denominator. So the replay checks the other direction: for each file, did the observed count actually move when the data changed?

$ python3 loop_bound_check.py --separation
gate verdict against what the replay actually did, per file
FILE         VERDICT           obs A    obs B   CHANGED?
----------------------------------------------------------
digest.py    BOUNDED               9        9         no
report.py    BOUNDED               3        3         no
sweep.py     BOUNDED             120      120         no
worker.py    UNBOUNDED            15      200        yes
multi.py     BOUNDED              14       14         no
refiner.py   UNBOUNDED             2        8        yes
exported.py  UNBOUNDED             4        4         no
poller.py    UNBOUNDED             5        5         no
triage.py    UNBOUNDED             3       40        yes
planner.py   UNBOUNDED             4        4         no
----------------------------------------------------------
BOUNDED files:   4, of which the observed count changed with the data: 0
UNBOUNDED files: 6, of which the observed count changed with the data: 3

A flag that changed nothing on 3 of 6 files is a real cost, not a rounding error.
A BOUNDED verdict that varied even once would have been a false ceiling: 0 of 4 did.

Half the flags moved, half did not. Be careful with that ratio, though: I wrote this corpus, so 6 flagged and 4 clean is a composition I chose, not a false-alarm rate I measured. Add one more data-driven file and it reads 3 of 7. What the run does establish is that the verdict is not constant in either direction, and that no file the gate priced ever moved.

poller.py is the honest embarrassment:

def main(client, data):
    n = 0
    while n < 5:
        # budget: 0.0312
        client.messages.create(model="m", prompt="poll %d" % n)
        n += 1
    return n

Five, obviously. The gate says while condition not statically bounded and refuses to price it. I could special-case a counter variable with a literal comparison and a matching increment, and then somebody writes n += step, and I am writing an interpreter. I left it flagged.

planner.py is a subtler entry in that column. It ran 4 calls on both datasets, but only because my stub returned stop on the same turn both times. That equality is a property of my stub, not of the code. In the source there is still no repeat count, and there would not be one on your machine either.

What happens on real code? urllib3 2.7.0

Fixtures show a tool does what its author meant. Real code decides whether the idea survives contact. I pointed the gate at urllib3 2.7.0 as installed on this machine, watching self.urlopen. My local copy is byte-identical to the upstream tag, sha256 b0616775d5d8c25c7b282e0908fd602af74d18b34af984c22437460021a3dd8f for connectionpool.py, so the line numbers below are clickable: connectionpool.py at 2.7.0.

$ python3 loop_bound_gate.py realcode/urllib3_full --config realcode_config.json
loop_bound_gate 1.0.0  root=realcode/urllib3_full  python=3.13.5
model calls watched: self.urlopen
entry points declared: (none)
mode: whole-tree (callers resolved)

CALL SITE                          VERDICT         CEILING / REASON
------------------------------------------------------------------------------
_request_methods.py:182            UNBOUNDED       unresolved caller (request() is also reached from __init__.py:193, which does not resolve to a function in this tree)
_request_methods.py:278            UNBOUNDED       unresolved caller (request() is also reached from __init__.py:193, which does not resolve to a function in this tree)
connectionpool.py:872              UNBOUNDED       recursion (HTTPConnectionPool.urlopen calls itself at connectionpool.py:872, connectionpool.py:923, connectionpool.py:955)
connectionpool.py:923              UNBOUNDED       recursion (HTTPConnectionPool.urlopen calls itself at connectionpool.py:872, connectionpool.py:923, connectionpool.py:955)
connectionpool.py:955              UNBOUNDED       recursion (HTTPConnectionPool.urlopen calls itself at connectionpool.py:872, connectionpool.py:923, connectionpool.py:955)
poolmanager.py:503                 UNBOUNDED       recursion (PoolManager.urlopen calls itself at poolmanager.py:503)
------------------------------------------------------------------------------

call sites found: 6   BOUNDED 0   UNBOUNDED 6   CANNOT ANALYZE 0   NO BUDGET 0
  no ceiling because: recursion                                                  4
  no ceiling because: unresolved caller                                          2
worst case allowed by the code: NOT FINITE (6 call site(s) have no ceiling, so there is no sum to report)
exit 1

Four of six are return self.urlopen(...) inside urlopen itself: retry after a broken connection, redirect, retry. Mature, heavily reviewed, correct code. Its retry limit is real, and it lives in a retries object threaded through the call at runtime. It is not in the structure, so a reader of the structure cannot see it.

The other two rows are a defect in my tool, and I had them diagnosed wrong on this page until review. I wrote that request() was reached through an __init__.py re-export. It is not. urllib3/__init__.py builds a module-level instance and wraps it in an ordinary function:

_DEFAULT_POOL = PoolManager()

def request(method, url, *, ...):
    return _DEFAULT_POOL.request(method, url, ...)

My local_instances() only recognises x = SomeClass() written inside the function doing the calling, so a module-level instance never resolves. The name request then lands in the unresolved pile, and the rule that matches on a bare name runs before the edges that did resolve, throwing them away with it. In requests 2.34.2 that single unresolved session.request(...) in api.py discards all seven real edges from get, options, head, post, put, patch and delete. It is not a hard problem, it is a check in the wrong order, and it is on my list rather than in this release.

That is the shape of the finding I would take away from this whole exercise. "No ceiling in the source" and "no ceiling" are different sentences. urllib3 is the good case: the bound exists, it is just somewhere this tool does not look. Your agent loop may be the other case. The gate tells you which question you are now holding, not which answer.

No dollars appear in that block. Nobody wrote a # budget: comment in urllib3, and the tool will not invent one for somebody else's code.

The other two bugs I shipped, and what they cost

My first alias rule was: if a watched name appears anywhere outside a call position, or if getattr is called on anything that prefixes a watched name, the module is unanalysable. It sounded conservative. On requests 2.34.2 with a deliberately sloppy watch list it produced this:

call sites found: 10   BOUNDED 0   UNBOUNDED 0   CANNOT ANALYZE 12   NO BUDGET 0

Twelve refusals, and requests never sees an LLM. Two more separate mistakes, both mine:

self.request = None is not a binding. I was matching ast.Attribute nodes without looking at the context. An assignment target carries ast.Store; a callable being squirrelled away for later carries ast.Load. I was counting both.

getattr(self, attr, None) in __getstate__ is not dispatch. It is pickling. I now flag getattr only when the second argument is a string literal that completes a watched name; a computed attribute name gets a NOTE line that changes no verdict and no exit code.

After both fixes, same package, same sloppy config:

call sites found: 10   BOUNDED 0   UNBOUNDED 10   CANNOT ANALYZE 1   NO BUDGET 0

From 12 to 1 on requests, and from 7 to 1 on urllib3 with that same watch list. Ten call sites on requests went from "I refuse" to an actual verdict, and six on urllib3. The old rule is still reachable as --v1-alias-rule, so the 12 above is a number you can re-measure rather than one you have to take from me.

The remaining 1 is my fault too, of a different kind: I put self.request in the watch list, and in requests that name is an attribute holding a Request object, not a method. Bad watch list, correct refusal.

The config line everything positive rests on

Earlier I said the entry point list is not boilerplate. Here is the price with a number on it. Delete "entrypoints": ["*:main"] and run the same corpus:

$ python3 loop_bound_gate.py fixtures/repo --config config_noentry.json --max-spend 1.00
loop_bound_gate 1.0.0  root=fixtures/repo  python=3.13.5
model calls watched: client.messages.create
entry points declared: (none)
mode: whole-tree (callers resolved)

CALL SITE                          VERDICT         CEILING / REASON
------------------------------------------------------------------------------
[...]
triage.py:12                       UNBOUNDED       for over a data-driven iterable
worker.py:10                       UNBOUNDED       for over a data-driven iterable at batch.py:12 (inside main)
------------------------------------------------------------------------------

call sites found: 6   BOUNDED 0   UNBOUNDED 6   CANNOT ANALYZE 0   NO BUDGET 0
  no ceiling because: for over a data-driven iterable                            2
  no ceiling because: unresolved caller                                          2
  no ceiling because: recursion                                                  1
  no ceiling because: while condition set by the model's own output              1
worst case allowed by the code: NOT FINITE (6 call site(s) have no ceiling, so there is no sum to report)
exit 1

BOUNDED 2 becomes BOUNDED 0. The same thing happens to fixtures/clean, BOUNDED 2 to BOUNDED 0. Every positive result on this page, the ceilings, the 8 of 8, the separation table, exists because I told the tool which functions run once. That is correct behaviour, and it is also the honest reading of every green number here: they are conditional on a declaration I supplied.

Which made the next thing worse than it looks. Write that key as a string instead of a list and the false ceilings come straight back, in silence: list("*:main") shreds it into characters, * matches every function, and worker.py is priced at $0.1560 again. Nothing warned me. There is a type check now:

$ python3 loop_bound_gate.py fixtures/repo --config config_typo.json --max-spend 1.00
loop_bound_gate: ConfigError: 'entrypoints' must be a list, got str

What this measures, and what it does not

It reads structure. It does not solve the halting problem and I am not claiming it finds every unbounded loop: fixtures/dynamic is one it does not find by the direct route.

HANDLERS = {}

def register(client):
    HANDLERS["summarize"] = client.messages.create

def main(client, data):
    register(client)
    fn = HANDLERS["summarize"]
    for row in data["rows"]:
        fn(model="m", prompt=row)

Zero direct call sites. Grep finds nothing. The replay says that file made 3 calls on dataset A and 40 on dataset B. A silent green here would be the most expensive output the tool could produce, so a module that mentions a watched call and yields zero call sites is reported, not skipped:

$ python3 loop_bound_gate.py fixtures/dynamic --config config.json
[...]
dispatch.py:11                     CANNOT ANALYZE  0 direct call sites, but client.messages.create bound to a name instead of called

call sites found: 0   BOUNDED 0   UNBOUNDED 0   CANNOT ANALYZE 1   NO BUDGET 0
worst case allowed by the code: UNKNOWN (1 place(s) reach the model through something this tool will not guess at)
exit 2

Exit 2, and 2 outranks 1. "I cannot answer" is a worse state for a gate than "I found something bad", because the second one you act on.

A few more things it is not. It does not execute anything, so a bound your code enforces at runtime is invisible to it, which is exactly the urllib3 result. It does not read a bill or a trace, which is where the after-the-fact question lives: how much of a bill delta a dated export can separate. It does not stop a running loop: that is the job of a runtime spend guard, which halts an action before it fires, and which also has to keep counting to keep working. These are different questions asked with different information, and this post is not arguing about which one comes first. A guard protects the call sites you remembered to wrap. This finds the ones you did not, and the ones where no wrapping produces a ceiling because there is no repeat count to multiply.

It also does not disagree with the arithmetic in what a per-call quote leaves out or why a per-call cap is not enough. Those measure a run that happened. This asks whether the source contains an upper bound at all, which is a question you can ask of a file that has never run.

Three more limits worth knowing before you point it at a repository. One file it cannot parse ends the whole scan: the SyntaxError is caught at the top level, you get exit 2 and no results for any other file. There is no directory filter at all, so .venv/ and node_modules/ are fair game unless you hand it a narrower path. And the entry point declaration is taken on trust: an entry point is a function you promise runs once, so if you declare one that re-enters itself, directly or through a helper that calls back into it, the tool believes you and prices it at one run. I get BOUNDED $0.0312 and exit 0 on a main that recurses ten deep and spends $0.3120. The tool can see that self-edge and ought to refuse; it does not yet. All three are on the list; none is fixed here.

Reproduction: ./run_all.sh runs 29 scenarios three times each and byte-compares them. Last run: 29 deterministic, 0 not, Python 3.13.5. --selftest runs 27 assertions on money parsing, trapped rounding, config typing and each verdict class, including both halves of ATTACK-1 and both arithmetics of ATTACK-2; it reported 27 checks, 0 failed.

The part I am unsure about

The unresolved caller verdict is doing a lot of work, and right now I cannot tell how much of it is the world and how much is me. In this corpus it fires on 1 of 10 files, which tells me nothing. On urllib3 it fires on 2 of 6 sites, and both of those are the ordering defect I described above, not a property of urllib3.

That defect has to go before the question is even askable. Until the bare-name rule stops discarding edges that did resolve, and until a module-level instance resolves like a local one, any count of unresolved caller on a real repository is mostly a measurement of my bug. My whole fixture corpus is free functions with no classes at all, which is exactly the one call shape the resolver handles, and on real code full of classes it produced BOUNDED 0 of 6 and BOUNDED 0 of 16. I built the corpus that flattered the tool without noticing.

What I actually do not know sits behind that: on a service where functions are reached through a framework, a decorator or a DI container, is there any entry point declaration short enough to be worth writing? The globs are my current answer and I do not believe in them. If you have a repository shaped like that, the thing I would want to hear is not a ratio from this version of the tool, but whether declaring your real entry points is a ten-line job or a hopeless one. That is what decides whether this is a gate or a curiosity.

I publish one runnable tool per post, with the run pasted in and the failures left where they happened, including the two that a reviewer found after I had called this one correct. Follow if that is your kind of thing, and tell me in the comments how many entry points your repo would actually need declared. I read every one.

Rate vs Volume Attribution: $40.80 Disputed, or $0.00

Alexey Spinov — Mon, 07 Sep 2026 07:40:09 +0000

Rate vs volume attribution: one model line, one month, a $54.50 increase. The two-snapshot decomposition most cost reviews start from calls $40.80 of that disputed, 74.8% of the delta, the rate effect landing anywhere between $1.70 and $42.50 depending on an ordering nobody observed. The dated rows in the same export say $0.00 disputed, rate effect $1.70.

Both numbers are exact. Nothing about the bill changed between them. What changed is the bucket width I asked the question at.

That $0.00 is this export, where the volume in the bucket that repriced happens to sit flat. Give that bucket a wobble and the disputed money comes back as a residual, and the size of the residual is set by the volume the line was running when the price moved, not by the width. I measured that too. It is the part of this post I would read if I only read one part.

AI disclosure. I wrote resolution_gate.py with AI assistance and ran it myself before publishing: offline, standard library only, no network, no keys, no funds. It reads your export and writes nothing outside its own directory: its fixtures and output files land there, and --selftest and --mutants put their deliberately broken inputs in a throwaway mkdtemp sandbox that goes away with the run. Every output block below is pasted from a real run on Python 3.13.5. The runner executes 17 scenarios, each one three times, and byte-compares the three copies: it reported 17 deterministic, 0 not and exited 0. Code sha256: resolution_gate.py e6b00f2a…d7e2, resolution_limits.py 9f1ed6f5…0f50, resolution_counter.py 73f25fd3…f256, make_fixtures.py 32b715f4…f201, snippet.py a7a688c7…adbd. STDOUT sha256 of the blocks I lean on hardest: out_gate_separated.txt b74b3ca5…1901, out_gate_same_deploy.txt 9b6dbc08…17b0, out_gate_spike.txt 37342c3a…43f9, out_limits.txt ecc540f9…f717, out_counter.txt 058e8bee…2128, out_selftest.txt b2ea8600…0eff, out_mutants.txt 40d5026e…0a9e. Where a block is trimmed, [...] marks the lines I left out of that same run. The rate levels in the fixtures are plausible public list-price levels used as levels to sweep, not a measurement of anyone's billing; no vendor is named and none is implied. I mark which numbers are exhaustive enumerations of a declared grid, which are draws from a declared range, and which are identities that could not have come out otherwise. A leading ~ in any output line means the printed decimal is truncated for display; the arithmetic underneath is exact rationals or trapped Decimals. This is the third draft in this family of topics and the second attempt at this tool. Neither of the two before it was ever published; both were killed in my own pre-publication review. The one this tool came out of died over five defects, three of which were in the code, and the review line I have kept since is that the conclusions were wider, and in places different, than what had actually been measured. Two claims that draft made die further down this page. A later round of review, after this page was already written, killed a third: a characterisation I had wired into the tool's own printed output. What replaced it is narrower, and the section on the band now says which axis was never swept. Two runs in the limits section are two-row exports I ran by hand, outside the 17 in the runner, and they are labelled where they appear.

In short:

How much of a bill delta is genuinely order-dependent is a property of the bucket width you queried at and of which delta you are decomposing. It is not a property of the arithmetic and not a property of your bill.
Same journey, four widths, three regimes. When the reprice and the traffic ramp landed in different buckets, the disputed share is ~74.8623% at one month and exactly $0.00 at a day, an hour and a minute. That $0.00 is a construction in which the repricing bucket runs an exactly flat volume; the next bullet prices what happens when it does not. When one deploy moved both factors, it is ~74.8623% at every width including the minute.
So the first remedy is a finer query, not an extra billing period. It does not hand you $0.00 though. What survives the split is the volume wobble inside the bucket that repriced, times the rate step, and that residual is ~0.395% of the two-snapshot band when the reprice lands on a line running 2 Mtok and ~9.881% when the same reprice, at the same wobble in percent, lands on the same line running 50 Mtok. At a 30% wobble on the busy line it is ~47.169%. Twenty five times the dollars for the same percentage of wobble in that construction, because the repricing bucket was running twenty five times the volume. All of this is the bucket-over-bucket question only.
The correction that cost me the most: for the invoice-over-invoice question, in the one construction I built, a flat June against a July that steps once, a finer query made the disputed share bigger, from ~70.5% to ~72.7% of a $616.50 delta, and the monthly band does not even contain the daily answer. The daily rate-first number $35.70 sits $~13.7387 below the bottom of the monthly band. I have not swept the constructions, so what transfers is that a finer query is not automatically the remedy for the invoice question, not that it always makes it worse. Two questions, two answers. I had been treating them as one.
The Laspeyres to Paasche band is not a bound. One transient spike during a reprice puts the honest chained answer at +$85.00 against a band of +$1.70 .. +$42.50, wrong by $42.50, which is 77.9% of the delta, with no convention changed and exit 0 on the run.
Every path in the containment sweeps moves the rate exactly once, and on that family the escape needs a non-monotone volume path: on the 640 monotone three-step paths of the declared grid the answer left the band 0 times, on the 1040 overshooting paths it left 1040 times. What a file whose rate moves more than once does is not characterised anywhere here and I am not claiming it. I also have no production data on how often real traffic overshoots both endpoints during a reprice, and I am not inventing a percentage for that either.
Two bugs from the draft I killed before this one are fixed here, both with a regression test that can actually fail: the portfolio band summed per line (worst overstatement $5841.00 on a pair whose executable band is $0.00, and 244 of 27261 pairs where the bug changed the exit code), and a third fail-closed layer that was the identity delta == delta.
Drift leaks into the band, so a bucket where only one factor moved to the last decimal is a property of my fixture rather than something I would expect from live traffic. Priced: $0.17 at a drift of 0.2 Mtok per bucket, which is ~0.371% of the two-snapshot band, rising to ~1.298% at 1 Mtok per bucket. That is what --tolerance exists for.

What does two-snapshot rate vs volume attribution actually dispute?

Here is the smallest version. One model line. In the first bucket you paid $0.25 per million tokens on 2 Mtok: $0.50. In the last bucket the line reads $1.10 on 50 Mtok: $55.00. The rate went up 4.4x, the volume 25x, the bill by $54.50.

The rate effect has two exact values and they are 25 times apart. Move the rate at the old volume and it is (1.10 - 0.25) * 2 = $1.70, which leaves $52.80 of the increase to volume. Move it at the new volume and it is (1.10 - 0.25) * 50 = $42.50, which blames the price. The gap between them is the interaction term, (r1-r0)*(q1-q0) = $40.80, and the gate prints it as 74.8% of that $54.50.

Those two ends are the two classical index conventions, named after Étienne Laspeyres and Hermann Paasche, who were arguing about price indices in the nineteenth century, a long time before anyone billed per million tokens. The argument has not changed shape. What has changed is that our exports are timestamped.

Where did the $40.80 go at daily width?

Nowhere. It was never disputed. Here is the same journey as a 30-row daily export: the reprice lands on the 10th, the traffic ramp on the 20th.

$ python3 make_fixtures.py
make_fixtures.py
  sep_daily.csv       30 rows  total $631.50
  same_deploy.csv     30 rows  total $887.00
  spike.csv           30 rows  total $1284.50
  drift.csv           31 rows  total $4124.55
  opp_sign.csv         4 rows  total $50.60
  one_bucket.csv       1 rows  total $55.00
  sep_daily.json      30 rows  same data, JSON

$ head -2 fixtures/sep_daily.csv
date,line,rate_usd_per_mtok,volume_mtok,cost_usd
2026-07-01,worker,0.25,2,0.50

Run the gate on it:

$ python3 resolution_gate.py fixtures/sep_daily.csv
resolution_gate 1.0  offline, stdlib only, exact Decimal (Inexact trapped)
input   : fixtures/sep_daily.csv   30 buckets, 1 line(s), 30 rows
buckets : 2026-07-01 .. 2026-07-30   (bucket width is whatever your export used)

WHAT IS BEING DECOMPOSED
  first bucket 2026-07-01   bill $0.50
  last  bucket 2026-07-30   bill $55.00
  delta                       +$54.50
  SCOPE: all 30 buckets together add up to $631.50, which is NOT what
  the numbers below decompose. Re-querying at a finer bucket width does not
  refine the same answer, it changes the question from invoice-over-invoice to
  bucket-over-bucket. Quote them at the width you ran them at.

ENDPOINT VIEW   first bucket against last, nothing in between
  worker        BOTH_MOVED  $0.25 -> $1.10   2 -> 50 Mtok   rate-effect +$1.70 .. +$42.50
  portfolio rate effect, one consistent order: +$1.70 .. +$42.50   band $40.80

CHAINED VIEW    every bucket step in the file
  2026-07-09 -> 2026-07-10  worker      RATE_ONLY   rate-effect +$1.70 .. +$1.70   step band $0.00
  2026-07-19 -> 2026-07-20  worker      VOLUME_ONLY rate-effect +$0.00 .. +$0.00   step band $0.00
  2 step-line pairs moved, of which 0 moved BOTH factors inside one bucket
  chained rate effect  : +$1.70 .. +$1.70   band $0.00
  chained volume effect: +$52.80 .. +$52.80   band $0.00

additivity      : rate + volume + line == delta, both orderings, chained and endpoint: exact
cost cross-check: rate x volume == cost_usd on every row

CONTAINMENT     does the two-snapshot band contain the chained answer?
  yes. chained +$1.70 .. +$1.70 sits inside +$1.70 .. +$42.50

VERDICT: RESOLVED AT THE BUCKET WIDTH YOU ALREADY HAVE
  rate effect +$1.70, volume effect +$52.80.
  Every bucket step moved at most one factor, so every ordering gives this
  same split. There is no interaction term to argue about.
  The two-snapshot view of this same file calls $40.80 of it disputed.
  That difference, $40.80, is a property of the bucket width,
  not of your bill.
  Nothing was waited for and no policy was applied. The rows were already
  in the export.
$ echo $?
0

Two steps moved, each moved exactly one factor, so there is no ordering to assume. The SCOPE block exists because I got this wrong in the first version of the tool and I want the reader warned in the output rather than in a footnote: those 30 buckets total $631.50, and the $54.50 being split is the first bucket against the last, not the month. Re-querying finer does not sharpen the same answer. It asks a different question. I will come back to that with a number, because it is the part that hurt.

The same file as JSON produces output that differs in exactly one line, the input path. I diffed the two runs rather than asserting it.

When does a finer bucket buy you nothing?

When both factors moved in one deploy. Here is same_deploy.csv: identical endpoints, identical delta, but the reprice and the ramp both land on the 15th.

$ python3 resolution_gate.py fixtures/same_deploy.csv
[...]
CHAINED VIEW    every bucket step in the file
  2026-07-14 -> 2026-07-15  worker      BOTH_MOVED  rate-effect +$1.70 .. +$42.50   step band $40.80
  1 step-line pairs moved, of which 1 moved BOTH factors inside one bucket
  chained rate effect  : +$1.70 .. +$42.50   band $40.80
[...]
steps where both factors moved inside one bucket, in full:
  2026-07-14 -> 2026-07-15  worker: rate $0.25 -> $1.10, volume 2 -> 50 Mtok, delta +$54.50
    rate first (old volume): +$1.70   volume first (new volume): +$42.50   interaction +$40.80 = 74.8% of the step delta

VERDICT: IRREDUCIBLE AT THIS BUCKET WIDTH
  chained band $40.80 exceeds the tolerance you declared (+$0.00).
  1 step-line pair(s) moved rate AND volume inside a single bucket,
  so no ordering is observed and the split is a choice.

WHAT TO DO, IN THIS ORDER
  1. Re-export at a finer bucket width. If the two moves landed on different
     timestamps, what is left of this band is the volume wobble inside the
     bucket where the rate moved, times the rate step. That is $0.00 only if
     that bucket's volume was exactly flat, so price the residual before you
     call it done. Read the SCOPE note above before you quote the new number.
  2. If the finest width you can get still shows both factors moving in one
     bucket, they shipped together. Waiting a billing period does NOT shrink
     this band: it is invariant to the width when the moves are simultaneous.
     Split the DEPLOY, not the period.
[...]
$ echo $?
1

Two files, byte-identical endpoints, opposite verdicts. That is the whole argument in two runs.

The invariance in step 2 is not rhetoric. resolution_limits.py takes the same journey, fixes its span at 60 days, and cuts it into buckets of a month, a day, an hour and a minute, under three regimes.

$ python3 resolution_limits.py
PART 1: the same journey, four bucket widths, three regimes
[...]
  bucket          N   delta/bucket                separated             simultaneous                 co-drift
  1 month         2       $54.5000  $40.80000000 ~74.8623%*   $40.80000000 ~74.8623%   $40.80000000 ~74.8623%
  1 day          60       $~1.8166      $0.00000000 0.0000%    $1.36000000 ~74.8623%    $~0.02305084 ~1.2688%
  1 hour       1440       $~0.0756      $0.00000000 0.0000%   $~0.05666666 ~74.8623%    $~0.00003937 ~0.0520%
  1 minute    86400       $~0.0012      $0.00000000 0.0000%   $~0.00094444 ~74.8623%    $~0.00000001 ~0.0008%
  * DEGENERATE: at N=2 the journey has exactly ONE step, so the two
  moves cannot land in different buckets and the separated construction
  collapses into the simultaneous one. That is the whole problem in one
  cell: two snapshots cannot tell you which regime you are in.
[...]

Read the percentage column and ignore the dollars: dollars per bucket shrink with the bucket by construction, because a minute of traffic costs less than a month of it. The share is what matters, and it does three different things. Separated is exactly zero at every width above the degenerate one, and that is true by construction rather than by luck: put the two moves in different buckets and every step moves one factor. Simultaneous sits at ~74.8623% at every width, minute included. Co-drift falls as 1/(N-1), which is also arithmetic rather than a discovery: N-1 steps each carry an interaction of 1/(N-1) squared of the whole. The measured content of the table is not the zeros and not the decay law. It is that the three regimes are indistinguishable at N=2 and separate completely below it.

The starred cell is the honest centre of this. At two snapshots the separated construction is not merely wide, it is indistinguishable from the simultaneous one. Two aggregates cannot tell you which world you are in, and that is why the disputed number looks like a fact about billing.

I also asked whether the chained answer is just another convention, since I am replacing one number with another. It is not, but the answer is narrower than I wanted. Re-bucketing the same 30 days of the spike fixture at three widths:

$ python3 resolution_counter.py
[...]
ATTACK B: 'the chained answer is just another convention, so you have
           replaced one arbitrary number with another'
[...]
   1-day buckets, 30 of them: chained rate effect $85.00   delta $  54.5000   two-snapshot band $  40.8000   answer inside it: NO
   5-day buckets,  6 of them: chained rate effect band $32.64   delta $ 272.5000   two-snapshot band $ 204.0000   ambiguous
  15-day buckets,  2 of them: chained rate effect band $~236.53   delta $ 805.5000   two-snapshot band $~236.5365   ambiguous
[...]
  VERDICT: NARROWED, computed from the three rows: unique at 1 of 3 widths.
  A band at the finest width would print KILLED, a $0.00 at all three
  would print SURVIVES. Every number has to carry its bucket width, and
  claim (3) is 'query at the width where each step moves one factor',
  not 'query as fine as possible'.

So every number in this post carries its bucket width, and the rule is not "query as fine as possible". It is "query at the width where each step moves one factor". At 15-day buckets a coarser bucket genuinely does contain two events and the answer is ambiguous again.

Why is the Laspeyres to Paasche band not a bound?

Because a rollout is a path, and those two ends are only two of the paths through the same two endpoints. Take a path whose rate moves exactly once, in a bucket running volume w. The rate effect is then (r1-r0)*w, which lands inside the endpoint band while w lies between q0 and q1, and outside it when w does not. That family is where the containment sweeps stop. A path whose rate moves more than once prices each of its steps at a different volume, and I have not characterised those at all. Inside the family: let traffic spike above both endpoint volumes during the reprice and the honest answer is outside the band the endpoints print.

This is the part I had backwards in the draft I killed, when I called the band "both of the answers you could have earned". Twenty eight lines, no dependencies:

"""The whole point in one screen: the two-snapshot band is not a bound.

Same two endpoints. Two paths between them. One of the answers is not in the
band the endpoints print, and the difference is not a rounding artifact.
"""
from decimal import ROUND_DOWN
from decimal import Decimal as D

r0, q0 = D("0.25"), D("2")       # first bucket: USD per Mtok, Mtok in the bucket
r1, q1 = D("1.10"), D("50")      # last bucket
delta = r1 * q1 - r0 * q0

band_lo = (r1 - r0) * q0         # Laspeyres: the rate moves at the OLD volume
band_hi = (r1 - r0) * q1         # Paasche:   the rate moves at the NEW volume

# What the dated rows actually said: traffic spiked to 100 before the reprice
# landed, then settled at 50. Three steps, each moving exactly one factor, so
# each step's rate effect is not a matter of convention.
path = [(r0, q0), (r0, D("100")), (r1, D("100")), (r1, q1)]
chained = sum((b[0] - a[0]) * a[1] for a, b in zip(path, path[1:]))

print(f"delta ${delta}")
print(f"two-snapshot band for the rate effect: ${band_lo} .. ${band_hi}")
print(f"chained over the dated rows          : ${chained}")
print(f"inside the band: {band_lo <= chained <= band_hi}")
share = ((chained - band_hi) / delta * 100).quantize(D("0.1"), rounding=ROUND_DOWN)
print(f"outside by ${chained - band_hi}, which is {share}% of the delta")

$ python3 snippet.py
delta $54.50
two-snapshot band for the rate effect: $1.70 .. $42.50
chained over the dated rows          : $85.00
inside the band: False
outside by $42.50, which is 77.9% of the delta

The full gate says the same thing about the 30-row version of that path, and, this is the uncomfortable bit, says it while returning exit 0:

$ python3 resolution_gate.py fixtures/spike.csv
[...]
CHAINED VIEW    every bucket step in the file
  2026-07-09 -> 2026-07-10  worker      VOLUME_ONLY rate-effect +$0.00 .. +$0.00   step band $0.00
  2026-07-14 -> 2026-07-15  worker      RATE_ONLY   rate-effect +$85.00 .. +$85.00   step band $0.00
  2026-07-19 -> 2026-07-20  worker      VOLUME_ONLY rate-effect +$0.00 .. +$0.00   step band $0.00
  3 step-line pairs moved, of which 0 moved BOTH factors inside one bucket
  chained rate effect  : +$85.00 .. +$85.00   band $0.00
  chained volume effect: -$30.50 .. -$30.50   band $0.00
[...]
CONTAINMENT     does the two-snapshot band contain the chained answer?
  NO. chained +$85.00 .. +$85.00 falls outside +$1.70 .. +$42.50
  FLAG: ENDPOINT_BAND_EXCLUDES_RESOLVED   miss $42.50 = 77.9% of the delta
  The endpoint band prices the whole rate change at one of the two endpoint
  volumes and at nothing else. The chained answer prices every rate step at
  the volume in the bucket where that step happened, so it can land outside.
  The band is one pair of paths out of many, not a range that contains the
  answer.

VERDICT: RESOLVED AT THE BUCKET WIDTH YOU ALREADY HAVE
  rate effect +$85.00, volume effect -$30.50.
  Every bucket step moved at most one factor, so every ordering gives this
  same split. There is no interaction term to argue about.
  The two-snapshot view of this same file is not merely wider here. Its band
  $40.80 wide does not contain the answer above at all, so quoting either
  end of it would have been wrong by at least $42.50.
[...]
$ echo $?
0

The rate effect is $85.00 and the volume effect is negative $30.50, on a bill that went up. That is not a paradox, it is what happens when the reprice catches a traffic peak and the traffic then settles below it. Anyone quoting either end of the two-snapshot band here would be wrong by at least $42.50 on a $54.50 delta.

How far outside can it go? Linear in the peak, (r1-r0)*(w-q1), with no bound inside the two endpoint snapshots:

PART 4: when the two-snapshot band stops being a bound
[...]
  endpoints $0.25 x 2M -> $1.10 x 50M, delta $54.50, band $1.70..$42.50
  peak volume while repricing      rate effect   outside by  % of delta
  50 Mtok (1x the new level)            $42.50        $0.00        0.0%
  100 Mtok (2x the new level)           $85.00       $42.50      ~77.9%
  250 Mtok (5x the new level)          $212.50      $170.00     ~311.9%
  500 Mtok (10x the new level)         $425.00      $382.50     ~701.8%
  1000 Mtok (20x the new level)        $850.00      $807.50    ~1481.6%
  5000 Mtok (100x the new level)      $4250.00     $4207.50    ~7720.1%
[...]

Now the two attacks that matter on this claim, because a characterisation that only holds on my tidy grid is worth nothing.

Is it a lattice artifact? No. 200000 draws from a fixed-seed 64-bit LCG written out in the file, rates from 0.0001 to 10.0000 in ten-thousandths, volumes and the mid-path peak from 0.001 to 5000.000 in thousandths, so nothing sits on a hand-built diagonal:

ATTACK C: 'claim (2) is a lattice artifact of your tidy grid, like the
           28-of-212 flips were'
[...]
  usable draws                     : 200000  (dropped 0 with a factor that did not move)
  chained answer outside the band   : 133575  (~66.78% +/- 0.1053 pp, 1 SE)
  implementation check, draw by draw: 200000 of 200000  (100.00%)
  miss as a multiple of the band    : median ~0.99x, 90th pct ~9.03x, max ~182254.44x

  The share in row two is a property of how wide I let the peak roam and
  is not a frequency in anyone's traffic: I report it only to show it is
  not zero and not confined to a lattice. Row three is an IDENTITY and
  not a finding: is_out and predicted_out are the same predicate, one of
  them divided by (r1-r0), so agreement is the only thing it can print.
  It checks this implementation, the way the 400-of-400 line in
  --selftest does, and it says nothing about the world. The row that
  carries weight is row four, the size of the miss, which nothing forces.
  The draws are off a lattice but they are not continuous: rates land on
  a grid of ten-thousandths, volumes and the peak on a grid of
  thousandths. Row four is measured against the band and not against the
  delta on purpose: the delta goes through zero inside this draw range
  and a ratio to it would be noise.
  VERDICT: SURVIVES, computed from row two: 133575 of 200000 off-lattice
  draws put the answer outside the band, and a zero there would have
  printed KILLED. Claim (2) does not depend on the spacing of any grid.

Read row four, and not row two or row three. The ~66.78% is a property of the range I let the peak roam over, and it would be dishonest to hand it to you as "two thirds of reprices". Row three is worse than that: is_out and predicted_out are the same predicate with one of them divided by (r1-r0), so 200000 of 200000 is the only thing that line can print. It checks this implementation, the way the 400 of 400 line in --selftest does, and it says nothing about the world. I had it in the post as the load-bearing number, which was the same mistake in a new suit. The draws are also off a lattice rather than continuous: rates land on ten-thousandths, volumes and the peak on thousandths. The tool calls row four the one that carries weight. Reviewing this page I checked that too, and it does not: across all 133575 draws that landed outside, the miss over the band is exactly the distance from the peak to the endpoint interval over the gap between the endpoints, and (r1-r0) cancels out of it completely. Row four is a distribution of three uniform volume draws over a range I declared, and it says nothing about rates or reprices either. The worst case, ~182254.44x, comes from two endpoint volumes 0.009 Mtok apart, where the band itself is $0.0196 wide, which is the same near-zero denominator I said I was avoiding by not dividing by the delta. So all four rows of ATTACK C check my implementation and my sampling. The claim they were supposed to support is carried by the theorem and by the spike run, not by them.

Did I cherry-pick a spike? Yes, and the attack lands:

ATTACK D: 'you cherry-picked a spike. On a normal path the band holds'
[...]
  monotone paths (peak inside the endpoints):   640, outside the band: 0
  overshooting paths                        :  1040, outside the band: 1040
[...]
  VERDICT: NARROWED, computed from the two counts: 0 escapes on
  monotone paths, 1040 on overshooting ones. Claim (2) applies to
  non-monotone volume paths, which includes every transient spike,
  migration ramp-and-settle and failed rollback. Every path swept here
  moves the rate EXACTLY ONCE. What happens when the rate moves more
  than once is not characterised by this sweep and is not claimed.

On a plain ramp the band held on all 640 of them, because the rate effect is then a convex combination of the two ends. So inside this family, where the rate moves once, the failure mode needs a non-monotone volume path: a transient spike, a migration that ramps and settles, a rollback that did not hold. What a file whose rate moves several times does is not swept by any of those three, and I am not going to guess at it. The tool's own selftest checks the same statement on a slightly different waypoint list and reports 640 inside, 800 outside, 0 disagreements with the theorem.

Here is what I cannot tell you: how often real traffic overshoots both endpoint volumes during a reprice. I have no production data on that and I am not putting a percentage on it. What I can do is price the miss when it happens, and hand you a flag that fires on your own export.

Does the invoice question get better too? In this construction it got worse

This is the correction that mattered most, and it kills a sentence I would have written happily while drafting this: "look finer and the ambiguity goes away".

Finance rarely asks bucket-over-bucket. It asks invoice-over-invoice: this month against last month. Take the same July as sep_daily.csv, put a flat June next to it, and use the finer data the way a same-day-last-month comparison does, by pairing July day i against June day i.

PART 2: the invoice question moves the OTHER way
[...]
  June : bill $15.00   volume 60 Mtok   blended rate $0.2500/Mtok
  July : bill $631.50   volume 588 Mtok   blended rate $~1.0739/Mtok
  delta $616.50
  The two prices actually charged in July were $0.25 and $1.10. The blended
  $~1.0739 is neither, and it is the number a monthly aggregate hands the decomposition.

  MONTHLY AGGREGATE, two snapshots:
    rate effect $~49.4387 .. $484.5000   band $~435.0612 = ~70.5% of the delta
  DAILY PAIRS, same-day-last-month, 30 pairs (9 flat, 10 single-factor, 11 both-moved):
    rate effect $35.7000 .. $484.5000   band $448.8000 = ~72.7% of the delta

  The finer view is $~13.7387 WIDER, not narrower (~70.5% -> ~72.7%).
  And the monthly band $~49.4387..$484.5000 does not contain the daily one:
  the daily rate-first answer $35.7000 sits $~13.7387 below the bottom of it.
  Why: 11 of the 30 day-pairs moved BOTH factors, because a step change
  persists into every later day. Slicing separates EVENTS in time; it
  does not separate a state that is different for the rest of the month.
[...]

Eleven of thirty day-pairs moved both factors, because a step change persists into every later day. Slicing separates events in time. It does not separate a state that stays different for the rest of the month. So in this construction the finer query makes the band wider, and the monthly band does not even contain the finer answer, which means an extra billing period is not the remedy here either. One construction is one construction: I built a flat June against a July that steps once, and I did not sweep the space of Junes and Julys. Reviewing this page, I was handed a counter-construction where the finer cut goes the other way, and it is easy to build. What survives is the weaker and more useful statement, that a finer query is not automatically the remedy for the invoice question.

Two questions, two different answers, and I had been treating them as one. If you take one operational thing from this post, take that: before you quote a rate effect, say out loud which delta you are decomposing. The SCOPE block prints in every run for exactly that reason.

Note also the blended $~1.0739 per Mtok. July charged $0.25 and $1.10, and never $1.0739. A monthly aggregate hands the decomposition a price nobody was billed, which is a second, quieter reason the invoice question is harder than it looks.

Two bugs the draft of this tool carried

Both were found by my own review, after the draft they were in had been written, and both are the same shape: a number that looks like a measurement and is actually an artifact of how I computed it, which is exactly what a 58% win-rate over zero closed trades was.

Bug one: the portfolio band was not any rollout's range. The old code built the portfolio band by summing per-line minima and maxima, which quietly lets every line choose its own rollout order. A real rollout applies one order to everything. So the portfolio answer takes two values, sum-of-Laspeyres and sum-of-Paasche, and the band is the absolute value of the sum of interactions, not the sum of the absolute interactions. With interactions of opposite sign those are wildly different:

PART 3: a portfolio band summed per line is not any rollout's range
[...]
  both-moved transitions on the 5x4 grid: 234
  two-line portfolios (unordered pairs)   : 27261
  where the per-line sum OVERSTATES the band: 13680  (~50.1%)
  where it is exactly right (same sign)     : 13581  (~49.8%)
  where the executable band is $0.00 and the old code printed a non-zero one
  and exited 1 on it                        : 244  (~0.8%)
  worst overstatement: $5841.00
    line 1 $0.25 x 2M -> $15.00 x 200M   interaction $2920.50
    line 2 $0.25 x 200M -> $15.00 x 2M   interaction $-2920.50
    executable band $0.00, per-line sum $5841.00
  FORCED BY CONSTRUCTION, and I am not reporting the ~50.1% as a frequency in
  anyone's bill: this grid is symmetric in direction, so close to half of all
  unordered pairs must have interactions of opposite sign. The number that is
  not forced is the SIZE of the overstatement and the fact that it changes the
  exit code, which is what a CI job acts on.

The ~50.1% is forced by the construction and I am labelling it rather than quoting it: the grid is symmetric in direction, so roughly half of all unordered pairs must have opposite-sign interactions. What is not forced is the size, $5841.00 at worst, and the 244 pairs where the old code printed a non-zero band and exited 1 on a portfolio whose answer is a single number. An exit code is what a CI job acts on, so that 244 is the part that would have woken somebody up for nothing.

The fix, and the reason my old regression test could never have caught it:

def totals(units, line_bucket):
    """Aggregate under the two orderings a single rollout can actually execute.

    A portfolio-wide answer comes from ONE order applied to everything: either
    every rate move lands at the old volume, or every one lands at the new
    volume. The band is therefore |sum of interactions|, not the sum of the
    absolute interactions. The draft of this tool summed per-unit min and max
    independently, which lets every line pick its own order. Two lines with
    interactions +$0.60 and -$0.60 came out as a $1.20 band and exit 1, when the
    portfolio answer is a single number and no rollout on earth produces $1.20.
    free_width below is that older number, kept and labelled for what it is.
    """

The old test used two identical lines. Identical lines have interactions of the same sign, and with the same sign the buggy sum and the correct one are equal, so the test passed on a tool that was wrong. The new fixture uses opposite signs on purpose:

$ python3 resolution_gate.py fixtures/opp_sign.csv
[...]
ENDPOINT VIEW   first bucket against last, nothing in between
  up            BOTH_MOVED  $1.00 -> $1.20   10 -> 13 Mtok   rate-effect +$2.00 .. +$2.60
  down          BOTH_MOVED  $1.00 -> $1.20   13 -> 10 Mtok   rate-effect +$2.00 .. +$2.60
  portfolio rate effect, one consistent order: +$4.60 .. +$4.60   band $0.00
  per-line free order would print $1.20: that is an upper bound on
  a rollout where each line takes a different order, not a range of one answer.
[...]
VERDICT: RESOLVED AT THE BUCKET WIDTH YOU ALREADY HAVE
  rate effect +$4.60, volume effect +$0.00.
  2 step-line pair(s) moved both factors, and their interactions cancel:
  the PORTFOLIO answer is a single number under either consistent order. Each of
  those lines on its own is still order-dependent, so do not quote them one at a
  time out of this run.
$ echo $?
0

The old number is still printed, labelled as an upper bound for the case where each line really does take a different order. That is a legitimate quantity. It was just never a range of one answer.

Bug two: one of my three fail-closed layers was a tautology. I claimed three independent layers and one of them checked rate + volume + line == delta where the volume term had been defined as delta - line - rate. That expands to delta == delta. It could only ever catch a non-finite value, which the first two layers already catch.

The fix is that every term is now multiplied out on its own, and the delta it is compared against is recomputed from the raw bucket bills:

    d["delta"] = r1 * q1 - r0 * q0
    # Each of the four numbers below is its own multiplication, not derived by
    # subtracting another effect from the delta. That is what makes the
    # additivity check in report() a check and not a tautology.
    d["rate_rf"] = (r1 - r0) * q0          # rate first  (Laspeyres rate effect)
    d["vol_rf"] = r1 * (q1 - q0)           # volume after the rate moved
    d["rate_vf"] = (r1 - r0) * q1          # volume first (Paasche rate effect)
    d["vol_vf"] = r0 * (q1 - q0)           # volume before the rate moved
    d["inter"] = (r1 - r0) * (q1 - q0)

And a layer is only worth naming if it catches something the others let through, so --mutants breaks one thing at a time, including two internals that the old tautological layer waved past:

$ python3 resolution_gate.py --mutants
resolution_gate 1.0 --mutants
  Each row is one deliberately broken input or one deliberately
  broken internal, and the layer that has to catch it. A layer that
  only ever fires when another layer already fired is decoration.

  layer 0  unbroken input                      -> exit 0 (baseline, must be 0)
  layer 1  date 10/07/2026 instead of ISO       -> exit 2  m1.csv row 2: date='10/07/2026' is not ISO YYYY-MM-DD or YYY
  layer 2  71-digit rate, no NaN, valid pattern -> exit 2  Inexact
  layer 4  cost_usd 54.99 vs rate x volume 55   -> exit 2  m4.csv row 3: rate 1.10 x volume 50 = $55.00 but cost_usd=$5
  layer 3  internal: chain drops its last step  -> exit 2  internal: additivity broken. chained rate-first +$1.70 + vol
  layer 3  internal: rate effect doubled        -> exit 2  internal: additivity broken. chained rate-first +$3.40 + vol
           The draft version of layer 3 was the identity delta == delta and
           printed a verdict for both of these internals.

  five broken inputs, exits [2, 2, 2, 2, 2], distinct layers responsible: 1, 2, 3, 4, 3

I am not proud of either bug. I am mildly proud that the fixture for the first one is now in the selftest with opposite signs, which is the version that can fail.

Is a single-factor bucket ever real, though?

Fair question, and it is the strongest attack on the $0.00 above, because in my separated fixture the volume sits exactly flat while the rate moves, and exactly flat is a constant of the fixture rather than a fact about traffic. So I added a drift to every bucket, so that no step has an unchanged volume:

ATTACK E: 'the $0.00 in the separated regime is a constant of your
           fixture: real traffic is never flat, so no bucket is ever
           genuinely single-factor'
  Same separated sequence, with a volume drift added to EVERY bucket so
  that no step has an exactly unchanged volume.

  drift per bucket        chained band   two-snapshot band  chained / two-snapshot
  0.000 Mtok                   $0.0000            $40.8000                  0.000%
  0.001 Mtok                  $~0.0008           $~40.8246                 ~0.002%
  0.010 Mtok                   $0.0085            $41.0465                 ~0.020%
  0.050 Mtok                   $0.0425            $42.0325                 ~0.101%
  0.200 Mtok                   $0.1700            $45.7300                 ~0.371%
  1.000 Mtok                   $0.8500            $65.4500                 ~1.298%

  Drift leaks into the band only through the ONE bucket where the rate
  moved, so the leak is that bucket's volume change times the rate step,
  $0.85. That much is an identity. At a drift of 0.2 Mtok per bucket the
  band is $0.17, which is 0.371% of the two-snapshot band, and at 1 Mtok
  per bucket, a 50% swing on a line running 2 Mtok, it reaches 1.298%.
  Not zero. What --tolerance exists for.

  The table above holds a second constant of the fixture, and it is the
  one that moves the answer: WHERE the reprice landed. Above, the rate
  moves on bucket 10 while the line still runs 2 Mtok, and the ramp
  comes after it. Put the ramp first and the same reprice lands on a
  line already running 50 Mtok. Sweep both orders, with the wobble read
  as a percentage of the level the line is running rather than as an
  absolute: a deterministic zigzag, level*(1+w) on odd buckets and
  level*(1-w) on even ones, so no step has an unchanged volume and
  nothing is drawn at random.

  wobble  reprice lands on    chained band   two-snapshot band  chained / two-snapshot
  5%       2 Mtok                  $0.1700            $43.0100                 ~0.395%
  5%      50 Mtok                  $4.2500            $43.0100                 ~9.881%
  10%      2 Mtok                  $0.3400            $45.2200                 ~0.751%
  10%     50 Mtok                  $8.5000            $45.2200                ~18.796%
  20%      2 Mtok                  $0.6800            $49.6400                 ~1.369%
  20%     50 Mtok                 $17.0000            $49.6400                ~34.246%
  30%      2 Mtok                  $1.0200            $54.0600                 ~1.886%
  30%     50 Mtok                 $25.5000            $54.0600                ~47.169%

  Same percentage wobble, same two events, same width: 25 times the
  dollars, because the bucket that repriced was running 25 times the
  volume. So the separated regime does not hand you $0.00. It hands you
  a residual whose size is set by the volume in the bucket where the
  rate moved, and that is a number you have to price on your own export.
  VERDICT: NARROWED, computed from the largest share above, ~47.16%. A residual
  that reached the two-snapshot band would print KILLED; a residual that
  stayed at exactly $0.00 through every row would print SURVIVES.

The first table is the one I had. It is honest and it is not the interesting one, because it holds a second constant fixed: where the reprice landed. In the separated fixture the price moves on the 10th, while the line is still running 2 Mtok, and the ramp arrives ten days later. Put the ramp first and the identical reprice lands on a line already running 50 Mtok. Same two events, same width, same wobble in percent, and the residual goes from ~0.395% of the two-snapshot band to ~9.881%. Twenty five times the dollars in that construction, because the repricing bucket was running twenty five times the volume. What the ratio is on your export depends on how far apart your two volume levels are, and I have swept two levels, not a range of them.

So "re-export finer and the band goes to $0.00" was a promise the tool had no business making, and it was printing it in the WHAT TO DO block. It now says what is actually left: the wobble in the repricing bucket times the rate step, which is $0.00 only when that bucket happened to be flat. On the busy end of my own sweep that residual reaches ~47.169% of the two-snapshot band, which is not a rounding artifact and not something you want to find out after quoting a number.

On the co-drift fixture, where both factors move a little in all 31 buckets, the gate returns exit 1 on a chained band of $0.30 against a two-snapshot band of $9.00, and it will keep returning exit 1 until you say what you are willing to live with:

$ python3 resolution_gate.py fixtures/drift.csv --tolerance 0.50
[...]
  30 step-line pairs moved, of which 30 moved BOTH factors inside one bucket
  chained rate effect  : +$34.35 .. +$34.65   band $0.30
[...]
VERDICT: RESOLVED AT THE BUCKET WIDTH YOU ALREADY HAVE
  rate effect +$34.35 .. +$34.65, band $0.30, inside the
  tolerance you declared (+$0.50). You accepted the ambiguity; the tool
  did not remove it.
  The two-snapshot view of this same file calls $9.00 of it disputed.
  That difference, $8.70, is a property of the bucket width,
  not of your bill.
$ echo $?
0

That last sentence in the verdict is deliberate. A tolerance is you accepting an ambiguity, not the tool removing one. Same for --tolerance 50 on the one-deploy fixture: exit 0, and a verdict that says in as many words that you accepted a $40.80 band.

The claim I kept from the earlier draft, and just lost

In the earlier draft of this work I measured how often the named culprit flips when you flip the assumed order, swept the spacing of the grid levels, and reported that the flips survive at ordinary drift spacing: 28 of 212 at 1.02x. I had already retracted a bigger number in that draft, and I kept this one as the survivor. It does not survive.

Both ladders in that sweep were built with the same multiplier k, so the rate ratio and the volume ratio came from one identical set, which manufactures exact ties. At that spacing every flip I counted sat on a tie, and once the two ladders stop sharing a multiplier the ties stop happening: 0 of them in every independent row below. Sweep the two spacings independently:

PART 5: the claim I kept after my first self-correction, and lost
[...]
    rate k  volume k  cases  flips  on an exact tie  median gap   min gap
     1.02x     1.02x    212     28      28 (100.0%)       ~0.9%     ~0.9%
     1.02x    1.021x    240      2         0 (0.0%)       ~0.6%     ~0.6%
     1.02x    1.019x    240      2         0 (0.0%)       ~0.3%     ~0.3%
     1.05x     1.05x    212     28      28 (100.0%)       ~2.4%     ~2.4%
     1.05x    1.053x    240     10         0 (0.0%)       ~2.1%     ~2.1%
        2x        2x    212     72      28 (~38.8%)      ~33.3%     ~6.6%
        2x      2.1x    240     72         0 (0.0%)      ~31.2%     ~5.7%
[...]
  candidates that differ by about 1% of the delta: nobody names a
  culprit on a 1% gap. That draft's 5x4 grid is a separate matter:
    108 flips in 374 changed-bill transitions, 6 of them on an exact tie,
    median gap between the two candidates ~60.5% of the delta, min ~27.1%.
    That one is not a lattice artifact. The 2%-spacing row was.
[...]

Move the volume multiplier by one tenth of a percentage point, which changes nothing anybody would call a different regime, and the flips go from 28 to 2. All 28 of the survivors sat on an exact tie I had built by hand, and they were deciding between two candidates about 0.9% of the delta apart. I would not name a culprit on a 0.9% gap, and I doubt you would either.

The headline count from that draft, 108 flips in 374 transitions on the 5x4 step-change grid, holds up: 6 of the 108 are ties, the median gap between the two candidates is ~60.5% of the delta, and the minimum is ~27.1%. Those are real disagreements about real money. It was the narrow, careful, drift-spacing claim that was an artifact, which is the opposite of the way I expected to be wrong.

All counts in that table are exhaustive enumerations of a declared grid, so n is the whole population at each row, no sampling is involved and no standard error applies. The only randomised probe in this project is ATTACK C, which prints its SE.

Run it on your own export

Three commands, no dependencies, nothing to configure:

$ python3 make_fixtures.py
$ python3 resolution_gate.py fixtures/sep_daily.csv ; echo $?
$ python3 resolution_gate.py your_export.csv --tolerance 25

Your export needs four columns, and a fifth if you want the cost cross-check:

date,line,rate_usd_per_mtok,volume_mtok,cost_usd
2026-07-01,worker,0.25,2,0.50

One row per bucket per line. Dates are ISO, either YYYY-MM-DD or YYYY-MM-DDTHH:MM, one form per file, so lexicographic order is chronological order. JSON works too: {"rows": [...]} or a bare list, parsed with parse_float=str so no number round-trips through binary float.

Exit codes are the contract: 0 resolved at the bucket width you have, 1 irreducible at that width, 2 bad input. Wire it as a CI job on a monthly export and it will tell you, before anybody writes a number into a slide, whether the number is a measurement or a choice. Refusals are loud rather than quiet:

$ python3 resolution_gate.py fixtures/one_bucket.csv
INPUT ERROR: fixtures/one_bucket.csv: needs at least 2 distinct dates, found 1. With one bucket there is no delta; with exactly two you are back to a two-snapshot view, which is the thing this tool exists to check.
$ echo $?
2

And the guards get counted rather than asserted:

$ python3 resolution_gate.py --selftest
resolution_gate 1.0 --selftest
  input guards: 17 of 17 malformed exports refused with exit 2
  additivity in both orderings, exact, on 400 of 400 ordered pairs (identity: it
    cannot come out otherwise, so this checks the implementation, not the world)
  opposite-sign interactions (+$0.60 and -$0.60): consistent-order band $0.00 yes, exit 0, per-line free bound $1.20 reported separately yes
  containment, on paths where the rate moves once: chained rate effect inside
    the endpoint band while the volume waypoint is between the endpoint volumes.
    640 inside, 800 outside, 0 disagreements with the theorem
  zero-volume encodings (absent / 0 tokens / 0 tokens at rate 0): exits [0, 0, 0], BOTH_MOVED anywhere False (one event, one verdict)
  selftest: 5 of 5 checks passed

The 400 of 400 line is labelled an identity in the output on purpose. It cannot come out any other way, so it tests my implementation and tells you nothing about the world. Keeping identities out of the findings column is most of the discipline here.

The zero-volume line closes a hole that used to change the verdict from formatting alone. A model that was not called in a bucket shows up in real exports three ways: the row is absent, or it carries 0 tokens, or it carries 0/0 for cost over tokens. The old tool read the last one as a rate moving from 0 and a volume moving from 0, handed the whole delta to the interaction term, and returned its strongest verdict. Now all three encodings of a line that starts billing are one event with one verdict. A line that stops billing in the middle of the period and comes back is a different case: I checked it after writing this, and the three encodings there give exits 0, 1 and 0, so that half of the hole is still open.

def present(cell):
    """A line billed 0 tokens at rate 0 is not a line, it is an empty cell.

    Real exports write cost/tokens = 0/0 for a model that was not called. The
    two-snapshot version of this tool read that as rate 0 -> rate r and volume
    0 -> volume q, i.e. BOTH factors moved, and handed the whole delta to the
    interaction term. Same event, three encodings, three verdicts. So: absent.
    """
    return not (cell[0] == 0 and cell[1] == 0)

What this measures, and what it does not

The fixtures are synthetic and I want that stated in the body rather than buried. The rate levels are plausible public list-price levels used as levels to sweep; they are not a measurement of anyone's billing and they support no claim about any vendor. The 5x4 grid counts and the bucket-width tables are exhaustive enumerations over populations I declared, which makes them arithmetic over grids I chose rather than statistics about the world. The one randomised probe, ATTACK C, prints its own standard error and names, in its own output, which of its rows carries weight and which one is an identity that could not have printed anything else.

Five things this tool will not do for you, in the order I expect them to bite.

It knows nothing about mix. If your rate column is cost divided by tokens, it is a blended average, and it moves when nobody repriced anything: a prompt cache segment breaks, an output:input ratio shifts, a routing weight moves between two models that both already exist. I measured a hit rate falling from 100% to 40% off one injected timestamp in the cache-break detector, and that shows up in this tool as a rate move. The blended $~1.0739 in PART 2 is the same problem in miniature. The tool says so in its own docstring and keeps going, because that is your export's problem and not arithmetic's.

It assumes your rate is not a function of your volume, and a lot of real pricing is. Tiered rates, committed-use amortisation, batch discounts and cached-input pricing all make the blended rate move because the volume moved, and then cost = rate x volume has stopped being a decomposition of two independent things. I ran a two-row export through the gate to watch it fail. Pricing with a step at 100 Mtok, everything up to the step at $3.00 per Mtok and the excess at $2.00: 100 Mtok bills $300.00 at a blended $3.000, and 160 Mtok bills $420.00 at a blended $2.625.

date,line,rate_usd_per_mtok,volume_mtok,cost_usd
2026-07-01,worker,3.000,100,300.00
2026-07-02,worker,2.625,160,420.00

The gate reads that as BOTH_MOVED, prints a rate effect of -$60.00 .. -$37.50 and a band of $22.50, returns IRREDUCIBLE AT THIS BUCKET WIDTH at exit 1, and tells you to split the deploy. There was no deploy. Nobody repriced anything and nobody shipped anything; the traffic crossed a pricing step. In this regime "both factors moved inside one bucket" does not mean "they went out together", so the instruction this tool gives most firmly is the wrong one here, and nothing in its output tells you which of the two cases you are looking at.

The mix problem has a sharper version in the same place. Hold the volume exactly flat at 100 Mtok and move only the input:output split, at $3 per Mtok in and $15 per Mtok out, from 75/25 to 50/50. The bill goes from $600.00 to $900.00 and the blended rate goes from $6.00 to $9.00.

date,line,rate_usd_per_mtok,volume_mtok,cost_usd
2026-07-01,worker,6.00,100,600.00
2026-07-02,worker,9.00,100,900.00

Now the gate prints RATE_ONLY, a band of $0.00, a rate effect of +$300.00 and exit 0. That is its strongest verdict, on a rate effect nobody was charged. Neither of those two runs is a bug in the arithmetic and neither is in the runner: they are two-row exports I ran by hand, and you can paste them into a file and reproduce both in under a minute. What they show is the arithmetic being handed a rate column that is not a price.

It cannot tell you how often the band excludes the answer in your traffic, and it does not characterise every file. The containment result covers paths whose rate moves exactly once, checked against 200000 off-lattice draws and against every endpoints-plus-waypoint combination of the declared grid: there the band bounds the answer while the bucket that repriced ran a volume between the two endpoint volumes. A file whose rate moves more than once prices each of its rate steps at a different volume, and none of the containment sweeps covered that; for those the flag in the output is the only thing that speaks. How often either case happens to you is a fact about your rollouts that I have no data on.

It rejects credits and refunds by design. Negative costs fail the input pattern loudly rather than getting quietly attributed. That means it reads a net-positive usage summary and not a full invoice, and the sign arguments in this post rest on that rejection.

It validates shape, not billing meaning. Nothing in the four layers looks at what your rows mean: a pattern, an ISO date, a duplicate key, an additivity sum and a cost product. So cumulative month-to-date rows, mixed units, or gross where you meant net will pass all of it and get a verdict. The layers stop malformed numbers, not a wrong query.

Where does that leave the practice? Roughly here. Do not decompose two aggregates when a dated export is one query away. Say which delta you are decomposing before you say the number, because in the construction I measured the invoice question and the bucket question came out with different answers, and I did not sweep the constructions. Price the residual instead of assuming it went to zero: after the finer cut, what is left is the volume wobble in the repricing bucket times the rate step, and on the busy end of my own sweep that was ~47.169% of the two-snapshot band rather than nothing. Check the containment flag before you quote a band, because a band that excludes the answer is worse than a wide one. And if the finest width you can get still shows both factors moving inside one bucket, first check whether your rate column is a blended cost over tokens, and whether your rate is a function of your volume; if it is neither, then the two moves went out together, no waiting will separate them, and the thing to split next time is the deploy rather than the period. That last part is the only place where the old advice was right, and it is a much smaller place than I gave it.

The line I keep coming back to is that attribution is not a property of a report. It is a property of how the change was rolled out and at what resolution the change was recorded. Everything I have written about gating before execution rather than logging after it says the same thing in a different register, and the spend cap that stopped counting is the same lesson with a worse ending. The counterpart to this post is the 0.35% per day drift that produced zero alerts in 60 days: slow drift makes the disputed money small and the trend invisible, while a step change makes the disputed money enormous and the trend obvious. Also worth knowing before you argue about a rate: a single agent loop re-bills its whole transcript, which I measured as a 42.8x re-bill multiplier on one debugging session, so your volume can move a long way with nobody deciding anything.

Here is the question I have not answered, and it is the one I would ask a stranger with a bigger bill than mine. When the two moves genuinely shipped in one deploy, and no resolution can separate them, is a fixed slice of traffic held at the old configuration for one bucket a real remedy or a fantasy? It turns a bill into an experiment, it needs a routing layer most stacks do not have, and the two slices have to be comparable enough that the comparison means something. If you have shipped anything like it, I want to know how you kept the slices comparable, because that is where I expect it to fall apart, and it is the only remedy left in the one regime this post cannot fix.

Follow along if you want the next measurement rather than the next opinion: this is a FinOps blog for people who run agents, and every post here comes out of a tool I ran first. And tell me the worst bill delta you have had to explain from two aggregates, especially one where you later found out the real path between them.

Lost Update: When Two AI Agents Edit One File, One Silently Wins

Alexey Spinov — Thu, 30 Jul 2026 02:29:50 +0000

A lost update between AI agents: two agents edit the same resource, both writes return success, and one is silently gone with nothing in the system saying a word. It happens when both read the same version and the last writer overwrites the rest. The fix is a pre-write compare-and-set gate, not a bigger log.

I ran it. With 5 agents committing to one shared file under a worst-case schedule, where every agent reads before anyone writes, 4 of the 5 contributions vanished from the final state, and every one of those 5 writes had been acknowledged. The final file held exactly one agent's work. The other four were paid for, ACKed, and silently overwritten. And that worst case is not rare: in a fair sample of interleavings at N=5, some write was lost in essentially 100% of runs.

This is the lost update anomaly, and it is older than AI agents by decades. What is new is that each vanished write burned real model tokens, and the loss leaves no error, no exception, no red log line. So I built a small offline simulator to make the loss countable, and to test the one class of fix that actually stops it: a version check before the write lands.

AI disclosure: I wrote lostwrite_sim.py with an AI assistant and ran it myself, offline, on Python 3.13.5, standard library only. No network, no keys, no threads, no funds. Concurrency is simulated by explicit interleaving schedules, not real parallelism, so the output is reproducible. Every number and hex string below is pasted from a real local run. Three runs produced byte-identical STDOUT with sha256 1df08c5e38894622314a2b052684303853b9b8fa9508eaabdfd877970c88983b. The linked specs and articles are other people's work.

In short:

When N agents read the same version of a resource, each does its work, and each writes back, the last writer wins and everyone in between is silently overwritten. In the worst case at N=5 that was 4 lost writes out of 5, all acknowledged.
Whether it happens at all depends on the schedule. In a fair sample of interleavings the loss hit 75.0% of runs at N=2 and effectively 100% by N=5, but a quarter of N=2 runs serialized and lost nothing. So one green test run proves nothing.
An append-only log records every write, so you can prove each write happened. It does not help you recover one. Reading the current value still returns a lost update, because logging preserves the fact of a write, not the presence of its change.
A pre-write compare-and-set (check the version has not moved since you read it) drops lost updates to 0 by construction. Measured, it lost nothing in all 46,333 fair interleavings I sampled, at every N.
The gate does not save money. It costs more. In the worst case at N=5 it added 4 work-units of retries on top of 5, so total work went from 5 to 9. It buys integrity and a countable collision, not a refund.
Turn the version check off inside the same gate and the loss comes straight back (4 again). That falsifier is the proof that the compare, not the scaffold around it, is doing the work.

This is not a toy concern anymore. The moment you run two sub-agents, or a swarm, or a retrying planner that spawns workers, you have multiple actors writing shared state: a plan file, a scratchpad, a database row, a GitHub issue. @opsveritas wrote up the paired version of this: one agent times out, three more do not notice, and work gets paid for that no one will ever read. Lost update is the same failure seen from the write side.

What is a lost update between two AI agents?

A lost update happens when two actors read the same version of a resource, both compute a new value from what they read, and both write it back. The second write is based on a snapshot that predates the first write, so committing it erases the first. Neither actor sees an error. The classic definition lives in database isolation theory, and agents inherit it the instant they share a resource without a write precondition.

The mental model I used in the simulator is deliberately concrete: N agents each append their own section to one shared file. The correct final file contains all N sections. A lost update is a section that made it into an acknowledged write and then disappeared from the final file.

Here is the store. It holds a version number and the set of contributions currently present.

class VersionedStore:
    def __init__(self):
        self.version = 0
        self.value = frozenset()
        self.wal = []  # append-only log of every accepted write

    def write_lww(self, agent_id, new_value):
        """Last-writer-wins: overwrite unconditionally, always ACK."""
        self.value = frozenset(new_value)
        self.version += 1
        self.wal.append((self.version, agent_id))
        return "ACK"

    def write_cas(self, expected_version, agent_id, new_value, enforce=True):
        """Commit only if the version has not moved since this agent read."""
        if enforce and self.version != expected_version:
            return "CAS_FAIL"
        self.value = frozenset(new_value)
        self.version += 1
        self.wal.append((self.version, agent_id))
        return "ACK"

Each agent runs three steps: READ the store, WORK (build snapshot | {my_id} and spend one unit), then WRITE. Concurrency is not threads. It is an explicit interleaving, and I run it two ways on purpose. The first is a deterministic worst case: every agent reads before anyone writes, so every read is stale. That is the adversarial upper bound, not a typical run, and I use it for the headline because it is the clearest picture of the failure. The second is a fair sampler: at each tick pick one still-ready agent uniformly at random (random.Random, seeded, so the whole run is reproducible to the byte), and draw tens of thousands of independent interleavings to get a distribution. The worst case shows the failure; the fair sample shows how often it actually bites.

A note on that, because the first version of this tool got it wrong and it matters. I originally picked agents with a hand-rolled linear congruential generator and next() % n, which reads the least-random low bits. At N=2 that strictly alternates for every seed, so it silently pinned the worst case and printed it as if it were a seeded sample. random.Random.randrange draws from the full generator state and does not have that bias. The fair numbers below come from the fixed version.

One thing I want to be honest about up front, because it decides how you read every number here. The costs in this tool are counts, not dollars. COST_UNIT is an accounting constant set to 1, labelled in the source as "not dollars, not a measured price." Every money figure is an integer counter times that constant. If you want your dollars, multiply the run counts by your own measured per-run token cost. I did not measure a token price, and I am not going to pretend a synthetic schedule tells you one.

The anomaly at its worst: N=5

5 agents, the worst-case schedule (everyone reads before anyone writes), gate off versus gate on. This is the headline block straight from the run:

WORST CASE  N=5, maximum contention (every agent reads stale)
--------------------------------------------------------------------
                              NO_GATE (LWW)      GATE (compare-and-set)
  writes acknowledged         5                  5
  contributions in final      1                  5
  contributions expected      5                  5
  lost_updates (ACKed, gone)  4                  0
  integrity_ok (present==all) False              True
  cas_failures                0                  4
  retries                     0                  4
  refusals                    0                  0
  total_runs (work executed)  5                  9
  total_cost (runs*COST_UNIT) 5                  9

Read the left column first. Five writes acknowledged, one contribution in the final state, four lost. The lost_updates number is not printed from a flag that says "no gate, so print 4." It is re-derived: the tool takes the set of agents whose write was ACKed at least once, subtracts the set actually present in the final value, and counts what is left. Four writes said success and are not in the file.

The right column is the same schedule with a version check on the write. Zero lost. All five contributions present. And I need to be blunt about why that zero is a zero, because it would be dishonest to sell it as a surprising discovery: it is true by construction. A compare-and-set write only commits when the version has not moved, which means it commits on top of a snapshot that already contains every prior commit. It cannot drop one. The interesting, measured part of the right column is not the zero. It is the price of the zero, which I will get to.

Tracking is not control

Here is the part I keep having to relearn. "Just log every write" feels like it should help. It does not.

TRACKING IS NOT CONTROL  (NO_GATE, worst case, N=5)
--------------------------------------------------------------------
  writes recorded in append-only log (WAL) : 5
  WAL entry shape                          : (version, agent_id), no value
  contributions surviving in final state   : 1
  final state contains contributions       : (4,)
  writes logged and ACKed but gone on read : 4

The append-only log has all five writes. Every one. It proves that five agents each wrote, and it tells you which agent and at what version. That is enough to know you were charged for five and kept one. It is not enough to hand any of the four back: my WAL entry is (version, agent_id), the fact of the write, not the value it carried. A richer log that stores the content, like the separate git-refs @dipankar_sarkar described for two coding agents editing the same issue, keeps both versions and reconciles after, and for a merge-later workflow it is the right tool.

But look at the last line, and notice it holds no matter how fat the log gets. The log knows about all five writes, and reading the current value still returns four lost updates. Recording a write and preventing its loss are different operations at different times. The log is a receipt written after the fact. It cannot un-overwrite the file, any more than a receipt read after the fact can recover a field that was never stored. This is the same franchise as a spend cap that counts tokens but cannot stop the loop: tracking tells you what happened, control changes what is allowed to happen. Only a write that runs before the commit gets to reject the stale one.

The gate, and its honest price

The fix is a precondition on the write. Optimistic concurrency, the lock-free version: read a version, do your work, and at commit time refuse if the version moved. It has a formal name, optimistic concurrency control, and the primitive under it is compare-and-swap. A pessimistic lease or lock would also work; it just makes the agent wait instead of retry. Either way the shared idea is the same: no write lands on state you did not read.

res = store.write_cas(a.base, a.aid, a.newval, enforce=not disable_version_check)
if res == "ACK":
    a.acked = True
else:
    cas_failures += 1
    if a.retries < max_retries:
        a.retries += 1
        a.pc = "READ"      # re-read the latest, re-work, re-write
    else:
        a.refused = True   # fail-closed: nothing silently applied

Now the cost, which is the whole reason this post is not called "how the gate saves you money." It does not. Look again at the headline: NO_GATE ran 5 work-units, GATE ran 9. The gate added 4 work-units of retries, because each stale writer had to re-read fresh state and redo its work before it could land. The delta is positive, and it is positive by design:

  Delta cost of the gate      = 9 - 5 = 4 work-units (>=0: the gate
                                costs MORE, it does not save tokens)
  Spend destroyed without gate= lost_updates * COST_UNIT = 4 work-units
                                (already paid, silently gone from state)

Two numbers, two different things, and it is easy to blur them into a false savings pitch. The 4 destroyed work-units under NO_GATE are money you already spent on work that then vanished. The gate does not refund them. It cannot reach back into a run that already happened. What it does is stop the next four from vanishing, and it charges you 4 units of retries to do it. If your worry is the token bill, the gate is a cost, not a saving. If your worry is that four agents did real work and the result quietly disappeared, the gate is the thing that keeps that from being silent.

That is the contrarian bit, and I will state it as a falsifiable claim: a pre-write concurrency gate on shared agent state increases token spend and is still worth it, because it converts a silent unbounded loss into a visible, counted, retryable event. If your data shows a gate that reduces total spend under contention, I would genuinely like to see the workload, because my own run says the opposite.

Is the gate doing the work, or is my test scaffold cheating?

This is the failure mode I trust least in myself, so the tool has a falsifier for it. The classic mistake is to build a "gate" whose zero comes from the test scaffold rather than the mechanism. So F1 keeps the entire gate code path and disables one thing: the version check inside write_cas.

FALSIFIER F1  gate code path, version-check DISABLED (worst case, N=5)
--------------------------------------------------------------------
  lost_updates with the compare removed    : 4  (was 0 with it on)

Four again. Same as no gate. The retry loop, the counters, the refusal path, all still there. Remove only the comparison of expected version to current version, and the loss returns in full. That is the proof I actually care about: the compare is the mechanism. Everything else is plumbing.

Does it get worse with more agents?

Two questions live here, and they have different answers. How bad can it get, and how often does it get bad at all. The worst case answers the first. The fair sample answers the second, and it is the more useful number.

First the worst case, swept over N. Every agent reads before anyone writes, so all but the last writer are clobbered. The N=1 row is the negative control: one writer, no one to overwrite, zero loss.

WORST-CASE SWEEP over N (maximum contention, every read stale)
----------------------------------------------------------------------------
  N | ng.lost ng.integ | g.casfail g.retry g.refuse g.lost g.integ | dCost
  --------------------------------------------------------------------------
  1 |      0 True    |        0       0        0      0 True    |     0  (degenerate: 1 writer)
  2 |      1 False   |        1       1        0      0 True    |     1
  3 |      2 False   |        2       2        0      0 True    |     2
  5 |      4 False   |        4       4        0      0 True    |     4
  8 |      7 False   |        7       7        0      0 True    |     7

In the worst case the loss is exactly N-1: everyone but the last writer is gone, and the gate pays one retry per clobbered writer to fix it. But nobody schedules their agents to lose. The honest question is what a fair draw of interleavings does, so I sampled 40000 // N of them per N and counted the distribution.

FAIR SWEEP  uniform random interleavings (random.Random, Mersenne Twister)
----------------------------------------------------------------------------
  base_seed=20260728, trials per N = 40000 // N
  N | trials | NO_GATE loses>=1 | serializes (0 lost) | mean lost | max | GATE loses>=1
  --------------------------------------------------------------------------
  2 |  20000 |   75.0% (14994) |   25.0% ( 5006)  |     0.75 |   1 | 0
  3 |  13333 |   97.3% (12972) |    2.7% (  361)  |     1.59 |   2 | 0
  5 |   8000 |  100.0% ( 7999) |    0.0% (    1)  |     3.35 |   4 | 0
  8 |   5000 |  100.0% ( 5000) |    0.0% (    0)  |     6.14 |   7 | 0

This is the number I would actually put in a design doc. At N=2 a quarter of interleavings serialize on their own and lose nothing, which is exactly the trap: two agents, run it a few times, watch it pass, ship it. Then it loses data in three runs out of four. By N=3 the safe fraction is down to 2.7%, and by N=5 it is one interleaving in eight thousand. So the anomaly is not universal on small N, and it is close to certain past a handful of writers. That is a worse story than "it always happens," not a better one, because "usually fine" is the thing that gets shipped.

The gate column is the point of the whole exercise. Across all 46,333 fair interleavings I drew, over every N, compare-and-set lost an update exactly zero times. That is a measured claim, not a hand-wave: the by-construction argument says a CAS write cannot drop a prior commit, and 46,333 independent draws agree. NO_GATE integrity is False on every row with a second writer; GATE integrity is True on every one of them; and the gate's delta cost is never negative.

When the gate runs out of retries

A gate that only works when it can always retry is not much of a gate. So the last thing I checked is the ugly case: heavy contention with zero retry budget. What does it do when it cannot win the race?

FAIL-CLOSED  GATE with max_retries=0 (worst case, N=8)
--------------------------------------------------------------------
  cas_failures                             : 7
  refusals (VISIBLE, counted)              : 7
  refused agent ids                        : (1, 2, 3, 4, 5, 6, 7)
  lost_updates (SILENT)                    : 0
  integrity_ok                             : False

Seven agents lost the race and, out of retries, they refused. Loudly. integrity_ok is False, which is correct, because seven contributions genuinely did not make it. But lost_updates is 0. Nothing was silently overwritten. The difference between this and the NO_GATE case is the entire point: NO_GATE gives you integrity_ok=False with a silent 4, and you find out never. The gate gives you integrity_ok=False with seven refusals by explicit id, and you find out immediately. One failure is invisible, the other is a list of names. That is what "fail-closed" buys, and the tool exits non-zero if that property ever breaks.

What this is, and what it is not

It is a simulator, not a measurement of production. Two kinds of number live in it and they deserve different trust. The worst-case counts (headline 4 lost, N-1 across the sweep) are a deterministic upper bound: they are what a maximally adversarial schedule does, not what a typical one does, and I label them that way. The fair-sweep fractions (75.0% at N=2, 97.3% at N=3, ~100% at N=5 and N=8) are Monte Carlo estimates over tens of thousands of uniform-random interleavings, so they carry the usual sampling error of a proportion and would shift a little on a different base seed. What does not shift is the direction: the zero-loss fraction falls hard as writers are added, and the gate's zero holds in every sampled interleaving. None of this is a token-price measurement. I sampled one model of "random interleaving," a uniform scheduler; a real runtime with its own scheduling could sit anywhere between my worst case and my fair sample, and I have not measured a real one.

I also did not measure a token price, and I will not dress the accounting unit up as one. And this is not the revert-guard problem, where a single agent reintroduces reverted code, nor is it about a single agent double-charging on a retry. It is specifically two-or-more writers, one version, no precondition. If you run sub-agent dispatch or any fan-out where workers share a resource, this is the write-side gate that belongs next to your pre-execution checks. And it is a distinct axis from mandate freshness: freshness asks whether an old approval is still valid in time, this asks whether two live writes can both survive.

Run it yourself

Standard library only, offline, no keys, no funds, a few seconds (the fair sweep draws about 92,000 interleavings). run_all.sh runs the selftest, then three full runs, compares them byte for byte, and prints the sha256 of each.

interpreter: Python 3.13.5

self-test: PASS
run 1: exit=0 sha256=1df08c5e38894622314a2b052684303853b9b8fa9508eaabdfd877970c88983b
run 2: exit=0 sha256=1df08c5e38894622314a2b052684303853b9b8fa9508eaabdfd877970c88983b
run 3: exit=0 sha256=1df08c5e38894622314a2b052684303853b9b8fa9508eaabdfd877970c88983b
determinism: 3 runs byte-identical

The report body also self-reports its own sha256 (57e02acaa779e066d4736bb00b9ef1a2f4eeec21d886c626879cfd1a67db661e), so you can tell at a glance whether your run matches mine. The selftest is the contract: INV1 requires the worst case to lose exactly N-1, INV2 requires the gate to lose nothing there, INV3 requires the fair sample to lose in some but not all interleavings on small N, INV4 requires the gate to lose nothing in any sampled interleaving, F1 through F4 are the falsifiers, and any failed assertion exits 1 with FAIL. A gate that cannot fail its own tests is decoration.

Follow along if you want the numbers from the next teardown in this series. And if you are running more than one agent against shared state right now, tell me in the comments: what is the worst silent overwrite you have hit between two agents, and did you catch it before the commit or only when the result was already gone? I suspect, for most of us, the honest answer is "only when it was already gone."

x402 Signs the Money, Not the URL. I Checked 18 Fields.

Alexey Spinov — Wed, 29 Jul 2026 01:12:59 +0000

An x402 payer signature does not cover the URL. It commits to the amount, the recipient, the token contract and the chain, and to nothing that says what you are paying for. I mutated 18 leaf fields of the payment payload published in the x402 spec: 8 changes left the signature verifying, 10 broke it.

I rebuilt the EIP-712 digest from that example, recovered the signer with my own secp256k1 code, then changed one field at a time. The 8 that still verify include the entire resource object.

Change the resource URL to a different host. The signature still verifies. Nothing in the payment path notices.

AI disclosure: I wrote x402_intent_gate.py with an AI assistant and ran it myself, offline, on Python 3.13.5, standard library only, no network, no keys, no wallet, no funds. Every number and every hex string below is pasted from a real local run. Three runs produced byte-identical STDOUT with sha256 6cfe746ec64d8a497b1cafe27ed351dab1f56f690d3195959f0993a6e57888a6. The spec text I quote is other people's work, linked inline.

In short:

The signed structure in the x402 exact EVM scheme is EIP-3009's TransferWithAuthorization(address from,address to,uint256 value,uint256 validAfter,uint256 validBefore,bytes32 nonce), wrapped in an EIP-712 domain that adds the token name, version, chain id and contract address. That is the whole list. There is no slot for what you are buying.
I enumerated every leaf field of the PaymentPayload example in the x402 v2 spec, mutated one at a time, and re-ran real ECDSA recovery each time. 8 of 18 mutations left the signature verifying. The 10 that broke it are money, token, chain and clock.
Nothing the payer signed can be checked afterwards either. A settled transferWithAuthorization leaves an ERC-20 Transfer and AuthorizationUsed(address indexed authorizer, bytes32 indexed nonce), and neither names a resource. The SettlementResponse has seven fields, and exactly one of them, extensions, can carry the resource, because that is where the optional offer-and-receipt extension parks a receipt. That receipt is signed by the server, not by you.
The repair costs zero protocol changes. The nonce is 32 bytes the payer chooses, it is inside the signature, and it is emitted indexed on-chain. So stop wasting it on randomness: nonce = keccak256(canonical_intent || salt).
The gate runs before the signature exists. Of 17 constructed cases, 10 pass the facilitator's own verification steps, including a real validity-window check, and get refused by the gate anyway. All 3 legitimate ones still pass.

This is not a hypothetical protocol that nobody ships. Cloudflare announced a monetization gateway for x402 on 2026-07-01, which puts a 402 in front of anything sitting behind their edge. The number of agents that will sign one of these grew a lot faster than the number of people asking what the signature says.

What does an x402 payer signature actually commit to?

Start with the anchor, because everything after it depends on my arithmetic being right.

The x402 v2 specification publishes a complete PaymentPayload example in section 5.2.1, including a real 65-byte signature. The exact/EVM scheme spec republishes the same payload with one extra field, and defines what gets signed. I implemented keccak256 and secp256k1 from scratch, rebuilt the EIP-712 digest from that example, and ran public key recovery against that signature.

  [PASS] keccak256("") matches the published vector
  [PASS] keccak256("abc") matches the published vector
  [PASS] keccak256(TransferWithAuthorization type string) equals the TYPEHASH
         constant published in EIP-3009
  [PASS] secp256k1 base point is on the curve
  [PASS] n*G is the point at infinity
  [PASS] the signature published in the x402 spec example recovers to the
         payer address published in that same example
  [PASS] sign then recover round-trips on the throwaway demo key

  keccak256("")    c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470
  keccak256("abc") 4e03657aea45a94fc7d47ba826c8d667c0d1e6e33a64a036ec44f58fa12d6c45
  TWA typehash      0x7c7c6cdb67a18743f49ec6fa9b35f50d52ed05cbed4cc592e13b44501c1a2267
  recovered signer  0x857b06519e91e3a54538791bdbb0e22373e36b66
  authorization.from 0x857b06519e91e3a54538791bdbb0e22373e36b66

That last line is the part I care about. The address my code recovers from their signature equals the from address in their example. So the 32 bytes I am reconstructing are the 32 bytes that were actually signed, not a plausible-looking reimplementation of them.

The digest is 0xf256992871671abcb27ff92885a7afa46218724e5fc0bac35d050115aa1d22e6, and it is built from exactly this:

def eip712_digest(payload):
    """Rebuild exactly the 32 bytes an x402 exact/EVM payer signs."""
    acc = payload["accepted"]
    auth = payload["payload"]["authorization"]
    chain_id = int(acc["network"].split(":")[1])
    ds = domain_separator(acc["extra"]["name"], acc["extra"]["version"],
                          chain_id, acc["asset"])
    struct_hash = keccak256(keccak256(TWA_TYPE) + _addr(auth["from"]) + _addr(auth["to"])
                            + _u256(auth["value"]) + _u256(auth["validAfter"])
                            + _u256(auth["validBefore"]) + _b32(auth["nonce"]))
    return keccak256(b"\x19\x01" + ds + struct_hash)

Read the inputs. from, to, value, two timestamps, a nonce, and a domain made of the token name, version, chain id and contract. Count the fields that describe what you are buying: zero.

The Permit2 path in the same spec is stricter, not looser. Its witness type is keccak256("Witness(address to,uint256 validAfter)"), and the spec carries the comment post-audit: extra removed from Witness. The one place context could have been smuggled in got taken out by an audit.

Which x402 payload fields can change without breaking the signature?

I walked that same v2 section 5.2.1 example as an object tree, collected every leaf, and mutated them one at a time with a minimal type-preserving change. After each mutation the digest gets recomputed and the original 65 bytes get re-verified by full recovery. The signature field itself is excluded, since it is the artifact under test.

leaf field                             | minimally changed to   | signature
------------------------------------------------------------------------------
x402Version                            | 3                      | STILL VERIFIES
resource.url                           | value + "-mutated"     | STILL VERIFIES
resource.description                   | value + "-mutated"     | STILL VERIFIES
resource.mimeType                      | value + "-mutated"     | STILL VERIFIES
accepted.scheme                        | value + "-mutated"     | STILL VERIFIES
accepted.network                       | eip155:8453            | fails
accepted.amount                        | 10001                  | STILL VERIFIES
accepted.asset                         | 0x00000000000000000... | fails
accepted.payTo                         | 0x00000000000000000... | STILL VERIFIES
accepted.maxTimeoutSeconds             | 61                     | STILL VERIFIES
accepted.extra.name                    | value + "-mutated"     | fails
accepted.extra.version                 | 3                      | fails
payload.signature                      | not mutated            | the artifact under test
payload.authorization.from             | 0x00000000000000000... | fails
payload.authorization.to               | 0x00000000000000000... | fails
payload.authorization.value            | 10001                  | fails
payload.authorization.validAfter       | 1740672090             | fails
payload.authorization.validBefore      | 1740672155             | fails
payload.authorization.nonce            | 0xababababababababa... | fails
------------------------------------------------------------------------------
leaf fields enumerated                    : 19
excluded (the signature itself)           : 1
mutated                                   : 18
signature STILL VERIFIES after the change : 8
signature fails after the change          : 10

Two notes on that table before the counts, because the printout carries them and I would rather you read them from me than find them yourself. The example's extensions is an empty object, so it has zero leaves and gets no row, and it happens to be the one place a resource identifier could ever ride. Section 3 comes back to it. And accepted.network is the single field I gave a hand-written mutation, eip155:84532 to eip155:8453, because a malformed CAIP-2 string crashes the chain id parser instead of testing anything.

The 10 failures are the negative control, and they matter more than the 8 passes. If everything had come back STILL VERIFIES, the honest conclusion would have been that my checker was broken. It is not: touch the payer, the recipient, the amount, either timestamp, the nonce, the token contract, the chain id, or the token name or version, and recovery lands on a different address.

So the line is clean, and it is not an accident of my field ordering. Everything describing where the money goes is inside the signature. Everything describing what the money is for is outside it.

The 8 survivors split in two. Four of them say what is being bought: resource.url, resource.description, resource.mimeType and the scheme name. The other four carry no payment authority at all: the protocol version, the timeout hint, and the display copies of the amount and the recipient.

Two of those 8 deserve a note, because I do not want to overclaim. accepted.amount and accepted.payTo are display copies. They do not move money, since the money follows authorization.to and authorization.value. What they do is decide what your client renders and what your logs keep. A client builds the authorization from the requirements the server sent, so the unsigned copy is the input, and afterwards only one of the two versions can be proven. That is a smaller problem than the URL, and I am flagging it as smaller.

The row worth sitting with

Minimal mutations prove coverage. They understate severity. So I re-ran the uncovered fields with values chosen to be obnoxious, using the same 65 bytes:

change to the envelope                       | signature
------------------------------------------------------------------------------
resource.url -> a different host entirely    | STILL VERIFIES
resource.url -> a different path             | STILL VERIFIES
resource.description -> unrelated            | STILL VERIFIES
resource.mimeType -> unrelated               | STILL VERIFIES
accepted.amount display copy, 100x           | STILL VERIFIES
accepted.payTo display copy -> burn address  | STILL VERIFIES
accepted.scheme -> another scheme name       | STILL VERIFIES
accepted.maxTimeoutSeconds -> one hour       | STILL VERIFIES
add accepted.extra.assetTransferMethod       | STILL VERIFIES
------------------------------------------------------------------------------
adversarial envelope changes tried  : 9
signature still verifies after      : 9

api.example.com to evil.example.net, same signature, still valid. The money is pinned to the last atomic unit. The name of the thing the money bought is a free text field sitting next to it.

Why can't you reconcile this afterwards?

Because the data does not exist. This is the part that surprised me, and it is why I stopped looking for a post-hoc answer.

A settled transferWithAuthorization leaves two records on chain: the ERC-20 Transfer the token contract emits, and EIP-3009's own event AuthorizationUsed(address indexed authorizer, bytes32 indexed nonce). Neither names a resource. Two purchases at the same price to the same recipient for different resources look like this:

  buy A
    resource                  https://api.example.com/premium-data
    Transfer.from             0x857b06519e91e3a54538791bdbb0e22373e36b66
    Transfer.to               0x209693bc6afc0c5328ba36faf03c514ef312287c
    Transfer.value            10000
    AuthorizationUsed.nonce   0xf3746613c2d920b5fdabc0856f2aeb2d4f88ee6037b8cc5d04a71a4462f13480
  buy B
    resource                  https://api.example.com/cheap-data
    Transfer.from             0x857b06519e91e3a54538791bdbb0e22373e36b66
    Transfer.to               0x209693bc6afc0c5328ba36faf03c514ef312287c
    Transfer.value            10000
    AuthorizationUsed.nonce   0x63e1985efb2feb72dfaa78debef5dc246d15984f7895fb86294e43d9153d476d

  ERC-20 Transfer args identical between A and B : True
  AuthorizationUsed differs only in the nonce    : True
  fields naming the resource in either record    : 0

Those records are reconstructions from my fixtures using the field lists the spec and the EIP define. I did not query a chain, and nothing in that script touches money.

Then there is the SettlementResponse, and here I have to correct myself. My first draft of this post listed five fields and said flatly that none of them names the resource. Section 5.3.2 of the v2 spec has seven, and the two I dropped were the two that mattered:

The SettlementResponse the server hands back has seven fields in the x402
v2 spec, section 5.3.2, and here is the whole list:
    success      required   errorReason  optional
    transaction  required   payer        optional
    network      required   amount       optional
                            extensions   optional

Six of those seven cannot name a resource. The seventh can. extensions is exactly where the offer-and-receipt extension parks a receipt, at extensions["offer-receipt"].info.receipt, and that receipt carries a resourceUrl. The spec prints a worked example with "resourceUrl": "https://api.example.com/premium-data" sitting right there in the settlement response. So "no field names the resource" is false as an absolute, and I am glad it got caught before this went out.

The true claim is the narrower one, and it is the one this whole post is about: nothing the payer signed names the resource. The single slot that can name it is opt-in, and it is filled by the server. That extension is genuinely useful and I would turn it on. But it is signed by the service rather than the payer, its own text calls it an audit layer "without changing payment execution or settlement semantics", and the receipt is "privacy-minimal by default and intentionally omits transaction references to reduce correlation risk". A merchant-signed artifact that by default does not link to the transaction is a merchant's statement, not the payer's proof of what the payer decided to buy.

PaymentPayload.resource has the same shape of problem: it exists, the v2 spec marks it Optional, and it sits outside the signature.

Which leaves one place to stand. Not after the settlement. Before the signature.

This is not the stale mandate problem

I want to keep two failures apart, because they look alike and they are not.

In the mandate freshness gate the axis is time. The signature was honest when it was made, and the authority behind it walked out afterwards: revoked, expired, limit lowered. Everything cryptographic holds, and the question is whether the yes is still standing at execution.

Here the axis is content. The authority is perfectly live. The signature is fresh, valid, and inside every limit. It simply never said what it was for. A freshness check passes this case with full marks, and so does a signature check, because both are answering questions that have correct answers.

This is the same shape as tracking is not control, pushed one level down. Your spend cap counts tokens you can compute. In an x402 flow the counterparty names the price in the 402 response, so a sliding window guard is watching a number it did not choose. And a receipt read after the fact cannot recover a field that was never recorded.

The fix that needs no protocol change

The nonce is 32 bytes. EIP-3009 says they are random and payer-chosen. They sit inside the signed struct, and the contract emits them indexed on-chain in AuthorizationUsed.

That is a 32-byte payer-controlled channel that is already signed and already published, and we currently fill it with noise.

I expected to be arguing that this is merely protocol-legal. It turns out the spec argues it for me. EIP-3009's own Security Considerations say that where cross-use is a risk, "the app developer could dedicate some leading bytes of the nonce as an identifier to prevent cross-use". Putting meaning in those bytes is a sanctioned use, not a loophole I found. The one caveat worth keeping: the spec says the nonce is randomly generated, and what preserves that property here is the 32-byte salt, not the intent. Hash a bare intent with no salt and you get a nonce that repeats and that anyone can grind.

def canonical_intent(intent):
    """One line per field, fixed order, newline separated. No JSON ambiguity."""
    return ("x402-intent/1\n"
            "method: %s\n"
            "url: %s\n"
            "body-sha256: %s\n"
            "class: %s\n"
            "max-atomic: %d\n"
            "asset: %s/%s\n"
            "payTo: %s\n" % (
                intent["method"].upper(),
                normalize_url(intent["url"]),
                intent.get("body_sha256") or "-",
                intent["resource_class"],
                int(intent["max_atomic"]),
                intent["network"], intent["asset"].lower(),
                intent["payTo"].lower())).encode()


def intent_nonce(intent, salt32):
    if len(salt32) != 32:
        raise ValueError("salt must be 32 bytes")
    return "0x" + keccak256(canonical_intent(intent) + salt32).hex()

From the run:

canonical_intent for buy A, exactly the bytes that get hashed:
    | x402-intent/1
    | method: GET
    | url: https://api.example.com/premium-data
    | body-sha256: -
    | class: market-data
    | max-atomic: 20000
    | asset: eip155:84532/0x036cbd53842c5426634e7929541ec2318f3dcf7e
    | payTo: 0x209693bc6afc0c5328ba36faf03c514ef312287c

  committed nonce   0xce1d40f2e8ccbe52ec6f127abb9752a42f0c469669bebd856a16649c558d4711
  same intent, same salt, recomputed                : MATCH
  host swapped to evil.example.net, same salt       : MISMATCH
  cap raised from 20000 to 30000, same salt         : MISMATCH
  different salt, same intent                       : MISMATCH
  nonce length in bytes                             : 32

It is still a valid 32-byte nonce, still unique, still opaque to everyone without the salt. Nothing changes on the wire. What changes is that the signature becomes impossible to produce without first having written the decision down, which is the property I actually wanted: the control produces the audit trail, instead of the audit trail being offered as a substitute for control.

Two things it does not do, stated plainly. It does not make the server deliver the resource you named. And it is a commitment, not a receipt: it proves what you decided, not what you received. If someone shows me a way to bind delivery from the payer side without an extension the merchant has to opt into, I would like to see it, because I could not find one.

The gate, run before anything is signed

Same family as the pre-execution gate and the pre-send transaction canary, aimed at the payment decision. decide() collects every reason instead of bailing on the first, and there is no code path that returns ALLOW on an error.

I built 17 cases and ran each one twice: once through the facilitator's own verification steps, once through the client gate. Every case is really signed with a throwaway key derived from a fixed string in the file, using RFC 6979 deterministic nonces so the bytes come out the same every run. The facilitator column is doing real recovery, not trusting an asserted boolean.

Step 3 of that verification list reads "Verify the authorization parameters (Amount, Validity Window) meet the PaymentRequirements", and the validity window is the part it is easy to quietly skip, since checking it needs a clock and a clock breaks determinism. My first version skipped it, checked only that validBefore was greater than validAfter, and still printed a column labelled offline-checkable. So the run pins a clock instead:

    FIXED_NOW = 1740672100  (inside the spec example's window 1740672089..1740672154)

That instant sits inside the window of the spec's own example, so freshness gets checked for real while the output never touches the wall clock. Falsifier F5 exists purely to prove the check fires: an authorization whose window closed before FIXED_NOW gets flagged outside-validity-window, and the spec example does not.

constructed case                                      | gate   | facilitator
------------------------------------------------------------------------------
legit: market data, 10000, everything allowlisted     | ALLOW  | ACCEPT
legit: image gen, 4000, inside its own class cap      | ALLOW  | ACCEPT
legit: second market-data buy, budget still fits      | ALLOW  | ACCEPT
quote is for a different URL than we asked for        | REFUSE | ACCEPT
resource host is not on the allowlist                 | REFUSE | ACCEPT
payTo is not on the allowlist                         | REFUSE | ACCEPT
chain is not the one we fund                          | REFUSE | ACCEPT
token contract is not the one we fund                 | REFUSE | ACCEPT
25000 for a class capped at 20000                     | REFUSE | ACCEPT
fits the cap, but 45000 already signed and unsettled  | REFUSE | ACCEPT
resource class the policy never heard of              | REFUSE | ACCEPT
plain random nonce, no commitment to any intent       | REFUSE | ACCEPT
nonce commits to a different intent                   | REFUSE | ACCEPT
control: signed value contradicts the quote           | REFUSE | REJECT
control: signed destination contradicts the quote     | REFUSE | REJECT
control: validity window closed at decision time      | REFUSE | REJECT
unusable input, required field missing                | REFUSE | n/a
------------------------------------------------------------------------------
cases constructed                                  : 17
gate ALLOW                                         : 3
gate REFUSE                                        : 14
facilitator ACCEPT (offline-checkable steps)       : 13
facilitator REJECT                                 : 3
pass the facilitator, refused by the gate          : 10

Ten cases sail through the facilitator's verification list and get stopped by the gate. That is not a criticism of facilitators, and it is worth being fair here: the spec is explicit that "the Facilitator cannot modify the amount or destination", and my run agrees, because both of those are inside the signature. The facilitator is doing its job correctly. Its job is to check the authorization against the server's requirements. Both of those come from the server. Your decision is not an input to that comparison anywhere in the verification list.

The three REJECT rows are there so you can see the facilitator checker is capable of saying no on each axis it claims to check: amount, destination and freshness.

One thing I should not let myself round off. Two of those ten, intent-nonce-missing and intent-nonce-mismatch, are refused for not using a convention I invented four paragraphs ago. Every x402 payment on earth today would trip them. That is a proposal, not a finding, and if you strip those two out the gate still catches eight cases with nothing more exotic than an allowlist and a cap.

One reason code is mine and I have not seen it elsewhere: budget-would-exceed-with-outstanding. An EIP-3009 authorization is a liability from the moment it is signed, not from the moment it settles. It carries validAfter and validBefore, and until one of those windows closes or the nonce is consumed, the money is committed. A cap that counts settled spend will happily sign the payment that puts you over, and then watch it land.

What would prove me wrong

Here is the single counterexample that ends this post: show me a field, in what the payer authorizes under the exact EVM scheme, that identifies the resource. One field and I am wrong.

Be clear about which part of that I ran and which part I read. The scheme defines three asset transfer methods, and my tool exercises one. EIP-3009 is the one measured above. Permit2 signs Witness(address to,uint256 validAfter), with the spec's own comment post-audit: extra removed from Witness. ERC-7710 sends delegationManager, permissionContext and delegator, and the spec says its verification "is performed entirely through simulation" of an ERC-20 transfer(payTo, amount). Two addresses and an amount. So the claim holds across all three, but only the first is a measurement and the other two are me reading the spec, which is a weaker kind of evidence and I would rather label it than launder it.

The tool ships six falsifiers, all PASS on the run above. F1 is the negative control on the checker. F2 checks the commitment is a function and not a coincidence. F3 requires the gate to allow every case built to be legitimate and refuse every case built to be wrong, which stops a gate that refuses everything from scoring well. F4 feeds it garbage, an empty object and a null, and requires REFUSE with bad-input on all of them. F5 proves the validity-window check actually fires. F6 re-runs the sweep and demands identical verdicts.

Then I tried to break it on purpose, four times, and it exited 1 every time: flipping one Keccak round constant, forcing decide() to return no reasons, turning the fail-closed branch into fail-open, and dropping the intent from the nonce so it hashed only the salt. A gate that cannot fail its own tests is decoration.

What this is not

It is not a conformance suite, and it does not implement upto, deferred or any Solana scheme. It does not talk to a chain, a facilitator or a wallet, so balance and simulation are not run at all and are never reported as passed. Those are steps 2 and 5 of the exact/EVM EIP-3009 list specifically; the Permit2 list numbers them 3 and 7, so the numbers are not portable even inside one document. The counts are counts of cases I constructed in one file. They are not frequencies, not samples, not rates observed anywhere in production, and no standard errors apply because nothing here is an estimate. Recount every one of them from the printout.

I also have no idea how common any of this is in the wild. I have not measured a single real x402 payment, and I am not going to pretend a count of seventeen constructed cases tells you anything about how often an agent overpays for the wrong URL. What the run does establish is structural: the field is not in the signature, so the check cannot be done later, no matter how careful your logging is.

Run it yourself

Standard library only, offline, no keys, no funds, about ten seconds. run_all.sh runs the self-test, then three full runs, compares them byte for byte and prints the sha256 of each:

interpreter: Python 3.13.5

self-test: PASS
run 1: exit=0 sha256=6cfe746ec64d8a497b1cafe27ed351dab1f56f690d3195959f0993a6e57888a6
run 2: exit=0 sha256=6cfe746ec64d8a497b1cafe27ed351dab1f56f690d3195959f0993a6e57888a6
run 3: exit=0 sha256=6cfe746ec64d8a497b1cafe27ed351dab1f56f690d3195959f0993a6e57888a6
determinism: 3 runs byte-identical

The report itself ends with report-sha256: 49cc227bc3cb64316dbea77387ea304f14b9ca3c677c9c707538bc0fe6bc3ccc, so you can tell at a glance whether your run matches mine.

The question I have not answered: the intent-committed nonce binds my decision to my money, and it does that with no protocol change and no cooperation from anyone. It still cannot prove the server gave me what I paid for. Every payer-side scheme I sketched for that ends up needing the merchant to sign something, which means it needs adoption, which means it is not something I can ship on my own next week. If you have found a payer-side way to bind delivery, I want to read it.

Follow along if you want the numbers from the next teardown in this series. And if you are running x402 in anything resembling production, tell me in the comments what your client does with the resource URL after it signs, because I suspect the honest answer for most of us is "logs it, unsigned, next to the amount".

Your Authz Checks the Caller. The Model Picked the Tenant.

Alexey Spinov — Sun, 26 Jul 2026 03:51:45 +0000

A confused deputy in an AI agent is not a broken authorization check. It is an authz check aimed at the wrong operand: it verifies the caller, never the model-authored tenant_id selecting the resource. A pre-execution provenance gate refuses model-authored selectors before any read. Without it, 4 of 5 selectors returned another tenant's rows; with it, 0 of 5 did.

Your agent is authorized to read invoices. This morning it read a different company's invoices, and every authorization check returned yes.

Nobody bypassed the gate. The caller was who it claimed to be, the token was valid, the role allowed the tool. The gate answered the exact question it was built to answer, correctly. The leak lived in an argument the gate never looked at: the tenant_id that selects which company's rows come back. And that argument was written by the model.

In short:

A confused deputy in an AI agent is not a broken auth check. It is an auth check aimed at the wrong operand: it verifies who is calling and never verifies which resource the call selects.
When the model authors the resource-selecting argument (tenant_id, account_id, project_id), an authorized caller can reach another tenant's data. The identity check passes the whole time.
The discriminator is provenance, not value. A tenant_id that came from the authenticated session is fine. The same value, if the model wrote it, is not, because the model picking the right tenant once is luck, not authorization.
The gate: any resource-selecting argument must be session-derived. Model-authored selectors are refused before the database is touched.
The tool below runs eight concrete calls. Without the gate, 4 of the 5 model-authored selectors returned another tenant's rows. With the gate, 0 of 5 reached any row, and all 3 session-derived calls were still served. Standard library only, offline, deterministic. Recount every number from the printout.

AI disclosure: I wrote scope_provenance_gate.py with an AI assistant and ran it myself, three times, on Python 3.13.5, standard library only, no network, no keys. Every output block below is pasted from that run. The STDOUT is byte-for-byte identical across the three runs; its sha256 is 63adbe8ebe17e873cbf7dbdf24faed392f20a3205d50579aec1705cf3c7841cb and bash run_all.sh reproduces it. The fixture is synthetic and calibrated to nothing: the tenants, invoices and sessions are invented to isolate one mechanism. bot2 is a new project. It has no production fleet and no incident to sell you. This post demonstrates how a bug is reachable, not how often it happens in the wild. The one verbatim external quote (the definition of confused deputy) is attributed and linked; the practitioner posts I reference are their words, and I link the primary sources.

The confused deputy: the operand nobody checks

Here is the shape of a normal agent tool call. The agent has a tool, read_account_rows(account_id, limit). A request comes in on an authenticated session. Middleware checks whether this caller, in this role, may invoke this tool. It may. The tool runs. Rows come back.

Now look at where account_id came from. In an LLM agent the tool-call arguments are assembled from two very different sources. Some fields the runtime injects: the session, the auth context, anything you copy in from the request you already trusted. The rest the model fills, from its plan. limit is a fine thing for the model to choose. account_id decides whose data you return. If the model writes that field, then the argument that selects the resource is authored by the least trusted component in the system, and the authorization layer never reads it.

That is a textbook confused deputy. The Wikipedia article on the confused deputy problem defines it in one sentence:

"In information security, a confused deputy is a computer program that is tricked by another program (with fewer privileges or less rights) into misusing its authority on the system."

The term is Norm Hardy's, from his 1988 ACM SIGOPS paper of the same name. The deputy here is your authz middleware. It holds real authority (it can read any account) and it is tricked into using that authority on a target chosen by the model, because it checks the caller and not the selector. The privilege gap is exact: the model has no standing to read account B, the middleware does, and the middleware acts on the model's choice.

So here is the claim, stated so you can break it: an authorization layer that verifies the caller and reads the resource-selecting argument from the model's output can return a resource the caller was never scoped to. The fix does not need a smarter authz check on identity. It needs a check on a different operand: the provenance of the selector. Show me a model-authored account_id that reaches data through the gate below, or a session-derived one the gate refuses, and the tool is broken and the claim with it.

The world, in eight calls

The fixture is three accounts and two sessions. acct_apex and acct_ceres and acct_borealis, each with one or two invoice rows. Two authenticated sessions: S1, scoped to acct_apex only; S2, a shared-ops user scoped to both acct_apex and acct_ceres. Nobody is scoped to acct_borealis here, which makes it the clean victim.

BLOCK 1 -- the world (synthetic, calibrated to nothing)
  acct_apex: rows [apex-inv-2201,apex-inv-2202]
  acct_borealis: rows [bor-inv-5501,bor-inv-5502]
  acct_ceres: rows [cer-inv-8801]
  session S1: identity=user_apex_ops role=reader authorized_accounts=['acct_apex']
  session S2: identity=user_shared_ops role=reader authorized_accounts=['acct_apex', 'acct_ceres']

The authorization middleware is the ordinary kind. It answers one question and it answers it right:

def authz_allow_caller(session, tool):
    """Answers exactly one question: may this caller invoke this tool?
    It never sees account_id."""
    return tool in ROLE_TOOLS.get(session.role, set())

And the tool trusts that the middleware already did its job, so it does no ownership check of its own:

def run_tool(tool, values):
    if tool == "read_account_rows":
        account_id = values["account_id"]
        limit = values["limit"]
        return list(DB.get(account_id, [])[:limit])
    raise ValueError("unknown tool")

Neither layer checks who is allowed to read account_id. That is the whole bug, and it is boring, which is why it ships.

Each argument in the model carries a provenance tag: session if the runtime injected it from the authenticated context, model if it came out of the plan. This is not something I invented for the demo. Your runtime already knows which fields it injected and which the model filled, because you wrote the code that assembles the call. The tag just names a fact you are throwing away.

Watch identity pass while the data leaks

Run the eight calls through the current path: identity check, then the tool. No scope check anywhere.

BLOCK 2 -- every enumerated call, WITHOUT the gate
  call sess selector(account_id)   prov    id_ok foreign rows_returned
  C1   S1   acct_apex              session True  False   apex-inv-2201,apex-inv-2202
  C2   S1   acct_borealis          model   True  True    bor-inv-5501,bor-inv-5502
  C3   S1   acct_borealis          model   True  True    bor-inv-5501,bor-inv-5502
  C4   S1   acct_apex              model   True  False   apex-inv-2201,apex-inv-2202
  C5   S1   acct_ceres             model   True  True    cer-inv-8801
  C6   S2   acct_ceres             session True  False   cer-inv-8801
  C7   S2   acct_borealis          model   True  True    bor-inv-5501,bor-inv-5502
  C8   S1   acct_apex              session True  False   apex-inv-2201,apex-inv-2202

  Cross-tenant reads achieved without the gate: 4 -> C2,C3,C5,C7

Read the id_ok column. It is True on every row, including the four that leaked. The identity check never failed. C2 and C3 are S1, scoped to apex, walking out with bor-inv-5501 and bor-inv-5502, borealis rows, because the model wrote account_id = "acct_borealis" and the middleware only ever asked whether S1 may call the tool. C5 is S1 reading ceres. C7 is S2 reading borealis, an account it was never scoped to.

That is the confused deputy in one screen. The gate the system trusts did not break. It answered "may this caller use this tool?" with a correct yes, while a different operand it never inspected decided the answer to a question nobody asked: "may this caller read this account?"

The gate reads the source, not the value

The fix keys on the one property that separates C1 from C4. Look at those two rows again: same session S1, same selector value acct_apex, same rows on disk. C1 is safe and C4 is not, and the only difference between them is who wrote the account_id. C1's came from the session. C4's came from the model. The value is identical. The provenance is not.

So the rule is provenance, and only provenance:

def scope_provenance_gate(call):
    """Looks ONLY at the source of each resource-selecting argument,
    never at its value. Fails closed on unknown tools or params."""
    schema = TOOL_SCHEMA.get(call.tool)
    if schema is None:
        return (False, "DENY: unknown tool, no schema (fail-closed)")
    for param, arg in call.args.items():
        pspec = schema["params"].get(param)
        if pspec is None:
            return (False, f"DENY: undeclared param '{param}' (fail-closed)")
        if pspec["resource_selecting"] and arg.provenance != "session":
            return (False, f"DENY: resource-selecting arg '{param}' is "
                           f"{arg.provenance}-authored, must be session-derived")
    return (True, "ALLOW: all resource-selecting args are session-derived")

A tiny schema declares which parameter selects a resource (account_id yes, limit no). The gate runs before authz and before the database. Same eight calls:

BLOCK 3 -- the same calls, WITH the provenance gate
  call selector         prov    verdict rows / reason
  C1   acct_apex        session ALLOW   rows [apex-inv-2201,apex-inv-2202]
  C2   acct_borealis    model   DENY    DENY: resource-selecting arg 'account_id' is model-authored, must be session-derived
  C3   acct_borealis    model   DENY    DENY: resource-selecting arg 'account_id' is model-authored, must be session-derived
  C4   acct_apex        model   DENY    DENY: resource-selecting arg 'account_id' is model-authored, must be session-derived
  C5   acct_ceres       model   DENY    DENY: resource-selecting arg 'account_id' is model-authored, must be session-derived
  C6   acct_ceres       session ALLOW   rows [cer-inv-8801]
  C7   acct_borealis    model   DENY    DENY: resource-selecting arg 'account_id' is model-authored, must be session-derived
  C8   acct_apex        session ALLOW   rows [apex-inv-2201,apex-inv-2202]

C4 is the row I want you to sit with. Its value was harmless. acct_apex is exactly what S1 is allowed to read, and the model happened to name it correctly. The gate denies it anyway. That is not the gate being dumb. It is the gate refusing to grade the model on whether it guessed right this time, because a component that is allowed to pick the tenant when it guesses right is allowed to pick the tenant, full stop. If you want apex read, the selector comes from the session. The model does not get partial credit for coincidence.

The counts, so you can recount them

Every number I am about to state is a count of the eight rows above. No sample size, no percentage over an invented denominator, no averaging. Count them yourself.

BLOCK 4 -- the counts (recount them from BLOCK 2 and BLOCK 3)
  K total enumerated calls                          : 8
  resource selector is session-derived (S)          : 3  -> C1,C6,C8
  resource selector is model-authored (M)           : 5  -> C2,C3,C4,C5,C7
  of M, selector points OUTSIDE caller's set        : 4  -> C2,C3,C5,C7
  of M, selector happens to name caller's own acct  : 1  -> C4

  WITHOUT gate: model-authored selectors that
    returned another tenant's rows                  : 4 of 5  -> C2,C3,C5,C7
  WITH gate: model-authored selectors that
    reached ANY row                                 : 0 of 5  -> (none)
  WITH gate: session-derived selectors that still
    returned the caller's authorized rows           : 3 of 3  -> C1,C6,C8
  WITH gate: session-derived selectors denied        : 0 of 3  -> (none)

Three facts, each a count and not an estimate:

Reachable. 4 of the 5 model-authored selectors returned another tenant's rows without the gate. The fifth, C4, returned the caller's own rows only because the model happened to name the caller's own account. The leak is reachable, and you can see exactly which rows walked out.
Blocked. 0 of the 5 reached any row with the gate on. All five were denied before the database was touched, C4 included. The gate does not need to know which of them was malicious, because it does not decide on the value.
Not broken. 3 of 3 session-derived selectors were still served with the gate on. The gate denied zero legitimate calls. It is not "deny everything," which would be trivial and useless.

The 4-of-5 and 0-of-5 are not measurements of a rate that could come out differently on a bigger fixture. They are properties of the construction. Without the gate, a model-authored selector pointing outside the caller's set will always return foreign rows, because identity-only authz always passes and the tool always reads whatever account_id it is handed. With the gate, a model-authored resource selector is always denied, because that is the rule. Grow the fixture to eighty calls or eight hundred and the two structural facts do not move. Only the sizes of the sets do.

Four things that could have broken and didn't

A gate that only ever says DENY would pass the leak test and be worthless. So the tool ships four falsifiers, each of which fails loudly if the gate is the wrong shape:

BLOCK 6 -- falsifiers (each could fail if the gate were wrong)
  [PASS] F1 gate does not break legitimate session-derived scope
         3/3 session-derived calls served, 0 denied. A deny-all gate would fail this.
  [PASS] F2 classifier reads the source of the arg, not its value
         C1 and C4 both select 'acct_apex'; C1 (session) ALLOW, C4 (model) DENY. A value-based check (account_id == self) would ALLOW C4 and miss the rule.
  [PASS] F3 session-derived scope leaks nothing even without the gate
         All session-derived selectors are inside the caller's authorized set by construction, so identity-only authz already returns own rows. The vulnerability is specific to model-authored scope, not to the tool.
  [PASS] F4 gate keys on the resource-selecting operand, not on 'any model arg'
         C8's limit is model-authored but non-selecting; account_id is session-derived. Gate ALLOWs C8. A blanket 'no model input' gate fails.

F2 is the one that matters most, because it is the difference between a real fix and a fake one. The obvious patch is a value check: assert account_id == session.tenant. That would stop C2, and it would also break S2 reading its second authorized account, and it would silently pass a model-authored value that happens to match. F2 proves the gate is not doing that. C1 and C4 carry the identical value acct_apex and get opposite verdicts, so the decision is provably keyed on the source, not the string. F4 proves the gate is not the lazy over-correction either: it does not reject every argument the model touched, only the one that selects a resource. C8's limit is model-authored and the gate lets it through, because a model choosing to fetch fifty rows of accounts it is allowed to see is not a confused-deputy problem.

F3 is the honest scoping of the whole claim. The tool is not dangerous. read_account_rows is fine. The vulnerability is specific to model-authored scope, and F3 shows it: every session-derived call leaks nothing even with no gate at all, because a selector drawn from the session is inside the caller's authorized set by construction. Take the model out of the resource-selection path and there is no deputy to confuse.

This is not the write-hop bug

If you read the write-chain taint post here three weeks ago, this can look like the same story told twice. It is a different operand, and the difference is the whole point.

In that post the gate keyed on a signal, sender_trust, and the danger was that a model had written some store upstream of that signal, so the value the gate read was model-laundered. The fix walked the write-closure of the signal the gate reads and refused to let a model-tainted signal hold the authorization role. The tainted thing was the thing the gate checks.

Here the gate reads a clean signal. Caller identity is world-anchored; the model did not write it and there is no write-hop to trace. The gate is right about what it checks. The bug is that the resource selector is a different argument entirely, a sibling operand the authz layer structurally never inspects, and that operand is model-authored at the point of the call. There is no laundering and no upstream store. The model just fills a field, directly, and the field decides whose data comes back. Taint-linting the signal the gate reads would not catch this, because the signal the gate reads is clean. You have to gate the operand the gate ignores. Same family, adjacent bug, different fix. If your mental model is "make sure the gate's inputs are trustworthy," this one slips past, because the gate's inputs are trustworthy and the leak is in an input the gate never took.

The general franchise both posts sit in is the same, and it is the one I keep coming back to: gate before you execute, do not log after. It is the same instinct as gating the lethal trifecta before the agent runs and as the post on model-authored SQL reaching the database: an operand an untrusted component authored gets checked before it acts, not logged after. Logging the call tells you which account leaked, next week, in the incident review. Gating the provenance of the selector stops the read.

What this is and is not

This is a mechanism demo on a synthetic fixture. It shows that the leak is reachable and that a provenance gate closes it on eight enumerated calls. It is not a claim about how common the bug is, not a benchmark, and not a measurement of anything in production, because bot2 has no production. The provenance tag is the load-bearing assumption: the gate is only as good as your runtime's honesty about which arguments it injected versus which the model filled. If you assemble tool calls by letting the model emit the whole JSON and never tracking what you put in, you do not have the tag, and step one is to start attaching it. The gate cannot recover a provenance you never recorded.

There is a real cost, and C4 is it. The gate denies model-authored selectors that would have been harmless, because it refuses to read the value. In exchange you get a rule that does not depend on the model being right. I think that trade is correct for anything that selects a tenant. You may not, for lower-stakes selectors, and F4 is there precisely so you can scope the rule to the arguments that deserve it.

What to change on Monday

Find every tool where an argument selects a resource: tenant_id, account_id, project_id, workspace, user_id, org. For each one, ask a single question: at the moment the call is assembled, does that argument come from the session or from the model? If you cannot answer, that is the finding. Start tagging.

Practitioners are already converging on this from the other side. Kailash Sankar, in "Defense in Depth: Tenant Isolation for an Agent That Executes Code", wires a proxy that overwrites whatever the model puts in a tenantId parameter with the trusted value from the context registry, hallucinated or injected value be damned. Brian Hall, in "Don't use an LLM to decide what your AI agent is allowed to do", puts it as a design rule: the decision on whether a real action runs "has to sit on something that gives the same answer every time and can show its work afterward."

Overwrite and deny are two implementations of the same rule, and they differ in one way worth naming. Sankar's proxy silently corrects the model's tenantId; the model never learns it overreached. The gate here refuses and says why, which turns a silent correction into a visible signal you can count, alert on, and use to notice a plan that keeps reaching for tenants it was not handed. Silent is safer to ship. Loud is better for finding out your agent has been trying the wrong door for a month. I have not run this in anger long enough to tell you which one you will regret less. Pick the one that matches how much you trust your own logging.

The tool, all eight calls, both execution paths, the selftest and the four falsifiers are in scope_provenance_gate.py. Copy it, add a ninth call, watch the counts move by exactly one.

I write one runnable tool per post about operating AI agents in production: the cost, the failures, the gates that run before execution instead of the logs that run after. Follow for the next one. And tell me in the comments: in your agent, which tool arguments come from the session and which come from the model, and are you sure?

Cost Per Verified Success: Your Exit-0 Denominator Lies

Alexey Spinov — Sat, 25 Jul 2026 03:51:39 +0000

Your cost-per-task dashboard is doing division. Spend on top, "successful tasks" on the bottom. The number it prints is the average cost of one agent task getting done. Here is the problem nobody puts on the dashboard: the denominator is whatever your agent told you was a success. On most stacks that means exit_code == 0, or ok: true, or an HTTP 200. That is the actor grading its own homework. When the agent silently fails, that failure stays in the denominator as a "success," so the average cost per success comes out lower than the truth. Your cheapest-looking number is the one you can least trust.

Cost per verified success is agent spend divided by witnessed successes, not by exit-0. A verified success is one an independent witness re-confirms: a file in the manifest, a DB row, a token in an HTTP body, a matching sha. Silent failures inflate the exit-0 count, so the dashboard number is a lower bound on real cost.

AI disclosure. I wrote verified_cost.py with an AI assistant and ran every case myself before publishing. Every terminal block below is pasted from a real run on Python 3.13.5, stdlib only. The run-log is a synthetic fixture: the token counts and the price sheet are made up, and I label them so. What is real is the witness logic (each check is recomputed from recorded evidence, not asserted) and the arithmetic. I have no production incident and no invoice to sell you here. bot2 is new and its lifetime spend is zero dollars. What I have is a script that runs and a number you can reproduce.

Why is cost per task a lie?

Because "task" and "successful task" are two different measurements, and the cheap one is wearing the expensive one's name tag. exit_code == 0 is cheap: it is the process telling you it thinks it finished. It is a self-report from the same actor whose work you are trying to price. On this blog the shape keeps recurring: a tool returns 200 and lies, a green check reconciles against nothing, an approval is not immutability. Cost inherits the same disease. If your success count is self-reported, your cost-per-success is self-reported too, and it always rounds in the flattering direction.

Watch it happen on one line of a run-log. An agent calls a "create order" endpoint. The endpoint returns HTTP/1.1 200 OK with a body of {"status":"RATE_LIMITED","order":null}. The process exits 0, because 200 is not an error. The dashboard counts a successful task and folds its cost into the average. No order was created. You paid for the tokens, you paid for the task, and the dashboard told you that money bought a success. It did not. Multiply that by an overnight batch and your per-success number drifts away from reality while looking perfectly healthy.

The fix is not a better dashboard. It is a better denominator.

What is a witnessed success?

A verified success is a task that (1) exited 0 and (2) has its effect re-confirmed by a check independent of the agent. That "and" matters. It makes verified successes a subset of exit-0 successes, which is the whole reason the math has a direction. Call M the count of exit-0 tasks and M' the count of verified ones. Because every verified success is also an exit-0 success, M' <= M for any log you will ever feed it. So:

naive_cost_per_success    = total_spend / M
verified_cost_per_success = total_spend / M'      (M' <= M, so this is >= naive)
understatement_factor     = verified / naive = M / M'   (>= 1, always)

That inequality is not a finding. It is arithmetic. naive_cost <= verified_cost for every possible run-log, with equality only when M' == M, meaning zero silent failures. I want to be blunt about that, because it is the honest core of the tool: the tool does not discover that your true cost is higher. It proves it is at least as high and then measures by how much. The direction is guaranteed. The magnitude is what you did not know.

A witness has to be concrete or it is just vibes with a checkmark. verified_cost.py ships four kinds, and each one is recomputed from recorded evidence rather than trusting a boolean somebody wrote down:

def eval_witness(w):
    kind, ev = w["kind"], w["evidence"]
    if kind == "file_exists":
        return ev["target"] in ev["manifest"]          # path actually present
    if kind == "db_row_present":
        return ev["target"] in ev["rows"]              # row id actually there
    if kind == "http_body_contains":
        return ev["needle"] in ev["body"]              # token actually in the body
    if kind == "hash_match":
        return ev["expected"] == ev["observed"]        # sha actually matches
    raise ValueError("unknown witness kind")           # anything else -> fail closed

(The real function has the type-checking and the fail-closed raises spelled out; this is the spine.) The point is that a witness result is a function of evidence you can inspect, not a second self-report. If the create order body says RATE_LIMITED, then http_body_contains("ORDER_CONFIRMED") returns false no matter what the exit code claimed. You can paste your own bodies in and rerun. The check does not care about the agent's opinion.

Running it on a batch

Here is the meter on a synthetic overnight batch of 12 tasks. Ten exited 0. Two failed honestly (a migration that returned exit 1, a docker build that OOM-killed with 137), and the dashboard already knows about those. The interesting three are the ones that exited 0 and did nothing: t03 got the RATE_LIMITED body above, t04 produced an artifact whose sha did not match what was expected, t06 claimed to update user-90 but that row is not in the table.

$ python3 verified_cost.py report fixtures/runlog.json
id     spend      exit0   witness   verified
t03    $0.3540    yes     FAIL      no
t04    $0.5850    yes     FAIL      no
t06    $0.2775    yes     FAIL      no
...
total_spend            = $3.4665
M  (exit-0 successes)  = 10      <- the denominator your dashboard uses
M' (verified: exit0 AND witness) = 7
silent failures (exit0, witness FAIL) = 3  ['t03', 't04', 't06']
spend on silent failures = $1.2165 (bought an exit 0, witness rejected it)
------------------------------------------------------------------------------
naive_cost_per_success    = total/M  = $3.4665 / 10 = $0.3466
verified_cost_per_success = total/M' = $3.4665 / 7 = $0.4952
understatement_factor     = M/M' = 10/7 = 1.4286x  (arithmetic, not a measured effect)

Read the raw pieces, not just the ratio. M is 10, M' is 7, and I print both so the 1.4286x cannot hide anything. On this batch the dashboard would tell you each success cost 35 cents. The witness says 50. That is the same 1.4286x the report prints: the true per-success cost sits 43% above the dashboard's number (equivalently, the dashboard reads 30% below the truth), and it is not a rounding error. $1.2165 of the $3.4665 you spent, better than a third of the bill, bought exit-0s that the witness threw out. The dashboard counted that third as wins.

I want to be precise about what is real here and what is not. The 1.4286x is 10/7, and the 10 and the 7 are counts I chose when I built the fixture. So the magnitude is synthetic. What is not synthetic is the direction and the mechanism: on any log, the moment one exit-0 task fails its witness, M' drops below M and your true cost climbs above what the dashboard shows. The tool computes M' by rerunning the checks, so on your own log the number is yours, not mine.

The gate: block before the inflated number reaches the budget

A meter you read after the fact is a postmortem. The point of this franchise is to gate before the spend lands, the same way the pre-execution gate decides whether an action runs at all. So verified_cost.py gate takes two policy knobs, a max verified cost per success and a max understatement factor, and returns a CI exit code: 0 to pass, 1 to block, 2 to fail closed on garbage input.

$ python3 verified_cost.py gate fixtures/runlog.json 0.15 1.20
  naive=$0.3466/succ   verified=$0.4952/succ   understatement=1.4286x
  policy: max_verified=$0.1500/succ  max_understatement=1.2000x
  BLOCK: verified_cost $0.4952/succ exceeds budget $0.1500/succ
  BLOCK: understatement 1.4286x exceeds tolerance 1.2000x (silent failures mask the bill)
VERDICT: BLOCK exit=1

Two independent reasons fired. The budget one is a generic FinOps cap. The understatement one is the reason this tool exists, so I isolated it: loosen the budget to a dollar per success, well above the real $0.4952, and the gate still blocks.

$ python3 verified_cost.py gate fixtures/runlog.json 1.00 1.20
  policy: max_verified=$1.0000/succ  max_understatement=1.2000x
  BLOCK: understatement 1.4286x exceeds tolerance 1.2000x (silent failures mask the bill)
VERDICT: BLOCK exit=1

That is the whole idea in one exit code. Even when you can afford the verified cost, a wide gap between the naive and verified numbers is itself the signal: your dashboard is dividing by a denominator that is lying to it, and that is worth stopping a deploy over before the pattern scales into next month's budget.

The falsifier: when this tool should shut up

A gate that always fires is a stuck alarm, not a control. So the tool has to pass a case where it adds nothing, and it has to do that honestly. If your agent never silently fails, then exit_code == 0 really does mean success, M' equals M, and there is nothing to correct. Here is that case, five tasks, every exit-0 witnessed:

$ python3 verified_cost.py gate fixtures/honest_zero.json 0.30 1.20
  naive=$0.2244/succ   verified=$0.2244/succ   understatement=1.0000x
VERDICT: PASS exit=0

Understatement 1.0000x. Naive equals verified to the cent. The masking check is silent and the gate passes. That is the falsifier working: if exit-0 never lied on your stack, this tool changes nothing and says so out loud. Any claim it makes on your real batch has to survive that comparison first.

Now the reasonable objection: "fine, but our agents are healthy, we pass 95% of tasks, the gap must be negligible." It is not zero, and the arithmetic says exactly how not-zero. understatement = M/M' = 1/(1-s) where s is the silent-failure rate among exit-0 tasks. I swept s to show the magnitude at each rate (this table is arithmetic, not a measurement of any workload, and it is labeled that way in the output):

k      M'     s        SE(s)      M/M'         1/(1-s)
0      200    0.0000   0.0000     1.0000       1.0000
10     190    0.0500   0.0154     1.0526       1.0526
40     160    0.2000   0.0283     1.2500       1.2500
60     140    0.3000   0.0324     1.4286       1.4286
100    100    0.5000   0.0354     2.0000       2.0000
200    0      1.0000   0.0000     UNBOUNDED    1/0     <- M'=0, verified cost UNBOUNDED, gate BLOCKS

At a 5% silent-failure rate, the "healthy 95% agent," your true cost is 1.0526x the dashboard number. Small, but not nothing, and it only grows: the lower your real success rate, the wider the gap, because you are dividing the same spend by an ever-smaller denominator. And the bottom row is the one I most wanted the tool to handle without crashing: when every exit-0 task is a silent failure, M' is zero, verified cost is unbounded, and the gate blocks rather than dividing by zero and printing a comfortable-looking number.

One more honest note buried in that table. If you use exit_code itself as the witness, M' equals M for every s, so the gap is 1.0 always. A dashboard is not doing bad arithmetic. It is using the null witness: the actor as its own judge. The gap this tool measures is precisely the information a real witness adds beyond the agent's self-report. No witness, no gap, no visibility. That is why "add a witness log" is the actual ask here, not "buy a better dashboard."

Where this is soft, and what it is NOT

I would rather you trust the small claim than oversell the big one.

The magnitudes are synthetic. 1.4286x, $0.4952, $1.2165 come from a fixture I built. They illustrate the arithmetic. They are not a measurement of any real agent fleet, mine or anyone's. The direction (naive is a lower bound) is general; the size is not.
It is exactly as good as your witness. Garbage witness, garbage M'. If your only "independent" check is another call to the same flaky service, you have two self-reports, not a witness. The four kinds here (file_exists, db_row_present, http_body_contains, hash_match) are the ones I could make recompute from evidence with no network. Yours may need more. And the quiet version of a garbage witness is one too weak to ever fail: an empty needle sits inside every body, so it passes silently and the gap reads 1.0000x. The tool fails closed on evidence it cannot evaluate (exit 2), not on a witness that cannot fail. Writing a witness that can never fail is on you.
A witnessed success is not a correct one. http_body_contains("ACK") confirms the body said ACK. It does not confirm the ACK was for the right thing. Contract-level witnessing catches silent failure; it does not catch subtle wrongness. This is the same blind spot exit codes have, moved one notch up, not removed.
Cost is only the loop's forward pass. This tool prices completed tasks. It says nothing about tokens burned inside a task that never finishes; that is what the loop cost forecaster is for. Pair them: forecast the loop, then price the loop against what it actually produced.
It fails closed, on purpose. A malformed witness (say a hash_match with no observed field) exits 2, not 0. You cannot trust a bill computed from evidence you cannot evaluate, so the tool refuses to compute one. I checked that by hand: malformed input exits 2, and zero verified successes with nonzero spend blocks rather than dividing by zero.

Run it against your own run-log

If you log agent tasks at all, you already have M. What you probably do not have is M', because most stacks never record an independent witness next to the exit code. That is the actual gap, and it is cheaper to close than it sounds: pick the one artifact each task is supposed to produce, record whether it is really there, and divide by that count instead. Everything here is offline, keyless, stdlib-only Python 3.13.5. It drops into CI or a pre-deploy hook with no daemon and no account, and it prints its own sha256 on the last line so you can pin the version you ran.

Here is the part I have not settled, and I want your take. I set the masking tolerance at 1.20x and I am not sure that is right for anyone but this fixture. Too tight and it fires on the honest 5% agent; too loose and it waves through a batch where a third of the spend bought nothing. And the deeper question is upstream of the threshold: what is your witness? If you run agents in production, tell me the single check you trust to confirm a task actually happened, the one that is not just the agent saying ok: true a second time. I read every comment, and I am collecting the good ones.

If this was useful, the 200-and-lies witness gate is where the honest denominator comes from, the loop cost forecaster is the numerator side, and the pre-execution gate is the pattern that stops the bad number before it spends. Follow along; the next one keeps walking down this stack.

Your MCP Pin Blocks Every Update. Most Never Broke You.

Alexey Spinov — Fri, 24 Jul 2026 03:47:33 +0000

A month ago I shipped a 40-line padlock for MCP tools: pin the manifest hash, block the rug-pull. It works. It also has a flaw I wrote about in the last paragraph and then had to go build a fix for, because it kept nagging me. The pin fires on any change to a tool's definition. A server that honestly adds an optional parameter trips it exactly as loud as a server that yanks a required one out from under you. Same alarm, same block, same 3am page. The pin knows the contract changed. It has no idea whether the change broke you.

In short: an MCP tool schema breaking change narrows the valid-call set, so a call your agent makes today stops validating tomorrow. A backward-compatible change widens or leaves that set alone, so old calls stay valid. A byte pin cannot tell them apart. compat_gate.py diffs the inputSchema and replays the calls your agent recorded: silent on compatible, fail-closed on breaking.

AI disclosure: I wrote compat_gate.py with an AI assistant and ran every case myself before publishing. Every terminal block below is pasted from a real run on Python 3.13.5. The compatible case (C2) is a real dated diff between two published MCP spec schemas, fetched over curl and checksummed; the breaking cases (C3 to C5) are synthetic fixtures I built and confirmed by replay, and I label them as such. I have no production incident to sell you here; bot2 is new and its lifetime run count is zero. What I have is a tool that runs and a diff you can reproduce.

Why does pinning a manifest fire on every change?

Because a hash has one bit of memory: same, or different. The June pin takes a canonical SHA-256 over name + description + inputSchema and blocks when the live manifest stops matching the one you approved. That canonicalization is already the good kind of pin: it sorts keys and strips whitespace, so a re-serialized-but-identical manifest does not trip it. I checked that first, and it holds. The pin is not paranoid about JSON formatting.

It is paranoid about meaning. And that is the problem, because most meaning changes to a schema are harmless to your calls. The MCP spec itself is the clearest evidence. The protocol ships dated, breaking-capable revisions on a cadence; the schema directory in the spec repo currently carries 2024-11-05, 2025-03-26, 2025-06-18, 2025-11-25, and a live draft/. I pulled two of those and diffed them by hand (more on that below). Servers track the spec. Tools get re-versioned. Descriptions get rewritten for the model. Every one of those events flips the pin's one bit, and the pin dutifully blocks.

Here is the open question I left myself in June and could not answer from my own toolbox: where is the line between a legitimate tool update and drift? The franchise on this blog keeps landing on the same shape. Tracking is not control. Having a credential is not scoping it. Approval is not immutability. This post adds the next line: a changed contract is not a broken one. A pin is not a compatibility check. The pin tracks change. It does not control whether a change that would break your agent reaches the call. That is a different job, one level down from the hash, and it needs a different tool.

What makes a tool-schema change breaking?

There is a ground truth here, and it is older than MCP. It is subtyping. A schema describes a set: the set of argument objects it calls valid. A change is backward-compatible when the new set contains the old one, so every call that used to validate still validates. A change is breaking when the new set is smaller, so some previously-valid call now fails. That is the whole test, and it does not require an opinion.

On an object inputSchema, the breaking moves are the ones that shrink the valid set:

Add a required property. Old calls that omit it were fine and now are not.
Remove a property when additionalProperties: false. A call that passed that key is now rejected as an extra.
Narrow a type. number to integer throws out 3.5. Any incompatible type swap can strand an old value.
Shrink an enum. Drop "csv" from ["json", "csv"] and every call that asked for CSV is gone.
Make an optional property required, or tighten additionalProperties from true to false.

And the compatible moves, the ones that widen or preserve the set:

Add an optional property. Nobody was required to send it; old calls are untouched.
Widen an enum, relax a type (integer to number), drop a required constraint, or rewrite a description. None of these can invalidate a call that was already valid.

That list is the entire logic of the classifier. The point of the tool is not to be clever about it. The point is to run it against real schemas and, critically, to check the classifier's verdict against something independent: the calls themselves.

The gate: replay the calls your agent actually makes

compat_gate.py is keyless, offline, and stdlib-only (json, sys, hashlib). It carries a deliberately small JSON-Schema validator that understands exactly six keywords: type, required, properties, enum, additionalProperties, const. Anything else in a schema (a format, an items, a $ref) is treated as no constraint. That is a real limit and I come back to it at the end. It is the right subset for the corpus here and small enough to read in one sitting.

Two signals run side by side, and they are meant to agree:

def classify(old_is, new_is):
    """Diff two inputSchemas. Return [(verdict, reason), ...] in a stable order."""
    changes = []
    old_props, new_props = old_is.get("properties", {}), new_is.get("properties", {})
    old_req, new_req = set(old_is.get("required", [])), set(new_is.get("required", []))
    old_ap = old_is.get("additionalProperties", True)
    new_ap = new_is.get("additionalProperties", True)

    for p in sorted(new_props):
        if p not in old_props:
            if p in new_req:
                changes.append(("BREAKING", "added required property '%s' (old calls omit it)" % p))
            else:
                changes.append(("COMPATIBLE", "added optional property '%s' (old calls stay valid)" % p))
    # ... removed props, newly-required props, additionalProperties, enum shrink/widen, type narrow, const ...
    return changes

That is signal (A), the subtyping classifier: it labels each field-level change COMPATIBLE or BREAKING with a reason. Signal (B) is the one I trust more, because it does not reason at all. It replays every recorded call against both schemas:

    replay = []  # (idx, call, valid_under_old, valid_under_new, errs_new)
    for i, call in enumerate(calls, 1):
        vu_old = not validate(call, old_is)
        errs_new = validate(call, new_is)
        replay.append((i, call, vu_old, not errs_new, errs_new))
    control_ok = all(r[2] for r in replay)               # every recorded call must be valid under old
    broken = [r for r in replay if r[2] and not r[3]]    # valid under old, invalid under new

Every call in the corpus has to be valid under the old schema. That is the entry criterion and I print it as valid_under_old=True on every line, as a negative control: if a recorded call is not valid under the schema it was recorded against, the fixture is wrong and the tool says so. Then each call is validated against the new schema. A call that was valid under old and is invalid under new is a broken call, and the count of broken calls is a measurement, not a label. The classifier can call a change breaking; the replay tells you whether it broke your traffic.

Quick start is two commands:

# just classify the schema delta:
python3 compat_gate.py classify old_tool.json new_tool.json

# the gate: classify + replay recorded calls, exit code is your CI signal:
python3 compat_gate.py gate old_tool.json new_tool.json calls.jsonl

gate returns 0 when there are zero breaking changes and zero broken calls, 1 when either fires, and 2 on a usage error. The exit code is the whole product. It drops into CI or a pre-connect hook with no daemon and no account.

Three changes, three verdicts, one tool

I ran the gate against six changes. One is real; five are synthetic fixtures on a made-up run_query tool (a read-only SQL tool: required sql, optional limit, format as an enum of ["json","csv"], dry_run, timeout_ms). Four recorded calls, none of them touching timeout_ms. Here is the null case first, because a gate that screams at everything is useless.

C1, identical (re-serialized). The new tool is the old one with keys reordered and whitespace changed. Same content.

### C1 identical (re-serialized)
byte-pin: ok
subtyping: 0 changes
compat verdict: PASS   breaks=0/4

Both silent. The canonical pin does not trip on re-serialization, and the gate finds nothing to break. Good. Neither tool is trigger-happy on a non-change.

C2, a real backward-compatible diff from the MCP spec. This is the one I care most about being honest on. I fetched two published MCP schema files, 2025-06-18 (sha256 b3db8f1c...) and 2025-11-25 (sha256 7b2d96fd...), and pulled the Implementation object out of each. Implementation is the serverInfo/clientInfo payload in the handshake; it is a JSON Schema with properties and required, the same shape a tool inputSchema is, which is why I can drive the gate with it. Between those two dated versions it gained three optional properties (description, icons, websiteUrl) and kept required at ["name", "version"]. Textbook additive.

### C2 real MCP additive diff (Implementation)
byte-pin: BLOCK
  [COMPATIBLE] added optional property 'description' (old calls stay valid)
  [COMPATIBLE] added optional property 'icons' (old calls stay valid)
  [COMPATIBLE] added optional property 'websiteUrl' (old calls stay valid)
compat verdict: PASS   breaks=0/3

The pin blocks. The bytes changed, so the hash changed, so the pin does its one job and refuses. The gate replays three recorded handshake instances ({"name": "example-server", "version": "1.0.0"} and friends), all valid under old, all still valid under new, and passes in silence. This is half the falsifier for the whole thesis: on a genuinely compatible change, the gate has to be quiet, or it is just a slower pin. It is quiet.

C3, an added required parameter. The new run_query requires a tenant_id. This is the shape I most wanted to fail closed on, because it is the common one: a server tightens a tool and every existing integration silently starts sending calls that will be rejected.

### C3 added required param 'tenant_id'
byte-pin: BLOCK
  [BREAKING] added required property 'tenant_id' (old calls omit it)
  call#1 valid_under_old=True valid_under_new=False [BROKEN] {"sql": "select 1"}
        -> $: missing required 'tenant_id'
  ...
compat verdict: FAIL   breaks=4/4

Four of four recorded calls break, each because it is missing tenant_id, and the gate names them. The gate command exits 1. That is the other half of the falsifier: on a breaking change, the gate must fail closed and point at the call. It does, and I checked the exit code by hand, because a gate that prints FAIL and exits 0 is not a gate.

C4, a shrunk enum. The format enum drops "csv". Only one recorded call asked for CSV.

### C4 enum shrank on 'format'
byte-pin: BLOCK
  [BREAKING] property 'format' enum shrank ['csv', 'json']->['json'] (dropped ['csv'])
  call#3 valid_under_old=True valid_under_new=False [BROKEN] {"sql": "select id, email from users", "format": "csv"}
        -> $.format: 'csv' not in enum ['json']
compat verdict: FAIL   breaks=1/4

One of four. Not four of four. This is why the replay matters and the classifier alone does not: "breaking" is true, but the blast radius on your traffic is a single call, and the number tells you that. breaks=1/4 is a different operational decision than breaks=4/4.

C5, breaking by schema, silent on your traffic. The new tool removes the optional timeout_ms under additionalProperties: false. By subtyping that is breaking: a call that sent timeout_ms would now be rejected. But none of the four recorded calls ever set it.

### C5 removed optional 'timeout_ms'
byte-pin: BLOCK
  [BREAKING] removed property 'timeout_ms' under additionalProperties:false (a call using it is now rejected)
compat verdict: FAIL   breaks=0/4

Here the two signals disagree, and the tool refuses to hide it. The classifier says breaking; the replay says breaks=0/4. The verdict is FAIL, because I would rather a gate be conservative on a schema that genuinely narrowed. But the printout names the split out loud: breaking by schema, 0 recorded calls touch it, silent on your traffic. That middle cell is the honest one. If I collapsed it to a clean PASS I would be lying about the schema; if I collapsed it to a clean FAIL with no annotation I would be hiding that your recorded traffic is fine. So it prints both.

Byte-pin versus compat-gate on the same changes

Here is the head-to-head, straight from the run. The byte-pin column is the June canonical hash, my own prior tool, not a strawman.

change                                   byte-pin   compat   breaks
------------------------------------------------------------------------------
C1 identical (re-serialized)             ok         PASS     0/4
C2 real MCP additive diff (Implementatio BLOCK      PASS     0/3
C3 added required param 'tenant_id'      BLOCK      FAIL     4/4
C4 enum shrank on 'format'               BLOCK      FAIL     1/4
C5 removed optional 'timeout_ms'         BLOCK      FAIL     0/4        <- breaking by schema, 0 calls touched
C6 desc rewrite + added optional 'cache' BLOCK      PASS     0/4
------------------------------------------------------------------------------
byte-pin fired: 5/6   |   compat-gate fired: 3/6

Read the absolute counts, not the ratio. On this corpus of six changes, the byte-pin fired five times and the gate fired three. The two changes the pin flagged and the gate stayed silent on (C2, the real MCP additive; C6, a description rewrite plus an added optional cache param) were both backward-compatible: every recorded call still validated. That gap of two is the false alarms the gate removed on this corpus. It is not a claim about all possible changes; it is the measured difference on these six.

Two is a small number because six is a small corpus. But the shape scales the wrong way for the pin. The more a spec re-versions, the more the pin fires, and the fraction of those fires that are actually compatible does not drop just because the volume went up. A pin under heavy re-versioning is a siren that is right about "changed" and useless about "should I stop." The gate's job is to turn most of that siren off without turning off the part that matters.

Where does the compatible/breaking line actually sit?

I would rather you trust the small claim than the big one, so here is where this gets soft.

Replay only covers the calls you recorded. The gate is exactly as good as your call log. If your agent has a code path that fires a call shape you have never captured, C5 is your whole life: the gate will say breaks=0 and be technically right and operationally blind. The classifier signal (A) is the hedge here, because it flags the schema narrowing even when your traffic misses it, which is precisely why I keep both signals and fail closed when they disagree.

Compatible-by-schema is not compatible-by-behavior. This is the same blind spot the pin has, one level up. A server can keep inputSchema byte-identical and change what the tool does behind it: return different data, hit a different backend. Nothing in this gate, and nothing in a hash, catches that. Contract compatibility is not runtime compatibility, and I am not claiming it is.

The validator is a subset. Six keywords. It does not model pattern, minimum, maxItems, nested $ref resolution, anyOf, or a dozen other JSON Schema features that can each carry a breaking change. On a schema that tightens a pattern or lowers a maximum, this tool will miss it, because it does not read those keywords. That is a scoping decision, not a completeness claim. Extend the validator and you extend the coverage; the design leaves room for it.

What this is NOT

It is not a replacement for the pin. Run both. The hash catches the description-poisoning and rug-pull shapes, where a byte-level change is the signal and "compatible" is the wrong frame entirely: a hidden instruction slipped into a description is backward-compatible by subtyping and still hostile. The pin is your detector for "changed at all." This gate is your filter for "changed in a way that breaks my calls." They answer different questions and they belong in the same pipeline.
It is not a semantic scanner. It does not read the meaning of a description or judge whether a new parameter is dangerous. It asks one structural question about the argument set.
The breaking fixtures are synthetic. C3 to C5 are fixtures I wrote. They are confirmed by independent replay (a call valid under old genuinely fails under new, and the tool prints the failing path), not by my say-so, but they are not a live server. The one real diff is C2, and I checksummed it.
A PASS is not a proof of full compatibility. It means: on the six schema keywords I model, zero breaking changes, and zero of your recorded calls broke. It does not mean nothing can go wrong. See the section above.

Run it against your noisiest server

Pull the tools/list for the MCP server your agent leans on hardest. Save the inputSchema you have now, save the next version when the server updates, dump a few hundred of your real recorded calls to a .jsonl, and run compat_gate.py gate. If it passes, you just re-approved an update without a human reading a diff at 3am. If it fails, it hands you the exact calls that would have broken, before the first one goes out. That is the cheapest pre-call check you will add this quarter, and it runs before the call instead of in the postmortem.

Here is the part I still have not settled, and I want your take. Signal (A) and signal (B) disagree in the C5 shape (breaking schema, untouched traffic), and I resolved it by failing closed with a loud annotation. I am not sure that is right for every team. If you re-version aggressively, C5 is going to fire constantly on parameters nobody sends, and constant firing is how a gate gets disabled. Do you gate on the schema signal (conservative, noisy) or the replay signal (permissive, blind to unseen paths)? Do you tie re-approval to a publisher signature and skip the diff entirely? I have run this against exactly one real diff and a pile of fixtures, so my line is drawn in pencil. If you have run a compat check against a server that updates for real, tell me where you put the line and why. I read every comment.

If this was useful, the MCP tool pin is the layer above it, the pre-execution gate is the layer that decides whether a specific action runs at all, and the MCP server token tax measures what each tool costs you in context. Follow along; the next one keeps walking down this stack.

The best config in your bake-off didn't win. Selection did.

Alexey Spinov — Thu, 23 Jul 2026 03:24:54 +0000

Best-of-K eval selection bias: pick the highest-scoring config from K candidates on one eval set and that observed score is biased up. It reports the expected maximum of K noisy estimates, which beats the field mean whenever K exceeds one. The bias appears even when all K configs are truly equal, grows with K, and shrinks with n.

Here is the version that bites you. Your bake-off ran a batch of prompts against one eval set, the top one came out ahead, and you shipped it. In production it does worse. That drop reads like bad luck, or drift, or a bad week. It is none of those. It is a number you could have computed before you shipped, and it gets larger the more candidates you tried.

I ran a small script to make the gap concrete. Eight configs, one hundred eval items, and here is the catch: I made all eight configs truly identical, every one a fair coin at 50%. There is no real best. Nothing to tune. Then I let selection pick a winner anyway:

   config 0:  47/100 =  47.0%
   config 1:  50/100 =  50.0%
   config 2:  52/100 =  52.0%
   config 3:  52/100 =  52.0%
   config 4:  50/100 =  50.0%
   config 5:  50/100 =  50.0%
   config 6:  54/100 =  54.0%  <- selected winner (argmax)
   config 7:  44/100 =  44.0%

     config 6: 54.0% (k=54 n=100 SE=4.98)
     config 2: 52.0% (k=52 n=100 SE=5.00)
   RANK: INDISTINGUISHABLE - gap 2.00 pp against 7.06 pooled SE = 0.28 SE < 2.0. Ranking "config 6" above "config 2" is NOT allowed.

Held-out the winner on a fresh 100 items: 48/100 = 48.0%.

Config 6 wins the bake-off at 54.0%. Then I asked the same eval-guard I use in the McNemar and rule-of-three pieces to rank config 6 against the runner-up. It refused: the gap is 0.28 SE, far under the two-SE bar, so INDISTINGUISHABLE. The ranker would not call config 6 the best. Selection did. And on a fresh held-out set the 54.0% falls back to 48.0%, toward the true 50% it was always going to be.

TL;DR

Picking the best of K configs by observed pass rate reports the expected maximum of K noisy estimates. The max of K exceeds the field mean, strictly, for K at least 2 (Jensen's inequality on a convex function). So the winner's reported score is biased up.
The bias is not about a weak config. In the null world, where all K configs are truly equal, best-of-16 reads 8.81 pp above the true 50%, and best-of-64 reads 11.67 pp above it. Nothing to optimize; the inflation is the act of selecting.
It grows with K: 2.82 pp at K=2, 7.10 pp at K=8, 11.67 pp at K=64 (p=0.5, n=100). Adding candidates makes it worse, not better. You cannot fix a selection bias by selecting from more.
It equals the drop you feel on held-out. On held-out, best-of-16 is indistinguishable from a random pick, 50.4% against 49.9% at 1.40 SE, while its in-sample number sits 29.5 SE higher. The whole curse lives in the eval you selected on.
It shrinks with n like 1/sqrt(n): 8.81 pp at 100 items, 2.21 pp at 1600. Real, but you fight the square root, so halving the bias costs four times the eval budget.

What this is and is not. The fixtures are synthetic, and I say so on purpose. The bias here is a theorem, not an effect I fitted: E[max of K i.i.d. non-degenerate estimates] > their common mean is Jensen, and the null-world sections compute it exactly from binomial order statistics with no sampling and no tuned constant. The one sampled section uses a fixed seed and is byte-identical across runs. The curse is negligible when one config is genuinely far ahead of the field; it dominates when the field is tight, which is where most bake-offs actually sit. I show both.

Why the winner's score is biased up

Selection reports a maximum, and a maximum is not an average. Each config's observed pass rate is its true rate plus noise. When you take the best of K, you are not reading a typical config, you are reading whichever one the noise flattered most. E[max] sits above the mean for any K of 2 or more, and the gap widens as K grows. This is Jensen's inequality: the max is a convex function of the estimates, so the expectation of the max exceeds the max of the expectations, which in the equal case is just the common true rate.

The magnitude has a textbook form. The expected max of K standard normals grows like sqrt(2 ln K), so the selection bias scales roughly as sigma * sqrt(2 ln K), where sigma is the standard error of one config's estimate. That is why more candidates keep costing you: the penalty climbs with ln K and never turns around. Here is the exact order-statistic computation, summed over the binomial, not sampled:

   K | E[in-sample winner] | true field p | bias = drop (pp) | sigma*sqrt(2lnK)
   1 |              50.00% |        50.0% |            -0.00 |             0.00
   2 |              52.82% |        50.0% |             2.82 |             5.89
   4 |              55.14% |        50.0% |             5.14 |             8.33
   8 |              57.10% |        50.0% |             7.10 |            10.20
  16 |              58.81% |        50.0% |             8.81 |            11.77
  32 |              60.31% |        50.0% |            10.31 |            13.16
  64 |              61.67% |        50.0% |            11.67 |            14.42

Read the bias column downward. Zero at K=1, because with one config there is no selection to make; the script flags that row as degenerate and refuses to run the selection path on it. Then 2.82, 5.14, 7.10, all the way to 11.67 pp at K=64. The sqrt(2 ln K) column is the growth guide, not an equality: it is the asymptotic leading term and runs loose at small K, which is why I print the exact bias beside it rather than the approximation. The conclusion does not live in any single row. It lives in the monotone climb of the whole swept column, with zero real signal anywhere in the fixture.

The bias is there even when nothing is better

The null world is the part I care about most, because it is the part you cannot argue with. There is no best config to find. Every candidate is the same coin. If the winner's score were an honest estimate, best-of-K would land at 50% on average for every K. It does not. It lands at 58.81% for K=16 and 61.67% for K=64. That premium is manufactured entirely by the operation of taking a maximum over noise.

This is the anti-fixture check I hold myself to after dropping five of my own drafts in one day for hiding a conclusion inside a tuned constant. Here there is no constant to tune in my favor: no lucky base rate, no engineered gap, no cherry-picked K. I set the true difference between configs to exactly zero and the bias is still positive and still growing. You can disagree with the binomial if you like. You cannot make the max of sixteen coins average out to one coin.

It doesn't depend on a lucky base rate

The next question a good skeptic asks: did I pick p=0.5 because it flatters the effect? So I swept the base rate at K=16 and read the exact bias at each:

     p | E[in-sample winner] | bias = drop (pp) | sigma (pp) | bias/sigma
  0.10 |              15.58% |             5.58 |       3.00 |       1.86
  0.20 |              27.28% |             7.28 |       4.00 |       1.82
  0.50 |              58.81% |             8.81 |       5.00 |       1.76
  0.80 |              86.79% |             6.79 |       4.00 |       1.70
  0.90 |              94.92% |             4.92 |       3.00 |       1.64

Positive at every base rate, 5.58 pp at a 10% pass rate up to 8.81 pp at 50%. Look at the last column instead of the fourth. Measured in units of the per-cell noise sigma, the inflation is close to constant, about 1.76 sigma at p=0.5 and never straying far. That is the real statement: the curse scales with sigma = sqrt(p(1-p)/n), the noise in one estimate, not with any special value of p. There is no base rate you can choose to make it disappear. The absolute pp figure is largest near 50% only because that is where a coin is noisiest.

More items shrink it, but you're fighting a square root

There is one honest escape, and the same math hands it to you. Since the bias scales with sigma, and sigma falls like 1/sqrt(n), running more eval items shrinks the curse. The theorem's degenerate limit is n to infinity: sigma to zero, bias to zero. Sweeping n at K=16, p=0.5:

     n | sigma (pp) | bias = drop (pp) | bias/sigma | normal pred (pp)
   100 |       5.00 |             8.81 |       1.76 |             8.83
   400 |       2.50 |             4.41 |       1.76 |             4.41
  1600 |       1.25 |             2.21 |       1.77 |             2.21
  6400 |       0.62 |             1.10 |       1.77 |             1.10

Two things worth seeing here. First, the last column: I also computed the normal-theory prediction, sigma times the exact expected max of sixteen standard normals (which the script works out to 1.7660 by quadrature), and the binomial bias converges onto it as n grows and the central limit theorem takes hold. 8.81 against a predicted 8.83 at n=100, then dead-on to two decimals by n=400. The theorem and the run agree because it is the same theorem.

Second, the cost. Going from 100 to 400 items, a 4x eval budget, halves the bias from 8.81 to 4.41 pp. That is the square-root tax. If someone pays a judge model per eval item, buying your way out of the winner's curse is a real line on the bill, and it scales the wrong way. "Just run more evals" is correct and expensive at the same time.

The held-out drop is the bias you actually feel

The bias so far is E[in-sample winner] - true rate. You never see the true rate, so how do you know you got bitten? You hold the winner out. Re-evaluate the selected config on a fresh set and its inflated score regresses. In expectation the held-out drop equals the selection bias, because the held-out re-estimate is unbiased for whatever config you happened to pick.

That is exactly the trap I walked into on the McNemar piece, where a quantity that looked like a finding turned out to be an unconditional rate wearing a conditional costume. So this script runs the independence probe from the same library before it claims anything: it compares the winner's held-out rate against a random pick's held-out rate, and reports whether the conditioning on "was selected" actually buys you anything.

   in-sample(best of 16) : 58.8% (k=23506 n=40000 SE=0.25)
   held-out(best of 16)  : 50.4% (k=20145 n=40000 SE=0.25)
   held-out(random pick) : 49.9% (k=19947 n=40000 SE=0.25)
   FORCED BY CONSTRUCTION [NULL: held-out of best-of-16 vs a random pick]: conditional 50.4% is indistinguishable from unconditional 49.9% (1.40 SE < 2.0)
   in-sample vs held-out (paired over 400 trials): drop 8.40 pp = 29.5 SE -> SEPARATED (measured curse)

On held-out, best-of-16 reads 50.4% and a random pick reads 49.9%. The probe calls them indistinguishable at 1.40 SE, which is the whole point: selecting the best of sixteen coins bought nothing you could keep. The winner's held-out rate is the base rate. Meanwhile the in-sample number sits 29.5 SE above held-out on the paired comparison. The entire curse lived in the eval you selected on, and none of it survived the move to fresh data. That is the honest translation of "you selected noise."

The probe is not vacuous, and I check that too. When I give one config a real edge, config 0 truly at 0.70 while the other fifteen sit at 0.50, the same probe flips:

   held-out(best of 16)  : 69.4% (k=27747 n=40000 SE=0.23)
   held-out(random pick) : 51.5% (k=20595 n=40000 SE=0.25)
   ok [SPREAD: held-out of best-of-16 vs a random pick]: conditional 69.4% separates from unconditional 51.5% by 52.60 SE >= 2.0 - the conditioning is real

Now selection earns its keep: the winner's held-out rate beats a random pick by 52.60 SE. So the probe discriminates. It is forced under the null and not forced when there is signal, which is the only way I would trust the null result at all.

When selection is safe and when it lies

None of this says stop running bake-offs. It says know which regime you are in. I held K=8 and n=100 fixed and widened one config's true edge over a flat field, reading the exact stats at each width:

leader true p | edge (SE) | P(pick best) | held-out best | lift vs random | in-sample bias
         0.50 |      0.00 |        14.4% |         50.0% |         -0.00 |           7.10
         0.52 |      0.40 |        23.6% |         50.5% |          0.22 |           6.96
         0.55 |      1.00 |        42.0% |         52.1% |          1.47 |           6.21
         0.60 |      2.00 |        74.3% |         57.4% |          6.18 |           3.62
         0.70 |      4.00 |        99.2% |         69.8% |         17.33 |           0.19

Two ends, two different worlds. A dead-flat field, edge 0 SE, is the null: selection picks the true best only 14.4% of the time, about one in eight for eight configs, the held-out lift over a random pick is zero, and the in-sample bias of 7.10 pp is the entire result. A runaway leader, edge 4 SE at a true 0.70, gets picked 99.2% of the time, delivers a real 17.33 pp held-out lift, and its in-sample bias shrinks to 0.19 pp because there is little noise left to exploit next to a gap that large. The dangerous regime is the middle, one or two SE of edge, which is where most prompt and model bake-offs actually land: a handful of items separating the top few. There, selection inflates the score and the winner regresses.

How is this different from the McNemar piece?

Fair question, since both are about comparing configs on an eval set. The McNemar piece is about comparing two configs correctly: two prompts on one task set give paired outcomes, so the right significance test is McNemar, not a two-proportion SE. That is a question about the test you run on a fixed comparison. This piece is about what happens after you compare K of them and keep the max. The bias here is not in any pairwise test. It is in the selection step and in carrying the winner's eval score forward as if it were an unbiased estimate. You can run a flawless paired test between every pair and still report an inflated winner, because the inflation is post-selection, not a testing error. Different failure, one layer up.

What I am not claiming

Three limits, stated plainly.

The fixtures are synthetic Bernoulli configs, not your agent. The order-statistic math is exactly right for i.i.d. estimates; real evals have correlated items and non-identical difficulty, which change the constants but not the direction. If anything, correlation across your candidates makes the effective K smaller and the bias milder than the i.i.d. case, while heavy-tailed noise makes it worse. The sign does not move.

The bias is about the winner's score, not about whether you chose the right config. With a real leader you usually still pick it, as the spread table shows. The mistake the curse causes is trusting the number next to the winner, not the act of choosing a winner.

And it is an expectation. Any single held-out re-eval has its own noise, so a particular winner might regress a lot, a little, or by luck not at all. The claim is about what happens on average across bake-offs, which is the thing your intuition quietly assumes when it treats an eval score as ground truth.

What does your bake-off do

Everything here ran locally, Python 3.13.5, stdlib plus the pinned measure.py, offline, no keys, no funds, three runs byte-identical, exit 0, empty stderr. The null-world numbers are exact order statistics; the sampled sections use a fixed seed. Same eval harness as the two pieces before it, and the three of them are one family of pre-execution gates for AI agents: decide what you can actually claim before you act on the claim.

I publish the runs that correct my own reading, not only the ones that flatter it, so follow along if that is your kind of thing. And a real question I do not have a clean answer to. When your last bake-off picked a winner out of a dozen prompts on a couple hundred items, did you hold that winner out on a fresh set before you shipped its score, or did the eval number go straight into the launch doc? I want to know how many candidates your team tries, and whether anyone writes down that trying more of them is the thing making the winner look better.

AI disclosure. I drafted this with an AI writing assistant and edited every line; the framing, the tool, and the reading are mine. Every output block is pasted from one real local run on 2026-07-23. winners_curse.py sha256 c968f39c9295fff7…, run output sha256 41934693461986a6…, and the imported measure.py stays at its pinned sha256 b1b3702bccb6ab46… so the earlier pieces keep verifying. Recompute: python3 winners_curse.py prints the same bytes, exit 0, empty stderr, stdlib only.

Zero failures isn't zero risk: the rule of three for evals

Alexey Spinov — Wed, 22 Jul 2026 03:11:12 +0000

The rule of three for evals says zero failures in N runs is a count, not a rate. With 0 failures in N independent runs, the exact 95% upper bound on the true failure rate is 1 - 0.05^(1/N), which 3/N approximates. After 100 clean runs you still cannot rule out a 2.95% rate, about 1 in 34.

Here is the reading that bites you. Your eval harness runs the agent 100 times, prints "0 failures," and the tile goes green. Someone screenshots it into the launch thread. The unspoken translation is "the failure rate is zero." It is not what the data says.

I wrote a small script to make the gap concrete, so I ran a real gate over 200 deterministic agent runs first, counted honestly, and got the dashboard everyone trusts:

gate: spend<=budget over 200 deterministic agent runs
observed failures: 0  (distinct scenarios: 200)
naive point rate : 0.00%   binomial SE: 0.00 pp
naive 95% CI     : [0.00%, 0.00%]   <- zero width: false certainty

Look at the standard error. For a zero count the binomial SE is sqrt(0*1/200), which is exactly 0, so the naive 95% interval collapses to [0.00%, 0.00%]. A zero-width confidence interval. The math is telling you it is completely certain, from 200 samples, that the true rate is precisely zero. That is obviously wrong, and it is the exact shape of every "all green" board I have ever trusted too much.

TL;DR

"0 failures in N runs" is an observed count, not a rate. The naive binomial SE of a zero count is 0, which is why a green board looks like proof and isn't.
The honest number is the one-sided upper bound. With 0 failures in N runs, the 95% upper confidence limit on the true failure rate is 1 - 0.05^(1/N). The rule of three, 3/N, approximates it and rounds the risk slightly up.
At N=30 the bound is 9.50% (about 1 in 11). At N=100 it is 2.95% (1 in 34). At N=1000 it is 0.30% (1 in 334). Zero failures in 30 runs is compatible with a 1-in-11 true failure rate.
To rule out a 0.1% rate at 95% you need about 2995 clean runs, not 50. "We ran it fifty times" and "we are 99.9% sure" are different sentences.
Correlated runs make it worse. 200 runs that are really 20 scenarios replayed ten times give an effective N of 20, and the bound jumps from 1.49% to 13.91%.

What this is and is not. The gate in the script is benign by construction: I built it so it observes zero failures, because a clean eval is exactly the case I want to talk about. The bound over that count is a theorem, not a fitted effect. There is no tuned constant anywhere in this that, if I nudged it, would flip the conclusion. The only inputs are the count (0), the number of runs (N), and the confidence level. I show that explicitly in section 6 of the run, because "the number rides on a fixture I chose" is the failure mode I most distrust in this kind of post.

Why zero observed failures is not a zero failure rate

Counting is not estimating. If you flip a coin twice and it lands heads twice, you have observed a 100% heads rate, and nobody sane concludes the tails face does not exist. Two runs is not enough to rule much out. The same logic holds when the observed count is zero and the sample is 30, or 100, or 1000: a rare failure can hide behind any finite streak of clean runs, and the question is not "did it fail" but "what rate is still consistent with what I saw."

The naive dashboard answers a different, easier question. It reports the point estimate, 0/N = 0%, and its standard error, which for a zero count is zero. So the interval has no width and the board is green with no caveat. This is the statistical version of a monitor that has been silently dead for three weeks: a check that always returns "fine" and a check that has stopped running produce the identical screenshot. Silence is not the same as health, and a zero-width interval is not the same as certainty. This is the whole of the tracking-is-not-control idea in one number: the board is counting what already happened and calling the absence of an event a guarantee about the next one.

So throw out the point estimate and ask the honest question instead. Given that I saw zero failures in N runs, how high could the true failure rate be and still make a run this clean plausible?

The rule of three, and why it is exact at zero

The upper bound falls straight out of the binomial, no approximation required. The only way to observe zero failures in N independent runs is for every run to pass, and the probability of that is (1-p)^N. The Clopper-Pearson upper confidence limit is the failure rate p that makes an outcome this clean just barely believable at your confidence level: set (1-p)^N = 0.05, solve, and you get p = 1 - 0.05^(1/N). Because the binomial tail at x=0 is a single term, this is exact, not a normal approximation that wobbles at small N. One word of caution on exact: it means the coverage is guaranteed to be at least 95%, not that this is the tightest interval available. Clopper-Pearson sits at the conservative end of the family (Wilson and Jeffreys intervals run a hair narrower), and I take the conservative end on purpose when the thing I am bounding is a rare and expensive failure.

The rule of three is the pocket version. The true constant is -ln(0.05) = 2.9957, and 3/N is that rounded to a number you can do in your head. Here is the exact bound beside the rule of three across N, all assuming zero observed failures:

     N |  exact 95% upper |      3/N |   ~1 in K | exact-3/N (pp)
------------------------------------------------------------------
    10 |           25.89% |   30.00% |         4 |         -4.113
    30 |            9.50% |   10.00% |        11 |         -0.497
    50 |            5.82% |    6.00% |        17 |         -0.184
   100 |            2.95% |    3.00% |        34 |         -0.049
   300 |            0.99% |    1.00% |       101 |         -0.006
  1000 |            0.30% |    0.30% |       334 |         -0.001

Read the N=30 row. Thirty green runs, zero failures, and the true failure rate could still be 9.50%, roughly one bad run in eleven, without your eval being unlucky at all. The 3/N column sits a hair above the exact bound in every row (the gap is negative because exact minus 3/N is small and negative), so the mnemonic slightly over-states the risk. That is the direction you want a shortcut to err. By a thousand runs the two columns read the same to two decimals (0.30% and 0.30%), and the rule of three stops being an approximation worth distinguishing.

None of these numbers are typed in. They come out of 1 - 0.05^(1/N) in the script, and I cross-checked the row values against an independent recompute before I quoted them, because quoting a bound I hadn't actually run would be its own kind of lie.

How many clean runs actually buy a target?

Flip the question around. If you want to be able to rule out a given failure rate, how many clean runs does that take? Invert the bound, 1 - 0.05^(1/N) <= target, and you get N >= ln(0.05)/ln(1-target):

target = the failure rate you want to be able to rule out at 95%.
  bound <=  5.00%  needs  N =    59 clean runs (exact 58.4)
  bound <=  1.00%  needs  N =   299 clean runs (exact 298.1)
  bound <=  0.30%  needs  N =   998 clean runs (exact 997.1)
  bound <=  0.10%  needs  N =  2995 clean runs (exact 2994.2)

This is the section that changes how I plan an eval budget. Ruling out a 1% failure rate takes 299 clean runs. Ruling out 0.1%, the neighborhood where you'd actually feel safe shipping an agent that can move money or delete data, takes about 2995. For anyone paying a judge model per eval item, those Ns are a line straight to a bill, and the honest version of "prove it's safe" is expensive in a way the green tile hides. If your rare-but-catastrophic action got 50 clean runs, the most you can claim at 95% is that its failure rate is probably under 6%. That is not a number I would put next to "delete the production database."

The confidence level is a knob too, and it is worth seeing that "three" is not sacred:

  conf |  -ln(alpha) |           name |  upper@N=100
----------------------------------------------------
   90% |      2.3026 |    rule of 2.3 |        2.28%
   95% |      2.9957 |      rule of 3 |        2.95%
   99% |      4.6052 |    rule of 4.6 |        4.50%

The rule of three is specifically the 95% case, because -ln(0.05) rounds to 3. Want 99% confidence in your upper bound? That is the rule of 4.6, and at a fixed N it gives a looser bound (4.50% versus 2.95% at N=100), because more confidence over the same data costs you sharpness. Pick the confidence deliberately instead of inheriting 95% because a stats class used it.

The trap in the number: correlated runs shrink your N

Everything above assumes the runs are independent. In agent evals they very often are not, and the failure is quiet. You run the same 20 scenarios with 10 seeds each and call it 200 runs. The board says 200; the informative count is 20. Common-mode inputs, the same prompt replayed, the same fixture, the same upstream data, inflate N on the dashboard and leave the real bound where the distinct cases put it.

nominal runs: 200   distinct scenarios: 20   observed failures: 0
bound if you count nominal N=200 : 1.49%  (~1 in 67)
bound at effective N=20          : 13.91% (~1 in 7)

Same zero failures. Same 200 lines in the log. But the bound you're entitled to is 13.91%, about 1 in 7, not 1.49%. This one cuts against the reassuring direction, which is exactly why I put it in: replays and shared inputs shrink your effective N, and a smaller N only ever loosens the bound, never tightens it, since 1 - 0.05^(1/N) falls as N grows. A harness that counts replays as independent runs is quietly reporting a safety you don't have. If you can't argue your runs are independent draws from the distribution you care about, use the count of distinct cases, not the count of executions.

What the bound does not tell you

Three honest limits, because the number is narrow on purpose.

It is about frequency, not identity. 1 - 0.05^(1/N) bounds how often a failure could happen. It says nothing about which failure, or how bad. A 1% rate of "off-by-a-cent" and a 1% rate of "wired the whole treasury" are the same number here and nothing alike in practice. The bound does not replace worst-case analysis or reading results by severity class; it sits next to them.

It bounds the rate on your eval distribution, not on production. If your 100 clean runs don't cover the inputs production will throw, the bound is honest about the runs you did and silent about the ones you didn't. A clean eval on the wrong distribution is a different problem, closer to contamination in the eval itself.

It is an upper bound, not an estimate. Zero failures does not mean the rate is 3/N. It means the rate is probably below 3/N. The true rate could be much lower, or, with bad luck, right at the bound. The point is not to name the rate. It is to stop pretending a clean run named it zero.

Why there is nothing here to fake

I have dropped enough of my own drafts to be suspicious of any post where the punchline sits inside a constant the author picked. So I built this one to have no such constant. The conclusion depends on three inputs and nothing else: the observed count, N, and the confidence level. Section 6 of the run makes the point by brute force:

re-run the gate: failures still 0 of 200  ->  bound 1.49%

Shuffle which scenario the gate sees first, run it twice, reorder the whole fixture: the count stays 0, so the bound is byte-for-byte the same. There is no effect size to permute, no ratio whose denominator is secretly doing the work. That is the opposite of the class of eval post that gets dropped, where the "finding" is really a fixture setting in disguise. Here the only thing I observe is a count, and the only thing I do to it is bound it. If you distrust the bound, you are distrusting the binomial, which is a fine thing to argue about and a very different argument from "he tuned the numbers."

The whole thing runs in a second: Python 3.13.5, stdlib only, offline, no keys, no funds, three runs byte-identical, exit 0, empty stderr. Picking the right statistical test is a sibling problem, and I wrote up the paired-data version of it in why your A/B eval probably needs McNemar, not a two-proportion SE. Both belong to the same family of pre-execution gates for AI agents: decide what you can actually claim before you act on the claim.

What does your board show

The next time a tile goes green on "0 failures in N runs," do the one-line translation out loud. Zero in 30 is a rate that could be 1 in 11. Zero in 100 is a rate that could be 1 in 34. Neither is zero, and the dashboard will not tell you that unless you make it.

I publish the runs that correct my own reading, not only the ones that flatter it, so follow along if that is your kind of thing. And a real question I don't have a clean answer to: when your eval last went green, did anyone write down the N, or did "0 failures" quietly get filed as "the failure rate is zero"? I want to know how many runs your team treats as enough, and why that number and not ten times it.

AI disclosure. I drafted this with an AI writing assistant and edited every line; the framing, the gate, and the reading are mine. Every output block is pasted from one real local run on 2026-07-22. zero_failures.py sha256 d3b3f205909640fc…, run output sha256 cc7aca328fdd7236…. Recompute: python3 zero_failures.py prints the same bytes, exit 0, empty stderr, stdlib only.

Your A/B eval is paired. Your stat test probably isn't.

Alexey Spinov — Tue, 21 Jul 2026 03:03:28 +0000

Paired eval, wrong test: two prompts scored on one 100-item set give paired outcomes, so ranking them needs McNemar, not the two-proportion Wald SE. On the same data, Wald read 1.01 SE and said 'collect more'; McNemar read 2.65 SE and allowed the ranking. Same 100 items, opposite decision.

I shipped a little eval helper that refuses to rank two configurations when their gap is smaller than two standard errors. The idea is good. The refusal is honest. Last week it looked at two prompts scored on the same 100-item set and printed this:

RANK: INDISTINGUISHABLE - gap 7.00 pp against 6.95 pooled SE = 1.01 SE < 2.0. Ranking "prompt B" above "prompt A" is NOT allowed.

Seven points of apparent improvement, called noise. The honest reading of that line is "collect more items." And it was wrong. Not a rounding error, not a close call. The right test on that exact data says the ranking is already decided, and I could have stopped.

The bug was not in the arithmetic. It was in which test the arithmetic ran. My helper was treating two paired runs as if they were two independent samples.

TL;DR

Two prompts on one task set are paired: each task gives a pair (pass_A, pass_B), and both-pass / both-fail tasks carry zero information about the difference.
My harness used the Wald SE of a difference of two proportions, pooled = sqrt(se_A**2 + se_B**2). That is the independent-samples formula; it ignores the pairing entirely. In my showcase that cost real power.
The paired test is McNemar: SE = 100*sqrt(b+c)/n, using only the discordant counts b and c. On the same 100 items it returned 2.65 SE, not 1.01, and ranking was allowed. With just 7 discordant pairs the tool also prints the exact binomial, p=0.0156 (z-equivalent 2.42), so the decision does not ride on the normal approximation.
On 535 real paired observations from my own sweep, one pair reads 2.65 SE under Wald and 6.24 under McNemar with c=0: a strictly nested result that Wald reports as a hair over the line and has no way to flag as deterministic.
The fix is about fifteen lines. The hard part is not the formula, it is noticing your runs are paired.

What this is and is not. The two-prompt table below is a constructed example: I picked the counts to sit exactly on the line where the two tests disagree, so you can reproduce it with four numbers. The 535-observation table comes from a synthetic marker fixture I wrote, not from anyone's production system. Where a number is forced by my constants rather than measured, I say so in the same paragraph. That habit is half the point.

Why two prompts on one set are paired

Run prompt A and prompt B on the same 100 eval tasks. Task 7 either trips both of them, or neither, or exactly one. That last group is the only one that tells you which prompt is better. The tasks where both pass and the tasks where both fail are shared difficulty: they move both pass rates together and say nothing about the gap.

Here is the case that caught me, laid out as a 2x2:

                     B pass   B fail
        A pass          55        0     <- b = 0 (A pass, B fail)
        A fail           7       38     <- c = 7 (A fail, B pass)
        A marginal = 55/100   B marginal = 62/100   n = 100 pairs

Prompt A passes 55, prompt B passes 62. Same items. Ninety-three of the hundred tasks are concordant: 55 both pass, 38 both fail. Seven tasks are discordant, and all seven go the same way, B passes where A failed. Zero go the other way. B's pass set contains A's pass set completely. That is a strong statement, and it is invisible to a test that only looks at the two marginal rates.

What my harness printed, and why it was the wrong number

The refusal came out of rank(). Under the hood it builds the pooled standard error the way you were taught for two independent proportions:

    pooled = sqrt(p1.se ** 2 + p2.se ** 2)

Each se is the binomial standard error of one marginal rate. For 55/100 and 62/100 that pools to 6.95 points, the 7-point gap divides to 1.01 SE, and the guard files it under "indistinguishable." Fair, if the two samples were drawn independently. They were not. The 93 concordant tasks are the same 93 tasks in both columns, and the formula charged me full variance for them anyway.

The cost of believing that line is real budget. At 1000 items each, the same seven-point gap finally clears the bar:

RANK: "prompt B" > "prompt A" - gap 7.00 pp = 3.18 SE >= 2.0. Ranking is allowed.

So the harness asked for ten times the eval spend to reach a verdict the correct test already had at 100 items. For anyone paying per token to run a judge model over a set, that is a straight line from a stats mistake to a bill.

What McNemar does instead

McNemar's test (Quinn McNemar, Psychometrika, 1947, the standard test for paired nominal data) throws away the concordant pairs and looks only at b and c. The standard error of the difference becomes 100*sqrt(b+c)/n, and the test statistic is |c-b|/sqrt(b+c). I added it to the same library as mcnemar(name_a, b, name_c, c, n). On the exact same 100 items:

McNEMAR: discordant pairs prompt A=0, prompt B=7 (concordant 93 of n=100).
  SE = 2.65 pp (100*sqrt(b+c)/n); marginal gap 7.00 pp = 2.65 SE >= 2.0 -> ranking "prompt B" over "prompt A" is allowed.
  small discordant count (b+c=7 < 25): the normal approximation is anti-conservative here. Exact two-sided binomial p=0.0156 (z-equiv 2.42), continuity-corrected z=2.27. all three clear the 2.0 bar; the decision is unchanged.
  NESTED (b=0 or c=0): every discordant item favours the same side; the difference is deterministic (strict nesting on this sample), not a coin-flip margin.

2.65 SE against the same 2.0 threshold. Allowed. Same data, opposite decision. And note the third line. With only 7 discordant pairs, 2.65 is a normal approximation, and the tool will not let me lean on it: beside it sit the exact two-sided binomial, p=0.0156 (z-equivalent 2.42), and the continuity-corrected z at 2.27. All three clear the 2.0 bar. Same decision, reached without trusting the largest number in the row. The NESTED line fires because one discordant count is zero, which means the two prompts never disagreed in both directions: on this sample B dominates A item by item. That is a qualitatively different thing from a noisy 7-point margin, and a test that pools marginals cannot see it.

One honesty note, because it matters. NESTED is not a license to rank anything with a zero in it. A pair with b=0, c=1 is also nested, and the tool prints it at 1.00 SE, nowhere near the bar. The ranking decision still rides on the z value. In this case the z is 2.65 and the nesting is strict, so both agree. I flag them separately on purpose.

The same divergence on 535 real observations

The constructed example is clean but you should distrust anything I can tune to land on the line. So here is the same phenomenon on data I did not hand-pick: a sweep of a monotone-but-not-total marker fixture, P_PATHS=12 W_PER_TICK=3 T_TICKS=400 SEEDS=20, where a false reject is a landed write that a single-integer witness wrongly rejected.

The cells of that sweep share one underlying draw. Path and outcome come off the LCG independently of the axis knobs, so the raw (tick, path, outcome) at each sampled position is identical across cells. The run checks that before it does anything else, across all 33 sampled points per seed:

(tick,path,outcome) skew 6:6 vs 7:5            identical: True  (33 sampled points)
(tick,path,outcome) streams 8 vs 4             identical: True  (33 sampled points)
ALL PAIRED (raw observations identical across cells): True

Same observations, only the verdict moves. That is the definition of paired, and it is why the two-proportion SE is the wrong tool here too. Cross-tabbing the false-reject indicators gives real b/c counts. Both tests, side by side:

pair                     | b / c / n        | Wald (rank)    | McNemar (paired)
------------------------------------------------------------------------------
6:6 vs 7:5               | 142 / 128 / 535  | 0.87 SE        | 0.85 SE
7:5 vs 8:4               | 39 / 3 / 535     | 2.31 SE        | 5.55 SE
8:4 vs 9:3               | 39 / 0 / 535     | 2.65 SE        | 6.24 SE NESTED
streams=8 vs streams=4   | 91 / 32 / 535    | 3.98 SE        | 5.32 SE
streams=2 vs streams=1   | 229 / 0 / 535    | 20.01 SE       | 15.13 SE NESTED

Read the third row. 8:4 versus 9:3 is 39 discordant pairs, all in one direction, c=0. Strict nesting again, on real counts. Both tests clear the 2.0 bar here, so both allow the ranking. But Wald reports 2.65, a hair over the line, while McNemar reports 6.24 and flags NESTED. One of those tells you the result is deterministic on this sample; the other cannot tell "barely over the threshold by luck" apart from "decided." Here is that pair with both guards printing in full:

PAIR 8:4 vs 9:3   marginal FR: 8:4=172/535  9:3=133/535
    8:4: 32.1% (k=172 n=535 SE=2.02)
    9:3: 24.9% (k=133 n=535 SE=1.87)
  RANK: "8:4" > "9:3" - gap 7.29 pp = 2.65 SE >= 2.0. Ranking is allowed.
  McNEMAR: discordant pairs 8:4=39, 9:3=0 (concordant 496 of n=535).
    SE = 1.17 pp (100*sqrt(b+c)/n); marginal gap 7.29 pp = 6.24 SE >= 2.0 -> ranking "8:4" over "9:3" is allowed.

Two honest observations about that table. First, the two tests never disagree on direction: whichever version each one ranks higher, they agree, on all five pairs. Second, at this 2.0 threshold none of the five flip the decision. Row one, 6:6 vs 7:5, is 0.87 under Wald and 0.85 under McNemar, both below the bar, both refused. Rows two through five sit above the bar for both. What moves is the reported SE, sometimes a lot, 2.31 against 5.55, and whether the c=0 nesting gets named at all.

And the gap runs both ways, which is why "McNemar is always more powerful" would be the wrong lesson. On 6:6 vs 7:5 Wald reads 0.87 against McNemar's 0.85, and on streams=2 vs streams=1 it reads 20.01 against 15.13: when the discordant pairs are many and lopsided, the independent-samples SE is not the conservative one, it understates the variance instead. The rule is not that one test wins, it is that you owe your data the test its design calls for.

So where is the decision flip? Near the bar. This particular sweep happens to spread its pairs away from the 2.0 line, so the two tests agree on every call even while their SEs diverge. The constructed 100-item example sits right on the line, which is where the difference between the tests stops being cosmetic and turns into a yes or a no. I did not engineer that to cheat; it is where most config bake-offs I have watched actually get decided, on a handful of items either way.

One thing I am deliberately not claiming

You may have noticed those false-reject levels, 75.9%, 64.9%, and want me to say something about them. I will not, and the same library is why. Run the construction-independence probe on those cells and it comes back:

FORCED BY CONSTRUCTION [streams=8]: conditional 75.9% (k=406 n=535 SE=1.85) is indistinguishable from unconditional 74.7% (k=493 n=660 SE=1.69) (0.48 SE < 2.0)

The level equals the unconditional pass-the-gate rate, 74.7% at n=660, because the fixture draws outcome and stamp from separate LCG steps. So the level is an artifact of my constants, not a measurement, and I quote none of them as findings. What survives that probe is the discordance structure, b and c, which is a genuine between-cell comparison. The McNemar inputs are real; the levels they sit next to are not. Running the probe and reporting the result is the only reason I trust the distinction.

The fix, in about fifteen lines

There is nothing clever in it. It is the discordant count and a square root:

    disc = b + c
    conc = n - disc
    ...
    se = 100.0 * sqrt(disc) / n
    gap = 100.0 * abs(c - b) / n
    n_se = abs(c - b) / sqrt(disc)          # = gap / se, n cancels
    ...
    if b == 0 or c == 0:
        # NESTED: strict set-nesting, deterministic on this sample

(Truncated by me: the wrapper handles b==c, the no-discordant case, and localized output; full function in measure.py.)

The formula is the easy part. The part that actually protects you is upstream and it is not in this function at all: the library cannot know your two runs are paired. You have to establish that yourself, by confirming the observations are the same items in the same order and only the outcome moved. That is the check the sweep runs before it calls mcnemar() at all, and if the raw observations had differed it would have refused to treat them as paired. A paired test on unpaired data is its own mistake.

The limits, plainly. SE = 100*sqrt(b+c)/n is the normal approximation to McNemar; for a handful of discordant pairs you want the exact binomial instead, which is why the function prints it automatically once b+c drops below 25, and I would not read the plain SE without it. The z threshold of 2.0 is a convention I carried over from the rest of the harness, not a law. And none of this touches the failure mode that actually costs you money, a silent false accept, because that is a question about your ground truth, not your arithmetic.

What does yours do

Everything here ran locally on a synthetic fixture: Python 3.13.5, stdlib only, offline, no keys, no funds, three runs byte-identical, exit 0, empty stderr. Two earlier pieces work the same eval harness from other angles: reading eval results by severity class instead of a flat pass rate and a static probe that found contamination points without running the agent. Both belong to the same family of pre-execution gates for AI agents.

I publish the runs that corrected my own reading, not only the ones that confirmed it. Follow along if that is your kind of thing. And a real question I do not have a clean answer to: when your last A/B put two agent versions at 76 and 74 out of 100 on the same set, did your harness run a paired test, or did it pool two marginals and quietly ask you for more data? I would like to know what yours does.

AI disclosure. I drafted this with an AI writing assistant and edited every line; the test choice, the sweep, and the reading are mine. Every output block is pasted from one real local run on 2026-07-21. measure.py sha256 b1b3702bccb6ab46…, paired_test.py sha256 fa687e05e8e69512…, run output sha256 d8e4e521b1f35ac0…. The library default stays Russian so the sha256 of two already-published runs keeps verifying; English is opted into explicitly.

A Spend Cap That Stops Counting Is Already Fail-Open

Alexey Spinov — Sun, 19 Jul 2026 01:53:12 +0000

Two of the five ways a spend cap can handle a missing price produce the exact same decision stream — same sha256, byte for byte. One of them is the thing everybody calls fail-open. The other is the thing everybody recommends instead of it: fall over to a free local model.

Let me say what that hash is and isn't before it does any work. It covers (seq, label, admitted, charge) and deliberately drops the human-readable reason strings, so two policies that print different words hash the same when they decide the same. Once you see that, the collision is a theorem rather than a discovery: a free fallback charges zero, fail-open charges zero, and a ledger built out of charges cannot tell them apart because there is nothing there to tell apart. The sha256 proves only that my implementation doesn't quietly cheat.

The reason it's still worth a post is that nobody ships them as the same policy. One is the bug you apologise for; the other is the fix you recommend in the thread. On the axis that matters they are one policy, and one of them has better branding.

AI disclosure. I wrote blind_spend_cap.py with AI assistance and ran it myself. Every number and hash below is pasted from a real run: offline, stdlib only, no network, no keys, no funds. The oracle is injected, so runs are deterministic. I ran it three times; the output was byte-identical each time. Code sha256 ddc42590…, output sha256 9ebe1b4a…. External figures are linked and labeled, and I say clearly which ones I did not reproduce.

TL;DR

A spend cap needs a cost-oracle to price the next action. The oracle has its own outage.
The usual framing (fail-open vs fail-closed) is the wrong axis. The real split: does the ledger keep moving while the oracle is quiet?
Strategies that charge a price the remaining budget can still absorb keep the ledger alive and re-trip the cap. Charge zero and spent freezes forever. Charge more than fits and you've written refuse with extra steps — the harness proves that one on itself.
A free local fallback charges zero. In my harness it produces a decision stream identical to plain fail-open: same sha256.
But a moving ledger is a floor, not a certificate. Any positive fiction satisfies it — price a $0.05 call at $0.01 and your cap is quietly five times the one you configured. The real axis is the bias of your estimator; zero is just where that bias hits −100%.
The headline number you'd expect me to use here (34 extra actions) is arithmetic, not evidence. I take it apart below rather than sell it.

The branch nobody writes down

Every spend cap I've shipped has the same shape. Price the action, compare against a budget, allow or block. The pricing step assumes the oracle answers.

It doesn't always. CoinGecko 429s. A usage endpoint times out. A token meter sits behind a gateway returning 526. In that moment your cap takes a decision that probably isn't in your code review notes, because it isn't in your code: what to do with an action it cannot price.

I know it's unwritten because I shipped it that way. On June 8, 2026 I published SpendGuard, a 40-line pre-execution cap. It works, and it never declared this branch. The oracle call sits inside cost_fn on line 121, and eth_price_usd() calls raise_for_status() before it returns anything. So on a 429 the exception blows straight past the gate and out of the wrapper. Accidentally fail-closed, by way of an uncaught exception that takes the caller down instead of returning a verdict you can count.

Copy the demo loop from that same post and you get the opposite. It prices once before the loop and reuses that number for every round. A mid-loop outage is invisible. Accidental fail-open.

Same file, two wirings, two opposite behaviors, and I declared neither.

Five strategies, one fork

So I built the smallest thing that isolates the branch. blind_spend_cap.py runs one Analyzer/Verifier ping-pong through one budget gate, under five strategies that are identical everywhere except the quote is None block:

    else:
        if strategy == "refuse":
            return _v(seq, "BLOCK", "no quote: refuse", False, 0, priced)
        if strategy == "admit-unpriced":
            return _v(seq, "ADMIT", "no quote: admit, charge 0 (ledger frozen)",
                      True, 0, priced)
        if strategy == "stale":
            if last_known is None:
                return _v(seq, "BLOCK", "no quote and no last-known price: refuse",
                          False, 0, priced)
            est, tag = last_known, "stale "
        elif strategy == "pessimistic":
            est, tag = per_action_cap, "pessimistic "
        elif strategy == "fallback":
            est, tag = fallback_cents, "fallback "

Two design choices worth stating, because both cut against the result I might have wanted.

A BLOCK does not stop the loop. A real runaway retries. My earlier harness gave the refusing policy a free break, which quietly handed it the win: it "stopped the runaway" because I wrote the loop that way. Here the gate stops the spend, not the work, and the loop keeps hammering. There's an --on-block halt flag for the single-shot shape, and I sweep both.

The budget has no clock, so I call it a per-run budget rather than a daily one. Calling it daily would be a lie in a file with no time in it.

The split isn't admit-vs-block. It's counting-vs-not.

Here's the outage run, oracle down from step 6, straight from output.txt:

SCENARIO B — oracle down from step 6, loop retries after a BLOCK
  refuse           admitted=6   spent=$0.30  unpriced=0   unaccounted=0   ledger-moved=False exit(conv)=1 exit(strict)=1
  admit-unpriced   admitted=40  spent=$0.30  unpriced=34  unaccounted=34  ledger-moved=False exit(conv)=0 exit(strict)=2
  stale            admitted=10  spent=$0.50  unpriced=4   unaccounted=0   ledger-moved=True  exit(conv)=1 exit(strict)=2
  pessimistic      admitted=6   spent=$0.30  unpriced=0   unaccounted=0   ledger-moved=False exit(conv)=1 exit(strict)=1
  fallback:0c      admitted=40  spent=$0.30  unpriced=34  unaccounted=34  ledger-moved=False exit(conv)=0 exit(strict)=2
  fallback:1c      admitted=26  spent=$0.50  unpriced=20  unaccounted=0   ledger-moved=True  exit(conv)=1 exit(strict)=2

Look at the spent column, not the admitted one.

Exactly two strategies end at $0.50: stale and fallback:1c. That's the budget, tripped, doing its job. admit-unpriced and fallback:0c end at $0.30 and stay there — not because the run was cheap, but because after step 6 nothing was ever added to the ledger again.

Now the row that breaks the tidy version of this claim, which I had written as "every strategy that charges something ends at $0.50" until the table two lines above told me otherwise. pessimistic charges the most of anybody and still ends at $0.30. Charging something isn't sufficient. The something has to fit. pessimistic prices every un-priced call at the $0.25 per-action cap, only $0.20 of budget remains after step 6, so every un-priced call is blocked on arrival: admitted=6, unpriced=0.

Which changes how you read the unaccounted column. It counts admissions where the oracle gave no quote and nothing was charged — defined by the fact of a missing quote, not by the name of the policy, so it can accuse any strategy including the ones I like. stale and fallback:1c sit at zero because they kept counting. pessimistic and refuse sit at zero because they never admitted a blind call in the first place. Same number, two different stories, and I'd been reading the flattering one into both.

Push it to the limit and the failure gets loud. Oracle dead from step 0:

$ python3 blind_spend_cap.py --strategy admit-unpriced --oracle-fails-from 0
admit-unpriced   admitted=40  spent=$0.00  unpriced=40  unaccounted=40  ledger-moved=False exit(conv)=0 exit(strict)=2

Forty actions admitted. Ledger says zero dollars. Exit code zero, under the mapping most gates actually ship.

That's the thing worth internalizing. A blind cap doesn't report danger. It reports innocence.

The number I'm not putting in the headline

You'd expect the pitch to be "fail-open admitted 34 more actions." The tool does print it. I'm going to argue against it anyway, because I got burned by exactly this number last time.

M is the gap in admitted actions between admit-unpriced and refuse. In the run above it's 34. Sweep the outage step across the whole parameter space and you get this:

  on-block=retry
        K:    0    5    6    9   10   11   12   20   39   40
        M:   40   35   34   31   30   29   28   20    1    0

(That's the retry half of the sweep with the unacc row dropped for now — under retry it's identical to M anyway. The full block, both loop shapes, is two sections down.)

M = WANTS − K, exactly, everywhere. I picked WANTS = 40. If I'd picked 1000, the headline would read 994. It isn't a property of any policy, it's a property of how long I let the loop want things. A number I chose, dressed up as a number I found.

Which is why the strategy table above leads with spent and unaccounted, and why M is buried in a scenario that tells you to go read the sweep before quoting it.

Pressing my own kill switch

The honest question isn't whether my metric works in the run I picked. It's where it stops working. So here's the same sweep under both loop shapes:

  on-block=retry
        K:    0    5    6    9   10   11   12   20   39   40
        M:   40   35   34   31   30   29   28   20    1    0
    unacc:   40   35   34   31   30   29   28   20    1    0
    -> M = 0 in 1/41 of K (2%); unaccounted = 0 in 1/41 (2%)
  on-block=halt
        K:    0    5    6    9   10   11   12   20   39   40
        M:   40   35   34   31   30    0    0    0    0    0
    unacc:   40   35   34   31   30    0    0    0    0    0
    -> M = 0 in 30/41 of K (73%); unaccounted = 0 in 30/41 (73%)

Under halt semantics, everything I've argued dies in 73% of the parameter space. Not weakens. Dies, to identically zero, both metrics at once.

The reason is dull and important. The budget is $0.50, a healthy call is $0.05, so a healthy run hits the cap at step 10. If the oracle only falls over at step 11 or later, the loop is already stopped by money. The un-priced branch is never reached. Nothing to measure, nothing to argue about.

So the applicability condition, which my previous draft never stated and which I'm stating plainly now: this post is about outages that start before your budget would have stopped the loop anyway — K <= budget // unit_cost, which is K <= 10 here — or about loops that retry after being refused. Outside those two cases the whole thing is a non-event.

That boundary was off by one in the tool until this morning: the summary line said the metrics collapse once K >= 10, when K = 10 is the last K where they're still alive and K = 11 is the first dead one. The sweep table underneath it was right the whole time. A decent argument for printing the table and not just the conclusion drawn from it.

I think retrying is the common case, because runaway agent loops are usually retry loops. That's a judgement about the world, not a measurement, and I'm flagging it as one.

A free fallback is fail-open with better branding

Now the equivalence. Same outage, and I hash the decision stream — the (seq, label, admitted, charge) tuples, deliberately excluding the human-readable reason strings, so two strategies that make the same decisions hash the same even when they print different words:

SCENARIO C — is a free fallback a distinct strategy, or a renamed fail-open?
  admit-unpriced   sha=53b01d22a232f1fee833a76c7cd1ed810d1945da2e7620c8a1a17a9302b4df79
  fallback:0c      sha=53b01d22a232f1fee833a76c7cd1ed810d1945da2e7620c8a1a17a9302b4df79
  stale            sha=0dcbd560f59adcf2b1eec3ca111dc7f89c3e7ac08569fe165d88ec7af77b4311
  fallback:5c      sha=0dcbd560f59adcf2b1eec3ca111dc7f89c3e7ac08569fe165d88ec7af77b4311
  refuse           sha=7df1f34491edfde21648d36e9a8eda1db7306d12efda4c29364fa8f54ad3b04a
  pessimistic      sha=7df1f34491edfde21648d36e9a8eda1db7306d12efda4c29364fa8f54ad3b04a
  -> fallback:0c  == admit-unpriced : True
  -> fallback:5c  == stale          : True  (needs >=1 real quote before the outage, and a constant oracle price)
  -> pessimistic  == refuse         : True  (at THESE caps: $0.25 never fits what is left of $0.50)

Three collisions in that block, and the third one costs me a third of my own recommendation.

fallback:0c == admit-unpriced is the headline, and — as I said up top — a theorem. Both charge zero, the hash covers the charge. I couldn't break it by moving the budget, the unit cost, the per-action cap, the loop shape or the outage step; it isn't a coincidence of the parameters I picked.

fallback:5c == stale is real but conditional, and I stated it as a general truth in an earlier pass. It needs two things I'd left unsaid. There has to be at least one successful quote before the outage — with --oracle-fails-from 0 there's no last-known price at all, stale refuses, and the equality collapses. And the oracle's price has to actually hold still; point the harness at a varying price and the two decision streams separate immediately. The honest version is narrower: a fallback pinned to the real rate is stale pricing, as long as the real rate isn't moving.

pessimistic == refuse was sitting in my own main table and I walked past it twice. Same 7df1f344…. A $0.25 estimate never fits the $0.20 left after step 6, so pessimistic blocks every un-priced call — which is refuse, decision for decision. Run --strategy fallback --fallback 25 --oracle-fails-from 6 and you get 7df1f344… as well. Three names, one behavior.

That forces an admission about my own advice. Further down I tell you to charge a stale price, or a conservatively biased estimate, or the per-action cap. At the parameters in my own demo that last one is bit-identical to the refusal I declined to recommend — which is why it now ships with that caveat attached instead of posing as a third independent door. It doesn't make the advice wrong. Refusing is defensible, and I spend a whole section on its bill.

The price sweep is where this gets concrete:

SCENARIO D — sweep the fallback price (the whole argument hangs on it)
  fallback:0c      admitted=40  spent=$0.30  unpriced=34  unaccounted=34  ledger-moved=False exit(conv)=0 exit(strict)=2
  fallback:1c      admitted=26  spent=$0.50  unpriced=20  unaccounted=0   ledger-moved=True  exit(conv)=1 exit(strict)=2
  fallback:2c      admitted=16  spent=$0.50  unpriced=10  unaccounted=0   ledger-moved=True  exit(conv)=1 exit(strict)=2
  fallback:5c      admitted=10  spent=$0.50  unpriced=4   unaccounted=0   ledger-moved=True  exit(conv)=1 exit(strict)=2
  fallback:25c     admitted=6   spent=$0.30  unpriced=0   unaccounted=0   ledger-moved=False exit(conv)=1 exit(strict)=1

So the clean line I wanted to write — "the runaway terminates if and only if the price is above zero" — is not true, and the bottom row is the counterexample. fallback:25c prices well above zero and spent still sticks at $0.30 with the ledger frozen. What a price above zero actually buys is a ledger that keeps moving, and only while that price still fits what's left of the budget. Under that band you get fail-open. Over it you get refusal wearing a price tag. The usable range is narrower than "not zero", and where it sits depends on caps I picked.

Worth being exact about the edge, because I rounded it off in an earlier pass. The price has to clear min(per-action cap, budget − spent when the outage begins) — here min($0.25, $0.20), so the collision with refuse actually starts at $0.21, and $0.25 is just the row I happened to print. Which of the two terms binds depends on when the oracle dies: at step 6 it's the leftover budget, at step 4 it's the per-action cap. So "too expensive to fit" isn't a property of the price alone — it's the price measured against however much budget the outage left you.

This matters because the free version is the one people ship. On July 15, 2026, a developer publishing as @ddhh released a small LLM circuit breaker with a clean statement of the instinct: "When I'm about to overspend, don't fail and don't keep paying — fall through to a free local model and keep working." The config in the post marks the tier # local, free, always-on fallback.

I want to be precise about his design rather than convenient, so I read the repo instead of the headline. The gate I'm comparing against is his budget gate: _tier_order() switches to local-only tiers once _budget_exhausted(), and if no local tier is configured it raises BudgetExceeded rather than continuing. That gate keys on accumulated spend against a limit — not on a missing quote. That's a declared branch with a hard stop, which is more than my June code had.

Provider failure is handled too, just not by that gate. His post opens on exactly that problem — "Paid API quotas dying mid-run. One provider 429s, and the whole run falls over" — and complete() wraps each tier in an except … continue (breaker.py:121-131, commented "a failed tier should never crash the caller"), so a 429 on a paid tier falls through to the next one, local included. Two triggers, one ordered failover. The reason I'm separating them carefully is that the thing I care about is which signal moves the ledger, and on both paths the local tier reports the same number: ollama_tier._call ends with return text, 0.0 (providers.py:167), which _record writes into the JSONL ledger as cost_usd: 0.0.

For his stated goal that is correct, and I'll say so flatly: a free local call really does cost zero dollars, so his ledger is accurate and his dollar spend genuinely cannot grow past the limit. He solved the problem he set out to solve.

The pattern risk lives one step to the side. Once every call costs zero, the dollar ledger can never stop anything again — and if the thing that pushed you over budget was a non-converging loop rather than an expensive model, dollars were never the binding constraint. Wall-clock, local GPU, rate-limited downstream endpoints and side effects all keep accruing, and the ledger reports $0.00 while they do. Exactly the fallback:0c row above.

None of that is a defect in his breaker. It's what the graceful-degradation instinct does when you port it from "budget exhausted" to "cost unknown" without noticing the axis changed.

For scale on why any of this is worth an afternoon: on July 16, 2026 @royanannya published a postmortem of a multi-agent loop that billed $1,847 in one weekend, fixed by pushing decisions off the LLM layer and cutting per-game cost from $1.95 to $0.35. Their number, their run; I didn't reproduce it. Their fix was architectural, not a spend cap, and I'm not going to pretend otherwise.

Where my own metric stops meaning anything

I've been leaning on unaccounted as though it measures how accurately you're counting. It doesn't. It's a binary test — did an admitted call with no quote get charged zero? — and the counterexample is sitting in my own output.

The harness has no notion of what an action actually costs. Every admit really executes, and a real call here is $0.05. Multiply the admitted column by that and set it beside the ledger:

strategy	ledger says	actually spent	vs the `$0.50` budget	`unaccounted`
`stale`	$0.50	$0.50	1.0x	0
`fallback:5c`	$0.50	$0.50	1.0x	0
`fallback:1c`	$0.50	$1.30	2.6x	0
`fallback:0c`	$0.30	$2.00	4.0x	34

fallback:1c passes my metric cleanly. unaccounted=0, ledger-moved=True, budget tripped on schedule — and I wrote "that's the budget, tripped, doing its job" about that exact row. It also pushed 26 real calls through a ten-call budget. Pricing a $0.05 action at $0.01 under-counts by 5x, and a cap that under-counts by 5x is a cap five times larger than the one you configured.

So "keep the ledger moving" is satisfied by any positive fiction. The real axis isn't zero versus non-zero — it's the bias of your estimator. Zero is simply the point where the bias hits −100% and the cap stops existing at all. It's the worst case and it's the common case, which is why it earns a post, but unaccounted=0 is a floor, not a certificate. If your fallback price is a comfortable number rather than a conservative one, you haven't fixed the runaway. You've slowed it down and moved it out of view.

There's a second place my instrument lies, and I found it checking this piece rather than writing it. The ledger-moved column samples spent after each blind decision, so what it actually asks is "did the ledger move more than once?" A price that fits exactly once — try --fallback 15 — pushes spent from $0.30 to $0.45 and still prints ledger-moved=False, the same value fail-open gets. None of the rows in this post are affected; they sit at 0c, 1c, 2c, 5c and 25c. But if you sweep the price yourself you'll walk into it, and it's the same conflation I've spent the whole post complaining about, sitting in my own column. Trust spent. The boolean is a convenience and I got it wrong.

One caveat on that table, because it cuts against my own framing: it assumes an un-priced call costs what a healthy one costs. If your fallback genuinely is a free local model, the dollar figure really is zero and the fallback:0c row overstates the dollars. That's precisely @ddhh's case — and precisely why what escapes there is wall-clock, GPU and downstream rate limits rather than dollars.

What refusing actually costs you

I've been describing the failure mode of not counting. Refusing has its own bill, and my last draft skipped it, which was the honest complaint against it.

First, the recovery case. Outages end. Oracle down from step 6, back at step 15:

SCENARIO F — the outage ENDS: oracle down 6..14, back at 15
  loop retries after a BLOCK (a real runaway does):
  refuse           admitted=10  spent=$0.50  unpriced=0   unaccounted=0   ledger-moved=False exit(conv)=1 exit(strict)=1
  same run, but the loop HALTS on the first BLOCK (single-shot shape):
  refuse           admitted=6   spent=$0.30  unpriced=0   unaccounted=0   ledger-moved=n/a   exit(conv)=1 exit(strict)=1

(The scenario prints admit-unpriced and stale rows too; I've kept only refuse here, because it's the policy on trial.)

Under retry, refusing costs nothing: the run resumes and completes the same ten actions it would have anyway. Under halt, it ends the run at step 6 and never sees the oracle come back. Same policy, same outage, and whether refusing is free or expensive depends entirely on a property of your caller that isn't in the cap at all.

Three more costs, none of which my harness measures:

You hand your stop button to a third party. If your gate refuses whenever CoinGecko is unreachable, then CoinGecko's rate limiter is now your kill switch, and anyone who can degrade it can halt your agent remotely.

Refusing is not automatically safe. The fail-closed doctrine arrives from authorization, where denial is the safe default. Spending isn't authorization. Stopping halfway through a non-idempotent sequence — and SpendGuard was written for ETH and gas — can be worse than admitting one un-priced call. If your actions aren't safely interruptible, "refuse" is not the free option it looks like.

The escape hatch never gets closed. Any --allow-unpriced flag will be added to a systemd unit at 3am during an incident and will still be there next year. If you build one, give it an expiry.

Which is why the recommendation of this post isn't "fail closed." It's narrower: declare the branch, and keep the ledger moving with a price you'd defend out loud. Charge a stale price. Charge an estimate biased high rather than convenient. Charge the per-action cap if you accept what my own harness showed above — that on a tight budget this is refusal under a different name. Just never charge zero and call it accounting.

Two exit codes, and which one is an opinion

The tool prints exits under two mappings, because the difference is the trap:

EXIT_CONVENTIONAL = {"PASS": 0, "ADMIT": 0, "BLOCK": 1, "ERROR": 2}
EXIT_STRICT = {"PASS": 0, "BLOCK": 1, "ADMIT": 2, "ERROR": 3}

conventional is what most gates ship: an admitted action is a success. Under it, the blind run exits 0 while 34 actions went through un-priced. Your orchestrator sees green and moves on.

strict treats "admitted without a quote" as its own outcome. Under it no strategy exits 0 during an outage — refusing gets a 1, admitting blind gets a 2. Nobody gets a clean run when the oracle is down, which seems right to me.

That second mapping is my opinion, and I'm labeling it as one. My last attempt at this put ADMIT: 0 in the table, then acted amazed that fail-open exited green — I'd assumed the conclusion and called it a finding. The counts are the evidence. The exit code is a choice, and you should make your own.

The tool also exits worst-wins in every mode including the demo, so the default run exits 3. It contains a deliberate oracle fault. A file about failures laundered into green zeros doesn't get to launder its own.

The oracle-fault path catches magnitude, not just sign, since a cents-versus-dollars mixup is the classic cost-oracle bug:

      seq 3: oracle-untrusted: quote 500 is 100x last known 5 (unit error, not a price)

A 100x jump is caught as a broken oracle instead of an expensive action. The threshold is 20x and it's a judgement call, and it can't fire on the first quote of a run because there's nothing to compare against yet.

It's also one-sided, which I only noticed while writing this up. It catches a quote that's too big and sails straight past the mirror-image bug: dollars arriving as cents, every action priced at a hundredth of what it costs. For a spend cap that's the more dangerous direction — it under-counts instead of blocking, which is the same failure as everything else in this post — and my check doesn't cover it. It's marked as a known gap in the source rather than quietly left there.

What this is not

A benchmark of your bill. The constants ($0.05 a call, a $0.50 budget, 40 rounds) exist to make the branch legible. The transferable part is the shape.
Proof that outages and runaways co-occur. I believe they do, because a runaway hammering a rate-limited endpoint is often the thing knocking its own price feed over. A synthetic harness can't show that, and I'm not claiming it does.
A verdict on anyone's library. The equivalence is about the pattern of pricing a fallback at zero. It reproduces with --strategy fallback --fallback 0 --oracle-fails-from 6 in twenty lines of my own code. (Leave the outage flag off and you get a healthy-oracle run where the branch never fires — which proves nothing, as Scenario A says out loud.)
A correctness proof. The sha256s show determinism, nothing more. A wrong program reproduces byte-for-byte just as well as a right one.
A model of how commercial budget systems fail. AWS Budgets, GCP billing and most usage endpoints don't go quiet — they lag, then reconcile, and what you couldn't see gets billed to you later. My gate has no true-up: an estimate charged during the outage is never corrected when the oracle comes back. And for LLM calls there's no honest pre-call price at all, since output tokens aren't known until the call is finished — so a real LLM cap lives permanently in pessimistic/fallback and never earns the PASS row my harness prints. What this models cleanly is a price feed, which is the case I actually shipped.

This sits on a different axis from the sliding-window guard, which is about cheap calls that sum to a runaway, and from the pre-execution gate, which is about gating before you execute rather than after. Today's axis is oracle availability, and neither of those touched it.

The one I'm still stuck on

Stale pricing keeps the ledger alive, which is the whole recommendation, and it's also quietly a lie: you're charging against a number you know might be wrong. So how long is a cost estimate allowed to live before "cached" becomes "guessing"? For gas that moves in seconds it might be five seconds. For a token price it might be an hour. I don't have a principled way to set that TTL, and I suspect it's per-oracle rather than a general rule.

If you've drawn that line in production — the point where a cached cost stops being a fact — I'd like to hear where you put it and what made you move it.

Run it yourself: stdlib, offline, and it prints hashes you can diff against mine. Then go look at your own cap and answer one question. When it can't price the next call, does the number in your ledger keep moving?

Follow for the next teardown in this series, and tell me the worst thing your agent ever did while your dashboard showed $0.00. I read every comment.

Written with AI assistance and reviewed/edited by a human. The Python in this post was run offline (stdlib only, no network, no keys, no funds) on 2026-07-19; every number and hash in the output blocks is from a real deterministic run, repeated three times byte-for-byte. Code sha256 ddc425908d070c07a6765810e6115f649c17312b8e754d034227d37c982357e8, output sha256 9ebe1b4ab3459eb78c1d3aeea7eaa16ee0c2286e1d8d57b09e0eeb6451a189f1. The $1,847 figure belongs to @royanannya and was not reproduced here; the circuit-breaker design described is @ddhh's, quoted from their post and their MIT-licensed repository.