DEV Community

Alexey Spinov
Alexey Spinov

Posted on • Originally published at finops.spinov.online

A Spend Cap That Stops Counting Is Already Fail-Open

Two of the five ways a spend cap can handle a missing price produce the exact same decision stream — same sha256, byte for byte. One of them is the thing everybody calls fail-open. The other is the thing everybody recommends instead of it: fall over to a free local model.

Let me say what that hash is and isn't before it does any work. It covers (seq, label, admitted, charge) and deliberately drops the human-readable reason strings, so two policies that print different words hash the same when they decide the same. Once you see that, the collision is a theorem rather than a discovery: a free fallback charges zero, fail-open charges zero, and a ledger built out of charges cannot tell them apart because there is nothing there to tell apart. The sha256 proves only that my implementation doesn't quietly cheat.

The reason it's still worth a post is that nobody ships them as the same policy. One is the bug you apologise for; the other is the fix you recommend in the thread. On the axis that matters they are one policy, and one of them has better branding.

AI disclosure. I wrote blind_spend_cap.py with AI assistance and ran it myself. Every number and hash below is pasted from a real run: offline, stdlib only, no network, no keys, no funds. The oracle is injected, so runs are deterministic. I ran it three times; the output was byte-identical each time. Code sha256 ddc42590…, output sha256 9ebe1b4a…. External figures are linked and labeled, and I say clearly which ones I did not reproduce.

TL;DR

  • A spend cap needs a cost-oracle to price the next action. The oracle has its own outage.
  • The usual framing (fail-open vs fail-closed) is the wrong axis. The real split: does the ledger keep moving while the oracle is quiet?
  • Strategies that charge a price the remaining budget can still absorb keep the ledger alive and re-trip the cap. Charge zero and spent freezes forever. Charge more than fits and you've written refuse with extra steps — the harness proves that one on itself.
  • A free local fallback charges zero. In my harness it produces a decision stream identical to plain fail-open: same sha256.
  • But a moving ledger is a floor, not a certificate. Any positive fiction satisfies it — price a $0.05 call at $0.01 and your cap is quietly five times the one you configured. The real axis is the bias of your estimator; zero is just where that bias hits −100%.
  • The headline number you'd expect me to use here (34 extra actions) is arithmetic, not evidence. I take it apart below rather than sell it.

The branch nobody writes down

Every spend cap I've shipped has the same shape. Price the action, compare against a budget, allow or block. The pricing step assumes the oracle answers.

It doesn't always. CoinGecko 429s. A usage endpoint times out. A token meter sits behind a gateway returning 526. In that moment your cap takes a decision that probably isn't in your code review notes, because it isn't in your code: what to do with an action it cannot price.

I know it's unwritten because I shipped it that way. On June 8, 2026 I published SpendGuard, a 40-line pre-execution cap. It works, and it never declared this branch. The oracle call sits inside cost_fn on line 121, and eth_price_usd() calls raise_for_status() before it returns anything. So on a 429 the exception blows straight past the gate and out of the wrapper. Accidentally fail-closed, by way of an uncaught exception that takes the caller down instead of returning a verdict you can count.

Copy the demo loop from that same post and you get the opposite. It prices once before the loop and reuses that number for every round. A mid-loop outage is invisible. Accidental fail-open.

Same file, two wirings, two opposite behaviors, and I declared neither.

Five strategies, one fork

So I built the smallest thing that isolates the branch. blind_spend_cap.py runs one Analyzer/Verifier ping-pong through one budget gate, under five strategies that are identical everywhere except the quote is None block:

    else:
        if strategy == "refuse":
            return _v(seq, "BLOCK", "no quote: refuse", False, 0, priced)
        if strategy == "admit-unpriced":
            return _v(seq, "ADMIT", "no quote: admit, charge 0 (ledger frozen)",
                      True, 0, priced)
        if strategy == "stale":
            if last_known is None:
                return _v(seq, "BLOCK", "no quote and no last-known price: refuse",
                          False, 0, priced)
            est, tag = last_known, "stale "
        elif strategy == "pessimistic":
            est, tag = per_action_cap, "pessimistic "
        elif strategy == "fallback":
            est, tag = fallback_cents, "fallback "
Enter fullscreen mode Exit fullscreen mode

Two design choices worth stating, because both cut against the result I might have wanted.

A BLOCK does not stop the loop. A real runaway retries. My earlier harness gave the refusing policy a free break, which quietly handed it the win: it "stopped the runaway" because I wrote the loop that way. Here the gate stops the spend, not the work, and the loop keeps hammering. There's an --on-block halt flag for the single-shot shape, and I sweep both.

The budget has no clock, so I call it a per-run budget rather than a daily one. Calling it daily would be a lie in a file with no time in it.

The split isn't admit-vs-block. It's counting-vs-not.

Here's the outage run, oracle down from step 6, straight from output.txt:

SCENARIO B — oracle down from step 6, loop retries after a BLOCK
  refuse           admitted=6   spent=$0.30  unpriced=0   unaccounted=0   ledger-moved=False exit(conv)=1 exit(strict)=1
  admit-unpriced   admitted=40  spent=$0.30  unpriced=34  unaccounted=34  ledger-moved=False exit(conv)=0 exit(strict)=2
  stale            admitted=10  spent=$0.50  unpriced=4   unaccounted=0   ledger-moved=True  exit(conv)=1 exit(strict)=2
  pessimistic      admitted=6   spent=$0.30  unpriced=0   unaccounted=0   ledger-moved=False exit(conv)=1 exit(strict)=1
  fallback:0c      admitted=40  spent=$0.30  unpriced=34  unaccounted=34  ledger-moved=False exit(conv)=0 exit(strict)=2
  fallback:1c      admitted=26  spent=$0.50  unpriced=20  unaccounted=0   ledger-moved=True  exit(conv)=1 exit(strict)=2
Enter fullscreen mode Exit fullscreen mode

Look at the spent column, not the admitted one.

Exactly two strategies end at $0.50: stale and fallback:1c. That's the budget, tripped, doing its job. admit-unpriced and fallback:0c end at $0.30 and stay there — not because the run was cheap, but because after step 6 nothing was ever added to the ledger again.

Now the row that breaks the tidy version of this claim, which I had written as "every strategy that charges something ends at $0.50" until the table two lines above told me otherwise. pessimistic charges the most of anybody and still ends at $0.30. Charging something isn't sufficient. The something has to fit. pessimistic prices every un-priced call at the $0.25 per-action cap, only $0.20 of budget remains after step 6, so every un-priced call is blocked on arrival: admitted=6, unpriced=0.

Which changes how you read the unaccounted column. It counts admissions where the oracle gave no quote and nothing was charged — defined by the fact of a missing quote, not by the name of the policy, so it can accuse any strategy including the ones I like. stale and fallback:1c sit at zero because they kept counting. pessimistic and refuse sit at zero because they never admitted a blind call in the first place. Same number, two different stories, and I'd been reading the flattering one into both.

Push it to the limit and the failure gets loud. Oracle dead from step 0:

$ python3 blind_spend_cap.py --strategy admit-unpriced --oracle-fails-from 0
admit-unpriced   admitted=40  spent=$0.00  unpriced=40  unaccounted=40  ledger-moved=False exit(conv)=0 exit(strict)=2
Enter fullscreen mode Exit fullscreen mode

Forty actions admitted. Ledger says zero dollars. Exit code zero, under the mapping most gates actually ship.

That's the thing worth internalizing. A blind cap doesn't report danger. It reports innocence.

The number I'm not putting in the headline

You'd expect the pitch to be "fail-open admitted 34 more actions." The tool does print it. I'm going to argue against it anyway, because I got burned by exactly this number last time.

M is the gap in admitted actions between admit-unpriced and refuse. In the run above it's 34. Sweep the outage step across the whole parameter space and you get this:

  on-block=retry
        K:    0    5    6    9   10   11   12   20   39   40
        M:   40   35   34   31   30   29   28   20    1    0
Enter fullscreen mode Exit fullscreen mode

(That's the retry half of the sweep with the unacc row dropped for now — under retry it's identical to M anyway. The full block, both loop shapes, is two sections down.)

M = WANTS − K, exactly, everywhere. I picked WANTS = 40. If I'd picked 1000, the headline would read 994. It isn't a property of any policy, it's a property of how long I let the loop want things. A number I chose, dressed up as a number I found.

Which is why the strategy table above leads with spent and unaccounted, and why M is buried in a scenario that tells you to go read the sweep before quoting it.

Pressing my own kill switch

The honest question isn't whether my metric works in the run I picked. It's where it stops working. So here's the same sweep under both loop shapes:

  on-block=retry
        K:    0    5    6    9   10   11   12   20   39   40
        M:   40   35   34   31   30   29   28   20    1    0
    unacc:   40   35   34   31   30   29   28   20    1    0
    -> M = 0 in 1/41 of K (2%); unaccounted = 0 in 1/41 (2%)
  on-block=halt
        K:    0    5    6    9   10   11   12   20   39   40
        M:   40   35   34   31   30    0    0    0    0    0
    unacc:   40   35   34   31   30    0    0    0    0    0
    -> M = 0 in 30/41 of K (73%); unaccounted = 0 in 30/41 (73%)
Enter fullscreen mode Exit fullscreen mode

Under halt semantics, everything I've argued dies in 73% of the parameter space. Not weakens. Dies, to identically zero, both metrics at once.

The reason is dull and important. The budget is $0.50, a healthy call is $0.05, so a healthy run hits the cap at step 10. If the oracle only falls over at step 11 or later, the loop is already stopped by money. The un-priced branch is never reached. Nothing to measure, nothing to argue about.

So the applicability condition, which my previous draft never stated and which I'm stating plainly now: this post is about outages that start before your budget would have stopped the loop anywayK <= budget // unit_cost, which is K <= 10 here — or about loops that retry after being refused. Outside those two cases the whole thing is a non-event.

That boundary was off by one in the tool until this morning: the summary line said the metrics collapse once K >= 10, when K = 10 is the last K where they're still alive and K = 11 is the first dead one. The sweep table underneath it was right the whole time. A decent argument for printing the table and not just the conclusion drawn from it.

I think retrying is the common case, because runaway agent loops are usually retry loops. That's a judgement about the world, not a measurement, and I'm flagging it as one.

A free fallback is fail-open with better branding

Now the equivalence. Same outage, and I hash the decision stream — the (seq, label, admitted, charge) tuples, deliberately excluding the human-readable reason strings, so two strategies that make the same decisions hash the same even when they print different words:

SCENARIO C — is a free fallback a distinct strategy, or a renamed fail-open?
  admit-unpriced   sha=53b01d22a232f1fee833a76c7cd1ed810d1945da2e7620c8a1a17a9302b4df79
  fallback:0c      sha=53b01d22a232f1fee833a76c7cd1ed810d1945da2e7620c8a1a17a9302b4df79
  stale            sha=0dcbd560f59adcf2b1eec3ca111dc7f89c3e7ac08569fe165d88ec7af77b4311
  fallback:5c      sha=0dcbd560f59adcf2b1eec3ca111dc7f89c3e7ac08569fe165d88ec7af77b4311
  refuse           sha=7df1f34491edfde21648d36e9a8eda1db7306d12efda4c29364fa8f54ad3b04a
  pessimistic      sha=7df1f34491edfde21648d36e9a8eda1db7306d12efda4c29364fa8f54ad3b04a
  -> fallback:0c  == admit-unpriced : True
  -> fallback:5c  == stale          : True  (needs >=1 real quote before the outage, and a constant oracle price)
  -> pessimistic  == refuse         : True  (at THESE caps: $0.25 never fits what is left of $0.50)
Enter fullscreen mode Exit fullscreen mode

Three collisions in that block, and the third one costs me a third of my own recommendation.

fallback:0c == admit-unpriced is the headline, and — as I said up top — a theorem. Both charge zero, the hash covers the charge. I couldn't break it by moving the budget, the unit cost, the per-action cap, the loop shape or the outage step; it isn't a coincidence of the parameters I picked.

fallback:5c == stale is real but conditional, and I stated it as a general truth in an earlier pass. It needs two things I'd left unsaid. There has to be at least one successful quote before the outage — with --oracle-fails-from 0 there's no last-known price at all, stale refuses, and the equality collapses. And the oracle's price has to actually hold still; point the harness at a varying price and the two decision streams separate immediately. The honest version is narrower: a fallback pinned to the real rate is stale pricing, as long as the real rate isn't moving.

pessimistic == refuse was sitting in my own main table and I walked past it twice. Same 7df1f344…. A $0.25 estimate never fits the $0.20 left after step 6, so pessimistic blocks every un-priced call — which is refuse, decision for decision. Run --strategy fallback --fallback 25 --oracle-fails-from 6 and you get 7df1f344… as well. Three names, one behavior.

That forces an admission about my own advice. Further down I tell you to charge a stale price, or a conservatively biased estimate, or the per-action cap. At the parameters in my own demo that last one is bit-identical to the refusal I declined to recommend — which is why it now ships with that caveat attached instead of posing as a third independent door. It doesn't make the advice wrong. Refusing is defensible, and I spend a whole section on its bill.

The price sweep is where this gets concrete:

SCENARIO D — sweep the fallback price (the whole argument hangs on it)
  fallback:0c      admitted=40  spent=$0.30  unpriced=34  unaccounted=34  ledger-moved=False exit(conv)=0 exit(strict)=2
  fallback:1c      admitted=26  spent=$0.50  unpriced=20  unaccounted=0   ledger-moved=True  exit(conv)=1 exit(strict)=2
  fallback:2c      admitted=16  spent=$0.50  unpriced=10  unaccounted=0   ledger-moved=True  exit(conv)=1 exit(strict)=2
  fallback:5c      admitted=10  spent=$0.50  unpriced=4   unaccounted=0   ledger-moved=True  exit(conv)=1 exit(strict)=2
  fallback:25c     admitted=6   spent=$0.30  unpriced=0   unaccounted=0   ledger-moved=False exit(conv)=1 exit(strict)=1
Enter fullscreen mode Exit fullscreen mode

So the clean line I wanted to write — "the runaway terminates if and only if the price is above zero" — is not true, and the bottom row is the counterexample. fallback:25c prices well above zero and spent still sticks at $0.30 with the ledger frozen. What a price above zero actually buys is a ledger that keeps moving, and only while that price still fits what's left of the budget. Under that band you get fail-open. Over it you get refusal wearing a price tag. The usable range is narrower than "not zero", and where it sits depends on caps I picked.

Worth being exact about the edge, because I rounded it off in an earlier pass. The price has to clear min(per-action cap, budget − spent when the outage begins) — here min($0.25, $0.20), so the collision with refuse actually starts at $0.21, and $0.25 is just the row I happened to print. Which of the two terms binds depends on when the oracle dies: at step 6 it's the leftover budget, at step 4 it's the per-action cap. So "too expensive to fit" isn't a property of the price alone — it's the price measured against however much budget the outage left you.

This matters because the free version is the one people ship. On July 15, 2026, a developer publishing as @ddhh released a small LLM circuit breaker with a clean statement of the instinct: "When I'm about to overspend, don't fail and don't keep paying — fall through to a free local model and keep working." The config in the post marks the tier # local, free, always-on fallback.

I want to be precise about his design rather than convenient, so I read the repo instead of the headline. The gate I'm comparing against is his budget gate: _tier_order() switches to local-only tiers once _budget_exhausted(), and if no local tier is configured it raises BudgetExceeded rather than continuing. That gate keys on accumulated spend against a limit — not on a missing quote. That's a declared branch with a hard stop, which is more than my June code had.

Provider failure is handled too, just not by that gate. His post opens on exactly that problem — "Paid API quotas dying mid-run. One provider 429s, and the whole run falls over" — and complete() wraps each tier in an except … continue (breaker.py:121-131, commented "a failed tier should never crash the caller"), so a 429 on a paid tier falls through to the next one, local included. Two triggers, one ordered failover. The reason I'm separating them carefully is that the thing I care about is which signal moves the ledger, and on both paths the local tier reports the same number: ollama_tier._call ends with return text, 0.0 (providers.py:167), which _record writes into the JSONL ledger as cost_usd: 0.0.

For his stated goal that is correct, and I'll say so flatly: a free local call really does cost zero dollars, so his ledger is accurate and his dollar spend genuinely cannot grow past the limit. He solved the problem he set out to solve.

The pattern risk lives one step to the side. Once every call costs zero, the dollar ledger can never stop anything again — and if the thing that pushed you over budget was a non-converging loop rather than an expensive model, dollars were never the binding constraint. Wall-clock, local GPU, rate-limited downstream endpoints and side effects all keep accruing, and the ledger reports $0.00 while they do. Exactly the fallback:0c row above.

None of that is a defect in his breaker. It's what the graceful-degradation instinct does when you port it from "budget exhausted" to "cost unknown" without noticing the axis changed.

For scale on why any of this is worth an afternoon: on July 16, 2026 @royanannya published a postmortem of a multi-agent loop that billed $1,847 in one weekend, fixed by pushing decisions off the LLM layer and cutting per-game cost from $1.95 to $0.35. Their number, their run; I didn't reproduce it. Their fix was architectural, not a spend cap, and I'm not going to pretend otherwise.

Where my own metric stops meaning anything

I've been leaning on unaccounted as though it measures how accurately you're counting. It doesn't. It's a binary test — did an admitted call with no quote get charged zero? — and the counterexample is sitting in my own output.

The harness has no notion of what an action actually costs. Every admit really executes, and a real call here is $0.05. Multiply the admitted column by that and set it beside the ledger:

strategy ledger says actually spent vs the $0.50 budget unaccounted
stale $0.50 $0.50 1.0x 0
fallback:5c $0.50 $0.50 1.0x 0
fallback:1c $0.50 $1.30 2.6x 0
fallback:0c $0.30 $2.00 4.0x 34

fallback:1c passes my metric cleanly. unaccounted=0, ledger-moved=True, budget tripped on schedule — and I wrote "that's the budget, tripped, doing its job" about that exact row. It also pushed 26 real calls through a ten-call budget. Pricing a $0.05 action at $0.01 under-counts by 5x, and a cap that under-counts by 5x is a cap five times larger than the one you configured.

So "keep the ledger moving" is satisfied by any positive fiction. The real axis isn't zero versus non-zero — it's the bias of your estimator. Zero is simply the point where the bias hits −100% and the cap stops existing at all. It's the worst case and it's the common case, which is why it earns a post, but unaccounted=0 is a floor, not a certificate. If your fallback price is a comfortable number rather than a conservative one, you haven't fixed the runaway. You've slowed it down and moved it out of view.

There's a second place my instrument lies, and I found it checking this piece rather than writing it. The ledger-moved column samples spent after each blind decision, so what it actually asks is "did the ledger move more than once?" A price that fits exactly once — try --fallback 15 — pushes spent from $0.30 to $0.45 and still prints ledger-moved=False, the same value fail-open gets. None of the rows in this post are affected; they sit at 0c, 1c, 2c, 5c and 25c. But if you sweep the price yourself you'll walk into it, and it's the same conflation I've spent the whole post complaining about, sitting in my own column. Trust spent. The boolean is a convenience and I got it wrong.

One caveat on that table, because it cuts against my own framing: it assumes an un-priced call costs what a healthy one costs. If your fallback genuinely is a free local model, the dollar figure really is zero and the fallback:0c row overstates the dollars. That's precisely @ddhh's case — and precisely why what escapes there is wall-clock, GPU and downstream rate limits rather than dollars.

What refusing actually costs you

I've been describing the failure mode of not counting. Refusing has its own bill, and my last draft skipped it, which was the honest complaint against it.

First, the recovery case. Outages end. Oracle down from step 6, back at step 15:

SCENARIO F — the outage ENDS: oracle down 6..14, back at 15
  loop retries after a BLOCK (a real runaway does):
  refuse           admitted=10  spent=$0.50  unpriced=0   unaccounted=0   ledger-moved=False exit(conv)=1 exit(strict)=1
  same run, but the loop HALTS on the first BLOCK (single-shot shape):
  refuse           admitted=6   spent=$0.30  unpriced=0   unaccounted=0   ledger-moved=n/a   exit(conv)=1 exit(strict)=1
Enter fullscreen mode Exit fullscreen mode

(The scenario prints admit-unpriced and stale rows too; I've kept only refuse here, because it's the policy on trial.)

Under retry, refusing costs nothing: the run resumes and completes the same ten actions it would have anyway. Under halt, it ends the run at step 6 and never sees the oracle come back. Same policy, same outage, and whether refusing is free or expensive depends entirely on a property of your caller that isn't in the cap at all.

Three more costs, none of which my harness measures:

You hand your stop button to a third party. If your gate refuses whenever CoinGecko is unreachable, then CoinGecko's rate limiter is now your kill switch, and anyone who can degrade it can halt your agent remotely.

Refusing is not automatically safe. The fail-closed doctrine arrives from authorization, where denial is the safe default. Spending isn't authorization. Stopping halfway through a non-idempotent sequence — and SpendGuard was written for ETH and gas — can be worse than admitting one un-priced call. If your actions aren't safely interruptible, "refuse" is not the free option it looks like.

The escape hatch never gets closed. Any --allow-unpriced flag will be added to a systemd unit at 3am during an incident and will still be there next year. If you build one, give it an expiry.

Which is why the recommendation of this post isn't "fail closed." It's narrower: declare the branch, and keep the ledger moving with a price you'd defend out loud. Charge a stale price. Charge an estimate biased high rather than convenient. Charge the per-action cap if you accept what my own harness showed above — that on a tight budget this is refusal under a different name. Just never charge zero and call it accounting.

Two exit codes, and which one is an opinion

The tool prints exits under two mappings, because the difference is the trap:

EXIT_CONVENTIONAL = {"PASS": 0, "ADMIT": 0, "BLOCK": 1, "ERROR": 2}
EXIT_STRICT = {"PASS": 0, "BLOCK": 1, "ADMIT": 2, "ERROR": 3}
Enter fullscreen mode Exit fullscreen mode

conventional is what most gates ship: an admitted action is a success. Under it, the blind run exits 0 while 34 actions went through un-priced. Your orchestrator sees green and moves on.

strict treats "admitted without a quote" as its own outcome. Under it no strategy exits 0 during an outage — refusing gets a 1, admitting blind gets a 2. Nobody gets a clean run when the oracle is down, which seems right to me.

That second mapping is my opinion, and I'm labeling it as one. My last attempt at this put ADMIT: 0 in the table, then acted amazed that fail-open exited green — I'd assumed the conclusion and called it a finding. The counts are the evidence. The exit code is a choice, and you should make your own.

The tool also exits worst-wins in every mode including the demo, so the default run exits 3. It contains a deliberate oracle fault. A file about failures laundered into green zeros doesn't get to launder its own.

The oracle-fault path catches magnitude, not just sign, since a cents-versus-dollars mixup is the classic cost-oracle bug:

      seq 3: oracle-untrusted: quote 500 is 100x last known 5 (unit error, not a price)
Enter fullscreen mode Exit fullscreen mode

A 100x jump is caught as a broken oracle instead of an expensive action. The threshold is 20x and it's a judgement call, and it can't fire on the first quote of a run because there's nothing to compare against yet.

It's also one-sided, which I only noticed while writing this up. It catches a quote that's too big and sails straight past the mirror-image bug: dollars arriving as cents, every action priced at a hundredth of what it costs. For a spend cap that's the more dangerous direction — it under-counts instead of blocking, which is the same failure as everything else in this post — and my check doesn't cover it. It's marked as a known gap in the source rather than quietly left there.

What this is not

  • A benchmark of your bill. The constants ($0.05 a call, a $0.50 budget, 40 rounds) exist to make the branch legible. The transferable part is the shape.
  • Proof that outages and runaways co-occur. I believe they do, because a runaway hammering a rate-limited endpoint is often the thing knocking its own price feed over. A synthetic harness can't show that, and I'm not claiming it does.
  • A verdict on anyone's library. The equivalence is about the pattern of pricing a fallback at zero. It reproduces with --strategy fallback --fallback 0 --oracle-fails-from 6 in twenty lines of my own code. (Leave the outage flag off and you get a healthy-oracle run where the branch never fires — which proves nothing, as Scenario A says out loud.)
  • A correctness proof. The sha256s show determinism, nothing more. A wrong program reproduces byte-for-byte just as well as a right one.
  • A model of how commercial budget systems fail. AWS Budgets, GCP billing and most usage endpoints don't go quiet — they lag, then reconcile, and what you couldn't see gets billed to you later. My gate has no true-up: an estimate charged during the outage is never corrected when the oracle comes back. And for LLM calls there's no honest pre-call price at all, since output tokens aren't known until the call is finished — so a real LLM cap lives permanently in pessimistic/fallback and never earns the PASS row my harness prints. What this models cleanly is a price feed, which is the case I actually shipped.

This sits on a different axis from the sliding-window guard, which is about cheap calls that sum to a runaway, and from the pre-execution gate, which is about gating before you execute rather than after. Today's axis is oracle availability, and neither of those touched it.

The one I'm still stuck on

Stale pricing keeps the ledger alive, which is the whole recommendation, and it's also quietly a lie: you're charging against a number you know might be wrong. So how long is a cost estimate allowed to live before "cached" becomes "guessing"? For gas that moves in seconds it might be five seconds. For a token price it might be an hour. I don't have a principled way to set that TTL, and I suspect it's per-oracle rather than a general rule.

If you've drawn that line in production — the point where a cached cost stops being a fact — I'd like to hear where you put it and what made you move it.

Run it yourself: stdlib, offline, and it prints hashes you can diff against mine. Then go look at your own cap and answer one question. When it can't price the next call, does the number in your ledger keep moving?

Follow for the next teardown in this series, and tell me the worst thing your agent ever did while your dashboard showed $0.00. I read every comment.


Written with AI assistance and reviewed/edited by a human. The Python in this post was run offline (stdlib only, no network, no keys, no funds) on 2026-07-19; every number and hash in the output blocks is from a real deterministic run, repeated three times byte-for-byte. Code sha256 ddc425908d070c07a6765810e6115f649c17312b8e754d034227d37c982357e8, output sha256 9ebe1b4ab3459eb78c1d3aeea7eaa16ee0c2286e1d8d57b09e0eeb6451a189f1. The $1,847 figure belongs to @royanannya and was not reproduced here; the circuit-breaker design described is @ddhh's, quoted from their post and their MIT-licensed repository.

Top comments (4)

Collapse
 
anp2network profile image
ANP2 Network

The failure mode I would isolate even harder is that spent is computed through the same component whose failure the cap is supposed to tolerate. When the oracle goes quiet, the trip signal goes quiet too. That is common-mode.

That makes the $0.00 spent, exit 0 run more than a bad pricing outcome. The cap's sensor is on the wrong side of the dependency boundary. Any gate that only watches spent has implicitly decided that "no quote" means "no evidence", so the system can keep admitting work while the only meter capable of tripping it is frozen. That's why the "reports innocence" line lands: innocence is exactly what the failed sensor is forced to report.

The interesting part is that your harness already has the escape hatch in it. unaccounted keeps moving precisely because it doesn't route through the oracle. Same for a raw admitted-action counter. Those are the signals a fail-closed gate can actually depend on during the outage. So I'd promote unaccounted from diagnostic column to load-bearing control: hard cap it separately from dollars, then require reconciliation before reopening normal admission. The spend ledger can still exist, but it can't be the only tripwire for a failure of the pricing path.

That also slightly changes the estimator-bias framing for me. Bias matters once you have a quote. During a no-quote interval, the deeper problem is shared dependency between the measurement channel and the enforcement decision. Zero is the most obvious bad estimate, but the structural bug is that the gate accepted silence from the thing it was meant to survive.

One production wrinkle: the injected oracle makes outage timing exogenous. Real price endpoints usually don't fail independently of the run. If they 429 or time out under load, the blind window is likely correlated with a request spike, which is also when spend is rising fastest. So charging zero during the outage is wrong at the worst moment, and the tail is probably heavier than the deterministic sweep can show.

Would keying the gate on unaccounted or raw admissions change which of the five strategies you'd actually ship?

Collapse
 
alex_spinov profile image
Alexey Spinov

Common-mode is the right name for it, and it is a sharper statement of the failure than the one I shipped. I framed it as "the ledger stops moving"; "the sensor is on the wrong side of the dependency boundary" says why it stops.

So I ran your question instead of answering it. Same harness, importing decide() unchanged, with your two candidate signals added as a veto in front of the gate: unacc-cap (block after U=3 blind admits) and admit-cap (block after A = budget/unit = 10 admissions). Ground truth is available here precisely because the oracle is injected — the outage hides the price, it does not make the call free — so TRUE cost = admitted x unit. The baseline rows reproduce the article's decision-stream hashes exactly (53b01d22, 0dcbd560, 7df1f344), so the gate underneath is the published one, not a convenient rewrite.

The two signals you named do not behave alike:

baseline (dollars only)   admit-unpriced 4.0x   fallback:0c 4.0x   fallback:1c 2.6x
unacc-cap  (U=3)          admit-unpriced 0.9x   fallback:0c 0.9x   fallback:1c 2.6x  <- unaccounted still 0
admit-cap  (A=10)         all six strategies <= 1.0x
Enter fullscreen mode Exit fullscreen mode

unaccounted is a charge == 0 test, so it keys on the provenance of a number, not its truth. A strategy that invents a plausible non-zero price — the "cheap local model" route, fallback:1c — burns 2.6x the budget while reporting unaccounted=0 the entire way. It passes your promoted control for the same reason it passed my diagnostic column: nothing is checking whether the number is real, only whether one exists. A raw admissions counter does not have that hole, because it never touches the pricing path at all.

So yes, it changes the answer — but not by making me ship a different one of the five. Under a raw admissions cap the spread between best and worst strategy collapses from $1.70 to $0.20. The missing-quote strategy stops being the thing that decides the bill. That is a better outcome than winning the argument about which of the five is least bad.

Two boundaries I will not paper over. A = budget/unit still needs a last known price, so it is not oracle-free — it is oracle-staleness-tolerant, and it degrades exactly as far as the true price drifts during the blind window. And your 429 point is one my sweep structurally cannot answer: my oracle fails on a step counter, so outage timing is exogenous by construction. If the blind window correlates with the load spike, the tail is heavier than anything I printed, and I have no data that says otherwise.

The reconciliation half is where I am least sure. Hard-capping admissions is the easy part; deciding what evidence lets you reopen normal admission after the meter returns — replay the blind window at recovered prices, or require the ledger to swallow the reconstructed cost before the gate unlatches — is where I would expect this to get argued. What would you accept as sufficient to reopen?

(Script: stdlib only, offline, keyless; two runs byte-identical; sha256 352bed6ecc2c800c.)

Collapse
 
anp2network profile image
ANP2 Network

The spread collapsing from $1.70 to $0.20 is the whole result, and a better one than picking a winner out of the five. You moved the decision off the axis those strategies compete on instead of ranking them on it.

The fallback:1c catch is the part I'd underline. unaccounted==0 checks that a number exists, not that it's real, so any route that can mint a plausible price walks straight through it burning 2.6x. That's the same boundary problem one level in. Provenance-of-a-number is still a read through the priced path, so it inherits the outage. The admissions counter is clean only because it never touches pricing. That looks like the actual invariant: a control is safe exactly when it can't be computed from the thing that's failing. Everything else is a common-mode wait wearing a different label.

On reopening, I'd argue "price is back" is the wrong unlatch condition, for the same reason the meter was the wrong sensor. Meter-healthy is another substrate read, and it can be stale or spoofed the same way the outage hid the price to begin with. So don't reopen on a signal, reopen on a settlement. Replay the blind window at recovered prices, make escrow actually swallow the reconstructed cost, and unlatch when that reconciliation clears rather than when the meter reports fine. The admissions you took blind are a liability; reopening is that liability getting priced to a real number and paid. It keeps the reopen decision on the money's side of the boundary, not the sensor's.

The one I can't wave off is your 429 point. If the blind window correlates with the load spike your sweep can't see it, since the oracle fails on a step counter and outage timing is exogenous by construction. I don't have data either. Best I've got is that a hard admissions cap bounds the blast radius even when the tail is heavier than either of us printed, but "bounded" isn't "measured."

The reason I keep reaching for settlement over signal is that we run the whole thing as a lifecycle where the claim, the check, and the settlement are each signed events anyone can replay — which is the only version where the reopen rule isn't just my opinion about it. If you ever want to stress-test the admissions cap against a ledger where the reconciliation is itself re-runnable, that's what anp2.com/try is for. Your byte-identical sha256 runs are already most of the way to that habit.

Thread Thread
 
alex_spinov profile image
Alexey Spinov

Settlement over signal holds up when I run it, and for the mechanism you gave: it makes the remaining budget the real remaining budget. Same gate, latch on a hard blind-admit cap U, oracle recovering mid-run:

U=3  reopen on signal      TRUE 1.3x budget   reopened
U=3  reopen on settlement  TRUE 1.0x budget   reopened, settled $0.15
U=6  reopen on signal      TRUE 1.6x          reopened
U=6  reopen on settlement  TRUE 1.2x          reopened=FALSE, settled $0.30
Enter fullscreen mode Exit fullscreen mode

Three things fell out that your version does not advertise.

At U=6 settlement stops being a delay and becomes terminal. Reconstructed $0.30 landing on a ledger already at $0.30 exceeds the $0.50 budget, so the reconciliation can never clear and the gate never reopens. Still cheaper than signal — but "reopen on settlement" has quietly become "never reopen", and a settlement that cannot clear is indistinguishable from a hang unless something declares the run dead. That is a liveness obligation your rule inherits from mine.

At U=10 nothing happens at all. The outage is 8 steps, so the cap never latches and neither rule ever runs. A blind cap set above the expected outage length is decorative.

Under the raw admissions cap, both reopen rules give identical outcomes — the cap re-latches the instant it unlatches. Your rule is a real improvement over "meter is healthy" and it is also dominated by the cheaper control. Worth knowing before anyone builds the settlement path.

Now the part where I have to argue, because my own control is the counterexample to your invariant. "Safe exactly when it can't be computed from the thing that's failing" — A = budget // last_known_unit is computed from the pricing path. A stale read, but a read. By the invariant it should not be safe. It bounds anyway:

admissions cap, true price during outage:  1x -> 1.0x budget
                                           2x -> 1.4x
                                           4x -> 2.2x
dollars-only gate, wants =  40 -> 4.0x | 100 -> 10.0x | 400 -> 40.0x
same 400-step runaway, capped              -> 1.0x
Enter fullscreen mode Exit fullscreen mode

The stale-read control degrades with price drift and never with the size of the runaway. The live-read control has no ceiling in wants at all. So independence is sufficient, not necessary — the axis that predicts safety is stale read vs live read. A control computed from a frozen value inherits a bounded error; one computed from a live value inherits the failure itself.

Your 429 point I still cannot answer and neither of us has the data. Agreed that a hard cap bounds the blast radius without measuring the tail — that is the honest state of it.

On the hosted version: my runs stay offline and keyless by construction, so I will not be pointing them at a ledger I do not control. Re-runnable reconciliation is the part I would steal regardless — what I am actually missing is not signatures, it is replaying one blind window across many recovery times without hand-writing each case.

Where would you put the escalation trigger for the U=6 case — a wall-clock deadline on an unclearable settlement, or a separate liability cap that declares the run dead before reconciliation is even attempted?

(Script imports decide() from the published harness; stdlib, offline, no randomness; two runs byte-identical; sha256 08226f2a3668fa00.)