Make Token Spend a Chunk-Level Ledger Before a Stream Cut Rewinds the Step

At 02:40 my nightly eval loop stopped mid-sentence, and at 09:00 I had two numbers that could not both be right.

The worker reported 6,400 tokens spent for the run; the gateway reported 11,900. Which one do you believe when the pass rate also dropped and nobody changed the prompts? My loop was not wrong about the model, it was wrong about accounting granularity, and free-tier workers make that mistake loud.

This post is an architecture review of one narrow question: when a stream is cut mid-step, where do you record the spend, and what invariant decides whether the step may be replayed? I will declare assumptions, model the data flow, enumerate failure domains, and build a small offline harness you can run before you trust any loop on a free tier.

The constraints I assume (and you should write yours down)

Assumptions. (1) The worker process can be killed without warning; there is no graceful drain. (2) The model gateway bills per delivered chunk, not per completed step. (3) A stream can end three ways: terminator, transport error, or silence followed by eviction. (4) Only durably flushed bytes are reusable after a restart. (5) The token budget is finite and shared across steps, so one orphaned step changes what the next step can afford. (6) I am not assuming exactly-once billing from anyone.

In my current setup the worker runs on MonkeyCode's free server option and the model calls go through its free model access. Those availability claims are operator-supplied, as is the advertised free allotment (my working figure is on the order of ten million tokens). Terms like this move, so verify the current ones on the project page before you size a budget against them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Could a step-level counter ever be enough? Only if streams never get cut, and preemptible free-tier workers are exactly the environment where that assumption fails.

Data flow, and the three places it can break

 control plane        worker (free-tier server)        gateway (free model access)
      |                        |                                  |
      |-- admit(step,est) ---->|                                  |
      |                        |----- stream request ------------>|
      |                        |<---- chunk k (billed) -----------|
      |                        |  flush? yes -> durable bytes     |
      |                        |  flush? no  -> orphan bytes      |
      |                        |                                  |
      |                        |   X eviction / 429 / reset       |
      |<--- cut(attempt) ------|                                  |
      |  reconcile, then decide: replay (new attempt) | abandon   |

The control plane never sees the chunks, so it can only reason about spend through whatever the worker managed to persist. That is the whole problem in one sentence: the billing authority is the gateway, the durability authority is the worker, and the admission authority is the control plane. Three authorities, no shared clock, and a worker that can vanish between the second and the third.

Failure domains

Failure class	Domain	Symptom	Naive fix	Invariant-preserving move
Eviction mid-stream	free-tier worker	chunks stop, no terminator	retry the same step id	mark attempt cut, recompute orphan spend, admit a fresh attempt
Rate limit or 5xx mid-stream	gateway	partial artifact, billed chunks	reuse the attempt id	new attempt id, never recycled
Redelivery from a stale queue	control plane	second dispatch after commit	trust a cache TTL	durable commit marker keyed by (step, attempt)
Out-of-order arrival	coordinator	DONE before the last chunk	trust arrival order	require a per-attempt durable flush watermark

Notice that only the first row is really about the free tier; the other three are the same bugs you would have on paid hardware, just quieter because you would not be watching the budget as closely.

Step 1: write the spend invariant before writing the loop

I use four rules, and I write them into the repository README so a reviewer can point at them.

1. S1 — Spend identity. spent == durable + orphan at every checkpoint, where orphan = billed tokens attached to bytes that are not durable.
2. S2 — No replay after commit. A step with a durable commit marker is never admitted again, even if a duplicate dispatch arrives.
3. S3 — Commit requires durable bytes. A terminator without a matching durable flush set does not commit; it becomes retryable or abandoned.
4. S4 — Admission is budget-aware. A new attempt is admitted only if spent + estimate <= budget; otherwise it is abandoned explicitly rather than started and starved.

S1 is the one people skip, because per-step counters satisfy it trivially until a cut happens, and then they satisfy it falsely.

Step 2: build an offline harness that reproduces the cut

I do not want my first encounter with this ordering to be at 02:40, so I rehearse it deterministically. The harness below makes no network calls: you hand it an event script, it maintains per-attempt accounting, and it tells you which invariant broke.

#!/usr/bin/env python3
"""chunk_ledger.py - offline rehearsal for chunk-level spend accounting."""
from __future__ import annotations
import argparse
from dataclasses import dataclass, field
from enum import Enum


class Kind(Enum):
    CHUNK = "chunk"
    CUT = "cut"      # stream ended without a terminator
    DONE = "done"    # terminator received, artifact is usable


@dataclass(frozen=True)
class Ev:
    kind: Kind
    key: tuple[str, int]      # (step, attempt)
    tokens: int = 0
    persisted: bool = False   # did the worker flush this chunk durably?


@dataclass
class Attempt:
    billed: int = 0
    durable: int = 0
    state: str = "open"       # open -> committed | cut | abandoned


@dataclass
class Ledger:
    budget: int
    attempts: dict[tuple[str, int], Attempt] = field(default_factory=dict)
    violations: list[str] = field(default_factory=list)

    @property
    def spent(self) -> int:
        return sum(a.billed for a in self.attempts.values())

    @property
    def durable(self) -> int:
        return sum(a.durable for a in self.attempts.values())

    @property
    def orphan(self) -> int:
        return self.spent - self.durable

    def admit(self, key: tuple[str, int], estimate: int) -> bool:
        step, _ = key
        if any(k[0] == step and a.state == "committed"
               for k, a in self.attempts.items()):
            self.violations.append(f"S2: {key} started after {step} committed")
            return False
        if self.spent + estimate > self.budget:
            self.violations.append(
                f"S4: {key} needs {estimate}, remaining {self.budget - self.spent}")
            return False
        return True

    def apply(self, ev: Ev) -> Attempt:
        a = self.attempts.setdefault(ev.key, Attempt())
        if ev.kind is Kind.CHUNK:
            a.billed += ev.tokens
            if ev.persisted:
                a.durable += ev.tokens
        elif ev.kind is Kind.DONE:
            if a.durable != a.billed:
                a.state = "cut"
                self.violations.append(
                    f"S3: {ev.key} done, durable {a.durable} != billed {a.billed}")
            else:
                a.state = "committed"
        elif ev.kind is Kind.CUT:
            a.state = "cut"
        return a

    def check(self) -> list[str]:
        errs = list(self.violations)
        for key, a in self.attempts.items():
            if a.durable > a.billed:
                errs.append(f"S1: {key} durable exceeds billed")
            if a.state == "committed" and a.durable != a.billed:
                errs.append(f"S1: committed {key} carries orphan bytes")
        return errs

    def report(self) -> None:
        print(f"{'attempt':<10}{'state':<11}{'billed':>7}{'durable':>9}{'orphan':>8}")
        for (step, n), a in self.attempts.items():
            print(f"{step}#{n:<8}{a.state:<11}{a.billed:>7}{a.durable:>9}"
                  f"{a.billed - a.durable:>8}")
        ratio = 0.0 if self.spent == 0 else self.orphan / self.spent
        print(f"spent={self.spent} durable={self.durable} orphan={self.orphan} "
              f"budget={self.budget} orphan_ratio={ratio:.2f}")


def cut_then_replay() -> list[Ev]:
    return [
        Ev(Kind.CHUNK, ("A", 1), 40, True),
        Ev(Kind.CHUNK, ("A", 1), 40, True),
        Ev(Kind.DONE, ("A", 1)),
        # attempt 1 dies: last chunk billed, never flushed
        Ev(Kind.CHUNK, ("B", 1), 60, True),
        Ev(Kind.CHUNK, ("B", 1), 60, False),
        Ev(Kind.CUT, ("B", 1)),
        # replay on a fresh attempt
        Ev(Kind.CHUNK, ("B", 2), 60, True),
        Ev(Kind.CHUNK, ("B", 2), 60, True),
        Ev(Kind.DONE, ("B", 2)),
    ]


def replay_after_commit() -> list[Ev]:
    return [
        Ev(Kind.CHUNK, ("A", 1), 40, True),
        Ev(Kind.CHUNK, ("A", 1), 40, True),
        Ev(Kind.DONE, ("A", 1)),
        Ev(Kind.CHUNK, ("A", 2), 40, True),   # stale queue redelivery
        Ev(Kind.DONE, ("A", 2)),
    ]


SCENARIOS = {
    "cut_then_replay": cut_then_replay,
    "replay_after_commit": replay_after_commit,
}


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--scenario", choices=sorted(SCENARIOS), default="cut_then_replay")
    ap.add_argument("--budget", type=int, default=1000)
    ap.add_argument("--estimate", type=int, default=200)
    args = ap.parse_args()

    led = Ledger(budget=args.budget)
    for ev in SCENARIOS[args.scenario]():
        if ev.key not in led.attempts and not led.admit(ev.key, args.estimate):
            continue
        led.apply(ev)

    led.report()
    errs = led.check()
    for e in errs:
        print("VIOLATION:", e)
    return 1 if errs else 0


if __name__ == "__main__":
    raise SystemExit(main())

One honest caveat: in this harness S1 holds by construction because totals are derived from per-attempt rows. The rules that actually bite are S2 and S3, and the orphan ratio, which is the number I want on a dashboard before I let a loop run unattended.

Step 3: inject the two orderings that matter

python3 chunk_ledger.py --scenario cut_then_replay --budget 1000 --estimate 200; echo "exit=$?"
python3 chunk_ledger.py --scenario replay_after_commit --budget 1000 --estimate 200; echo "exit=$?"

The first run exits 0 and prints one cut attempt carrying 60 orphan tokens against 200 durable ones. The second run exits 1 with S2: ('A', 2) started after A committed, which is the counterexample I want in CI: a duplicate dispatch from a stale queue must be rejected by the ledger, not absorbed because the cache had already expired.

Note that the retry in scenario one is admitted because spent + estimate still fits; shrink --budget to 250 and watch S4 fire instead, which is the correct outcome on a finite free allotment.

Step 4: turn the checks into an acceptance rule

Define the denominator before you argue about results. Mine is eight cut positions across an eight-chunk step, crossed with two retry policies (immediate replay, and replay gated on a durable commit marker), for 16 scenario runs. Acceptance is: zero S2 and S3 violations, and orphan_ratio <= 0.25 under gated replay. If a policy cannot hold both, I change the policy, not the threshold.

Tradeoffs, stated plainly

Option	Spend accuracy under cuts	Added latency	New failure surface	Wins when
Per-step counter	drifts silently	none	wrong retry decisions	you are still prototyping
Reserve per step	overshoots on cut	none	stranded reservations	streams almost never cut
Chunk-level ledger	bounded by orphan ratio	one durable write per chunk	write amplification	long steps on preemptible workers
Gateway-side reconciliation	highest	batch delay	needs a billing export	you have provider billing APIs

Who should not use this approach

If your gateway bills per completed step with exactly-once semantics, the ledger buys you little and costs you a write per chunk. If your steps are two seconds long, the orphan window is too small to justify the bookkeeping. And if you have no durable store other than the worker's disk, do not pretend a ledger exists: you have a counter, and it will drift the first time a free-tier worker is evicted while holding your accounting state.

What I would change next

The next revision is an admission control that estimates step cost from the last N attempts of the same step type, so S4 fires before a doomed attempt starts rather than after it burns the tail of the budget. That estimate is itself a small model, and it deserves its own failure injection.

So here is the question I would put to you: which event order breaks your invariant — the cut before the flush, the terminator before the durable set, or the redelivery after the commit — and when it does, should your loop reject the work, replay it as a fresh attempt, or compensate the effect it already caused? If you want a cheap place to rehearse those three, the free server option and free model access are worth a look; bring a workload that actually cuts, and check the current terms before you size anything against them.