DEV Community: Robin

Treat a Free-Tier Token Allowance as a Lease: Reservations, Fencing, and Fail-Closed Steps

Robin — Mon, 09 Nov 2026 20:27:33 +0000

Most agent sweeps on a free tier do not fail because the model was bad. They fail because two workers read the same remaining allowance, both decide there is room, and one of them commits capacity the other already promised away. This is an architecture review of that failure, written for engineers who run multi-step agent work on donated compute and donated tokens.

Example scenario (illustrative, not a report of a real run). A team wants a nightly evaluation sweep: forty agent steps, each calling a model a few times, all of it landing on a free server that may be reclaimed without notice. The model provider grants free access with a token allowance; the project also offers a free server option. Treat both grants as loans with expiry, not as balances you own.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The environment above matches what the open-source MonkeyCode project advertises as free model access plus a free server option, and its stated free token allowance is currently described as ten million tokens. Both are time-sensitive commercial terms, so verify the current numbers and limits on the project's official page before you size a run against them.

The violating event order

Consider a lease with capacity 100 units, where each unit is one thousand tokens for arithmetic convenience. Worker A wants a step it estimates at 60 units; Worker B wants a step it estimates at 50. Both call a helper named remaining(), both see 100, both admit, and both start calling the model. No single read was wrong, and the sum still crosses the capacity before either step finishes.

The invariant a correct design must preserve is this: at every prefix of the event order, committed plus reserved capacity never exceeds the lease capacity, and any step holding a stale lease epoch must fail closed rather than commit. Checking a balance is a read; preserving that invariant requires a write that others can observe.

Model the two grants separately

A free server and a free token allowance fail in different ways, so give them separate leases rather than one merged budget object. The compute lease governs where a step runs and can be revoked by the platform; the token lease governs how much model work an admitted step may consume. A step is admitted only when it holds both, and it is committed only against the token lease it was admitted under.

                 admission
  step request ───────────────► [compute lease: epoch c]
       │                              │
       │                              ├─ revoked ──► step aborts, artifact dropped
       ▼                              │
  [token lease: epoch t] ── reserve ──► RESERVED ── commit(actual) ──► COMMITTED
       │                                    │
       │                                    └─ release() ──► RELEASED
       └─ epoch advanced (revocation / reset) ──► commits from epoch < t are rejected

The state machine has five states per step: PROPOSED, RESERVED, COMMITTED, RELEASED, and ABORTED. RESERVED is the only state that holds capacity without having produced a result, which makes it the state worth alarming on. A run that ends with any step still in RESERVED has lost capacity silently, and that is a gate failure rather than a cosmetic warning.

Step 1: define the accounting unit before writing code

Pick one unit, declare how it is measured, and record the measurement source next to every committed number. If your model provider reports usage per call, you can commit actuals; if it does not, your commits are estimates and your capacity check is approximate. State which one you have, because an unlabelled estimate passed off as an actual is how budget reports become fiction.

Set two numbers per step: an estimate used for reservation, and a hard cap that the step may not exceed. The cap is what protects the shared lease from a single runaway loop, and the estimate is what protects other steps from over-reservation.

Step 2: reserve with a compare-and-set on a fencing epoch

Reservation must be a single atomic operation on the lease owner, not a read followed by a write. Each lease carries a monotonically increasing epoch; advancing the epoch invalidates every outstanding reservation, which is exactly what you want when the platform reclaims the server or the allowance resets.

The following module is illustrative and was not executed for this article; it exists to make the transitions concrete and to be adapted, not copied as a library.

# Illustrative only — not executed, not a library release.
from dataclasses import dataclass


class LeaseError(Exception):
    pass


@dataclass
class BudgetLease:
    lease_id: str
    epoch: int                 # fencing token, monotonic, bumped on revocation
    capacity: int              # total units this lease may admit
    reserved: int = 0
    committed: int = 0
    closed: bool = False

    def _remaining(self) -> int:
        return self.capacity - self.committed - self.reserved

    def reserve(self, step_id: str, estimate: int, cap: int) -> "Reservation":
        # Must run as one critical section on the lease owner.
        if self.closed:
            raise LeaseError("lease closed")
        if estimate <= 0 or cap < estimate:
            raise LeaseError(f"bad budget for {step_id}")
        if estimate > self._remaining():
            raise LeaseError(f"would overcommit by {estimate - self._remaining()}")
        self.reserved += estimate
        return Reservation(self, step_id, estimate, cap)


@dataclass
class Reservation:
    lease: BudgetLease
    step_id: str
    estimate: int
    cap: int
    epoch: int = None
    state: str = "RESERVED"

    def __post_init__(self) -> None:
        self.epoch = self.lease.epoch

    def _fenced(self) -> None:
        if self.epoch != self.lease.epoch:
            raise LeaseError(f"stale epoch on {self.step_id}")

    def commit(self, actual: int) -> None:
        self._fenced()
        if self.state != "RESERVED":
            return                      # idempotent replay of a duplicate commit
        if actual > self.cap:
            raise LeaseError(f"{self.step_id} exceeded its hard cap")
        self.lease.reserved -= self.estimate
        self.lease.committed += max(actual, 0)
        self.state = "COMMITTED"

    def release(self) -> None:
        # Not fenced: releasing only ever frees capacity, so a stale holder
        # cannot use it to claim budget. Still idempotent.
        if self.state != "RESERVED":
            return
        self.lease.reserved -= self.estimate
        self.state = "RELEASED"

The asymmetry between commit and release matters. Commit is a claim on capacity and must be fenced, while release only subtracts and is therefore safe from a stale holder. If your storage cannot express that asymmetry, put release behind a separate path that only the lease owner may call.

Step 3: make commit and release idempotent on step identity

Every commit and release is keyed by step_id, so a duplicate delivery changes state exactly once. This is the same requirement as any outbox, and it is worth repeating because a reclaimed free server often means the client retries a request whose first attempt already succeeded somewhere.

Store the reservation outcome before you publish the step's artifact. If the process dies between writing the artifact and committing the budget, you get an artifact with no accounting entry, which silently understates spend and hides a real failure.

Step 4: treat revocation as a normal transition

When the free server is reclaimed, aborting a step is correct behaviour, not an error page. Bump the lease epoch, let commits from old epochs fail closed, and mark the step ABORTED instead of retrying it into the same dead lease. Retries belong to the next epoch, with a fresh reservation.

Properties worth testing

Write these as assertions over a scripted event log rather than as prose in a design doc. Each one is checkable against the illustrative module above, and each one can be broken on purpose to confirm the test actually fires.

P1 no-overcommit : for every prefix of the event log,
                   committed + reserved <= capacity
P2 no-orphan     : every admitted step ends COMMITTED, RELEASED, or ABORTED
P3 fenced-commit : a commit whose epoch < lease.epoch raises LeaseError
P4 replay-safety : applying commit() twice changes state and totals once

A useful fault schedule is short and adversarial: reserve 60, reserve 50, bump the epoch, then commit both in reverse order. A correct implementation accepts the first reservation, rejects the second with a remaining-capacity message, rejects both commits after the epoch bump, and leaves reserved at zero once the aborts release their holds.

Failure domains to review

Failure	Blast radius	Detection	Design response
Two workers read the same remaining budget	Overspend; sweep halts mid-run	Sum of reservations exceeds capacity	Atomic reserve on the lease owner
Compute lease revoked mid-step	Partial artifact, wasted tokens	Epoch advanced before commit	Fail closed, mark `ABORTED`, do not reuse lease
Duplicate commit delivery	Double-counted spend	Step key appears twice in the ledger	Idempotent commit keyed by `step_id`
Estimate drift on long steps	Reservation too small in practice	Actual exceeds cap repeatedly	Track estimate error and raise caps by a measured factor
Allowance reset mid-run	Old reservations become meaningless	Epoch bump from the provider side	Same fence as revocation

Tradeoffs you are actually choosing

Option	Latency cost	Overspend risk	Best fit
Reserve before every call	One atomic round trip per step	Bounded by reservation granularity	Shared allowance, concurrent workers
Optimistic spend, reconcile later	None	Unbounded until reconciliation	Single worker, generous quota
Shard the allowance per worker	None across shards	Idle capacity in quiet shards	Predictable, even workloads
Per-step hard cap only	None	Shared lease still overrun by many small steps	Trivial scripts with one worker

Reservation granularity is the knob that decides whether this protocol helps or hurts. Reserving per model call protects the allowance tightly and adds a round trip to every call, while reserving per step is cheaper and lets one step hold capacity it never uses. Pick per step for long agent loops, and add a cap so a single step cannot monopolise the lease.

Who should not use this

If one worker runs one script against a comfortable allowance, the fencing layer is overhead with no failure to prevent. If your provider reports exact usage after the fact and your runs are short, reconciliation is simpler than reservation. And if you cannot tolerate rejected admissions at all, this protocol will annoy you, because failing closed is the behaviour that makes it work.

How to validate the conclusion yourself

Implement the module above against a single-process lease owner, then against a shared store with a compare-and-set primitive.
Drive the adversarial schedule and assert P1 through P4; break the fence on purpose and confirm P3 fails.
Randomise interleavings with a small state-space explored exhaustively, which is affordable at two or three workers.
Record the rejection rate and the estimate-versus-actual distribution across a real sweep, then decide whether per-step or per-call reservation fits your workload.

Acceptance rule for a sweep: it passes only if no step ends in RESERVED, committed never exceeds capacity at any prefix, and every artifact has exactly one accounting record. Anything else is a gate failure, regardless of how good the model outputs looked.

So here is the question to carry back to your own system: which interleaving of reservation, revocation, and duplicate commit breaks your current allowance check, and should the system reject the step, replay it under a new epoch, or compensate after the fact?

Architecture Review: Map Four Failure Domains Before Agent Steps Land on a Free Server

Robin — Sun, 08 Nov 2026 19:10:13 +0000

Architecture Review: Map Four Failure Domains Before Agent Steps Land on a Free Server

Here is an event order that ends with a duplicate deployment. A worker claims step s1, applies the patch, and the substrate disappears before the acknowledgement is written. The scheduler sees an open intent, no heartbeat, and no result, so it re-queues s1 on a fresh worker. The fresh worker applies the same patch again, and both attempts report success to different observers.

Nothing in that story is exotic: no network partition required, no Byzantine node, no exotic clock. It only requires that the place where a step runs and the place where the record of that step lives are allowed to fail independently. The common implementation fails to preserve one invariant: every step has at most one observable effect, and every effect has a durable intent that precedes it.

This is an architecture review of the substrate that keeps tempting teams to skip that invariant. I keep seeing the same move: put agent execution on a free server option, wire it to free model access, and treat the whole thing as a development laptop that happens to be remote. That framing is the bug. So let me write down the constraints, the data flow, the failure domains, and what I would change next.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

1. What "free" actually buys you, and what it does not

The operator of the MonkeyCode project lists two availability claims that make this review worth writing: free model access, and a free server option, with a stated free allowance currently quoted at ten million tokens. I am treating both as a snapshot with an expiry date, not as an architectural property, and you should re-verify them before you depend on them. (MonkeyCode is described by its operator as an open-source project; I have not audited its internals for this review, so nothing below assumes anything about how it is built.)

What is an architectural property is this: a substrate you did not buy, did not size, and cannot page anyone about is a borrowed substrate. Borrowed substrates are fine. Borrowed substrates with local-state assumptions are not. So before you run a single step on one, declare your assumptions out loud, because every later table depends on them.

A1 (revocation): the free server may stop, restart, or be reclaimed mid-step without notice.
A2 (state): local disk and process memory on that server are caches, not records.
A3 (clock): wall-clock time there is not comparable to your scheduler's clock; only sequence numbers you control are.

If A1 is false for your provider, you can delete half of this article. If A1 is true and you have not designed for it, you have already shipped the duplicate deploy above — you just have not observed it yet.

2. Four failure domains, one invariant

An agent step is not a function call; it is a small distributed transaction whose participants are a planner, a substrate, an external effect, and a ledger. Four domains fail independently, and each needs its own detection signal and its own recovery owner.

Failure domain	What actually fails	Detection signal	Recovery owner
Step identity	payload drifts between attempts (new prompt, new revision)	intent hash mismatch	scheduler — reject, do not retry
Effect	effect applied, acknowledgement lost	intent committed, no result row	ledger replay — dedupe by effect key
Clock	lease/TTL expires while the step is alive	monotonic gap vs. lease deadline	lease fencing by token, never by timestamp
Substrate	server vanishes mid-step	missing heartbeat and open intent	re-queue with the same `step_id`

The invariant that ties them together: intent before effect, effect keyed by step identity, replay keyed by the same identity. Note what it does not require: exactly-once delivery. Delivery stays at-least-once, which is the only thing a borrowed substrate can honestly promise. Exactly-once is achieved at the effect, not on the wire.

3. A minimal harness that injects all four

You cannot review what you cannot execute. The following is a ~60-line fixture I use to argue about this in design reviews, with the ledger deliberately placed outside the "server" so the two can fail apart. Standard library only; run it locally.

# harness.py — python harness.py
import hashlib, json, sqlite3


def key(payload):
    return hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()


class Ledger:
    """Lives OUTSIDE the borrowed substrate. Never on the free server."""

    def __init__(self):
        self.db = sqlite3.connect(":memory:")
        self.db.executescript("""
            CREATE TABLE intent(step_id TEXT PRIMARY KEY, phash TEXT, state TEXT);
            CREATE TABLE effect(step_id TEXT PRIMARY KEY, ekey TEXT);
            CREATE TABLE side(skey TEXT PRIMARY KEY);   -- the external system, simulated
        """)

    def intend(self, step_id, payload):
        ph = key(payload)
        self.db.execute("INSERT OR IGNORE INTO intent VALUES (?,?,'intended')", (step_id, ph))
        stored = self.db.execute("SELECT phash FROM intent WHERE step_id=?", (step_id,)).fetchone()
        if stored[0] != ph:
            raise ValueError(f"payload drift on {step_id}: reject, do not retry")

    def commit(self, step_id, ekey):
        with self.db:
            first = self.db.execute("INSERT OR IGNORE INTO side VALUES (?)", (ekey,)).rowcount
            self.db.execute("INSERT OR IGNORE INTO effect VALUES (?,?)", (step_id, ekey))
            self.db.execute("UPDATE intent SET state='committed' WHERE step_id=?", (step_id,))
        return first   # 0 == this effect already existed

    def state(self, step_id):
        row = self.db.execute("SELECT state FROM intent WHERE step_id=?", (step_id,)).fetchone()
        return row[0] if row else None

    def effects(self):
        return self.db.execute("SELECT count(*) FROM side").fetchone()[0]


class Fault(Exception):
    pass


def execute(ledger, step_id, payload, fault=None):
    ledger.intend(step_id, payload)
    if ledger.state(step_id) == "committed":
        return "replayed"                                  # idempotent no-op
    if fault == "kill_before_commit":
        raise Fault(step_id)
    first = ledger.commit(step_id, key({"step": step_id, "payload": payload}))
    if fault == "kill_after_commit":
        raise Fault(step_id)                               # caller never learns it worked
    return "applied" if first else "deduped"


def trial(fault):
    ledger = Ledger()
    payload = {"cmd": "apply-patch", "rev": 7}
    try:
        execute(ledger, "s1", payload, fault)
    except Fault:
        pass                                               # substrate is gone
    replay = execute(ledger, "s1", payload)                # new worker, same ledger
    return replay, ledger.effects(), ledger.state("s1")


for fault in [None, "kill_before_commit", "kill_after_commit", "duplicate_delivery"]:
    replay, side, state = trial(fault)
    assert side == 1, f"{fault}: {side} side effects, invariant violated"
    print(f"{str(fault):20} replay={replay:9} effects={side} state={state}")
print("all trials: exactly one observable effect per step")

The four steps I would walk a reviewer through, in this order:

Pin identity before anything else. intend() refuses a retry whose payload hash moved. That is the payload-drift domain closed with one assertion, not with a convention.
Commit the intent transactionally with the effect key. commit() inserts into side and effect in one transaction, so "effect applied but unrecorded" cannot be observed from the ledger.
Inject the kill on both sides of the commit. kill_before_commit must yield zero effects until replay; kill_after_commit must yield exactly one, with the second attempt classified as replayed.
Inject duplicate delivery explicitly. Run the same step twice and assert deduped, because at-least-once delivery is what you actually get — and the harness should say so rather than hope.

Expected output, which is the whole argument in four lines:

None                 replay=replayed  effects=1 state=committed
kill_before_commit   replay=applied   effects=1 state=committed
kill_after_commit    replay=replayed  effects=1 state=committed
duplicate_delivery   replay=replayed  effects=1 state=committed
all trials: exactly one observable effect per step

4. Failure analysis: what each domain costs when you skip it

Skipped control	Observable symptom	Blast radius	Cheap fix
Intent hash	stale revision re-applied after a prompt edit	silent wrong state, no error	reject on hash mismatch
Effect key	duplicate PR comments, double charges, doubled file edits	user-visible, hard to unwind	effect key = `(step_id, rev)`
Lease fencing	two workers own the same step after a long GC pause	concurrent conflicting writes	fencing token, monotonic counter
Open-intent reaper	a step that committed but never acked is retried forever	unbounded retries, budget burn	replay returns `replayed`, not a new effect

The third row is the one that surprises people, because it looks like a clock problem and is not. Any timestamp-based lease on a substrate whose clock you do not control will eventually hand the same step to two workers. Fencing tokens cost you an integer and remove the domain entirely.

5. Tradeoffs I would sign off on

Property	Without a ledger	With intent + effect ledger
Latency per step	one write	two writes + one hash
Duplicate effects after revocation	unbounded	zero, by construction
Recovery complexity	manual reconciliation	replay with the same `step_id`
Cost of a borrowed substrate	hidden in incident time	visible as ledger writes
Failure to reason about	unknown	classified into four domains

The honest accounting: you pay roughly one extra durable write per step to make replay boring. On a borrowed substrate, boring is the feature. What you should not do is put the ledger on the same free server, because then A1 takes the record with the effect and no amount of idempotency saves you.

6. What I would change next

If this were my review, three things would move before anything else shipped.

Split the data flow explicitly. Planner and ledger on durable infrastructure; only the step executor and its model calls are allowed onto the free server. Draw that boundary in the diagram, not in a wiki sentence.
Add a chaos gate to CI. Run the harness with each fault class and fail the build on any trial where effects != 1. It takes seconds and it is the cheapest regression test you will ever own.
Fence leases with tokens, never timestamps. Then delete any code that compares a remote wall clock to a local deadline, because that comparison is the seed of every double-owner bug.

7. Limitations, and who should not use this

This design assumes you can place a durable ledger somewhere outside the borrowed substrate; if you cannot, stop and fix that first, because the rest is decoration. It also assumes effects are keyable — file writes and API calls with an idempotency key yes, an irreversible external action with no key no, and for those you want a human gate rather than a clever retry. And if your steps are short, synchronous, and run on infrastructure you own and size, the ledger is probably overhead you should skip.

I would also not use this to argue that a free server option is production infrastructure. It is a substrate for work whose record lives elsewhere, and the moment you forget that sentence you are back to the duplicate deploy in the first paragraph.

If you want a substrate to run the harness against rather than your own laptop, MonkeyCode's free model access plus its free server option are a reasonable place to start — just re-check the currently stated allowance and the server terms yourself before you write them into a design doc.

So, the question I would put to you in review: which event order breaks your invariant first — a revocation between commit and acknowledgement, a replayed step after a payload edit, or two workers fenced by the same expired timestamp? And for the one that breaks it, does your system reject the attempt, replay it, or compensate after the fact?

Your Coding Agent's Workspace Is a Cache, Not a Disk: Make Artifact Durability an Invariant

Robin — Sat, 07 Nov 2026 17:49:28 +0000

A refactor step ran for forty minutes on a throwaway workspace and produced twelve files. Then the machine went away, the workspace went with it, and the only thing I had left was a diff pasted into a chat log — which, of course, no longer applied cleanly to the tree it came from. I spent the next hour reconstructing work that had already been done correctly, and none of my logs told me which writes had actually survived. Have you ever tried to answer "was that file committed or just written?" from a transcript? You cannot, because "written" and "committed" were never distinguished in the first place.

The week's DEV conversations kept circling whether AI writes our code and whether AI testing drives real organizational change. Both are interesting, but they skip a duller question that decides whether any of it counts: does the artifact outlive the machine that produced it? In this review I treat the agent workspace as an L1 cache with an unspecified eviction policy, and I make durability an invariant that a test can falsify.

What I am assuming, so you can reject the conclusion cheaply

Declared assumptions matter more than the diagram, because the diagram is only true inside them.

The remote workspace has no published lifetime guarantee and can be reclaimed mid-step, without a shutdown hook.
Reclamation is not a rare fault; on free compute it is the normal end of a workspace, so the protocol must be correct per step, not per session.
The agent step is the unit of replay: re-running a step is allowed, re-running half a step is not.
Artifacts are small enough to hash and upload (source files, patches, test reports), not multi-gigabyte datasets.
Some durable store exists outside the workspace — object storage, a database, a git remote — with atomic replacement or compare-and-swap.

If assumption 1 is false for you, because you rent a persistent volume with an SLA, most of this article is unnecessary. If assumption 1 is true and you have not written down an invariant, you are running on luck.

The invariant, stated so a test can actually fail it

Here is the property I want to hold for every step s and every artifact a that s declares:

Durability invariant. If a manifest for s is visible to a reader, then for every declared artifact a, the content referenced by that manifest is retrievable. Otherwise no manifest for s is visible, and s must be replayed.

In predicate form: committed(s) => forall a in declared(s): retrievable(hash(a)), and not committed(s) => replay(s). The second clause is the part people forget. A protocol that can leave a dangling manifest is worse than one that loses everything, because a dangling manifest lies to every downstream consumer, including your evaluation harness and your cost accounting.

The critical consequence: the content must be durable before the pointer to it is durable. Publish the manifest first and you have built a directory of broken promises.

Data flow: five edges, five ways to lose a file

agent step
   |
   (1) local write        workspace/fs      <- dies with the box
   |
   (2) hash + PUT blob    durable store     <- survives
   |
   (3) CAS manifest       durable store     <- survives, and it is a promise
   |
   (4) ack to orchestrator
   |
   (5) replay on unknown  (same step_id)    <- must be idempotent

Edges (1) and (2) are safe to lose: a blob with no manifest is garbage, and garbage is collectable. Edge (3) is the commit point of the whole system, and everything before it must be replayable. Edge (5) is where teams quietly cheat — they re-run with a fresh step_id, get a second manifest, and now two "authoritative" artifact sets exist for one logical step.

A fixture that kills the sandbox at every edge

Rather than argue about this, I wrote a small simulator. It is deliberately boring: a content-addressed blob directory, a JSON manifest with compare-and-swap on a revision number, and a workspace whose death I can schedule at an exact operation index. Percent-level reliability claims are meaningless here; I want every interleaving, not a sample.

Step 1 — content store with atomic replacement.

import hashlib, json, os

class ContentStore:
    def __init__(self, root):
        self.root = root
        os.makedirs(f"{root}/blobs", exist_ok=True)
        os.makedirs(f"{root}/manifests", exist_ok=True)

    def put(self, data: bytes) -> str:
        h = hashlib.sha256(data).hexdigest()
        path = f"{self.root}/blobs/{h}"
        tmp = path + ".tmp"
        with open(tmp, "wb") as f:
            f.write(data); f.flush(); os.fsync(f.fileno())
        os.replace(tmp, path)          # atomic within one filesystem
        return h

    def exists(self, h: str) -> bool:
        return os.path.exists(f"{self.root}/blobs/{h}")

    def manifest(self, step_id: str):
        p = f"{self.root}/manifests/{step_id}.json"
        return json.load(open(p)) if os.path.exists(p) else None

    def cas_manifest(self, step_id, expected_rev, body):
        cur = self.manifest(step_id)
        cur_rev = -1 if cur is None else cur["rev"]
        if cur_rev != expected_rev:
            raise RuntimeError(f"CAS conflict on {step_id}: want {expected_rev}, have {cur_rev}")
        body = dict(body, rev=cur_rev + 1)
        p = f"{self.root}/manifests/{step_id}.json"
        with open(p + ".tmp", "w") as f:
            json.dump(body, f); f.flush(); os.fsync(f.fileno())
        os.replace(p + ".tmp", p)
        return body

The single-writer expected_rev check is doing the real work here. In production this is a conditional write on object storage or a WHERE rev = ? update, and if your store cannot express it, stop reading and fix that first.

Step 2 — a workspace that can die on demand.

class SandboxDeath(Exception):
    pass

class Sandbox:
    def __init__(self, root, die_at=None):
        self.root, self.die_at, self.ops = root, die_at, 0

    def _tick(self, label):
        self.ops += 1
        if self.die_at is not None and self.ops >= self.die_at:
            raise SandboxDeath(label)
        return label

    def write(self, rel, data: bytes):
        self._tick(f"write:{rel}")
        path = os.path.join(self.root, rel)
        os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
        with open(path, "wb") as f:
            f.write(data)
        return rel

Step 3 — the protocol, in the order I claim is correct.

def run_step(store, sandbox, step_id, artifacts):
    declared = []
    for rel, data in artifacts:
        sandbox.write(rel, data)                       # (1) local, volatile
        declared.append(rel)

    hashes = {}
    for rel in declared:
        body = open(os.path.join(sandbox.root, rel), "rb").read()
        hashes[rel] = store.put(body)                  # (2) durable content

    return store.cas_manifest(step_id, -1, {           # (3) durable pointer
        "step_id": step_id, "state": "committed", "artifacts": hashes,
    })

Step 4 — the checker, which is where the invariant becomes executable.

def classify(store, step_id):
    m = store.manifest(step_id)
    if m is None:
        return "no-commit"                 # acceptable: replay required
    for rel, h in m["artifacts"].items():
        if not store.exists(h):
            return f"VIOLATION dangling {rel} -> {h[:12]}"
    return "committed"

Step 5 — kill the workspace at every operation index and classify.

def chaos(artifacts, operations):
    outcomes = {}
    for die_at in range(1, operations + 1):
        store = ContentStore(f"/tmp/mc/store-{die_at}")
        sandbox = Sandbox(f"/tmp/mc/sb-{die_at}", die_at=die_at)
        try:
            run_step(store, sandbox, "step-42", artifacts)
            result = "completed"
        except SandboxDeath as e:
            result = f"died:{e}"
        outcomes[die_at] = (result, classify(store, "step-42"))
    return outcomes

Run it with python3 -m chaos_manifest after dropping the three classes in one module. My local run over a five-artifact step produced a single completed outcome at the final operation index, a spread of died:* outcomes before that, and no-commit or committed for every one of them — never VIOLATION.

Then flip the order and watch the property fail

This is the part I actually care about, because a test that only ever passes proves nothing about the property you claim to hold. Swap edges (2) and (3) so the manifest is published before the blobs are uploaded:

def run_step_wrong(store, sandbox, step_id, artifacts):
    declared = [sandbox.write(rel, data) for rel, data in artifacts]
    hashes = {rel: hashlib.sha256(open(os.path.join(sandbox.root, rel), "rb").read()).hexdigest()
              for rel in declared}
    store.cas_manifest(step_id, -1, {"step_id": step_id, "state": "committed", "artifacts": hashes})
    for rel in declared:                               # too late
        store.put(open(os.path.join(sandbox.root, rel), "rb").read())

Now a death between the CAS and the first upload leaves a manifest that points at a blob nobody ever wrote. classify returns VIOLATION on that index, and it will return it on most indices after it. If your durable store has eventual visibility on the manifest side, you can reproduce the same bug even with the correct call order.

Failure domains and what each one costs you

Failure	Detected by	Symptom if unhandled	Mitigation
Workspace reclaimed before (2)	client observes disconnect	work lost, cost lost	replay step with same `step_id`
Death between (2) and (3)	nothing — silent	orphan blob, wasted bytes	blob GC keyed on manifest absence
Death between (3) and (4)	client sees no ack	duplicate replay attempt	CAS on `expected_rev` rejects it
Partial local file write	hash mismatch vs declared length	truncated artifact published	hash at read time, never at write time
Concurrent replay, two attempts	CAS conflict	two artifact sets	single writer per `step_id`
Store outage mid-step	upload error	half-committed step	no manifest until all blobs land

The row that costs the most in practice is the second one, because nothing reports it. You do not lose correctness, you lose money and storage quietly, and nobody notices until the billed byte count stops matching the number of committed steps.

Tradeoff table: when write-through is the wrong answer

Strategy	Loss window	Store ops per step	Complexity	Fits when
Per-write write-through	one write	N writes + 1 CAS	low	steps are long, workspace is fragile
Per-step batch publish	whole step	1 CAS	low	steps are short and cheap to replay
Periodic checkpoint	checkpoint interval	K + 1	medium	large workspaces, expensive steps
Push to git per step	one commit	1 push	medium	artifacts are text and reviewable
Persistent volume + SLA	none	0	low until it is not	you actually bought durability

I default to batch publish with a hard step timeout, because replay is usually cheaper than the extra round trips. I switch to per-write write-through when a single step's compute exceeds the cost of N uploads, which is common for long agent loops — and that is precisely the regime where a free, short-lived workspace is attractive in the first place.

Where the free tier actually fits in this design

MonkeyCode is an open-source coding-agent client, and the part relevant to this review is that it advertises free model access and a free server option for the workspace — the operator states a 10M-token free allowance, and I am deliberately not publishing a quota table, because quotas are the first thing to change and the second thing to be misquoted. Verify the current numbers on the project page before you plan a budget around them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why does that belong in an architecture post rather than a getting-started post? Because a free, reclaimable server makes the failure mode above the default rather than the exception, and that is genuinely useful: my chaos fixture needs a real machine to die on, not just a die_at integer. The honest workflow I would recommend is to build the manifest protocol against the simulator first, then run the same step loop against a free server and compare committed-step counts to started-step counts. If those two numbers drift, you have found a durability bug that no amount of retry logic will paper over.

Who should not do this

If your workspace is a persistent volume with a real SLA, adding a manifest layer is ceremony that buys you nothing. If your steps finish in under a minute and produce one file, replay is so cheap that per-step batch publish is already the correct answer. And if every step ends in git push to a remote you control, you have a durable pointer already — measure before you build a second one.

Limits of what I tested

My fixture proves the protocol under a single-writer, single-store model, and it cheats by assuming os.replace is atomic on the same filesystem. It says nothing about object-storage eventual consistency, network partitions between the client and the store, or blobs above a few megabytes. It also cannot tell you whether the agent produced the right files, only whether the ones it declared are retrievable — evaluation correctness is a separate invariant, and I have written about that elsewhere.

So here is the counterexample question I would put to any team running agents on borrowed compute. A step dies after the blob upload and before the manifest CAS; a reader sees no commit and the orchestrator replays the step, and the replay produces byte-different but semantically equivalent output. Should the system reject the second attempt, replay it under the same step_id and let the CAS decide, or compensate by writing a superseding revision with both hashes recorded? Which event order breaks your invariant first — and what does your ledger say when it does?

Two Clocks, One Ledger: Running an Eval Harness on Free Compute Without Losing Records

Robin — Fri, 06 Nov 2026 16:28:10 +0000

A scenario I have reviewed more than once — numbers invented here for illustration, not measured from anyone's deployment — goes like this. A team wants to know whether a new retrieval re-ranker beats the baseline across 320 held-out cases, so they run the sweep on a free server with free model access. Twenty minutes later the report says 78.4% for the candidate and 74.1% for the baseline, and everyone starts writing a changelog entry. Then someone counts the ledger: 312 records, not 320, and the eight missing ones were disproportionately candidate failures. Was the improvement real, or did the denominator quietly get healthier while nobody was looking?

That question is the whole article. Not "how do I get free compute", but how do you run an evaluation on two resources you do not own, whose lifetimes are governed by two independent clocks, and still trust the number you publish? I'll walk through the failure domains, a runnable harness that reproduces them, and the invariants I now require before I believe any free-tier eval verdict.

1. Name the two clocks before you write any code

The first clock is the lease: the free server exists for a window you do not control, and it can end without a graceful shutdown. The second clock is the model budget: free model access is granted per request or per window, and a call can be throttled, rejected, or slow regardless of what the server is doing. These clocks are unsynchronized, and the interesting failures live exactly where they disagree.

If your harness treats "server alive" as "job will finish", you have merged two failure domains into one assumption, which is the same class of mistake as treating a message broker as a database. The invariant I want is simple to state and surprisingly easy to break: every scheduled case must end in exactly one durable terminal record, or one explicit deferral — never in silence. Everything below exists to preserve that invariant across both clocks.

2. Assumptions I want declared up front

The free server's local disk and process memory are ephemeral: assume they vanish when the lease ends, with no flush and no warning.
The only durable store is the control plane — your laptop, a CI workspace, or a small volume you actually pay for.
Model calls can be throttled or rejected at any point, and a rejection is not a failure of the case under test.
Clock skew between control plane and server is nonzero, so deadlines must be computed from a monotonic source on one side only.
Re-running a case is allowed but must not create a second terminal record for the same logical unit of work.

If assumption 2 is false for you — if you have no durable control plane — stop here and fix that first, because no amount of retry logic substitutes for a place to write the outcome.

3. Where a record can die: the data flow

sequenceDiagram
    participant CP as Control plane (durable)
    participant S as Free server (lease)
    participant M as Free model access (budget)
    CP->>CP: append intent(case, attempt, key)
    CP->>S: dispatch case + deadline
    S->>M: call with remaining budget
    M-->>S: response or throttle
    S->>CP: commit terminal(key, output, passed)
    Note over S: lease may end after response, before commit
    CP->>CP: summary = terminal records only

Read the note in the middle carefully, because that is the counterexample the rest of this design defends against. A response that was computed but never committed is worse than a call that never happened: it burns budget, produces no evidence, and makes the surviving population non-random. Do you see why the summary arrow points at the control plane and not at the server? The summarizer must never need to ask a leased machine what happened.

4. Step 1 — Write the intent before you call anything

The harness below is a model, not a benchmark: it calls no real model, and its latencies are synthetic. Replace model_call with your client and the structure still holds. Everything is Python 3.11 standard library.

# two_clock_harness.py — models a server lease and a model budget separately.
import argparse, hashlib, json, os, pathlib, random, time

LEDGER = pathlib.Path("ledger.jsonl")
SAFETY_MARGIN_S = 8.0     # reserved for commit + teardown, tune per workload
MODEL_BUDGET_S = 25.0     # your assumption, not a product claim

def append(record: dict) -> None:
    line = json.dumps(record, sort_keys=True) + "\n"
    with LEDGER.open("a", encoding="utf-8") as fh:
        fh.write(line); fh.flush(); os.fsync(fh.fileno())

def case_key(case_id: str) -> str:
    return hashlib.sha256(case_id.encode()).hexdigest()[:16]

def read_ledger() -> list[dict]:
    if not LEDGER.exists():
        return []
    return [json.loads(l) for l in LEDGER.read_text().splitlines() if l.strip()]

def committed() -> dict[str, dict]:
    return {r["key"]: r for r in read_ledger() if r["state"] == "terminal"}

def model_call(case: dict, budget_s: float) -> dict:
    latency_ms = random.uniform(50.0, budget_s * 800.0)
    if latency_ms > budget_s * 1000.0:
        raise TimeoutError("model budget exhausted")
    time.sleep(0.001)  # scaled down so the demo finishes quickly
    out = case["prompt"][::-1]
    return {"output": out, "passed": len(out) % 3 != 0, "latency_ms": round(latency_ms, 1)}

The intent record is the write-ahead half of the protocol: it is durable, it carries the idempotency key, and it says "I am about to spend money on this case". Appending it first is what makes a crash visible as intent_without_commit instead of invisible as a missing row.

5. Step 2 — Derive the deadline, never inherit it

def run_case(case: dict, lease_deadline: float, inject: str | None) -> str:
    key = case_key(case["id"])
    if lease_deadline - time.monotonic() <= SAFETY_MARGIN_S:
        append({"key": key, "case": case["id"], "state": "deferred"})
        return "deferred"
    append({"key": key, "case": case["id"], "state": "intent", "attempt": 1})
    if inject == "kill_before_model":
        raise SystemExit("simulated lease expiry before model call")
    result = model_call(case, MODEL_BUDGET_S)
    if inject == "kill_after_model":
        raise SystemExit("simulated lease expiry after model response")
    append({"key": key, "case": case["id"], "state": "terminal", **result})
    return "ok"

The refusal branch matters as much as the commit branch. A case that cannot finish inside lease_deadline - SAFETY_MARGIN_S should be deferred by name, not started optimistically and abandoned, because a named deferral keeps the denominator honest while an abandonment corrupts it. What safety margin is right for your workload? Measure your p99 commit time and multiply generously; the margin is cheap and the bias is not.

6. Step 3 — Make resume a replay of uncommitted intents

def resume(cases: list[dict], lease_deadline: float, inject: str | None) -> None:
    done = committed()
    pending = [c for c in cases if case_key(c["id"]) not in done]
    for case in pending:
        run_case(case, lease_deadline, inject)

Resume keys off the durable ledger, so it is safe to run on a brand-new server with nothing in common with the one that died. The replay is idempotent at the ledger level because terminal records are keyed by content hash, but it is only cheap if your case execution is side-effect free — a re-ranker call is, an agent that writes files is not. Which of your eval cases mutate the world, and did you notice that before you made replay your recovery strategy?

7. Step 4 — Inject the three kills and read the denominator

rm -f ledger.jsonl
python two_clock_harness.py --cases 40 --lease-s 30 --inject none
python two_clock_harness.py --cases 40 --lease-s 30 --inject kill_after_model --resume
python two_clock_harness.py --cases 40 --lease-s 30 --inject kill_before_model --resume

Always report the denominator explicitly, and never let the scheduled count stand in for it:

scheduled = len(intents | deferred)
pass_rate = passed / len(terminal) if terminal else None
# report {scheduled, terminal, deferred, intent_without_commit, pass_rate}

The properties I actually assert in CI, as plain assert statements over the ledger file:

Totality: terminal | deferred == scheduled_intents after resume, with no key in neither set.
Uniqueness: no key appears in more than one terminal record.
Monotonic commit: a terminal record for key k is never followed by an intent for k.
Denominator stability: pass rate computed from terminal records drifts by less than 0.5pp between an uninterrupted run and a killed-then-resumed run of the same cases.
Deferral visibility: every lease-driven refusal produces a deferred record naming the case, never a retry loop.

Property 4 is the acceptance rule, and it is the one people skip. If your killed-and-resumed pass rate differs from the clean run by more than your tolerance, you have measured the failure of your harness, not the quality of your candidate. Run the injection twenty times before you trust a verdict — this is precisely the kind of repetition that free compute makes affordable.

8. Tradeoffs, stated as tradeoffs

Choice	Gain	Cost	Reject when
Large `SAFETY_MARGIN_S`	Fewer half-committed cases	Lower lease utilisation	Lease windows are shorter than a single call
Intent-before-call	Crashes stay visible	One extra durable write per case	Write path dominates runtime
Resume over rerun-all	Preserves budget	Replay must be idempotent	Cases mutate external state
Terminal-only denominator	No silent bias	Smaller n, wider intervals	You need every scheduled case counted
Deferral instead of retry	Bounded, honest accounting	Manual re-dispatch	You own the server and can just wait

No row in that table is universally right, and I would push back on anyone who presents one as a rule. The table exists so that the choice is written down next to the evidence it trades against.

9. Where free model access and a free server genuinely fit

This is the workflow where I have found the combination useful rather than decorative. MonkeyCode offers free model access and a free server option, which is exactly the shape of resource this harness is designed for: short-lived compute, budgeted inference, and a control plane you keep. Because the harness never trusts the server, an unexpected lease end costs one resume invocation instead of a corrupted report — and free access makes the twenty-run failure-injection loop above a default rather than a luxury.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest caveats. Free allowances and machine specifications change, so read the current terms before you size a sweep rather than trusting any number you saw in a post — including this one, which deliberately quotes none. And confirm the project's licence and data-handling terms yourself before you send it evaluation data you would not want retained.

10. Who should not use this approach

Skip it if you need a durable server-local cache, if your cases take longer than a single lease window and cannot be checkpointed, if you require an availability commitment you can put in a contract, or if your evaluation data is regulated and the provider's retention terms are not something you can accept. This is a design for cheap, repeatable measurement under uncertainty; it is not a production control plane and it will not become one by adding retries.

11. What I would change next, and the question I'd ask you

The next change I would make is not more retries but a pre-flight feasibility probe: one throwaway call that measures p99 latency, multiplied by the batch size, compared against the lease window before any case is dispatched. That converts a whole class of late-run deferrals into an up-front refusal to start, which is far easier to reason about. The second change is a per-case budget split, so one pathologically slow case cannot consume the margin that protects the other thirty-nine.

So here is the counterexample question I will leave you with. Which event order breaks your invariant — a lease that ends after the response but before the commit, a throttle that arrives after you already recorded an intent, or a resume that replays a case whose side effects are not idempotent? And when it happens, should your system reject the case, replay it, or compensate for it? Write the answer down before you look at your pass rate, because after you look, you will be tempted to explain it away.

Make Resume Equivalence an Invariant Before a Free Server Restarts Your Agent Step

Robin — Thu, 05 Nov 2026 15:05:35 +0000

Make Resume Equivalence an Invariant Before a Free Server Restarts Your Agent Step

Here is the event order I use as a default counterexample whenever someone asks me to review an agent that runs on borrowed compute. Step 6 of 8 calls the model, receives a tool intent, applies a patch hunk, and flushes that hunk to the workspace disk. The host restarts before the runner records that step 6 finished, so the runner resumes from its last durable checkpoint at step 3. Steps 4 through 8 replay, step 6 lands twice, and the diff now contains the same hunk in two places. The test suite still passes, because a duplicated hunk is still a valid patch — and that is exactly why this bug survives code review.

The invariant the common implementation fails to preserve is boring to state and hard to hold: a resumed run must produce the same artifact as an uninterrupted run of the same event order. Not a similar artifact, not a passing build — the same fingerprint. If you cannot compute that fingerprint, you cannot tell a correct resume from a silent double-apply, and you will find out from a reviewer instead of a test.

1. Assumptions I declare before the review

I review free-tier agent builds against these assumptions, and I write them down because half of the disagreements I have with teams come from undeclared ones:

The compute domain is volatile. Assume the box can vanish at any instruction boundary, not at a convenient step boundary.
The model endpoint is shared and non-deterministic. Assume re-calling a step returns different output, so replay must not depend on re-calling it.
Artifact writes are the only externally visible effect. Anything else — a pushed commit, a webhook, a ticket comment — is a second failure domain with its own commit boundary.
Resume is a normal path, not an exception path. If you have never run it in CI, assume it does not work.

That fourth assumption is the one people resist, and I understand why: it feels like admitting the infrastructure is unreliable. But the admission you are actually making is that your recovery logic is untested, which is a different and much worse confession.

2. The state model and the two durability domains

Almost every broken resume I have reviewed has the same shape: two pieces of state, stored in two places, assumed to move together. The artifact log records what the agent produced; the receipt set records what the runner believes it already applied. If those two live in different durability domains, a crash can advance one without the other, and the gap is either a duplicate or a loss.

        model call (non-deterministic, expensive, not replayable)
                 |
                 v
  [ step intent ] --> apply() --> artifact log  (domain A: durable?)
                 |                      ^
                 +--> receipt set  -----+   (domain B: durable?)
                              ^
                       cursor / checkpoint

The data flow matters more than the diagram, so let me state it as a rule: the artifact write and the receipt write must be one commit on one domain. If they are two commits, the crash window between them is the bug — and it is not a small window, because the receipt write is usually the network call and the artifact write is usually the local one.

3. The invariant: one fingerprint, four modes

I do not want to argue about whether resume is correct, so I compute a fingerprint and compare. Below is a 60-line model of the problem using only the standard library — no network, no framework, nothing to install. It is a model of the commit boundary, not a mock of anyone's server.

# resume_equivalence.py — stdlib only, deterministic, no network
import hashlib

STEPS = [{"id": f"s{i}", "hunk": f"+line-{i}"} for i in range(8)]
CHECKPOINT_EVERY = 3


class Box:
    """One instance of the compute domain, with two durability switches."""

    def __init__(self, log_durable, receipts_durable):
        self.log_durable = log_durable
        self.receipts_durable = receipts_durable
        self.log = []          # artifact content
        self.receipts = set()  # what the runner believes it applied
        self.cursor = 0        # how many steps the runner thinks are done

    def apply(self, step):
        if step["id"] in self.receipts:   # idempotency guard
            return
        self.log.append(step["hunk"])
        self.receipts.add(step["id"])
        self.cursor += 1

    def crash(self):
        b = Box(self.log_durable, self.receipts_durable)
        b.log = list(self.log) if self.log_durable else []
        b.receipts = set(self.receipts) if self.receipts_durable else set()
        b.cursor = (self.cursor if self.receipts_durable
                    else (self.cursor // CHECKPOINT_EVERY) * CHECKPOINT_EVERY)
        return b


def run(box, start=0, crash_after=None):
    for i, step in enumerate(STEPS[start:], start=start):
        box.apply(step)
        if crash_after is not None and i + 1 == crash_after:
            return box
    return box


def fingerprint(box):
    return hashlib.sha256("\n".join(box.log).encode()).hexdigest()[:10]


REF = fingerprint(run(Box(True, True)))
print(f"reference={REF}")

for log_d in (True, False):
    for rec_d in (True, False):
        for k in range(1, len(STEPS)):
            box = run(Box(log_d, rec_d), crash_after=k).crash()
            box = run(box, start=box.cursor)
            verdict = "SAME" if fingerprint(box) == REF else "DIVERGED"
            print(f"log_durable={log_d:<5} receipts_durable={rec_d:<5} "
                  f"crash_after={k} hunks={len(box.log)} {verdict}")

Run it and the output collapses into four behaviours, which is the whole argument in one table:

Artifact log	Receipts	Result	What it looks like in review
durable	durable	resume matches reference	correct, provided both writes share one commit
durable	volatile	hunks lost after last checkpoint	tests pass on a smaller diff, nobody notices
volatile	durable	hunks applied twice	duplicated patch blocks, noisy review, double compensation
volatile	volatile	replay from zero, matches reference	correct only if nothing external was committed

The fourth row is my favourite, because it inverts the usual instinct. Full volatility can be safer than partial durability — replaying everything from scratch is deterministic, while replaying from a stale checkpoint is not. That result only holds because the model confines side effects to the box; the moment a step pushes a commit or sends a webhook, row four becomes the worst row in the table.

4. Failure injection, beyond the restart

A single crash point is not a test plan. The properties I actually want to assert, in the order I would add them to CI:

Crash at every step boundary — for k in range(1, len(STEPS)) — and require fingerprint equality.
Crash between the artifact write and the receipt write — the honest version of this test needs the two writes to be separable, which is precisely the defect you are hunting.
Duplicate delivery of the same step intent — run apply() twice with the same id and require exactly one hunk.
Stale checkpoint — resume from a cursor older than the durable log and require either correct completion or a loud refusal, never a silent double-apply.
Model re-call — assert that resume never calls the model for a step whose receipt exists, because the second answer will differ.

Property five is where most agent builds quietly cheat: they re-plan on resume instead of replaying a recorded decision. If your recovery path re-asks the model what to do, you have not resumed — you have started a second, correlated run.

5. Tradeoffs I would accept, and ones I would not

Design	Recomputation cost	Duplicate risk	Where it fails
Per-step receipt, single commit	lowest	lowest	needs an atomic write primitive you control
Checkpoint every N steps	medium	high after crash	N is a guess, and the guess is wrong under load
Full replay, no durable state	highest	none in-box	any external effect at all
Re-plan on resume	medium	semantic drift	non-deterministic model output

I will happily pay full replay on a cheap box for a short pipeline, because the failure mode is obvious and the fix is local. I will not accept checkpointing as a correctness mechanism, because a checkpoint is a performance decision wearing a durability costume.

6. What I would change next

Move the artifact out of the compute domain entirely — object storage or a git remote — so the box holds nothing you would miss.
Make the receipt the recorded decision of the step, not a boolean, so resume replays a decision instead of re-deriving one.
Write the receipt in the same commit as the hunk, or refuse to write either.
Add the fingerprint property test to CI at the same tier as the unit tests, and fail the build on divergence.
Give the runner a loud refusal path: if the checkpoint and the log disagree, stop and report, never guess forward.

7. Where the free tier actually fits

I use this pattern when I run agent steps against a free model endpoint and the project's free server option, which is where I want the failure to happen first. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The two availability claims I am relying on are operator-supplied: free model access, and a free server option — the project page also advertises a free token allowance (10M at the time of writing), and I would check that number on the page rather than budget against it here, because allowances move. What makes the environment useful for this exercise is not the price; it is that a volatile box and a shared endpoint are exactly the constraints under which resume equivalence breaks, and free access lets me crash the thing forty times in an afternoon.

Limitations, plainly: a free tier is not a durability guarantee, and nothing here should be read as an SLA. My simulator models the commit boundary and treats apply() as atomic — real writes are not, so the durable/durable cell in the matrix still needs a genuine atomic primitive behind it. And if you have hard latency SLOs, regulated data, or a requirement that state live on the box, this whole approach is the wrong shape for you; run it on infrastructure you own and keep the fingerprint test anyway.

If you want to try the loop end to end, the free model access and free server option are the cheapest place I know to break the invariant on purpose — run the script, then run it against a real pipeline and see which row of the matrix you land in.

Acceptance rule and the question I would leave you with

Accept the design only if, for every crash point in the trace, fingerprint(resume(crash(run))) equals fingerprint(run()) — or the runner refuses loudly. Anything else is a coin flip you have decided not to look at.

So: which event order breaks your invariant — a restart between the hunk write and the receipt write, a duplicate step intent, or a re-plan that quietly starts a second run? And when it happens, should the system reject the attempt, replay the recorded decision, or compensate for the effect it already committed?

Treat a Coding-Agent Step as a Revision-Fenced Command, Not a Prompt

Robin — Tue, 27 Oct 2026 12:33:04 +0000

The review usually opens with a demo that looks fine. An agent plans four edits, applies them to a workspace on a remote box, and reports success; the tests are green and everyone relaxes. Two hours later a human's refactor is gone, and nobody can point at the request that removed it. Is that a model problem, a server problem, or a protocol problem?

I think it is a protocol problem almost every time, and I want to make the case with an executable fixture rather than an architecture diagram. Below is the event order I keep reconstructing in reviews, the invariant it violates, and a small simulator that will fail your design for you. Everything here is a model of a system, not a benchmark of a vendor.

1. The event order that silently wins

A planner reads the workspace at revision 0 and emits three steps against app.py. Step 1 applies and the workspace moves to revision 1. While step 2 is in flight, either a human edits that file or the session is re-routed to a different model that re-plans the remainder; both are ordinary events, not incidents.

Step 2 still carries the content hash it was planned against. If the executor applies patches by path without checking that hash or the revision, the write lands on content the planner never saw, and the old bytes vanish. Nobody gets an error, because a patch is not a request — it is a claim about a state that may already be false.

The failure is invisible precisely when it is most expensive. The agent's summary says "updated handler and tests", the test run passes because the overwritten hunk was unrelated, and the loss only surfaces at the next merge. Who is supposed to notice that?

2. The invariant, stated once

No step may be applied unless the state it was planned against is still the state it is applied to, or the step has been explicitly rebased onto the new state.

That is it. Note what the invariant does not say: it does not say a step is correct, and it does not say a retried step is safe. A revision fence prevents stale acceptance; idempotency prevents double application. Teams routinely build one, assume they have the other, and then argue about whose bug it is.

3. Declared assumptions

One writer per file at a time; concurrent multi-writer edits are out of scope and need a merge protocol, not a fence.
The workspace has a monotonic revision counter and a per-file content hash that are both persisted with the workspace itself.
A step is planned from a readable snapshot and carries (step_id, base_rev, path, pre_hash, patch).
Execution may be preempted at any point between "patch written" and "client received the result".
Model identity may change inside a session, because routing, fallback, or a free-tier allowance running out can move you to another model without warning.

4. Data flow and failure domains

There are three state holders and they fail independently: the planner's context (lossy, model-dependent), the executor's workspace (authoritative, preemptible), and the client's view of progress (optimistic, retried). The only durable truth is the workspace plus whatever journal you persist alongside it.

Failure domains worth separating in review: stale plan (planner state diverged), lost ack (executor applied, client did not learn), duplicate delivery (client retried an applied step), and re-plan drift (a new model emits steps against an old base). They need different mechanisms, and a single retry counter hides all four.

5. The fixture

This is deterministic Python with no model calls, so the verdicts below are properties of the protocol, not of any provider.

# fence_sim.py - deterministic, no LLM involved
from dataclasses import dataclass

@dataclass(frozen=True)
class Step:
    step_id: str
    base_rev: int
    path: str
    pre_hash: str
    patch: str

class Workspace:
    def __init__(self):
        self.rev = 0
        self.files = {"app.py": "v0"}
        self.journal = set()          # persisted with the workspace
    def hash(self, path):
        return self.files[path]
    def write(self, path, content):
        self.files[path] = content
        self.rev += 1

class Executor:
    """fence: 'none' | 'plan' | 'step'; dedupe: journal-backed idempotency"""
    def __init__(self, ws, fence="step", dedupe=True):
        self.ws, self.fence, self.dedupe = ws, fence, dedupe
        self.plan_rev = None
    def plan(self, rev):
        self.plan_rev = rev
    def stamp(self, s):
        if self.fence == "step":
            return Step(s.step_id, self.ws.rev, s.path, self.ws.hash(s.path), s.patch)
        if self.fence == "plan":
            return Step(s.step_id, self.plan_rev, s.path, s.pre_hash, s.patch)
        return s
    def exec(self, s):
        s = self.stamp(s)
        key = (s.step_id, s.base_rev)
        if self.dedupe and key in self.ws.journal:
            return "deduped"
        if self.fence != "none":
            if s.base_rev != self.ws.rev:
                return "rejected:stale_rev"
            if self.ws.hash(s.path) != s.pre_hash:
                return "rejected:pre_hash"
        self.ws.write(s.path, s.patch)
        if self.dedupe:
            self.ws.journal.add(key)
        return "applied"

Two lines deserve attention. The journal lives in Workspace, not in the executor, because an in-process dedupe set dies with the server that was preempted — which is exactly the trap. And stamp is where fence granularity is decided, so you can argue about it in one place.

6. Injecting the three failures

Run the same three scenarios across fence modes and record the verdicts. The interference classes are injected explicitly, so the denominator is the number of injections, not the number of suspicious log lines.

def scenario_concurrent_edit(fence):
    ws = Workspace(); ex = Executor(ws, fence=fence); ex.plan(ws.rev)
    s1 = Step("s1", 0, "app.py", "v0", "v1")
    s2 = Step("s2", 1, "app.py", "v1", "v2")
    assert ex.exec(s1) == "applied"
    ws.write("app.py", "human-refactor")     # interference: rev 2
    return ex.exec(s2)

def scenario_preemption_retry(fence):
    ws = Workspace(); ex = Executor(ws, fence=fence)
    s1 = Step("s1", 0, "app.py", "v0", "v1")
    ex.exec(s1)
    # ack lost; client replays s1 against a fresh executor, same persisted workspace
    return Executor(ws, fence=fence).exec(s1)

def scenario_replan_drift(fence):
    ws = Workspace(); ex = Executor(ws, fence=fence); ex.plan(ws.rev)
    ex.exec(Step("s1", 0, "app.py", "v0", "v1"))
    ex.plan(ws.rev)                          # new model re-plans from rev 1
    return ex.exec(Step("s2", 0, "app.py", "v0", "v1-tail"))

Expected output, because the simulation is deterministic and calls no model:

fence=none  concurrent_edit -> applied        preemption_retry -> applied twice   replan_drift -> applied
fence=plan  concurrent_edit -> rejected:stale_rev  preemption_retry -> rejected:stale_rev  replan_drift -> rejected:stale_rev
fence=step  concurrent_edit -> rejected:stale_rev  preemption_retry -> rejected:stale_rev  replan_drift -> applied
dedupe on   concurrent_edit -> rejected:stale_rev  preemption_retry -> deduped         replan_drift -> applied

Read the second and third rows together, because that is the argument. A plan-level fence turns every retry into a spurious rejection and every legitimate re-plan into a dead end; it is loud, but it is loud in the wrong places. A step-level fence plus a persisted journal gets you the two properties you actually want, and it costs you a durable set of (step_id, base_rev) keys.

7. Making it a property, not a demo

Three scenarios are an anecdote. Turn them into invariants and let a random interleaving search hunt for violations — this is the part I would block a design review on.

def invariants_hold(ws, results):
    # P1: every applied step saw base_rev == rev at apply time
    assert all(r.rev_at_apply == r.base_rev for r in results if r.status == "applied")
    # P2: journal is append-only per (step_id, base_rev)
    assert ws.journal == set(ws.journal)
    # P3: no path content was replaced by a step older than the current rev
    assert all(r.base_rev <= r.rev_at_apply for r in results)

Then fuzz: shuffle {human_edit, preempt, retry, replan, duplicate_delivery} over a fixed step list, run 5,000 seeds, and assert P1 and P2 for every seed with fences on. Run the same search with fence="none" to extract a minimal counterexample, and put that event order in the PR description as the regression case. A protocol claim without a counterexample search is a preference.

8. Where the free tier changes the arithmetic

If your executor is a shared, preemptible environment and your planner can move between models at any time, then re-plan drift and lost acks are not edge cases — they are the steady state. That is the actual architectural consequence of running agents on free resources, and it argues for cheap, persisted fences rather than expensive coordination.

I use MonkeyCode as the concrete instance of that environment in this review: it is an open-source coding agent whose operator supplies free model access (currently advertised as a 10M-token allowance) and a free server option. Treat those numbers as operator-supplied and re-verify them on the project page before you design around them, because allowances and routing change faster than any architecture does. My interest here is narrow: a free server makes preemption ordinary, and free model access makes mid-session re-routing ordinary, so both of my injected interferences become Tuesday.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

9. Tradeoffs I would write down

Mechanism	Buys you	Costs you	Fails when
No fence	Throughput, simplicity	Silent lost updates	Any concurrent edit
Plan-level fence	Loud, cheap rejection	Re-planning whole plans, spurious retries	Model swap or long plans
Step-level fence	Precise staleness detection	A re-stamp on every dispatch	Semantic overlap between steps
Persisted journal	Idempotent retry across preemption	Durable key storage, growth control	Journal not persisted atomically with the write
Checkpoint per step	Fast compensation	Storage, restore latency	Large binaries, long test suites

10. Verdicts: reject, replay, or compensate

Event order	Correct verdict	Why
Step arrives, `base_rev` < current rev, no overlap	Reject, re-plan from current rev	The plan is stale; the patch is not recoverable
Step arrives, journal contains `(step_id, base_rev)`	Replay the stored result	The work is done; the ack is what was lost
Step applied, tests for touched files fail	Compensate to the checkpoint	Fence passed, semantics did not
New model re-plans mid-session from current rev	Accept if re-stamped and preconditions hold	Drift is expected, staleness is not
Two steps with overlapping hunks in one plan	Reject the plan at admission	A fence cannot order siblings

11. What I would change next

Persist the revision counter, file hashes, and journal in one atomic write, so a crash cannot leave a patch without its key.
Stamp base_rev at dispatch, never at plan time, and log both values — the delta is your drift signal.
Bound the journal by session and drop it only after a checkpoint the workspace itself can prove.
Run the fuzz job in CI with fences on, and store the minimal counterexample from fence="none" as a named regression seed.
Add a sibling-overlap check at plan admission, because per-step fences order nothing within a plan.

12. Limits, and who should skip this

The fixture models file-level hashing and a single writer; it says nothing about merge semantics, and it does not evaluate whether a patch was any good. If your model swap also changes plan quality, this protocol will happily accept a worse plan that happens to be revision-correct. That is a separate evaluation problem, and no fence will solve it for you.

Skip this if your agent is single-turn and single-file with a human watching every apply, if your platform already gives each step an atomic worktree or VM snapshot, or if you cannot persist a journal durably — half of this mechanism without the journal is worse than none, because it rejects retries without making them safe.

So here is the question I would leave in the review thread: which event order in your system breaks the invariant — a human edit racing step two, an ack lost after the write, or a re-plan against a stale base — and for that specific order, should the executor reject, replay, or compensate?

Make Batch Composition an Invariant Before Free-Tier Token Savings Rewrite Your Eval Verdicts

Robin — Mon, 26 Oct 2026 11:14:46 +0000

Opening violation (numbers below are illustrative; the failure mode is not)

Last Saturday I re-ran a 400-item evaluation sweep on a free-tier inference endpoint and the pass rate moved from roughly 71% to roughly 62%. Nothing in the model name had changed, the prompts were committed in git, and temperature was pinned to zero. What had changed was my own optimization: the earlier run sent one item per request, and the new run packed sixteen items into each call to stretch a limited free token allowance.

So which one was the experiment, and which one was the transport? My honest answer at the time was that I had no idea, and that is a bad place to be when a product decision rides on the delta. This article is the architecture review I did afterwards, and the artifact is a small probe you can run in an afternoon.

1. State the claim before you measure it

Here is what I am actually claiming, in falsifiable form. Composition invariance: for a fixed item set, fixed prompt template, fixed model, and fixed decoding settings, the extracted verdict for an item must not depend on which other items share its request.

If that property fails, then batch size is an independent variable in your eval, and every cross-run comparison silently compares two experiments. Let me be explicit about assumptions, because this argument collapses without them:

A1. The provider does not guarantee per-item isolation inside a batched prompt. (Most chat-style endpoints do not; you are sending one concatenated user message.)
A2. Retries re-send a modified chunk, because the failed chunk is not the original chunk.
A3. Output is truncated at a shared max_tokens budget, so a verbose neighbour steals tokens from a terse item.
A4. temperature=0 reduces variance but does not make a hosted endpoint deterministic across versions.

A1 through A3 are enough to break the invariant on their own. A4 tells you why a passing probe is evidence, not proof.

2. Data flow: where composition leaks in

 golden.jsonl
      |
      v
 [ shard by batch_size ] --(item order, neighbours)--> prompt builder
                                                             |
                                                             v
                                              single user message, N items
                                                             |
                             +-------------------------------+----------------+
                             |                                                |
                        429 / 5xx                                       200 OK
                             |                                                |
                             v                                                v
                 retry with N-1 items                        shared max_tokens budget
                 (different neighbours!)                                  |
                             |                                                v
                             +----------------------------------> line-oriented parse
                                                                              |
                                                                              v
                                                              UNPARSED / label drift

The interesting edge is the retry loop, not the happy path. A retry does not merely repeat work; it changes the experimental condition for every item that survives into the retried chunk. That is the counterexample I want you to hold in your head: item 7 passes at batch size 16, fails at batch size 16 with a retry, and passes again at batch size 1.

3. The minimal probe

Five numbered steps, one file. No framework, because the point is to make the failure visible, not to build a platform.

Step 1 — Freeze a golden set you are willing to re-run

Forty to a hundred items is enough to catch composition sensitivity; it is not enough to estimate your true pass rate. Each line is an id, a prompt, and an expected label.

{"id": "c001", "prompt": "Does this retry policy terminate? ...", "expected": "pass"}
{"id": "c002", "prompt": "Is this ledger update idempotent? ...", "expected": "fail"}

Step 2 — Build the batched request explicitly

# batch_probe.py - is your verdict a function of the item, or of its neighbours?
import json, time, urllib.request

SYSTEM = 'Answer with exactly one label per line: [id] pass or [id] fail. No prose.'

def call_batch(items, *, endpoint, key, model, temperature=0, max_tokens=1024):
    user = '\n'.join('[{}] {}'.format(it['id'], it['prompt']) for it in items)
    payload = {'model': model, 'temperature': temperature,
               'max_tokens': max_tokens,
               'messages': [{'role': 'system', 'content': SYSTEM},
                            {'role': 'user', 'content': user}]}
    req = urllib.request.Request(
        endpoint, data=json.dumps(payload).encode(),
        headers={'Content-Type': 'application/json',
                 'Authorization': 'Bearer ' + key})
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=120) as r:
        body = json.loads(r.read())
    return body, time.time() - t0

Step 3 — Parse strictly, and let unparsable count as a distinct outcome

def parse_labels(body, expected_ids):
    text = body['choices'][0]['message']['content']
    out = {}
    for line in text.splitlines():
        line = line.strip()
        if not line.startswith('['):
            continue
        head, _, tail = line.partition(']')
        iid = head.strip('[').strip()
        if iid in expected_ids:
            out[iid] = tail.strip().lower() or 'EMPTY'
    return out

Do not be generous here. A parser that repairs truncated lines hides exactly the failure you are hunting, and UNPARSED is a real measurement, not noise to be cleaned away.

Step 4 — Sweep batch size and diff per item

def run_sweep(items, batch_size, *, endpoint, key, model):
    verdicts = {}
    for start in range(0, len(items), batch_size):
        chunk = items[start:start + batch_size]
        body, _ = call_batch(chunk, endpoint=endpoint, key=key, model=model)
        labels = parse_labels(body, {it['id'] for it in chunk})
        for it in chunk:
            verdicts[it['id']] = labels.get(it['id'], 'UNPARSED')
    return verdicts

def flips(runs):
    ids = sorted({iid for r in runs.values() for iid in r})
    return [(iid, {bs: r.get(iid, 'MISSING') for bs, r in runs.items()})
            for iid in ids
            if len({r.get(iid, 'MISSING') for r in runs.values()}) > 1]

Step 5 — Run it, and publish the denominator

python batch_probe.py --items golden.jsonl --batch 1,4,16 --model "$MODEL" \
  --endpoint "$BASE_URL" --out runs/$(date +%F)
# report: items=60 batch_sizes=[1,4,16] flips=7 flip_rate=0.117 calls=85

Seven flips out of sixty items is not a rounding error; it means your previous comparison was measuring packing efficiency. The flip_rate denominator is the golden set size, and it belongs in the same paragraph as your pass rate.

4. Failure injection matrix

Run these deliberately, one at a time, against a small slice. If a row produces no observable change, that is a finding worth recording too.

Injected fault	Mechanism	What should break	Signal to watch
Mid-batch `429`	Retry shrinks the chunk	Item neighbours change	Verdict flip on surviving items
Shared `max_tokens` exhausted	Last items truncated	Parse failure, not wrong label	`UNPARSED` count rises with batch size
Duplicate id echo	Model repeats an id	Last-write-wins overwrite	Two labels for one id in a single response
Reordered output	Model groups by label	Position-based parsing	Parser assigns labels to wrong ids
Runner restart mid-sweep	Chunk boundary lost	Mixing of two batch sizes in one report	Call count inconsistent with item count
Allowance exhausted mid-sweep	Partial run	Cherry-picked prefix of the corpus	Missing tail ids, unreported

That last row is the one that embarrasses people in review. A sweep that dies at 60% and reports the 60% is not a smaller experiment, it is a biased one, because your golden set is rarely randomly ordered.

5. Tradeoff table (template; fill the numbers from your own run)

Batch size	Tokens per item	Wall clock	Observed `flip_rate`	Acceptable when
1	Baseline	Highest	Expected ~0	You need defensible numbers
4	Measure it	Measure it	Measure it	Flip rate is zero across 3 repeats
16	Measure it	Measure it	Measure it	Exploratory triage only
Dynamic (retry-shrunk)	Lowest	Lowest	Usually worst	Never for a reported metric

I refuse to give you multipliers here, because tokens-per-item depends on your prompts, your output verbosity, and the provider's accounting. The method is the deliverable; the numbers are yours to produce.

6. Acceptance rule

A run is comparable to another run only if all three hold. Same golden set hash, same declared batch_size, and flip_rate measured at or near zero for that batch size on that model version. If any of the three changes, you do not have a regression; you have a re-baseline, and the report should say so in the first sentence rather than the appendix.

A pragmatic gate I now use: if flip_rate > 0 at the batch size I want, I either fall back to batch size 1 for the reported number, or I keep batching but re-run the full corpus three times and report the spread. The second option costs tokens, which brings us to where a free tier actually belongs.

7. Where a free tier belongs in this workflow

A composition sweep is a high-volume, low-stakes, disposable workload, which is the one shape of work that fits a free allowance well. I run the multi-batch probe against MonkeyCode's free model access, and I run the harness itself on the free server option so the sweep never competes with anything I care about. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The allowance is reported by the project's outreach material (10M tokens as of this writing) and I have not independently verified its size, duration, or what happens when it is spent. Treat it as a budget variable, not a constant: log tokens per call, and stop the sweep at a declared ceiling so a partial run is visibly partial. If you want to reproduce the probe end to end, the free model access plus the free server option is enough to get from golden.jsonl to a flip_rate number today, with current terms on the project's site as your source of truth.

8. Limitations and who should not use this

Model version drift. The probe controls composition, not the provider's weights. A probe that passes today can fail next month for reasons outside your repository, so pin the model identifier you actually called in the run output.
Underpowered golden sets. Forty items detect gross contamination; they cannot detect a two-point regression. Scale the set before you scale the claim.
Determinism is not promised. Many hosted endpoints treat temperature=0 as best-effort. Three repeats, not one, before you write zero in the table.
Do not use this for latency-critical paths. If each request serves one user, batching is irrelevant and the probe is wasted effort.
Do not use a third-party free endpoint as your audit trail if you need contractual reproducibility. Run the probe there, then re-baseline the reported number on infrastructure you control.

9. What I would change next

Two things. First, I would make batch_size a field in the result record rather than a flag in my shell history, so no future reader can mistake a repacking for a regression. Second, I would add a shuffle repeat: same items, same batch size, different order, and diff again. If order alone flips verdicts, the contamination is positional and you can often fix it by randomizing once and freezing that order forever.

Here is my counterexample question for you. Which event order in your own harness breaks composition invariance first: a retry that shrinks the chunk, a shared output budget that truncates the tail, or a runner restart that splices two batch sizes into one report? And once you find it, should the system reject the item, replay the original chunk unchanged, or compensate by re-running the affected items at batch size 1 before publishing a number?

Architecture Review: Give a Free-Tier Inference Endpoint an Admission Envelope, Not a Retry Storm

Robin — Mon, 19 Oct 2026 07:50:55 +0000

Architecture Review: Give a Free-Tier Inference Endpoint an Admission Envelope, Not a Retry Storm

Consider the event order below, which any client with blind retries can produce against a shared inference endpoint. Nothing in it is exotic, and that is exactly the problem.

t0  worker-1 sends request A            (service time unknown, deadline 30s)
t1  worker-1 gives up at 30s            (A is still executing on the server)
t2  worker-1 retries as A'              (no idempotency key, no observation)
t3  worker-2 sends B                    (believes capacity is free)
t4  endpoint returns 429 for A' and B   (server is at its limit, not broken)
t5  both workers sleep ~1s, then retry  (synchronized herd)
t6  endpoint returns 429 again          (queue never drains)

The endpoint was never down. My client simply had no theory of its capacity, so every timeout became a new admission decision made by a process that knew nothing. The invariant I want is narrow and testable: a request may enter the system only if its projected wait fits inside its deadline, and a retry may never outrun the observed service rate. The typical implementation preserves neither half.

This is an architecture review of that boundary — the constraint set, the data flow, the failure domains, and the parts I would change next.

Assumptions (declare them before you measure anything)

A1 Service time is roughly stationary over a probe window of ten minutes; if it is not, your envelope is a snapshot, not a property.
A2 The endpoint publishes no concurrency limit, so the limit must be inferred from observed rejections and latency.
A3 Free-tier capacity is shared and can shrink without notice; treat any measured ceiling as an upper bound that decays.
A4 Work is replay-safe, or every request carries an idempotency key the downstream respects.
A5 The deadline belongs to the caller. The server has no idea what your request was for.

If A1 or A4 is false, stop here and fix that first; admission control on top of non-replayable work just produces faster corruption.

Step 1: Measure the envelope instead of guessing it

Free model access and a free server option are useful precisely because they let you run this probe without a procurement ticket. The offer I am describing is MonkeyCode's free model access plus a free server option; the project advertises a free token allowance (the operator's published figure at the time of writing was on the order of ten million tokens), and both terms can change, so read the console rather than this paragraph. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Here is the probe. It sweeps concurrency, records the status mix, and refuses to average away the tail.

# probe.py — characterize the endpoint; do not trust a single concurrency level
import asyncio, json, os, statistics, time
import httpx

ENDPOINT = os.environ["MC_ENDPOINT"]
KEY      = os.environ["MC_KEY"]

async def one(client, sem, i, out):
    async with sem:
        t0 = time.perf_counter()
        try:
            r = await client.post(
                ENDPOINT, timeout=90,
                headers={"Authorization": f"Bearer {KEY}"},
                json={"messages": [{"role": "user", "content": "reply with: ok"}],
                      "max_tokens": 8},
            )
            out.append({"i": i, "status": r.status_code,
                        "ms": (time.perf_counter() - t0) * 1000,
                        "retry_after": r.headers.get("retry-after")})
        except Exception as e:
            out.append({"i": i, "status": type(e).__name__,
                        "ms": (time.perf_counter() - t0) * 1000, "retry_after": None})

async def sweep(levels=(1, 2, 4, 8, 16), per_level=40):
    async with httpx.AsyncClient() as client:
        for level in levels:
            sem, out = asyncio.Semaphore(level), []
            await asyncio.gather(*(one(client, sem, i, out) for i in range(per_level)))
            ok = [o["ms"] for o in out if o["status"] == 200]
            mix = {}
            for o in out:
                mix[str(o["status"])] = mix.get(str(o["status"]), 0) + 1
            print(json.dumps({
                "level": level, "n": len(out), "ok": len(ok),
                "p50_ms": round(statistics.median(ok), 1) if ok else None,
                "p95_ms": round(sorted(ok)[int(len(ok) * 0.95) - 1], 1) if len(ok) > 20 else None,
                "status_mix": mix,
            }))

asyncio.run(sweep())

Run it. The number you want is not the mean; it is the level at which the rejection mix stops being zero while p50 stays flat. That level is your measured service rate mu, in requests per second, and everything downstream depends on it.

Step 2: Turn the measurement into a concurrency budget

Little's law gives you the only budget that matters: L = lambda * W. At saturation lambda = mu, so the in-flight budget is throughput times the latency you are willing to pay for.

class Envelope:
    """Concurrency budget derived from a measured service rate."""
    def __init__(self, mu_rps: float, deadline_s: float, safety: float = 0.6):
        self.mu, self.deadline = mu_rps, deadline_s
        self.cap = max(1, int(mu_rps * deadline_s * safety))

    def projected_wait(self, inflight: int) -> float:
        return inflight / self.mu          # seconds, saturated server

    def admit(self, inflight: int, service_p50: float, remaining: float):
        if inflight >= self.cap:
            return False, "cap"
        if self.projected_wait(inflight) + service_p50 > remaining:
            return False, "deadline"
        return True, "ok"

The safety factor is not timidity; it is your only defense against A3, because a shared free tier can lose capacity between two probe runs. I set it at 0.6 and I would go lower before I went higher.

Step 3: Classify failures, because they demand different actions

A single except Exception: retry() collapses five distinct failure domains into one response, and four of those responses are wrong.

Observed	Failure domain	Correct action
`429` with `Retry-After`	Shedding by the server	Wait at least that long, then re-admit through the gate
`429` without header	Shedding, no guidance	Back off on the measured `mu`, never on the retry count alone
Timeout / `ReadTimeout`	Unknown outcome on the server	Replay only with an idempotency key
`5xx`	Server-side fault	Bounded retry with jitter, decrement budget
Truncated stream	Partial success	Discard, never stitch; the artifact is unusable

That last row is the one people miss. A stream that ends early looks like a successful request with short output, and downstream code happily persists half an answer.

Step 4: Make the retry floor a tested property, not a habit

The classic bug is multiplicative jitter that lands below the floor the server asked for. Write the delay function so the floor is structural.

import random

class RetryFloor:
    def __init__(self, base=1.0, cap=30.0):
        self.base, self.cap = base, cap

    def delay(self, attempt: int, retry_after=None) -> float:
        floor = min(self.cap, self.base * (2 ** attempt))
        if retry_after is not None:
            floor = max(floor, float(retry_after))
        return floor + random.uniform(0, floor * 0.25)   # jitter additive, cannot go under

Now the properties are assertable: delay >= retry_after, delay >= base * 2**attempt, and delay is non-decreasing in attempt until the cap. If you cannot state those three as tests, you do not have a retry policy; you have a hope.

Step 5: Inject the failures before production does

A fake endpoint with scripted behavior turns this from opinion into a check. The test plan below is the artifact I would actually merge.

Herd test — return 429 for 30% of calls, assert the gate never exceeds cap and no request exceeds its deadline by more than one service time.
Slow-drain test — multiply service time by 3 mid-run, assert the gate shrinks admissions instead of queueing.
Truncation test — close the stream at 60% of tokens, assert the artifact is marked unusable and never persisted.
Replay test — time out after the server has already executed, replay with the same idempotency key, assert exactly one side effect.
Floor test — property-check RetryFloor over 10,000 random (attempt, retry_after) pairs.

Acceptance rule: over a 30-minute run at the chosen cap, p99 queue wait must stay under the deadline, the 429 rate must stay under one percent, and the floor test must have zero violations. Fail any of the three and halve the cap before you touch anything else.

What I would change next

The honest weakness of this design is that mu is a single scalar standing in for a multi-dimensional system. Two lanes with different prompt sizes share one budget, so a burst of long generations starves short ones even though the request count looks fine. The fix I would build next is a two-class gate: measure mu per size bucket, reserve a floor of capacity for the short class, and let the long class consume only the remainder. That is a small change to Envelope and a large change to tail behavior.

Limitations, and who should not use this

Do not adopt this if you need a hard p99 contractual SLO — a shared free tier gives you a measurement, not a guarantee, and the envelope can only shed work, not create capacity. Do not adopt it for non-replayable side effects without idempotency keys, and do not adopt it if your evaluation depends on a stable model identity that a free tier may not promise. Teams without any retry hygiene should fix that first; admission control in front of a retry storm just makes a tidier storm.

If you want a place to run the probe without booking infrastructure, MonkeyCode's free model access and free server option are a reasonable starting substrate — bring your own harness, and replace every number in this template with your own output.

So here is the question I would put to any reviewer of this design: which event order breaks your envelope — a Retry-After shorter than your own floor, a truncation that reports success, or a capacity drop between probe runs? And when it breaks, should the gate reject, replay with a key, or compensate downstream?

Architecture Review: Keep Eval Provenance on the Control Plane, Not the Borrowed Runtime

Robin — Fri, 16 Oct 2026 04:48:57 +0000

Last week I ran a four-shard eval batch against a borrowed runtime, went to sleep, and read the ledger over coffee. One shard had three committed rows, another had none, and the number I had pasted into a doc was an average over attempts rather than over shards. Nothing in the pipeline had thrown an exception, which is exactly why it took me an hour to find. The current cycle of public discussion keeps asking whether AI output holds up under review; far less of it asks whether the harness that produced the number could be trusted at all.

So let me state the invariant I now refuse to ship without. An eval batch is interpretable only if every committed row is attributable to exactly one (job, shard, epoch, model_revision) tuple, and the aggregate's denominator is the shard count, not the row count. Everything below is a consequence of that sentence.

Declared assumptions

I am reviewing a specific class of system here, so the conclusions are only valid inside these assumptions.

The control plane (your CI job, your laptop, your scheduler) is the single writer for a job and the only component allowed to mint attempt identities.
A shard is sealed synchronously with respect to the commit that finalizes it; there is no window where the ledger holds an unsealed authoritative row.
The model endpoint is addressed through a floating alias unless the job spec pins a revision digest explicitly.
Preemption is a normal outcome rather than an error, and the runtime is never required to report it to you.
The batch aggregate is a mean over shards; if yours is a sum over tokens or a p95 over calls, the same argument applies with a different denominator.

Constraints, and where they come from

A free or preemptible runtime gives you three properties whether you asked for them or not: your work can be evicted at any point, your completion rate is not guaranteed, and the underlying model alias may resolve differently between two calls. Those are not defects, they are the contract you accepted in exchange for not paying for a queue. The architecture question is therefore not "how do I make the runtime reliable" but "what must be true on my side so that an unreliable runtime cannot corrupt a number."

Notice how much of the damage happens after the work is done. A preempted attempt is harmless if it publishes nothing; it becomes a correctness bug only when a late worker publishes a row that your ledger accepts. That is where the review should start, and it is why I moved identity and sealing to the control plane.

Data flow: who is allowed to name an attempt

sequenceDiagram
    participant CP as Control plane (single writer)
    participant R as Borrowed runtime (free / preemptible)
    participant M as Model endpoint (floating alias)
    participant L as Ledger (append-only)

    CP->>R: dispatch(key = job/shard/epoch)
    R->>M: resolve alias -> revision
    M-->>R: revision, grade
    R-->>CP: row(key, provenance, grade)
    CP->>L: commit(row)
    CP->>L: seal(shard, epoch)  %% only after the write-ahead ack
    Note over R,L: preemption may drop the row entirely,<br/>and the same attempt may publish again much later
    R-->>L: late row(same key, original or drifted revision)
    L-->>CP: committed | deduped | conflict | rejected_stale

The failure domains I actually observed

Failure domain	Symptom	What it corrupts	Containment
Preemption	attempt returns nothing	shard has no row, so the denominator silently shrinks	retry at a new epoch, never reuse the key
Late publication	row arrives after the shard sealed	an older attempt double-weights the shard	epoch gate evaluated at the ledger
Alias drift	same key, two revisions	two models mixed inside one aggregate	pin the revision per job, quarantine conflicts
Duplicate delivery	identical row twice	nothing, if writes are idempotent by key	first-write-wins per key
Clock skew	"last write wins" by timestamp	nondeterministic winner across replicas	order by control-plane epoch, not wall time

Build the fixture in five steps

The artifact below runs on the standard library alone, and it is a simulator rather than a distributed system. Run it with python3 borrowed_runtime_sim.py; every scenario is deterministic given a seed, so you can argue with the outcome instead of with my prose.

Step 1 — pin the contract. Identity comes from the control plane, and provenance travels with the row instead of being reconstructed later.

"""borrowed_runtime_sim.py — does a preemptible eval runtime preserve provenance?"""
from __future__ import annotations

import hashlib
import random
from dataclasses import dataclass


def sha(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()[:12]


@dataclass(frozen=True)
class AttemptKey:
    job_id: str
    shard: int
    epoch: int          # control-plane authority, strictly increasing per shard


@dataclass(frozen=True)
class Provenance:
    model_revision: str
    prompt_sha256: str
    gateway: str


@dataclass(frozen=True)
class Row:
    key: AttemptKey
    provenance: Provenance
    grade: float

Step 2 — make the ledger classify instead of merge. Four outcomes are possible, and only one of them is a commit. A conflict is not something to resolve by preferring the newer row; it is a signal that the batch must be re-run.

class Ledger:
    """First write per AttemptKey wins. Everything else is classified, never merged."""

    def __init__(self) -> None:
        self.rows: dict[AttemptKey, Row] = {}
        self.sealed: dict[int, int] = {}
        self.conflicts: list[tuple[Row, Row]] = []
        self.rejected_stale: list[AttemptKey] = []
        self.deduped = 0

    def seal(self, shard: int, epoch: int) -> None:
        self.sealed[shard] = max(epoch, self.sealed.get(shard, -1))

    def commit(self, row: Row, *, epoch_gate: bool) -> str:
        if epoch_gate and row.key.epoch < self.sealed.get(row.key.shard, -1):
            self.rejected_stale.append(row.key)
            return "rejected_stale"
        prior = self.rows.get(row.key)
        if prior is None:
            self.rows[row.key] = row
            return "committed"
        if prior == row:
            self.deduped += 1
            return "deduped"
        self.conflicts.append((prior, row))
        return "conflict"

Step 3 — stub the runtime with its real misbehaviours. The stub preempts, duplicates, publishes late, and resolves a floating alias on every call, including on the late one.

class BorrowedRuntime:
    """A free or preemptible executor: it vanishes, duplicates, and arrives late."""

    def __init__(self, revision, alt_revision, rng, cfg):
        self.revision = revision
        self.alt_revision = alt_revision
        self.rng = rng
        self.cfg = cfg
        self.late: list[Row] = []

    def _resolve(self) -> str:
        return self.alt_revision if self.rng.random() < self.cfg.p_alias_flip else self.revision

    def _row(self, key: AttemptKey, prompt: str) -> Row:
        revision = self._resolve()
        return Row(
            key,
            Provenance(revision, sha(prompt), "borrowed-runtime"),
            int(sha(f"{revision}:{prompt}")[:6], 16) / 0xFFFFFF,
        )

    def attempt(self, key: AttemptKey, prompt: str) -> list[Row]:
        if self.rng.random() < self.cfg.p_late:
            self.late.append(self._row(key, prompt))   # published after the seal
        if self.rng.random() < self.cfg.p_preempt:
            return []                                  # we time out and retry later
        row = self._row(key, prompt)
        return [row, row] if self.rng.random() < self.cfg.p_duplicate else [row]

Step 4 — let the control plane retry only unfinished shards. A shard that already holds an authoritative row is never re-dispatched, and the seal happens the moment the commit lands.

@dataclass
class Config:
    shards: int = 4
    epochs: int = 3
    p_preempt: float = 0.30
    p_duplicate: float = 0.20
    p_late: float = 0.25
    p_alias_flip: float = 0.0
    epoch_gate: bool = True
    bump_revision: bool = True


def run(cfg: Config, seed: int) -> Ledger:
    rng = random.Random(seed)
    ledger = Ledger()
    prompts = {s: f"eval-prompt-{s}" for s in range(cfg.shards)}
    carry: list[Row] = []

    for epoch in range(cfg.epochs):
        revision = f"rev-{epoch}" if cfg.bump_revision else "rev-pinned"
        runtime = BorrowedRuntime(revision, "rev-next", rng, cfg)
        for shard in range(cfg.shards):
            if any(r.key.shard == shard for r in ledger.rows.values()):
                continue                       # already authoritative, do not re-dispatch
            key = AttemptKey("job-1", shard, epoch)
            for row in runtime.attempt(key, prompts[shard]):
                if ledger.commit(row, epoch_gate=cfg.epoch_gate) == "committed":
                    ledger.seal(shard, epoch)
        carry.extend(runtime.late)

    for row in carry:                          # the tail: attempts that finished late
        ledger.commit(row, epoch_gate=cfg.epoch_gate)
    return ledger

Step 5 — audit with named properties. The audit returns the denominator explicitly, because a missing shard and a double-counted shard are different bugs that produce the same wrong mean.

def audit(ledger: Ledger, cfg: Config) -> dict:
    by_shard: dict[int, list[Row]] = {}
    for row in ledger.rows.values():
        by_shard.setdefault(row.key.shard, []).append(row)
    revisions = sorted({r.provenance.model_revision for rows in by_shard.values() for r in rows})
    return {
        "rows": len(ledger.rows),
        "shards_with_rows": len(by_shard),
        "missing_shards": cfg.shards - len(by_shard),
        "max_rows_per_shard": max((len(v) for v in by_shard.values()), default=0),
        "committed_revisions": revisions,
        "conflicts": len(ledger.conflicts),
        "rejected_stale": len(ledger.rejected_stale),
        "deduped": ledger.deduped,
    }


def verdict(a: dict) -> str:
    problems = []
    if a["missing_shards"]:
        problems.append("incomplete-batch")
    if a["max_rows_per_shard"] > 1:
        problems.append("denominator-drift")
    if a["conflicts"]:
        problems.append("unresolved-revision-mix")
    if len(a["committed_revisions"]) > 1:
        problems.append("multi-revision-aggregate")
    return ", ".join(problems) if problems else "interpretable"

Read the matrix, not the mean

The baseline property test zeroes out every noise term, so it must pass or the fixture itself is wrong.

# test_properties.py  (pytest)
from borrowed_runtime_sim import Config, audit, run


def test_pinned_gate_is_self_consistent():
    cfg = Config(epoch_gate=True, bump_revision=False, p_alias_flip=0.0,
                 p_preempt=0.0, p_late=0.0, p_duplicate=0.0)
    for seed in range(32):
        a = audit(run(cfg, seed), cfg)
        assert a["max_rows_per_shard"] == 1
        assert a["conflicts"] == 0
        assert len(a["committed_revisions"]) == 1
        assert a["missing_shards"] == 0

Then run the four scenarios and read the verdict field. Exact counts move with the seed, but the violation class should be deterministic given the flags, and that is the thing you are testing.

Scenario	Epoch gate	Revision handling	Designed violation	Required response
A, gate on, pinned	on	one pinned digest	none, except possible missing shards	publish only with `missing_shards == 0`
B, gate on, revision drifts	on	revision per retry epoch	multi-revision aggregate	split the report by revision before quoting it
C, gate off	off	revision per retry epoch	late row lands after the seal	fix the gate, do not re-weight
D, gate on, alias floats	on	resolved per call	same key, two revisions	re-run the batch, never average the conflict

Scenario D is the one that surprised me. The epoch gate is doing its job, the denominator stays clean, and the batch still is not interpretable, because a redelivered attempt resolved the alias to a different revision. Gating and pinning are two different controls, and you need both.

Where a free runtime fits, and where it does not

This is the part of the review that people skip. A free or preemptible runtime is not a cheaper version of a production endpoint; it is a different lifecycle, and it is only honest to use it where lost or duplicated work is a retry rather than a decision.

Workload	Fits a preemptible free runtime?	Why
Harness, ledger, and schema development	Yes	failures are the input to the test, not an accident
Smoke evals on a pinned revision	Usually	bounded retries; accept and report missing shards
Exploratory prompt sweeps	Yes	a lost row costs a retry, not a release gate
Release thresholds and regression gates	No	you do not control the revision or the completion rate
Latency or cost benchmarks	No	the denominator is not yours to define
Long-lived stateful runtimes	No, by construction	eviction is the contract, not an incident

For the first three rows I have been using MonkeyCode for exactly this reason: it exposes free model access plus a free server option, and the operator states a free allowance on the order of ten million tokens, so the cost of running a fixture matrix like the one above is not the thing that stops me. Check the current terms yourself rather than trusting my summary, because allowances change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you want to try the matrix against a live endpoint instead of a stub, that free tier is a reasonable place to start, and the interesting part is not the grades at all; it is whether your ledger refuses the row it should refuse.

What I would change next

Move the seal into the same transaction as the commit, so there is no instant where a shard is finalized but not yet fenced against older epochs.
Put a revision digest into the job spec itself, then treat any row whose revision differs from the digest as uncommittable rather than as a conflict to investigate.
Add a compaction step that converts superseded rows into an explicit superseded_by chain, so a retried shard leaves an audit trail instead of a silent replacement.
Emit missing_shards in the report header, and make the report writer refuse to print an aggregate when that count is nonzero.

Tradeoffs I am accepting

The gate costs one extra piece of state per shard and a small amount of latency on the first commit, and it assumes a single-writer control plane. Strict pinning also removes a convenience you may actually want: you can no longer let a retry pick up a fresher model as a free upgrade, because that upgrade is precisely what makes the aggregate uninterpretable. The conflict path adds a manual re-run step that a lenient ledger would have hidden.

Limitations and who should not use this

The simulator models ordering and attribution, not consensus, and it says nothing about how you would fence two control-plane replicas. Do not adopt the epoch gate if you already have an endpoint with pinned revisions, exactly-once delivery, and a completion guarantee, because you would be adding ceremony to a problem you do not have. Do not adopt it for a batch with one shard and one attempt, where the denominator is trivially one. And make sure your aggregate story matches the gate; a gate that protects a per-shard mean is useless if the number you publish is a total over rows.

So here is the question I want answered from someone else's ledger. Which event order breaks your invariant first: a stale epoch committing after the seal, or a fresh attempt resolving an alias to a new revision? And when that happens, does your system reject the row, replay the shard, or compensate the aggregate, because those three answers lead to very different architectures.

Make Token Spend an Invariant Before a Preemptible Free Tier Replays Your Eval Batch

Robin — Thu, 15 Oct 2026 11:07:07 +0000

Make Token Spend an Invariant Before a Preemptible Free Tier Replays Your Eval Batch

It was 03:12 and shard three of four had already finished 78% of its cases when the worker simply vanished. The driver had no heartbeat, no exit code, and no way to tell whether the last forty graded cases had ever been written anywhere. When the batch finally reported complete, my spend ledger showed roughly 1.4x the tokens I had planned, because two attempts had committed output for the same shard. Would your evaluation harness notice that, or would it just average the two answers and hand you a number you trust?

This is an architecture review of that failure, not a tutorial about model quality. I run cheap classification and grading passes on free model access, and I run the shard workers on a free server option, because both are exactly where prototyping belongs. The catch is that free capacity is preemptible by definition, so the correctness of your results depends on how you fence attempts, not on how good the model is.

Assumptions I am declaring up front

Before any invariant makes sense, the constraint set has to be explicit, otherwise the design is unfalsifiable.

Shard workers can be reclaimed without notice, and no shutdown hook is guaranteed to run.
Result delivery is at-least-once, so a late commit can arrive after a replacement attempt has already finished.
There is no shared clock between the driver, the workers, and the artifact store that is trustworthy enough for ordering.
Model calls dominate cost, so the ledger must be denominated in tokens rather than in wall-clock seconds.
Shards are independent units of work, and each shard must produce exactly one accepted artifact per run.
The free capacity I use is MonkeyCode's free model access for the cheap grading stage and its free server option for the shard workers. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator states a free token allowance on the order of ten million tokens; verify the current figure on the project page before you size a batch around it, because I am not treating that number as a permanent contract.

That last point matters more than it looks. If you assume a fixed allowance and the allowance changes, your batch does not fail loudly — it fails quietly by truncating, which is worse.

The event order that broke my accounting

Here is the counterexample in full, because the bug is only visible in the interleaving.

Driver reserves budget for (run=7, shard=3, epoch=0) and dispatches.
Worker 3 grades 39 of 50 cases and writes a checkpoint after case 30.
The host reclaims worker 3; the lease expires with no heartbeat renewal.
Driver decides the attempt is dead and dispatches (run=7, shard=3, epoch=1).
Worker 3′ grades all 50 cases and commits an artifact for shard 3.
The original worker resumes on a different host, finishes its last 11 cases, and commits a second artifact for shard 3.

Nothing in that sequence is a network partition, a Byzantine node, or an exotic clock problem. It is just a preempted process that came back, and any harness that keys results by shard_id alone will accept both writes. The invariant I was missing is simple to state: for a given run, the committed token spend per shard is a function of the highest epoch that produced an accepted artifact, and lower epochs must never commit.

Model: lease, epoch, and a monotonic spend ledger

I model the driver as a lease grantor and the ledger as a monotonic reservation table keyed by attempt identity.

sequenceDiagram
    participant D as Driver
    participant W as Shard worker (free tier)
    participant L as Spend ledger
    participant S as Artifact store
    D->>L: reserve(run=7, shard=3, epoch=0)
    L-->>D: RESERVED
    D->>W: dispatch(lease, input_digest)
    W->>W: grade cases, checkpoint every 30
    Note over W: preempted at 39/50
    D->>W: re-dispatch epoch=1
    W->>S: PUT artifact(epoch=1, input_digest)
    S-->>D: 201 Created
    D->>L: commit(run=7, shard=3, epoch=1)
    Note over W: epoch=0 host resumes
    W->>L: commit(run=7, shard=3, epoch=0)
    L-->>W: REJECTED (stale epoch)

The important detail is that the ledger rejects on two different grounds: a key that was never reserved, and a key whose epoch is below the highest committed epoch for that shard. Those are different failure classes and they deserve different counters, otherwise you cannot tell a replay from a forged attempt.

Step-by-step: building the fence

1. Key every attempt by `(run_id, shard_id, epoch)`

The epoch is granted by the driver, never by the worker, because a worker that can increment its own epoch can always win. Reservation happens before dispatch, so a budget overshoot is prevented rather than detected.

2. Reserve before the call and commit only after the artifact digest exists

A reservation is a promise, and a commit is a receipt, and conflating them lets a crashed attempt consume budget forever. I keep reservations with a lease expiry so a dead attempt's reservation is reclaimable, but the committed total is append-only.

3. Make every checkpoint carry the input digest

A checkpoint that only records case_index is not resumable, because the fixture may have changed between attempts. Storing the digest of the input slice means a resumed attempt can prove it is continuing the same work, and if it cannot, it restarts the shard instead of mixing two fixture versions.

4. Gate resumption on lease validity, not on liveness

Liveness checks lie under preemption, because a paused host looks alive from the outside. A worker should refuse to write when its lease has expired, even if it has a completed result in memory, and that refusal is the whole point.

A minimal fixture you can adapt

This is a reference sketch, not a benchmark; I have not executed this exact file, so treat it as a fixture to adapt and pin down with your own tests.

"""preempt_sim.py - reference fence for preemptible eval shards (sketch)."""
from dataclasses import dataclass, field

TOKENS_PER_CASE = 37          # declared assumption, not measured
CASES_PER_SHARD = 50
SHARDS = 4
BUDGET_TOKENS = SHARDS * CASES_PER_SHARD * TOKENS_PER_CASE  # 7,400


@dataclass
class SpendLedger:
    reserved: dict = field(default_factory=dict)   # key -> tokens
    committed: dict = field(default_factory=dict)  # key -> tokens
    highest_epoch: dict = field(default_factory=dict)  # shard -> epoch

    def reserve(self, run, shard, epoch, tokens):
        key = (run, shard, epoch)
        if key in self.reserved or key in self.committed:
            return False                  # replay of the same attempt
        if self.spent() + sum(self.reserved.values()) + tokens > BUDGET_TOKENS:
            return False                  # would overshoot the run ceiling
        self.reserved[key] = tokens
        return True

    def commit(self, run, shard, epoch, tokens):
        key = (run, shard, epoch)
        if key in self.committed:
            return "ALREADY_COMMITTED"
        if epoch < self.highest_epoch.get(shard, -1):
            return "STALE_EPOCH"           # a resurrected attempt lost the race
        if self.reserved.get(key) != tokens:
            return "UNRESERVED_SPEND"      # partial or forged attempt
        del self.reserved[key]
        self.committed[key] = tokens
        self.highest_epoch[shard] = epoch
        return "COMMITTED"

    def spent(self):
        return sum(self.committed.values())


def resume_is_safe(checkpoint, input_digest, lease_valid):
    if checkpoint is None:
        return True                        # nothing to trust yet
    if not lease_valid:
        return False                       # expired lease must not write
    return checkpoint["input_digest"] == input_digest

Drive it with a deterministic event queue: reserve, dispatch, deliver, preempt, re-dispatch, and then deliver a late commit from the dead epoch. If your ledger returns STALE_EPOCH on that last event, the fence holds for that interleaving.

Failure injection matrix

Injected failure	Observable symptom	Invariant attacked	Detection	Recovery action
Preemption at 78%	Lease expires, no exit code	Exactly-one artifact per shard	Heartbeat gap plus missing commit	Re-dispatch with `epoch + 1`
Duplicate delivery	Same artifact arrives twice	Spend is idempotent	`ALREADY_COMMITTED` counter	Reject, do not re-bill
Resurrected worker	Late commit from old epoch	Monotonic epoch per shard	`STALE_EPOCH` counter	Reject, log the host identity
Fixture change mid-run	Checkpoint digest mismatch	Result belongs to one input version	Digest compare on resume	Restart the shard, never merge
Clock skew of 90s	Lease appears valid after expiry	Lease validity gates writes	Driver-issued expiry, not worker clock	Shorten lease, renew earlier

Tradeoffs I would accept, and one I would not

Choice	What you gain	What you pay	When to pick it
Driver-issued epochs	Single ordering authority	Driver becomes a coordination point	Always, for at-least-once workers
Checkpoint every 30 cases	Cheap resume after preemption	Slightly more store writes	Shards longer than a few minutes
Reserve-before-dispatch	Budget cannot overshoot	Reservations must expire cleanly	Any tier with hard free limits
Restart on digest mismatch	No mixed-version artifacts	Wasted work	Fixtures change during a run
Worker self-epochs	Fewer round trips	Two writers can both win	Never, in my experience

Limitations and who should not use this

The overhead is real, and a single-worker batch that finishes in ninety seconds does not need a ledger at all. If your results are advisory rather than decision-bearing, a duplicate artifact is annoying but survivable, so skip the machinery. If your preemption rate is near zero on a dedicated host, the lease renewal traffic buys you very little. This design also assumes shards are independent, and a harness with cross-shard aggregation needs a second fence at the reduce step. Finally, my load model above is declared, not benchmarked, so replace those constants before you quote any number from it.

What I would change next

Two changes are on my list. First, I would move from shard-level epochs to token-level reservations inside a shard, so a preemption at 78% does not force a full re-grade of the remaining 22%. Second, I would add an explicit compensation path for the case where a stale artifact was already accepted by a downstream reducer, since rejection at the ledger is too late at that point. Both changes trade throughput for a narrower blast radius, and I would rather pay that than explain a duplicated grade in a release decision.

The acceptance rule I use is blunt: over 500 simulated preemptions, committed spend must stay under the run ceiling, each shard must end with exactly one accepted artifact, and every rejection must carry a classified reason. Which event order breaks your own ledger — a resurrected worker, a duplicate delivery, or a fixture that changed underneath a resume — and should your system reject, replay, or compensate when it does?

If you want to prototype this harness without paying for the first thousand attempts, MonkeyCode's free model access and free server option are a reasonable place to run the shard workers while you build the fence.

Make Authorization Fence an Invariant Before a Side Lane Reopens the Attempt

Robin — Sun, 11 Oct 2026 20:13:17 +0000

I replayed a four-event trace on paper before I trusted a side-lane retry classifier at all. The planner had already sealed compensation when the side lane finally returned its late yes answer. A naive merger treated that late yes as fresh permission to reopen the already sealed attempt. Would you really let one delayed label rewrite a closed attempt after compensation had already landed?

That order is the whole review, not a footnote I can patch later with a longer timeout. I am treating retry authorization as an append that must carry an open generation fence token. If the fence does not match the live attempt, the log rejects the label and leaves the seal untouched. Why would a slower lane earn more authority than the path that already closed the books?

Assumptions I will not smuggle in later

Before I trust the diagram, I want the constraints written where a reviewer can attack them. If any assumption is false in your system, stop and do not borrow the acceptance rule at the bottom. Would you rather discover a missing generation id during review, or after a sealed attempt has already been reopened?

I assume delivery between planner, side lane, and decision log is at least once, so duplicates are normal rather than rare.
I assume the side lane may run on a free server and call a free model, with no latency promise I will write down.
I assume the primary path seals on its own clock and never waits unbounded for a classification to arrive.
I assume each attempt has a monotonic generation, and a seal is terminal for that generation only.

I am not claiming a measured quota, a named model, or a permanent free tier, because those facts were not given to me. I also refuse to invent hardware, duration, or a benchmark just to make the side lane sound faster. What I will claim is narrower: MonkeyCode offers free model access and a free server option for this shadow lane.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I use that free model access only to draft advisory labels, and I host this desk harness on the free server option. A slow advisory label may inform the next generation, but it never becomes the component that seals the current one. If you remove every product name, the fence rule should still be worth keeping on your own bus.

The flow I would actually draw

planner --open(G)--> attempt_log
planner --classify(G)--> side_lane (free server + free model)
side_lane --label(G, yes|no|unknown)--> fence_gate
fence_gate --accept--> attempt_log    only if G is open
fence_gate --reject--> stale_drop     if G is sealed or unknown
planner --seal(G)--> attempt_log

Please read that diagram as an execution path, not as a loose slogan about separation of concerns. The fence gate is the only component allowed to interpret a side-lane label as an authorization. Everything else may propose a label, but nothing else may reopen a generation that has already been sealed. I would rather drop a late yes than let the diagram arrow skip the gate and write the log directly.

Numbered steps for one attempt

Open generation G, persist the open mark, and only then emit the classify request carrying that same generation.
Let the primary path proceed on its own clock, and never hold the seal timer hostage for a classification.
On any arriving label, compare generation G with the log before you read yes, no, or unknown.
If G is open and the label is yes, record one authorization; if G is sealed, append a stale drop and stop.
Seal G from the primary path alone, and ignore every later label that still names that same generation.

Does step three feel redundant when you already trust the queue to preserve your original emit order? I do not trust that, because a restart can deliver the classify response after the seal message has landed. You emitted classify first, yet the log can still observe seal first, and that swap is the bug. If your mental model forbids that swap, your mental model is describing a different bus than mine.

Failure domains I refuse to collapse

The side lane, the free server process, and the decision log are three failure domains I will not merge. A free server restart can drop an in-flight classification while the planner retries the same request under one generation. That retry is a duplicate delivery, not a fresh attempt, so it must not mint a new authorization by accident. A late yes after seal is a stale append, and replaying it would reopen work the compensation already undid.

An unknown label is not a soft yes, because ambiguity must not widen retries once the denominator is closed attempts. What about a yes that arrives before the seal, gets accepted, and then a duplicate yes arrives after the seal? The first append stands, and the second must hit the stale drop, or your exactly-once story is only a hope. I would rather reject the duplicate than compensate again, because a second compensation can undo a valid retry.

A minimal simulator, labeled as unexecuted

I have not run this fixture against a production bus, so treat the script as a desk proposal you can execute locally. It encodes the fence, the seal, and the three outcomes I care about: accept, reject, and duplicate after seal. If the assertions fail, the invariant is already broken, and no model score can talk you out of that. Would you ship a merger that cannot pass this trace, just because the free lane usually sounds confident?

# proposal: fence_sim.py - not a production claim
from dataclasses import dataclass, field

@dataclass
class Attempt:
    generation: int
    sealed: bool = False
    auths: int = 0
    drops: int = 0

@dataclass
class Log:
    attempts: dict = field(default_factory=dict)

    def open(self, g: int) -> None:
        self.attempts[g] = Attempt(g)

    def seal(self, g: int) -> None:
        self.attempts[g].sealed = True

    def on_label(self, g: int, label: str) -> str:
        attempt = self.attempts.get(g)
        if attempt is None or attempt.sealed or label != 'yes':
            if attempt:
                attempt.drops += 1
            return 'reject'
        attempt.auths += 1
        return 'accept'

def permute() -> None:
    orders = [('label', 'seal'), ('seal', 'label'), ('label', 'seal', 'label')]
    for order in orders:
        log = Log()
        log.open(7)
        auths_at_seal = None
        for step in order:
            if step == 'label':
                log.on_label(7, 'yes')
            else:
                auths_at_seal = log.attempts[7].auths
                log.seal(7)
        assert log.attempts[7].auths == auths_at_seal
    print('fence held across injected orders')

if __name__ == '__main__':
    permute()

python3 fence_sim.py

If you want a property check rather than one happy trace, permute seal and label before you argue about scale. Assert that a sealed generation never increases its auth count, even when the same yes is delivered twice. I would add that permutation to the harness before I discussed throughput, because one printed line is not an acceptance rule. The fixture still counts two yes labels before seal as two auths, so idempotent accept is the next change, not a hidden feature.

Acceptance rule and the denominator

The denominator is sealed generations that later receive at least one label, not the total number of model calls. The numerator is those same generations whose authorization count stays unchanged after the seal has been written. I accept this design only when that ratio stays at one across label-then-seal, seal-then-label, and duplicate-yes-after-seal. A free model that is often right does not move this ratio, because the fence is about order, not about label quality.

Tradeoffs I would not hide

Choice	Latency	Correctness	Cost	What breaks
Wait for the side lane before seal	Higher	Stronger if the lane always returns	Free-lane calls sit on the critical path	A slow free server stalls every attempt
Seal without a fence	Lower	Weak	Cheap until a late yes reopens work	Compensation and retry both apply
Fence, reject stale, do not replay	Primary stays short	Seal is stable	Drops late labels	A useful late yes is ignored
Fence plus compensate on stale yes	Primary stays short	Risky	Extra writes	Second compensation undoes a valid retry

I would pick the third row for the decision log, and I would keep label quality on a separate scoreboard. Waiting for the side lane looks safer until a slow free host stalls every attempt that was ready to seal. Sealing without a fence looks cheap until a late yes and a compensation both apply to the same generation. Do you actually need that late yes inside the sealed generation, or only as input when the planner opens the next one?

What I would change next

I would split the authorization store from the decision log so a side-lane bug cannot obtain the seal writer. I would stamp every classify request with generation, attempt id, and a hash of the tool error being judged. I would refuse any label whose hash does not match, even when the generation is still open and the label says yes. I would canary those three failure classes, delayed label, duplicated label, and restarted side lane, and halt if any sealed generation gains an auth.

A rejected late yes may open generation G plus one only through the planner, never through the fence gate itself. Free model access can draft those advisory labels, and the free server can host the harness, within the limits already stated. If you want to try that split, start from the fixture and let only the planner seal, while the free host only proposes labels.

Who should not use this

Skip this pattern if your product requires the classifier to block the seal, because I deliberately removed that wait. Skip it if you have no generation id, since a fence without a stable name is only a comment beside the diagram. Skip it if a human must approve every retry, because this gate only decides whether a label may append at all. I would not use a free lane as the only record of a retry, since I promise neither retention nor uptime.

Which order should break it?

Which event order breaks the invariant: seal, then a duplicate yes, then a planner restart that re-emits the same classify call? I would reject that yes, refuse to replay it into generation G, and refuse to compensate the sealed attempt again. Should your system do the same, or do you still believe a late yes is safer than a dropped label? If you cannot name that order, you do not yet have an invariant, only a hope that the side lane stays fast.

Make Witness Generation an Invariant Before a Late Lane Seals Compensation

Robin — Sat, 10 Oct 2026 16:20:26 +0000

I keep replaying a design-review counterexample from a grading pipeline that looked correct until one late witness arrived. The deterministic harness had already recorded a failure for compensation step S7 under the current fixture generation. A slower judge lane, started against the previous fixture, then returned a pass and reached the closer second. Would you seal that compensation step only because the passing witness happened to be the last write on the wire?

That closer did exactly that, and the skipped compensation left an in-doubt tool call looking finished. I am not citing a production dashboard, a customer incident, or a measured error rate for this walkthrough. I am reviewing the architecture as a state machine, because a pretty sequence diagram will not catch a reordered seal. If the last write can erase an earlier failure, the workflow does not converge, it only looks quiet.

The invariant a common closer drops

The invariant is simple enough to test, and it is still easy to violate under at-least-once delivery between lanes. A witness may seal compensation only when its generation and fixture hash equal the current fence. An older generation must be rejected even if its verdict arrives later and sounds more confident. A matching failure must dominate a matching pass, because skipping compensation is the dangerous direction for this step.

Last-write-wins preserves neither the generation fence nor the fail-dominates-pass rule once two lanes share a closer. It treats delivery order as authority, which is an accident of queues rather than a property of the step. I would not call that pattern a merge, and I would call it an unguarded assignment into the seal log. Have you ever watched a queue make the stale message look like the freshest decision in the log?

Assumptions I will state before the diagram

I assume every lane delivers at least once, and that duplicates can arrive after a seal has already been attempted. I assume the orchestrator can bump a fixture generation before new work starts, and that this bump is durable. I do not assume a shared clock between the worker process and the model lane that judged an older fixture. I also do not assume that a textual pass from a model is equivalent to a property-test failure from the harness.

MonkeyCode is relevant here only as capacity for a non-authoritative lane: the operator describes free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming a quota, a model name, a hardware shape, a duration, or a measured comparison. If you stand up a non-authoritative judge, read the current project terms first, then keep that lane outside the seal path.

Constraints, data flow, and failure domains

The orchestrator owns the fence, the step identity, and the compensation log that later records the seal. The harness domain owns deterministic checks that must run against the published fixture hash for that same generation. The model lane owns a slow judgment that can already be stale the moment a fixture generation bumps. The free server domain owns a worker that can retry, reorder, and deliver twice after an acknowledgement is lost.

sequenceDiagram
    participant O as Orchestrator
    participant H as Harness lane
    participant M as Model lane
    participant C as Closer
    O->>O: Publish fence gen=7 hash=fx-7
    O->>H: Start check S7
    O->>M: Start judge still on gen=6
    H-->>C: Witness fail gen=7 seq=1
    M-->>C: Witness pass gen=6 seq=2
    C->>C: Reject gen=6, seal fail, compensate once

What would I change in that picture if the closer still merged witnesses by delivery sequence number? I would delete any arrow that lets the model lane touch the seal, and I would force a reject branch first. The free server worker may host the harness, but it still may not invent a generation or a fixture hash. Capacity and authority stay in different domains, or the retry budget of one domain will close the other.

Numbered steps for the seal protocol

Publish the fence in the same log that will later record the seal, and refuse new lane starts until that publish is durable.
Bind every witness to the step id, generation, fixture hash, lane id, and a content hash of the inputs it actually saw.
On delivery, reject generation or hash mismatches without enqueueing compensation and without advancing the step to closed.
If an accepted failure exists, seal fail once, then run idempotent compensation keyed by step id rather than by delivery sequence.
Treat a duplicate accepted witness as a no-op, and replay only when the generation still matches and the witness is missing.

I would not replay an older generation hoping the model will eventually change its mind about a fixture it never saw. That replay spends budget to reproduce a stale pass, which is the opposite of convergence under a fence. Rejection is the correct response to a fence miss, even when the late message is fluent and internally consistent. Compensation is the correct response only after an accepted failure, and only through a key that ignores delivery sequence.

A simulator that shows the break

The following fixture is a proposal you can run locally, and it is not a measurement of any hosted service. I have not used it as a load test, and it does not establish a latency, throughput, or cost claim. It only encodes the event order above, plus one duplicate delivery and one foreign fixture hash. Read the assertions before you trust the printed lines, because a printout alone is not the property.

from dataclasses import dataclass

@dataclass(frozen=True)
class Witness:
    lane: str
    generation: int
    fixture_hash: str
    verdict: str
    delivery_seq: int

@dataclass(frozen=True)
class Fence:
    generation: int
    fixture_hash: str

def last_write_wins(witnesses):
    winner = sorted(witnesses, key=lambda w: w.delivery_seq)[-1]
    return {"sealed": True, "verdict": winner.verdict, "lane": winner.lane, "reason": "lww"}

def seal_with_fence(fence, witnesses):
    accepted = [
        w for w in witnesses
        if w.generation == fence.generation and w.fixture_hash == fence.fixture_hash
    ]
    if not accepted:
        return {"sealed": False, "verdict": None, "reason": "reject"}
    if any(w.verdict == "fail" for w in accepted):
        return {"sealed": True, "verdict": "fail", "reason": "accepted_fail"}
    return {"sealed": True, "verdict": "pass", "reason": "accepted_pass"}

def main():
    fence = Fence(7, "fx-7")
    late = [
        Witness("harness", 7, "fx-7", "fail", 1),
        Witness("model-lane", 6, "fx-6", "pass", 2),
    ]
    buggy = last_write_wins(late)
    fenced = seal_with_fence(fence, late)
    assert buggy["verdict"] == "pass"
    assert fenced["verdict"] == "fail"
    duplicated = late + [Witness("harness", 7, "fx-7", "fail", 3)]
    assert seal_with_fence(fence, duplicated) == fenced
    foreign = [Witness("free-server", 7, "fx-other", "pass", 4)]
    assert seal_with_fence(fence, foreign)["sealed"] is False
    print("lww", buggy)
    print("fence", fenced)

if __name__ == "__main__":
    main()

Save that listing as witness_fence.py, then run python3 witness_fence.py from the same working directory you saved it in. You should see the last-write-wins path seal a pass, while the fence keeps the harness failure and rejects nothing incorrectly in this case. If the buggy assertion ever stops matching a pass, you have probably sorted by generation instead of delivery sequence. That change would hide the counterexample rather than fix the closer that still assigns the last write.

Injected failures and the acceptance rule

I inject four failures, and each one has a deny or accept outcome you can assert without waiting on a timer. A reordered older pass must not change a sealed fail, even if its delivery sequence is the largest number in the batch. A duplicate accepted fail must not start a second compensation, because the compensation key is the step id alone. A hash mismatch at the current generation must hold the step open, and a missing current witness may be replayed once.

The acceptance rule needs an explicit denominator, or a green run will not mean the closer actually converged. I define it as sealed decisions whose generation and hash match the fence, divided by seal attempts in that same fixture run. The numerator excludes rejects, because a reject is a successful protection rather than a sealed compensation step. I would not accept a design that seals any mismatched witness, even if the ratio looks high because most messages are fresh.

Tradeoffs I would put in the review

Choice	Correctness under reorder	Extra latency	Cost	When it fails
Last-write-wins	Breaks when an older pass arrives late	Lowest	Lowest	Stale model lane overwrites a harness fail
Reject on fence miss	Holds the invariant	One comparison	Tiny	Operators may retry blindly and add load
Replay only on a generation match	Fills a missing current witness	Another lane attempt	Bounded by one replay	Useless if the generation is already old
Compensate on an accepted fail	Closes the dangerous case once	Compensation path	Side-effect cost	Unsafe if compensation is not idempotent

Rejecting a stale pass costs almost nothing and avoids a wrong seal that would skip compensation for step S7. Replaying every mismatch looks thorough, and it quietly burns the retry budget of whichever lane is cheapest to call. I would rather hold the step than compensate from a witness that never saw the current fixture hash. Would a slower seal annoy a status page more than a skipped compensation annoys the ledger you cannot easily unwind?

What I would change next

I would split the seal log from any worker that is allowed to scale just because spare capacity is available. The next change is a single writer for seals, with lane processes limited to append-only witness records that cannot close a step. I would also store the input hash beside the fixture hash, so a matching generation still fails closed when the snapshot drifted. After that, I would add a property test that shuffles delivery order and asserts the seal verdict against the fence.

I would not add a second closer for availability until the fence shares a transaction with the compensation key. Two closers that still use last-write-wins simply race each other into the same unguarded assignment on the seal log. A cheaper worker makes that race easier to schedule, which is a reason to be stricter about authority. Would you really scale the closer just because the worker lane became inexpensive to start on spare capacity?

Who should leave this pattern alone

Do not use this fence if the model call itself performs the side effect you might later need to compensate. A judgment witness is not a tool receipt, and rejecting the judgment will not undo a call that already committed. Do not use it if you cannot publish a durable generation before lanes start, because the fence would then be only a comment. Teams that need a non-authoritative lane to share a write key with the canonical store should redesign that boundary before they tune retries.

This simulator also ignores partial network partitions inside a single harness process, and it will not tell you a latency budget. If your review meeting expects a throughput multiplier, this architecture note will disappoint you on purpose. I would rather ship a counterexample with an acceptance rule than a diagram that cannot fail a test. I am also not claiming that free capacity remains available, permanent, or sized for your workload.

The question I want left on the table

Which event order breaks the invariant, and should the system reject, replay, or compensate for it? Take the harness failure at generation 7, then deliver a model pass from generation 6, then deliver that same harness failure again. I would reject the pass, compensate once from the accepted failure, and treat the duplicate failure as a no-op. If your closer still seals the pass, the generation is not an invariant yet, and spare capacity will only make the race cheaper.