DEV Community: Robin

Architecture Review: Give a Free-Tier Inference Endpoint an Admission Envelope, Not a Retry Storm

Robin — Mon, 19 Oct 2026 07:50:55 +0000

Architecture Review: Give a Free-Tier Inference Endpoint an Admission Envelope, Not a Retry Storm

Consider the event order below, which any client with blind retries can produce against a shared inference endpoint. Nothing in it is exotic, and that is exactly the problem.

t0  worker-1 sends request A            (service time unknown, deadline 30s)
t1  worker-1 gives up at 30s            (A is still executing on the server)
t2  worker-1 retries as A'              (no idempotency key, no observation)
t3  worker-2 sends B                    (believes capacity is free)
t4  endpoint returns 429 for A' and B   (server is at its limit, not broken)
t5  both workers sleep ~1s, then retry  (synchronized herd)
t6  endpoint returns 429 again          (queue never drains)

The endpoint was never down. My client simply had no theory of its capacity, so every timeout became a new admission decision made by a process that knew nothing. The invariant I want is narrow and testable: a request may enter the system only if its projected wait fits inside its deadline, and a retry may never outrun the observed service rate. The typical implementation preserves neither half.

This is an architecture review of that boundary — the constraint set, the data flow, the failure domains, and the parts I would change next.

Assumptions (declare them before you measure anything)

A1 Service time is roughly stationary over a probe window of ten minutes; if it is not, your envelope is a snapshot, not a property.
A2 The endpoint publishes no concurrency limit, so the limit must be inferred from observed rejections and latency.
A3 Free-tier capacity is shared and can shrink without notice; treat any measured ceiling as an upper bound that decays.
A4 Work is replay-safe, or every request carries an idempotency key the downstream respects.
A5 The deadline belongs to the caller. The server has no idea what your request was for.

If A1 or A4 is false, stop here and fix that first; admission control on top of non-replayable work just produces faster corruption.

Step 1: Measure the envelope instead of guessing it

Free model access and a free server option are useful precisely because they let you run this probe without a procurement ticket. The offer I am describing is MonkeyCode's free model access plus a free server option; the project advertises a free token allowance (the operator's published figure at the time of writing was on the order of ten million tokens), and both terms can change, so read the console rather than this paragraph. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Here is the probe. It sweeps concurrency, records the status mix, and refuses to average away the tail.

# probe.py — characterize the endpoint; do not trust a single concurrency level
import asyncio, json, os, statistics, time
import httpx

ENDPOINT = os.environ["MC_ENDPOINT"]
KEY      = os.environ["MC_KEY"]

async def one(client, sem, i, out):
    async with sem:
        t0 = time.perf_counter()
        try:
            r = await client.post(
                ENDPOINT, timeout=90,
                headers={"Authorization": f"Bearer {KEY}"},
                json={"messages": [{"role": "user", "content": "reply with: ok"}],
                      "max_tokens": 8},
            )
            out.append({"i": i, "status": r.status_code,
                        "ms": (time.perf_counter() - t0) * 1000,
                        "retry_after": r.headers.get("retry-after")})
        except Exception as e:
            out.append({"i": i, "status": type(e).__name__,
                        "ms": (time.perf_counter() - t0) * 1000, "retry_after": None})

async def sweep(levels=(1, 2, 4, 8, 16), per_level=40):
    async with httpx.AsyncClient() as client:
        for level in levels:
            sem, out = asyncio.Semaphore(level), []
            await asyncio.gather(*(one(client, sem, i, out) for i in range(per_level)))
            ok = [o["ms"] for o in out if o["status"] == 200]
            mix = {}
            for o in out:
                mix[str(o["status"])] = mix.get(str(o["status"]), 0) + 1
            print(json.dumps({
                "level": level, "n": len(out), "ok": len(ok),
                "p50_ms": round(statistics.median(ok), 1) if ok else None,
                "p95_ms": round(sorted(ok)[int(len(ok) * 0.95) - 1], 1) if len(ok) > 20 else None,
                "status_mix": mix,
            }))

asyncio.run(sweep())

Run it. The number you want is not the mean; it is the level at which the rejection mix stops being zero while p50 stays flat. That level is your measured service rate mu, in requests per second, and everything downstream depends on it.

Step 2: Turn the measurement into a concurrency budget

Little's law gives you the only budget that matters: L = lambda * W. At saturation lambda = mu, so the in-flight budget is throughput times the latency you are willing to pay for.

class Envelope:
    """Concurrency budget derived from a measured service rate."""
    def __init__(self, mu_rps: float, deadline_s: float, safety: float = 0.6):
        self.mu, self.deadline = mu_rps, deadline_s
        self.cap = max(1, int(mu_rps * deadline_s * safety))

    def projected_wait(self, inflight: int) -> float:
        return inflight / self.mu          # seconds, saturated server

    def admit(self, inflight: int, service_p50: float, remaining: float):
        if inflight >= self.cap:
            return False, "cap"
        if self.projected_wait(inflight) + service_p50 > remaining:
            return False, "deadline"
        return True, "ok"

The safety factor is not timidity; it is your only defense against A3, because a shared free tier can lose capacity between two probe runs. I set it at 0.6 and I would go lower before I went higher.

Step 3: Classify failures, because they demand different actions

A single except Exception: retry() collapses five distinct failure domains into one response, and four of those responses are wrong.

Observed	Failure domain	Correct action
`429` with `Retry-After`	Shedding by the server	Wait at least that long, then re-admit through the gate
`429` without header	Shedding, no guidance	Back off on the measured `mu`, never on the retry count alone
Timeout / `ReadTimeout`	Unknown outcome on the server	Replay only with an idempotency key
`5xx`	Server-side fault	Bounded retry with jitter, decrement budget
Truncated stream	Partial success	Discard, never stitch; the artifact is unusable

That last row is the one people miss. A stream that ends early looks like a successful request with short output, and downstream code happily persists half an answer.

Step 4: Make the retry floor a tested property, not a habit

The classic bug is multiplicative jitter that lands below the floor the server asked for. Write the delay function so the floor is structural.

import random

class RetryFloor:
    def __init__(self, base=1.0, cap=30.0):
        self.base, self.cap = base, cap

    def delay(self, attempt: int, retry_after=None) -> float:
        floor = min(self.cap, self.base * (2 ** attempt))
        if retry_after is not None:
            floor = max(floor, float(retry_after))
        return floor + random.uniform(0, floor * 0.25)   # jitter additive, cannot go under

Now the properties are assertable: delay >= retry_after, delay >= base * 2**attempt, and delay is non-decreasing in attempt until the cap. If you cannot state those three as tests, you do not have a retry policy; you have a hope.

Step 5: Inject the failures before production does

A fake endpoint with scripted behavior turns this from opinion into a check. The test plan below is the artifact I would actually merge.

Herd test — return 429 for 30% of calls, assert the gate never exceeds cap and no request exceeds its deadline by more than one service time.
Slow-drain test — multiply service time by 3 mid-run, assert the gate shrinks admissions instead of queueing.
Truncation test — close the stream at 60% of tokens, assert the artifact is marked unusable and never persisted.
Replay test — time out after the server has already executed, replay with the same idempotency key, assert exactly one side effect.
Floor test — property-check RetryFloor over 10,000 random (attempt, retry_after) pairs.

Acceptance rule: over a 30-minute run at the chosen cap, p99 queue wait must stay under the deadline, the 429 rate must stay under one percent, and the floor test must have zero violations. Fail any of the three and halve the cap before you touch anything else.

What I would change next

The honest weakness of this design is that mu is a single scalar standing in for a multi-dimensional system. Two lanes with different prompt sizes share one budget, so a burst of long generations starves short ones even though the request count looks fine. The fix I would build next is a two-class gate: measure mu per size bucket, reserve a floor of capacity for the short class, and let the long class consume only the remainder. That is a small change to Envelope and a large change to tail behavior.

Limitations, and who should not use this

Do not adopt this if you need a hard p99 contractual SLO — a shared free tier gives you a measurement, not a guarantee, and the envelope can only shed work, not create capacity. Do not adopt it for non-replayable side effects without idempotency keys, and do not adopt it if your evaluation depends on a stable model identity that a free tier may not promise. Teams without any retry hygiene should fix that first; admission control in front of a retry storm just makes a tidier storm.

If you want a place to run the probe without booking infrastructure, MonkeyCode's free model access and free server option are a reasonable starting substrate — bring your own harness, and replace every number in this template with your own output.

So here is the question I would put to any reviewer of this design: which event order breaks your envelope — a Retry-After shorter than your own floor, a truncation that reports success, or a capacity drop between probe runs? And when it breaks, should the gate reject, replay with a key, or compensate downstream?

Architecture Review: Keep Eval Provenance on the Control Plane, Not the Borrowed Runtime

Robin — Fri, 16 Oct 2026 04:48:57 +0000

Last week I ran a four-shard eval batch against a borrowed runtime, went to sleep, and read the ledger over coffee. One shard had three committed rows, another had none, and the number I had pasted into a doc was an average over attempts rather than over shards. Nothing in the pipeline had thrown an exception, which is exactly why it took me an hour to find. The current cycle of public discussion keeps asking whether AI output holds up under review; far less of it asks whether the harness that produced the number could be trusted at all.

So let me state the invariant I now refuse to ship without. An eval batch is interpretable only if every committed row is attributable to exactly one (job, shard, epoch, model_revision) tuple, and the aggregate's denominator is the shard count, not the row count. Everything below is a consequence of that sentence.

Declared assumptions

I am reviewing a specific class of system here, so the conclusions are only valid inside these assumptions.

The control plane (your CI job, your laptop, your scheduler) is the single writer for a job and the only component allowed to mint attempt identities.
A shard is sealed synchronously with respect to the commit that finalizes it; there is no window where the ledger holds an unsealed authoritative row.
The model endpoint is addressed through a floating alias unless the job spec pins a revision digest explicitly.
Preemption is a normal outcome rather than an error, and the runtime is never required to report it to you.
The batch aggregate is a mean over shards; if yours is a sum over tokens or a p95 over calls, the same argument applies with a different denominator.

Constraints, and where they come from

A free or preemptible runtime gives you three properties whether you asked for them or not: your work can be evicted at any point, your completion rate is not guaranteed, and the underlying model alias may resolve differently between two calls. Those are not defects, they are the contract you accepted in exchange for not paying for a queue. The architecture question is therefore not "how do I make the runtime reliable" but "what must be true on my side so that an unreliable runtime cannot corrupt a number."

Notice how much of the damage happens after the work is done. A preempted attempt is harmless if it publishes nothing; it becomes a correctness bug only when a late worker publishes a row that your ledger accepts. That is where the review should start, and it is why I moved identity and sealing to the control plane.

Data flow: who is allowed to name an attempt

sequenceDiagram
    participant CP as Control plane (single writer)
    participant R as Borrowed runtime (free / preemptible)
    participant M as Model endpoint (floating alias)
    participant L as Ledger (append-only)

    CP->>R: dispatch(key = job/shard/epoch)
    R->>M: resolve alias -> revision
    M-->>R: revision, grade
    R-->>CP: row(key, provenance, grade)
    CP->>L: commit(row)
    CP->>L: seal(shard, epoch)  %% only after the write-ahead ack
    Note over R,L: preemption may drop the row entirely,<br/>and the same attempt may publish again much later
    R-->>L: late row(same key, original or drifted revision)
    L-->>CP: committed | deduped | conflict | rejected_stale

The failure domains I actually observed

Failure domain	Symptom	What it corrupts	Containment
Preemption	attempt returns nothing	shard has no row, so the denominator silently shrinks	retry at a new epoch, never reuse the key
Late publication	row arrives after the shard sealed	an older attempt double-weights the shard	epoch gate evaluated at the ledger
Alias drift	same key, two revisions	two models mixed inside one aggregate	pin the revision per job, quarantine conflicts
Duplicate delivery	identical row twice	nothing, if writes are idempotent by key	first-write-wins per key
Clock skew	"last write wins" by timestamp	nondeterministic winner across replicas	order by control-plane epoch, not wall time

Build the fixture in five steps

The artifact below runs on the standard library alone, and it is a simulator rather than a distributed system. Run it with python3 borrowed_runtime_sim.py; every scenario is deterministic given a seed, so you can argue with the outcome instead of with my prose.

Step 1 — pin the contract. Identity comes from the control plane, and provenance travels with the row instead of being reconstructed later.

"""borrowed_runtime_sim.py — does a preemptible eval runtime preserve provenance?"""
from __future__ import annotations

import hashlib
import random
from dataclasses import dataclass


def sha(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()[:12]


@dataclass(frozen=True)
class AttemptKey:
    job_id: str
    shard: int
    epoch: int          # control-plane authority, strictly increasing per shard


@dataclass(frozen=True)
class Provenance:
    model_revision: str
    prompt_sha256: str
    gateway: str


@dataclass(frozen=True)
class Row:
    key: AttemptKey
    provenance: Provenance
    grade: float

Step 2 — make the ledger classify instead of merge. Four outcomes are possible, and only one of them is a commit. A conflict is not something to resolve by preferring the newer row; it is a signal that the batch must be re-run.

class Ledger:
    """First write per AttemptKey wins. Everything else is classified, never merged."""

    def __init__(self) -> None:
        self.rows: dict[AttemptKey, Row] = {}
        self.sealed: dict[int, int] = {}
        self.conflicts: list[tuple[Row, Row]] = []
        self.rejected_stale: list[AttemptKey] = []
        self.deduped = 0

    def seal(self, shard: int, epoch: int) -> None:
        self.sealed[shard] = max(epoch, self.sealed.get(shard, -1))

    def commit(self, row: Row, *, epoch_gate: bool) -> str:
        if epoch_gate and row.key.epoch < self.sealed.get(row.key.shard, -1):
            self.rejected_stale.append(row.key)
            return "rejected_stale"
        prior = self.rows.get(row.key)
        if prior is None:
            self.rows[row.key] = row
            return "committed"
        if prior == row:
            self.deduped += 1
            return "deduped"
        self.conflicts.append((prior, row))
        return "conflict"

Step 3 — stub the runtime with its real misbehaviours. The stub preempts, duplicates, publishes late, and resolves a floating alias on every call, including on the late one.

class BorrowedRuntime:
    """A free or preemptible executor: it vanishes, duplicates, and arrives late."""

    def __init__(self, revision, alt_revision, rng, cfg):
        self.revision = revision
        self.alt_revision = alt_revision
        self.rng = rng
        self.cfg = cfg
        self.late: list[Row] = []

    def _resolve(self) -> str:
        return self.alt_revision if self.rng.random() < self.cfg.p_alias_flip else self.revision

    def _row(self, key: AttemptKey, prompt: str) -> Row:
        revision = self._resolve()
        return Row(
            key,
            Provenance(revision, sha(prompt), "borrowed-runtime"),
            int(sha(f"{revision}:{prompt}")[:6], 16) / 0xFFFFFF,
        )

    def attempt(self, key: AttemptKey, prompt: str) -> list[Row]:
        if self.rng.random() < self.cfg.p_late:
            self.late.append(self._row(key, prompt))   # published after the seal
        if self.rng.random() < self.cfg.p_preempt:
            return []                                  # we time out and retry later
        row = self._row(key, prompt)
        return [row, row] if self.rng.random() < self.cfg.p_duplicate else [row]

Step 4 — let the control plane retry only unfinished shards. A shard that already holds an authoritative row is never re-dispatched, and the seal happens the moment the commit lands.

@dataclass
class Config:
    shards: int = 4
    epochs: int = 3
    p_preempt: float = 0.30
    p_duplicate: float = 0.20
    p_late: float = 0.25
    p_alias_flip: float = 0.0
    epoch_gate: bool = True
    bump_revision: bool = True


def run(cfg: Config, seed: int) -> Ledger:
    rng = random.Random(seed)
    ledger = Ledger()
    prompts = {s: f"eval-prompt-{s}" for s in range(cfg.shards)}
    carry: list[Row] = []

    for epoch in range(cfg.epochs):
        revision = f"rev-{epoch}" if cfg.bump_revision else "rev-pinned"
        runtime = BorrowedRuntime(revision, "rev-next", rng, cfg)
        for shard in range(cfg.shards):
            if any(r.key.shard == shard for r in ledger.rows.values()):
                continue                       # already authoritative, do not re-dispatch
            key = AttemptKey("job-1", shard, epoch)
            for row in runtime.attempt(key, prompts[shard]):
                if ledger.commit(row, epoch_gate=cfg.epoch_gate) == "committed":
                    ledger.seal(shard, epoch)
        carry.extend(runtime.late)

    for row in carry:                          # the tail: attempts that finished late
        ledger.commit(row, epoch_gate=cfg.epoch_gate)
    return ledger

Step 5 — audit with named properties. The audit returns the denominator explicitly, because a missing shard and a double-counted shard are different bugs that produce the same wrong mean.

def audit(ledger: Ledger, cfg: Config) -> dict:
    by_shard: dict[int, list[Row]] = {}
    for row in ledger.rows.values():
        by_shard.setdefault(row.key.shard, []).append(row)
    revisions = sorted({r.provenance.model_revision for rows in by_shard.values() for r in rows})
    return {
        "rows": len(ledger.rows),
        "shards_with_rows": len(by_shard),
        "missing_shards": cfg.shards - len(by_shard),
        "max_rows_per_shard": max((len(v) for v in by_shard.values()), default=0),
        "committed_revisions": revisions,
        "conflicts": len(ledger.conflicts),
        "rejected_stale": len(ledger.rejected_stale),
        "deduped": ledger.deduped,
    }


def verdict(a: dict) -> str:
    problems = []
    if a["missing_shards"]:
        problems.append("incomplete-batch")
    if a["max_rows_per_shard"] > 1:
        problems.append("denominator-drift")
    if a["conflicts"]:
        problems.append("unresolved-revision-mix")
    if len(a["committed_revisions"]) > 1:
        problems.append("multi-revision-aggregate")
    return ", ".join(problems) if problems else "interpretable"

Read the matrix, not the mean

The baseline property test zeroes out every noise term, so it must pass or the fixture itself is wrong.

# test_properties.py  (pytest)
from borrowed_runtime_sim import Config, audit, run


def test_pinned_gate_is_self_consistent():
    cfg = Config(epoch_gate=True, bump_revision=False, p_alias_flip=0.0,
                 p_preempt=0.0, p_late=0.0, p_duplicate=0.0)
    for seed in range(32):
        a = audit(run(cfg, seed), cfg)
        assert a["max_rows_per_shard"] == 1
        assert a["conflicts"] == 0
        assert len(a["committed_revisions"]) == 1
        assert a["missing_shards"] == 0

Then run the four scenarios and read the verdict field. Exact counts move with the seed, but the violation class should be deterministic given the flags, and that is the thing you are testing.

Scenario	Epoch gate	Revision handling	Designed violation	Required response
A, gate on, pinned	on	one pinned digest	none, except possible missing shards	publish only with `missing_shards == 0`
B, gate on, revision drifts	on	revision per retry epoch	multi-revision aggregate	split the report by revision before quoting it
C, gate off	off	revision per retry epoch	late row lands after the seal	fix the gate, do not re-weight
D, gate on, alias floats	on	resolved per call	same key, two revisions	re-run the batch, never average the conflict

Scenario D is the one that surprised me. The epoch gate is doing its job, the denominator stays clean, and the batch still is not interpretable, because a redelivered attempt resolved the alias to a different revision. Gating and pinning are two different controls, and you need both.

Where a free runtime fits, and where it does not

This is the part of the review that people skip. A free or preemptible runtime is not a cheaper version of a production endpoint; it is a different lifecycle, and it is only honest to use it where lost or duplicated work is a retry rather than a decision.

Workload	Fits a preemptible free runtime?	Why
Harness, ledger, and schema development	Yes	failures are the input to the test, not an accident
Smoke evals on a pinned revision	Usually	bounded retries; accept and report missing shards
Exploratory prompt sweeps	Yes	a lost row costs a retry, not a release gate
Release thresholds and regression gates	No	you do not control the revision or the completion rate
Latency or cost benchmarks	No	the denominator is not yours to define
Long-lived stateful runtimes	No, by construction	eviction is the contract, not an incident

For the first three rows I have been using MonkeyCode for exactly this reason: it exposes free model access plus a free server option, and the operator states a free allowance on the order of ten million tokens, so the cost of running a fixture matrix like the one above is not the thing that stops me. Check the current terms yourself rather than trusting my summary, because allowances change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you want to try the matrix against a live endpoint instead of a stub, that free tier is a reasonable place to start, and the interesting part is not the grades at all; it is whether your ledger refuses the row it should refuse.

What I would change next

Move the seal into the same transaction as the commit, so there is no instant where a shard is finalized but not yet fenced against older epochs.
Put a revision digest into the job spec itself, then treat any row whose revision differs from the digest as uncommittable rather than as a conflict to investigate.
Add a compaction step that converts superseded rows into an explicit superseded_by chain, so a retried shard leaves an audit trail instead of a silent replacement.
Emit missing_shards in the report header, and make the report writer refuse to print an aggregate when that count is nonzero.

Tradeoffs I am accepting

The gate costs one extra piece of state per shard and a small amount of latency on the first commit, and it assumes a single-writer control plane. Strict pinning also removes a convenience you may actually want: you can no longer let a retry pick up a fresher model as a free upgrade, because that upgrade is precisely what makes the aggregate uninterpretable. The conflict path adds a manual re-run step that a lenient ledger would have hidden.

Limitations and who should not use this

The simulator models ordering and attribution, not consensus, and it says nothing about how you would fence two control-plane replicas. Do not adopt the epoch gate if you already have an endpoint with pinned revisions, exactly-once delivery, and a completion guarantee, because you would be adding ceremony to a problem you do not have. Do not adopt it for a batch with one shard and one attempt, where the denominator is trivially one. And make sure your aggregate story matches the gate; a gate that protects a per-shard mean is useless if the number you publish is a total over rows.

So here is the question I want answered from someone else's ledger. Which event order breaks your invariant first: a stale epoch committing after the seal, or a fresh attempt resolving an alias to a new revision? And when that happens, does your system reject the row, replay the shard, or compensate the aggregate, because those three answers lead to very different architectures.

Make Token Spend an Invariant Before a Preemptible Free Tier Replays Your Eval Batch

Robin — Thu, 15 Oct 2026 11:07:07 +0000

Make Token Spend an Invariant Before a Preemptible Free Tier Replays Your Eval Batch

It was 03:12 and shard three of four had already finished 78% of its cases when the worker simply vanished. The driver had no heartbeat, no exit code, and no way to tell whether the last forty graded cases had ever been written anywhere. When the batch finally reported complete, my spend ledger showed roughly 1.4x the tokens I had planned, because two attempts had committed output for the same shard. Would your evaluation harness notice that, or would it just average the two answers and hand you a number you trust?

This is an architecture review of that failure, not a tutorial about model quality. I run cheap classification and grading passes on free model access, and I run the shard workers on a free server option, because both are exactly where prototyping belongs. The catch is that free capacity is preemptible by definition, so the correctness of your results depends on how you fence attempts, not on how good the model is.

Assumptions I am declaring up front

Before any invariant makes sense, the constraint set has to be explicit, otherwise the design is unfalsifiable.

Shard workers can be reclaimed without notice, and no shutdown hook is guaranteed to run.
Result delivery is at-least-once, so a late commit can arrive after a replacement attempt has already finished.
There is no shared clock between the driver, the workers, and the artifact store that is trustworthy enough for ordering.
Model calls dominate cost, so the ledger must be denominated in tokens rather than in wall-clock seconds.
Shards are independent units of work, and each shard must produce exactly one accepted artifact per run.
The free capacity I use is MonkeyCode's free model access for the cheap grading stage and its free server option for the shard workers. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator states a free token allowance on the order of ten million tokens; verify the current figure on the project page before you size a batch around it, because I am not treating that number as a permanent contract.

That last point matters more than it looks. If you assume a fixed allowance and the allowance changes, your batch does not fail loudly — it fails quietly by truncating, which is worse.

The event order that broke my accounting

Here is the counterexample in full, because the bug is only visible in the interleaving.

Driver reserves budget for (run=7, shard=3, epoch=0) and dispatches.
Worker 3 grades 39 of 50 cases and writes a checkpoint after case 30.
The host reclaims worker 3; the lease expires with no heartbeat renewal.
Driver decides the attempt is dead and dispatches (run=7, shard=3, epoch=1).
Worker 3′ grades all 50 cases and commits an artifact for shard 3.
The original worker resumes on a different host, finishes its last 11 cases, and commits a second artifact for shard 3.

Nothing in that sequence is a network partition, a Byzantine node, or an exotic clock problem. It is just a preempted process that came back, and any harness that keys results by shard_id alone will accept both writes. The invariant I was missing is simple to state: for a given run, the committed token spend per shard is a function of the highest epoch that produced an accepted artifact, and lower epochs must never commit.

Model: lease, epoch, and a monotonic spend ledger

I model the driver as a lease grantor and the ledger as a monotonic reservation table keyed by attempt identity.

sequenceDiagram
    participant D as Driver
    participant W as Shard worker (free tier)
    participant L as Spend ledger
    participant S as Artifact store
    D->>L: reserve(run=7, shard=3, epoch=0)
    L-->>D: RESERVED
    D->>W: dispatch(lease, input_digest)
    W->>W: grade cases, checkpoint every 30
    Note over W: preempted at 39/50
    D->>W: re-dispatch epoch=1
    W->>S: PUT artifact(epoch=1, input_digest)
    S-->>D: 201 Created
    D->>L: commit(run=7, shard=3, epoch=1)
    Note over W: epoch=0 host resumes
    W->>L: commit(run=7, shard=3, epoch=0)
    L-->>W: REJECTED (stale epoch)

The important detail is that the ledger rejects on two different grounds: a key that was never reserved, and a key whose epoch is below the highest committed epoch for that shard. Those are different failure classes and they deserve different counters, otherwise you cannot tell a replay from a forged attempt.

Step-by-step: building the fence

1. Key every attempt by `(run_id, shard_id, epoch)`

The epoch is granted by the driver, never by the worker, because a worker that can increment its own epoch can always win. Reservation happens before dispatch, so a budget overshoot is prevented rather than detected.

2. Reserve before the call and commit only after the artifact digest exists

A reservation is a promise, and a commit is a receipt, and conflating them lets a crashed attempt consume budget forever. I keep reservations with a lease expiry so a dead attempt's reservation is reclaimable, but the committed total is append-only.

3. Make every checkpoint carry the input digest

A checkpoint that only records case_index is not resumable, because the fixture may have changed between attempts. Storing the digest of the input slice means a resumed attempt can prove it is continuing the same work, and if it cannot, it restarts the shard instead of mixing two fixture versions.

4. Gate resumption on lease validity, not on liveness

Liveness checks lie under preemption, because a paused host looks alive from the outside. A worker should refuse to write when its lease has expired, even if it has a completed result in memory, and that refusal is the whole point.

A minimal fixture you can adapt

This is a reference sketch, not a benchmark; I have not executed this exact file, so treat it as a fixture to adapt and pin down with your own tests.

"""preempt_sim.py - reference fence for preemptible eval shards (sketch)."""
from dataclasses import dataclass, field

TOKENS_PER_CASE = 37          # declared assumption, not measured
CASES_PER_SHARD = 50
SHARDS = 4
BUDGET_TOKENS = SHARDS * CASES_PER_SHARD * TOKENS_PER_CASE  # 7,400


@dataclass
class SpendLedger:
    reserved: dict = field(default_factory=dict)   # key -> tokens
    committed: dict = field(default_factory=dict)  # key -> tokens
    highest_epoch: dict = field(default_factory=dict)  # shard -> epoch

    def reserve(self, run, shard, epoch, tokens):
        key = (run, shard, epoch)
        if key in self.reserved or key in self.committed:
            return False                  # replay of the same attempt
        if self.spent() + sum(self.reserved.values()) + tokens > BUDGET_TOKENS:
            return False                  # would overshoot the run ceiling
        self.reserved[key] = tokens
        return True

    def commit(self, run, shard, epoch, tokens):
        key = (run, shard, epoch)
        if key in self.committed:
            return "ALREADY_COMMITTED"
        if epoch < self.highest_epoch.get(shard, -1):
            return "STALE_EPOCH"           # a resurrected attempt lost the race
        if self.reserved.get(key) != tokens:
            return "UNRESERVED_SPEND"      # partial or forged attempt
        del self.reserved[key]
        self.committed[key] = tokens
        self.highest_epoch[shard] = epoch
        return "COMMITTED"

    def spent(self):
        return sum(self.committed.values())


def resume_is_safe(checkpoint, input_digest, lease_valid):
    if checkpoint is None:
        return True                        # nothing to trust yet
    if not lease_valid:
        return False                       # expired lease must not write
    return checkpoint["input_digest"] == input_digest

Drive it with a deterministic event queue: reserve, dispatch, deliver, preempt, re-dispatch, and then deliver a late commit from the dead epoch. If your ledger returns STALE_EPOCH on that last event, the fence holds for that interleaving.

Failure injection matrix

Injected failure	Observable symptom	Invariant attacked	Detection	Recovery action
Preemption at 78%	Lease expires, no exit code	Exactly-one artifact per shard	Heartbeat gap plus missing commit	Re-dispatch with `epoch + 1`
Duplicate delivery	Same artifact arrives twice	Spend is idempotent	`ALREADY_COMMITTED` counter	Reject, do not re-bill
Resurrected worker	Late commit from old epoch	Monotonic epoch per shard	`STALE_EPOCH` counter	Reject, log the host identity
Fixture change mid-run	Checkpoint digest mismatch	Result belongs to one input version	Digest compare on resume	Restart the shard, never merge
Clock skew of 90s	Lease appears valid after expiry	Lease validity gates writes	Driver-issued expiry, not worker clock	Shorten lease, renew earlier

Tradeoffs I would accept, and one I would not

Choice	What you gain	What you pay	When to pick it
Driver-issued epochs	Single ordering authority	Driver becomes a coordination point	Always, for at-least-once workers
Checkpoint every 30 cases	Cheap resume after preemption	Slightly more store writes	Shards longer than a few minutes
Reserve-before-dispatch	Budget cannot overshoot	Reservations must expire cleanly	Any tier with hard free limits
Restart on digest mismatch	No mixed-version artifacts	Wasted work	Fixtures change during a run
Worker self-epochs	Fewer round trips	Two writers can both win	Never, in my experience

Limitations and who should not use this

The overhead is real, and a single-worker batch that finishes in ninety seconds does not need a ledger at all. If your results are advisory rather than decision-bearing, a duplicate artifact is annoying but survivable, so skip the machinery. If your preemption rate is near zero on a dedicated host, the lease renewal traffic buys you very little. This design also assumes shards are independent, and a harness with cross-shard aggregation needs a second fence at the reduce step. Finally, my load model above is declared, not benchmarked, so replace those constants before you quote any number from it.

What I would change next

Two changes are on my list. First, I would move from shard-level epochs to token-level reservations inside a shard, so a preemption at 78% does not force a full re-grade of the remaining 22%. Second, I would add an explicit compensation path for the case where a stale artifact was already accepted by a downstream reducer, since rejection at the ledger is too late at that point. Both changes trade throughput for a narrower blast radius, and I would rather pay that than explain a duplicated grade in a release decision.

The acceptance rule I use is blunt: over 500 simulated preemptions, committed spend must stay under the run ceiling, each shard must end with exactly one accepted artifact, and every rejection must carry a classified reason. Which event order breaks your own ledger — a resurrected worker, a duplicate delivery, or a fixture that changed underneath a resume — and should your system reject, replay, or compensate when it does?

If you want to prototype this harness without paying for the first thousand attempts, MonkeyCode's free model access and free server option are a reasonable place to run the shard workers while you build the fence.

Make Authorization Fence an Invariant Before a Side Lane Reopens the Attempt

Robin — Sun, 11 Oct 2026 20:13:17 +0000

I replayed a four-event trace on paper before I trusted a side-lane retry classifier at all. The planner had already sealed compensation when the side lane finally returned its late yes answer. A naive merger treated that late yes as fresh permission to reopen the already sealed attempt. Would you really let one delayed label rewrite a closed attempt after compensation had already landed?

That order is the whole review, not a footnote I can patch later with a longer timeout. I am treating retry authorization as an append that must carry an open generation fence token. If the fence does not match the live attempt, the log rejects the label and leaves the seal untouched. Why would a slower lane earn more authority than the path that already closed the books?

Assumptions I will not smuggle in later

Before I trust the diagram, I want the constraints written where a reviewer can attack them. If any assumption is false in your system, stop and do not borrow the acceptance rule at the bottom. Would you rather discover a missing generation id during review, or after a sealed attempt has already been reopened?

I assume delivery between planner, side lane, and decision log is at least once, so duplicates are normal rather than rare.
I assume the side lane may run on a free server and call a free model, with no latency promise I will write down.
I assume the primary path seals on its own clock and never waits unbounded for a classification to arrive.
I assume each attempt has a monotonic generation, and a seal is terminal for that generation only.

I am not claiming a measured quota, a named model, or a permanent free tier, because those facts were not given to me. I also refuse to invent hardware, duration, or a benchmark just to make the side lane sound faster. What I will claim is narrower: MonkeyCode offers free model access and a free server option for this shadow lane.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I use that free model access only to draft advisory labels, and I host this desk harness on the free server option. A slow advisory label may inform the next generation, but it never becomes the component that seals the current one. If you remove every product name, the fence rule should still be worth keeping on your own bus.

The flow I would actually draw

planner --open(G)--> attempt_log
planner --classify(G)--> side_lane (free server + free model)
side_lane --label(G, yes|no|unknown)--> fence_gate
fence_gate --accept--> attempt_log    only if G is open
fence_gate --reject--> stale_drop     if G is sealed or unknown
planner --seal(G)--> attempt_log

Please read that diagram as an execution path, not as a loose slogan about separation of concerns. The fence gate is the only component allowed to interpret a side-lane label as an authorization. Everything else may propose a label, but nothing else may reopen a generation that has already been sealed. I would rather drop a late yes than let the diagram arrow skip the gate and write the log directly.

Numbered steps for one attempt

Open generation G, persist the open mark, and only then emit the classify request carrying that same generation.
Let the primary path proceed on its own clock, and never hold the seal timer hostage for a classification.
On any arriving label, compare generation G with the log before you read yes, no, or unknown.
If G is open and the label is yes, record one authorization; if G is sealed, append a stale drop and stop.
Seal G from the primary path alone, and ignore every later label that still names that same generation.

Does step three feel redundant when you already trust the queue to preserve your original emit order? I do not trust that, because a restart can deliver the classify response after the seal message has landed. You emitted classify first, yet the log can still observe seal first, and that swap is the bug. If your mental model forbids that swap, your mental model is describing a different bus than mine.

Failure domains I refuse to collapse

The side lane, the free server process, and the decision log are three failure domains I will not merge. A free server restart can drop an in-flight classification while the planner retries the same request under one generation. That retry is a duplicate delivery, not a fresh attempt, so it must not mint a new authorization by accident. A late yes after seal is a stale append, and replaying it would reopen work the compensation already undid.

An unknown label is not a soft yes, because ambiguity must not widen retries once the denominator is closed attempts. What about a yes that arrives before the seal, gets accepted, and then a duplicate yes arrives after the seal? The first append stands, and the second must hit the stale drop, or your exactly-once story is only a hope. I would rather reject the duplicate than compensate again, because a second compensation can undo a valid retry.

A minimal simulator, labeled as unexecuted

I have not run this fixture against a production bus, so treat the script as a desk proposal you can execute locally. It encodes the fence, the seal, and the three outcomes I care about: accept, reject, and duplicate after seal. If the assertions fail, the invariant is already broken, and no model score can talk you out of that. Would you ship a merger that cannot pass this trace, just because the free lane usually sounds confident?

# proposal: fence_sim.py - not a production claim
from dataclasses import dataclass, field

@dataclass
class Attempt:
    generation: int
    sealed: bool = False
    auths: int = 0
    drops: int = 0

@dataclass
class Log:
    attempts: dict = field(default_factory=dict)

    def open(self, g: int) -> None:
        self.attempts[g] = Attempt(g)

    def seal(self, g: int) -> None:
        self.attempts[g].sealed = True

    def on_label(self, g: int, label: str) -> str:
        attempt = self.attempts.get(g)
        if attempt is None or attempt.sealed or label != 'yes':
            if attempt:
                attempt.drops += 1
            return 'reject'
        attempt.auths += 1
        return 'accept'

def permute() -> None:
    orders = [('label', 'seal'), ('seal', 'label'), ('label', 'seal', 'label')]
    for order in orders:
        log = Log()
        log.open(7)
        auths_at_seal = None
        for step in order:
            if step == 'label':
                log.on_label(7, 'yes')
            else:
                auths_at_seal = log.attempts[7].auths
                log.seal(7)
        assert log.attempts[7].auths == auths_at_seal
    print('fence held across injected orders')

if __name__ == '__main__':
    permute()

python3 fence_sim.py

If you want a property check rather than one happy trace, permute seal and label before you argue about scale. Assert that a sealed generation never increases its auth count, even when the same yes is delivered twice. I would add that permutation to the harness before I discussed throughput, because one printed line is not an acceptance rule. The fixture still counts two yes labels before seal as two auths, so idempotent accept is the next change, not a hidden feature.

Acceptance rule and the denominator

The denominator is sealed generations that later receive at least one label, not the total number of model calls. The numerator is those same generations whose authorization count stays unchanged after the seal has been written. I accept this design only when that ratio stays at one across label-then-seal, seal-then-label, and duplicate-yes-after-seal. A free model that is often right does not move this ratio, because the fence is about order, not about label quality.

Tradeoffs I would not hide

Choice	Latency	Correctness	Cost	What breaks
Wait for the side lane before seal	Higher	Stronger if the lane always returns	Free-lane calls sit on the critical path	A slow free server stalls every attempt
Seal without a fence	Lower	Weak	Cheap until a late yes reopens work	Compensation and retry both apply
Fence, reject stale, do not replay	Primary stays short	Seal is stable	Drops late labels	A useful late yes is ignored
Fence plus compensate on stale yes	Primary stays short	Risky	Extra writes	Second compensation undoes a valid retry

I would pick the third row for the decision log, and I would keep label quality on a separate scoreboard. Waiting for the side lane looks safer until a slow free host stalls every attempt that was ready to seal. Sealing without a fence looks cheap until a late yes and a compensation both apply to the same generation. Do you actually need that late yes inside the sealed generation, or only as input when the planner opens the next one?

What I would change next

I would split the authorization store from the decision log so a side-lane bug cannot obtain the seal writer. I would stamp every classify request with generation, attempt id, and a hash of the tool error being judged. I would refuse any label whose hash does not match, even when the generation is still open and the label says yes. I would canary those three failure classes, delayed label, duplicated label, and restarted side lane, and halt if any sealed generation gains an auth.

A rejected late yes may open generation G plus one only through the planner, never through the fence gate itself. Free model access can draft those advisory labels, and the free server can host the harness, within the limits already stated. If you want to try that split, start from the fixture and let only the planner seal, while the free host only proposes labels.

Who should not use this

Skip this pattern if your product requires the classifier to block the seal, because I deliberately removed that wait. Skip it if you have no generation id, since a fence without a stable name is only a comment beside the diagram. Skip it if a human must approve every retry, because this gate only decides whether a label may append at all. I would not use a free lane as the only record of a retry, since I promise neither retention nor uptime.

Which order should break it?

Which event order breaks the invariant: seal, then a duplicate yes, then a planner restart that re-emits the same classify call? I would reject that yes, refuse to replay it into generation G, and refuse to compensate the sealed attempt again. Should your system do the same, or do you still believe a late yes is safer than a dropped label? If you cannot name that order, you do not yet have an invariant, only a hope that the side lane stays fast.

Make Witness Generation an Invariant Before a Late Lane Seals Compensation

Robin — Sat, 10 Oct 2026 16:20:26 +0000

I keep replaying a design-review counterexample from a grading pipeline that looked correct until one late witness arrived. The deterministic harness had already recorded a failure for compensation step S7 under the current fixture generation. A slower judge lane, started against the previous fixture, then returned a pass and reached the closer second. Would you seal that compensation step only because the passing witness happened to be the last write on the wire?

That closer did exactly that, and the skipped compensation left an in-doubt tool call looking finished. I am not citing a production dashboard, a customer incident, or a measured error rate for this walkthrough. I am reviewing the architecture as a state machine, because a pretty sequence diagram will not catch a reordered seal. If the last write can erase an earlier failure, the workflow does not converge, it only looks quiet.

The invariant a common closer drops

The invariant is simple enough to test, and it is still easy to violate under at-least-once delivery between lanes. A witness may seal compensation only when its generation and fixture hash equal the current fence. An older generation must be rejected even if its verdict arrives later and sounds more confident. A matching failure must dominate a matching pass, because skipping compensation is the dangerous direction for this step.

Last-write-wins preserves neither the generation fence nor the fail-dominates-pass rule once two lanes share a closer. It treats delivery order as authority, which is an accident of queues rather than a property of the step. I would not call that pattern a merge, and I would call it an unguarded assignment into the seal log. Have you ever watched a queue make the stale message look like the freshest decision in the log?

Assumptions I will state before the diagram

I assume every lane delivers at least once, and that duplicates can arrive after a seal has already been attempted. I assume the orchestrator can bump a fixture generation before new work starts, and that this bump is durable. I do not assume a shared clock between the worker process and the model lane that judged an older fixture. I also do not assume that a textual pass from a model is equivalent to a property-test failure from the harness.

MonkeyCode is relevant here only as capacity for a non-authoritative lane: the operator describes free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming a quota, a model name, a hardware shape, a duration, or a measured comparison. If you stand up a non-authoritative judge, read the current project terms first, then keep that lane outside the seal path.

Constraints, data flow, and failure domains

The orchestrator owns the fence, the step identity, and the compensation log that later records the seal. The harness domain owns deterministic checks that must run against the published fixture hash for that same generation. The model lane owns a slow judgment that can already be stale the moment a fixture generation bumps. The free server domain owns a worker that can retry, reorder, and deliver twice after an acknowledgement is lost.

sequenceDiagram
    participant O as Orchestrator
    participant H as Harness lane
    participant M as Model lane
    participant C as Closer
    O->>O: Publish fence gen=7 hash=fx-7
    O->>H: Start check S7
    O->>M: Start judge still on gen=6
    H-->>C: Witness fail gen=7 seq=1
    M-->>C: Witness pass gen=6 seq=2
    C->>C: Reject gen=6, seal fail, compensate once

What would I change in that picture if the closer still merged witnesses by delivery sequence number? I would delete any arrow that lets the model lane touch the seal, and I would force a reject branch first. The free server worker may host the harness, but it still may not invent a generation or a fixture hash. Capacity and authority stay in different domains, or the retry budget of one domain will close the other.

Numbered steps for the seal protocol

Publish the fence in the same log that will later record the seal, and refuse new lane starts until that publish is durable.
Bind every witness to the step id, generation, fixture hash, lane id, and a content hash of the inputs it actually saw.
On delivery, reject generation or hash mismatches without enqueueing compensation and without advancing the step to closed.
If an accepted failure exists, seal fail once, then run idempotent compensation keyed by step id rather than by delivery sequence.
Treat a duplicate accepted witness as a no-op, and replay only when the generation still matches and the witness is missing.

I would not replay an older generation hoping the model will eventually change its mind about a fixture it never saw. That replay spends budget to reproduce a stale pass, which is the opposite of convergence under a fence. Rejection is the correct response to a fence miss, even when the late message is fluent and internally consistent. Compensation is the correct response only after an accepted failure, and only through a key that ignores delivery sequence.

A simulator that shows the break

The following fixture is a proposal you can run locally, and it is not a measurement of any hosted service. I have not used it as a load test, and it does not establish a latency, throughput, or cost claim. It only encodes the event order above, plus one duplicate delivery and one foreign fixture hash. Read the assertions before you trust the printed lines, because a printout alone is not the property.

from dataclasses import dataclass

@dataclass(frozen=True)
class Witness:
    lane: str
    generation: int
    fixture_hash: str
    verdict: str
    delivery_seq: int

@dataclass(frozen=True)
class Fence:
    generation: int
    fixture_hash: str

def last_write_wins(witnesses):
    winner = sorted(witnesses, key=lambda w: w.delivery_seq)[-1]
    return {"sealed": True, "verdict": winner.verdict, "lane": winner.lane, "reason": "lww"}

def seal_with_fence(fence, witnesses):
    accepted = [
        w for w in witnesses
        if w.generation == fence.generation and w.fixture_hash == fence.fixture_hash
    ]
    if not accepted:
        return {"sealed": False, "verdict": None, "reason": "reject"}
    if any(w.verdict == "fail" for w in accepted):
        return {"sealed": True, "verdict": "fail", "reason": "accepted_fail"}
    return {"sealed": True, "verdict": "pass", "reason": "accepted_pass"}

def main():
    fence = Fence(7, "fx-7")
    late = [
        Witness("harness", 7, "fx-7", "fail", 1),
        Witness("model-lane", 6, "fx-6", "pass", 2),
    ]
    buggy = last_write_wins(late)
    fenced = seal_with_fence(fence, late)
    assert buggy["verdict"] == "pass"
    assert fenced["verdict"] == "fail"
    duplicated = late + [Witness("harness", 7, "fx-7", "fail", 3)]
    assert seal_with_fence(fence, duplicated) == fenced
    foreign = [Witness("free-server", 7, "fx-other", "pass", 4)]
    assert seal_with_fence(fence, foreign)["sealed"] is False
    print("lww", buggy)
    print("fence", fenced)

if __name__ == "__main__":
    main()

Save that listing as witness_fence.py, then run python3 witness_fence.py from the same working directory you saved it in. You should see the last-write-wins path seal a pass, while the fence keeps the harness failure and rejects nothing incorrectly in this case. If the buggy assertion ever stops matching a pass, you have probably sorted by generation instead of delivery sequence. That change would hide the counterexample rather than fix the closer that still assigns the last write.

Injected failures and the acceptance rule

I inject four failures, and each one has a deny or accept outcome you can assert without waiting on a timer. A reordered older pass must not change a sealed fail, even if its delivery sequence is the largest number in the batch. A duplicate accepted fail must not start a second compensation, because the compensation key is the step id alone. A hash mismatch at the current generation must hold the step open, and a missing current witness may be replayed once.

The acceptance rule needs an explicit denominator, or a green run will not mean the closer actually converged. I define it as sealed decisions whose generation and hash match the fence, divided by seal attempts in that same fixture run. The numerator excludes rejects, because a reject is a successful protection rather than a sealed compensation step. I would not accept a design that seals any mismatched witness, even if the ratio looks high because most messages are fresh.

Tradeoffs I would put in the review

Choice	Correctness under reorder	Extra latency	Cost	When it fails
Last-write-wins	Breaks when an older pass arrives late	Lowest	Lowest	Stale model lane overwrites a harness fail
Reject on fence miss	Holds the invariant	One comparison	Tiny	Operators may retry blindly and add load
Replay only on a generation match	Fills a missing current witness	Another lane attempt	Bounded by one replay	Useless if the generation is already old
Compensate on an accepted fail	Closes the dangerous case once	Compensation path	Side-effect cost	Unsafe if compensation is not idempotent

Rejecting a stale pass costs almost nothing and avoids a wrong seal that would skip compensation for step S7. Replaying every mismatch looks thorough, and it quietly burns the retry budget of whichever lane is cheapest to call. I would rather hold the step than compensate from a witness that never saw the current fixture hash. Would a slower seal annoy a status page more than a skipped compensation annoys the ledger you cannot easily unwind?

What I would change next

I would split the seal log from any worker that is allowed to scale just because spare capacity is available. The next change is a single writer for seals, with lane processes limited to append-only witness records that cannot close a step. I would also store the input hash beside the fixture hash, so a matching generation still fails closed when the snapshot drifted. After that, I would add a property test that shuffles delivery order and asserts the seal verdict against the fence.

I would not add a second closer for availability until the fence shares a transaction with the compensation key. Two closers that still use last-write-wins simply race each other into the same unguarded assignment on the seal log. A cheaper worker makes that race easier to schedule, which is a reason to be stricter about authority. Would you really scale the closer just because the worker lane became inexpensive to start on spare capacity?

Who should leave this pattern alone

Do not use this fence if the model call itself performs the side effect you might later need to compensate. A judgment witness is not a tool receipt, and rejecting the judgment will not undo a call that already committed. Do not use it if you cannot publish a durable generation before lanes start, because the fence would then be only a comment. Teams that need a non-authoritative lane to share a write key with the canonical store should redesign that boundary before they tune retries.

This simulator also ignores partial network partitions inside a single harness process, and it will not tell you a latency budget. If your review meeting expects a throughput multiplier, this architecture note will disappoint you on purpose. I would rather ship a counterexample with an acceptance rule than a diagram that cannot fail a test. I am also not claiming that free capacity remains available, permanent, or sized for your workload.

The question I want left on the table

Which event order breaks the invariant, and should the system reject, replay, or compensate for it? Take the harness failure at generation 7, then deliver a model pass from generation 6, then deliver that same harness failure again. I would reject the pass, compensate once from the accepted failure, and treat the duplicate failure as a no-op. If your closer still seals the pass, the generation is not an invariant yet, and spare capacity will only make the race cheaper.

Make Fragment Epoch an Invariant Before a Restarted Scorer Publishes a Grade

Robin — Fri, 09 Oct 2026 12:13:46 +0000

I still replay a grading-lane fixture I built for this review, because a green dashboard can hide a lying grade. A worker wrote fragment one, the eval server restarted, and a new worker wrote another fragment one. The delayed fragment two from the first epoch then arrived, and the publisher appended it anyway. Would you trust that mixed score inside a canary gate, or would you halt the lane?

The common implementation treats each partial score as an append-only log keyed only by attempt id. That key alone is not enough, because a restart creates a new generation that still shares the attempt. The invariant I want is narrower than idempotent writes: a grade may contain fragments from only one epoch. If two epochs mix, should the publisher reject the late fragment, or should it compensate the grade?

Assumptions I will not quietly expand

I am reviewing an architecture here, and I am not claiming a measured production outage or latency. Assume one attempt id, many fragments, at-least-once delivery, and a scorer that can restart mid-run. Assume the free model call is remote, slow enough to overlap a restart, and not a local deterministic function. Assume you will re-check current product terms, because I will not invent hardware, duration, or a permanence promise.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I would use this open-source project's free model access and free server as the fixture's remote scoring plane. The outreach brief states a free allowance of ten million tokens, which I did not measure myself. Have you confirmed that figure in current docs before sizing fan-out, or are you planning from a screenshot?

Constraints, then the data flow

I am constraining this review to one attempt stream, one publisher, and one remote scoring worker at a time. Cross-attempt fairness, semantic model quality, and multi-region consensus all stay outside the scope of this review. A restart may reorder fragments, duplicate a sequence number, or drop the worker without a clean cancel. The publisher is the only component allowed to make a finished grade visible to the canary gate.

The attempt store mints an epoch, the queue carries that epoch on every fragment, and the worker must echo it. The free server may execute the model call, but it must not choose a new epoch after a crash. A late fragment with a stale epoch is a different failure domain from a duplicate inside the live epoch. Do you see why a write keyed only by attempt id collapses those two domains into one silent append?

sequenceDiagram
    participant Store as Attempt store
    participant Queue as Fragment queue
    participant Worker as Scoring worker
    participant Fence as Epoch fence
    participant Gate as Canary gate
    Store->>Queue: enqueue(attempt, epoch=1, seq=0)
    Worker->>Fence: fragment(epoch=1, seq=0)
    Note over Worker: process restarts
    Store->>Queue: bump epoch=2 and re-enqueue
    Worker->>Fence: fragment(epoch=2, seq=0)
    Queue->>Fence: late fragment(epoch=1, seq=1)
    Fence-->>Gate: reject stale epoch, do not publish mix

Numbered protocol I want in front of publish

I want the protocol to be boring, because boring fences are the ones reviewers can actually test. A fence that lives only in a diagram will not survive the first restarted worker you inject. So I put the check in the publisher, and I also put the epoch on the queue message itself. Why keep the check in both places, if a single publisher check already covers the happy path?

Mint a new epoch when the attempt is admitted, and persist that epoch before the first model call is enqueued.
Stamp the attempt id, the epoch, and the sequence number on every fragment the worker emits, including retries.
Reject any arriving fragment whose epoch does not equal the open epoch currently stored for that attempt.
Treat a repeated epoch and sequence pair as a duplicate acknowledgement, not as a second sample to append.
Publish only when the expected sequence set is complete and the set of epochs inside the grade has size one.
On restart, bump the stored epoch and drop every unpublished fragment that still belongs to the previous epoch.
Surface rejects on a side log so the halt gate can see loss, instead of hiding that loss inside a retry counter.

Failure domains the fence does not magically erase

I split the lane into five failure domains, because a single retry policy will smear them together. The model plane can time out after tokens have already been spent, and that waste is not the same thing as a wrong epoch. The worker process can restart or get preempted, which is the domain this fence is actually aimed at. The queue can reorder or duplicate, and the gate can still read a grade if you publish before the fence runs.

There is a nasty overlap I do not want you to miss when you draw the next box. A zombie worker from epoch one can still be inside a model call after epoch two is opened. The fence can reject its late fragment, but it cannot unspend the call that already left the process. Is that a correctness bug in the grade, or a cost leak I should not hide inside the publisher check?

A fixture you can run before you trust the plane

This next block is an unexecuted local fixture, not a benchmark I collected from a shared server. I want you to run it on your laptop before you point the same state machine at any remote worker. The asserts encode the property, and a green printout only means the fixture held, not that a vendor plane is durable. If a line fails, fix the fence before you debate which model should sit behind the worker.

#!/usr/bin/env python3
'''Local epoch-fence fixture. Not a product benchmark.'''

from dataclasses import dataclass, field


@dataclass(frozen=True)
class Fragment:
    attempt_id: str
    epoch: int
    seq: int
    body: str


@dataclass
class GradeBook:
    open_epoch: dict[str, int] = field(default_factory=dict)
    rows: dict[str, list[Fragment]] = field(default_factory=dict)
    rejected: list[Fragment] = field(default_factory=list)

    def begin(self, attempt_id: str, epoch: int) -> None:
        self.open_epoch[attempt_id] = epoch
        self.rows.setdefault(attempt_id, [])

    def restart(self, attempt_id: str) -> int:
        nxt = self.open_epoch[attempt_id] + 1
        self.rows[attempt_id] = []
        self.open_epoch[attempt_id] = nxt
        return nxt

    def append(self, fragment: Fragment) -> str:
        current = self.open_epoch.get(fragment.attempt_id)
        if current is None or fragment.epoch != current:
            self.rejected.append(fragment)
            return 'reject'
        seen = {(row.epoch, row.seq) for row in self.rows[fragment.attempt_id]}
        if (fragment.epoch, fragment.seq) in seen:
            return 'duplicate'
        self.rows[fragment.attempt_id].append(fragment)
        return 'accept'

    def publish(self, attempt_id: str, expected: int) -> str:
        got = self.rows.get(attempt_id, [])
        epochs = {row.epoch for row in got}
        seqs = {row.seq for row in got}
        if epochs != {self.open_epoch[attempt_id]}:
            return 'halt'
        if seqs != set(range(expected)):
            return 'incomplete'
        return 'publish'


def counterexample() -> None:
    book = GradeBook()
    book.begin('a1', 1)
    assert book.append(Fragment('a1', 1, 0, 'old-sample')) == 'accept'
    assert book.restart('a1') == 2
    assert book.append(Fragment('a1', 2, 0, 'new-sample')) == 'accept'
    assert book.append(Fragment('a1', 1, 1, 'late-old')) == 'reject'
    assert book.publish('a1', expected=1) == 'incomplete'
    assert book.append(Fragment('a1', 2, 1, 'new-tail')) == 'accept'
    assert book.publish('a1', expected=2) == 'publish'
    epochs = {row.epoch for row in book.rows['a1']}
    assert epochs == {2}, epochs
    print(
        'property=single_epoch status=publish rejected=1 '
        'denominator=published_grades numerator=mixed_grades value=0'
    )


if __name__ == '__main__':
    counterexample()

python3 epoch_fence.py
python3 -m py_compile epoch_fence.py

I expect that command to print a single property line, and I expect the late fragment to land in rejected rather than in rows. If your wrapper shells out to a remote worker, keep this process as the publisher and let the worker only propose fragments. Would you let the worker publish directly just because the server was free to start, or would you keep the fence local? A free start is only an availability convenience, and it is not an assignment of epoch ownership.

Injected failures, denominator, and the acceptance rule

I inject four orders before I talk about scale, because scale will only hide the mix. First, deliver sequence zero, restart, deliver a new sequence zero, then deliver the old sequence one late. Second, deliver the same epoch and sequence twice, and demand a duplicate ack instead of a second row. Third, publish before the expected sequence set is complete, and demand incomplete rather than a short grade.

Fourth, leave a zombie fragment in flight across the bump, and demand a reject even if its body looks newer. The denominator is the count of grades that reached a publish status inside this injected suite. The numerator is how many of those published grades contain more than one fragment epoch inside them. Incomplete attempts stay out of the denominator, or a crash will look like a perfect score.

The acceptance rule is a numerator of zero, with every stale-epoch fragment present in the reject log. I would fail the review if a mixed grade is published even once inside that injected suite. I would also fail it if a stale fragment is missing from the reject log after the late delivery. What good is a green gate if the denominator quietly dropped the only attempt that mixed?

Tradeoffs I would write on the review

Choice	Latency effect	Correctness	Cost effect	What still breaks
Reject a stale epoch	One side-log write	Keeps a single-epoch grade	Drops already spent calls	Zombie spend after the bump
Append by attempt id only	Fewest checks	Mixes two samples	Looks cheap until the gate lies	Restart plus reorder
Compensate after publish	Extra rewrite	Works only if the gate has not read	Double write on the grade	A reader that already consumed the mix
Wait for quiescence before publish	Adds a heartbeat wait	Fewer late mixes	Holds a worker slot longer	A server that never sends the final beat

I would take the reject row for this lane, and I would delay compensation until a later review. Replay is the wrong default here, because replaying a stale fragment is how the mix gets built. Does that feel strict when the late body happens to look better than the new one? It should feel strict, because looks better is not an epoch check you can defend in review.

What I would change next

The next change is a fencing token the worker must present on the model call, not only on the fragment. Without that token, epoch two can be open while epoch one is still spending a remote call. I would also split the reject log from the retry counter, so backpressure is a visible depth and not a hidden loop. I would not add a second publisher until this fixture stays at numerator zero under reorder.

Who should leave this pattern alone

Do not use this fence if your side effect is a payment, an email, or any ledger this grade book does not own. Do not use it if you cannot persist the epoch before enqueue, because a minted-but-lost epoch recreates the mix. Do not treat a stated token allowance as a throughput SLO, and do not skip the local fixture. Teams that need semantic judging of model output still need a separate eval, since this protocol only protects fragment membership.

The question I want left open

A remote worker can host the restart injection if you confirm the current free-model and free-server offer first. I would keep the epoch fence in your process either way, and I would halt publish when the numerator moves off zero. I am not asking you to trust a vendor grade, and I am asking you to trust a property you can rerun. Which concrete event order breaks that single-epoch invariant, and should the system reject, replay, or compensate?

Canary Fixture Binding With Identity Failure Classes and a Halt Gate

Robin — Thu, 08 Oct 2026 09:08:48 +0000

The order that already left the lab

Last Tuesday I replayed an agent evaluation that was supposed to remain inside one synthetic lab scope. The third tool call departed carrying a hostname that I had never written into the fixture contract. The planner had already consumed a tool result naming a company domain that looked production-real to me. Why did that worker request another call before any halt gate had a chance to refuse it?

That sequence is not merely a logging gap, and it is not a prompt-quality issue either. The common implementation records a warning, rewrites nothing, and lets the next attempt proceed anyway unchecked. I still want a stricter invariant than one warning line buried inside a long trace file. No tool attempt may leave the worker unless the bound target is a closed fixture contract.

Assumptions I will not quietly expand

I am reviewing the binding path, not the model weights and not the surrounding pager workflow. I assume tool calls are at-least-once, and a timeout does not prove the call never landed. I assume a config refresh can change the host while the run id itself stays stable. I also assume a free isolated runner is available, but I do not assume its size, tenure, or model list.

Those assumptions matter because a stable run id tempts people to treat every refresh as the same world. If the host can change under a stable run id, identity is not a label on the job. Would you still call that binding closed if only the generation field had quietly moved underneath? I would not call it closed, and the simulator below encodes that refusal as an explicit decision.

Constraints that shape the protocol

The fixture contract has four fields that must fully agree before any tool call is issued. The scope must be a reserved non-routable name, and the host must end in .invalid as RFC 2606 defines. The marker must equal the expected synthetic token, and the generation must match the value the issuer holds. A missing field is not a fixture, and a contradictory field is not a fixture either.

RFC 2606 is the factual anchor for the name constraint, not a product claim and not a benchmark. I am not asserting that a string suffix stops a determined worker on a routed network. I am only asserting that the gate must fail closed whenever that string contract actually breaks. If your worker can already resolve production names, this naming protocol is not sufficient by itself.

Data flow from bind to halt

I draw the path as a short sequence, because a box diagram without an execution path has fooled me before. The planner proposes a binding, the gate classifies it, and only then may the worker touch the target. A late refresh is a new event, not a silent mutation of the previous bound event. I keep that late refresh on the diagram so a mutation cannot hide between the boxes.

sequenceDiagram
    participant P as Planner
    participant G as HaltGate
    participant W as Worker
    participant T as FixtureTarget
    P->>G: Bind(run, gen, host, marker)
    G-->>P: FIXTURE or HALT
    Note over P,G: Refresh may swap host, gen unchanged
    P->>W: Issue(call, gen)
    W->>G: Recheck(gen, host, marker)
    alt closed fixture
        W->>T: Call(.invalid only)
        T-->>W: Fixture result
    else live, unknown, or stale gen
        W-->>P: HALT or REJECT, no rewrite
    end

Read that alt branch as the real execution path, not as mere decoration beside the prose. If the recheck happens after the call, the diagram is a lie and the invariant is already gone. I would rather add one extra hop than discover the live host inside a result payload. Does your current worker recheck that binding, or does it simply trust the planner's first bind?

Failure domains I separate on purpose

I split this system into four domains so a fault in one cannot masquerade as health in another. The planner domain owns intent and may retry, but it does not own permission to touch a host. The gate domain owns classification, and it must not share a mutable config object with the issuer. The worker domain owns each attempt, and it must carry the exact generation it was given.

The target domain is either a closed fixture sink or it is entirely out of bounds. A refresh that edits the host sits in the gate domain, even when a planner thread performed the write. A model completion that mentions a real company sits in the planner domain, and it must not become a tool argument. Why would I compensate by rewriting the host and then calling that same target again?

That compensation hides the bad bind and can double-apply a side effect on the far side. I keep the numbered steps explicit because compensation feels helpful and is how the invariant dies. Reject means that a stale generation is discarded, and never replayed against the newly bound host. Halt means a separate report path must look, while the tool path itself stays fully closed.

Numbered steps to install the halt gate

Freeze the binding as an immutable value object before the planner is allowed to request a tool attempt.
Classify that value as fixture, live, or unknown, and treat unknown as halt rather than as a soft allow.
Compare the issued generation with the binding generation, and reject the attempt when they differ at all.
Issue the call only on allow, and refuse any rewrite-and-retry path that tries to launder a live host.
Record the decision against the attempt id so a later report cannot flip an earlier allow.

Replay is the wrong verb once a live or unknown class has already been seen here. I would rather drop the in-doubt attempt than invent a second call that merely looks cleaner. Have you ever watched a quiet retry helper turn a hard halt into a successful-looking trace? That helper is the first change I would delete before I trusted any larger worker fan-out.

A simulator you can execute locally

The following program is a proposal I have not treated as a benchmark, and it carries no latency or quota claims. Please run the file locally with Python on any interpreter that already accepts the syntax below. I use it strictly as the local fixture, not as evidence about any hosted model's accuracy. If your local interpreter rejects the type unions, rewrite those hints and keep every decision identical.

"""Unexecuted until you run it: fixture-binding halt gate."""
from dataclasses import dataclass
from enum import Enum

class TargetClass(Enum):
    FIXTURE = "fixture"
    LIVE = "live"
    UNKNOWN = "unknown"

class Decision(Enum):
    ALLOW = "allow"
    HALT = "halt"
    REJECT = "reject"

MARKER = "synthetic-fixture-v1"
SCOPES = {"lab.invalid", "fixture.local"}

@dataclass(frozen=True)
class Binding:
    run_id: str
    generation: int
    scope: str
    host: str
    marker: str | None
    target_class: TargetClass

def classify(b: Binding) -> TargetClass:
    host_ok = b.scope in SCOPES and b.host.endswith(".invalid")
    marker_ok = b.marker == MARKER
    if host_ok and marker_ok and b.target_class is TargetClass.FIXTURE:
        return TargetClass.FIXTURE
    if b.target_class is TargetClass.LIVE or not host_ok:
        return TargetClass.LIVE
    return TargetClass.UNKNOWN

def decide(b: Binding, issued_gen: int) -> Decision:
    if issued_gen != b.generation:
        return Decision.REJECT
    if classify(b) is not TargetClass.FIXTURE:
        return Decision.HALT
    return Decision.ALLOW

def main() -> None:
    base = Binding(
        "run-7", 3, "lab.invalid", "job.lab.invalid", MARKER, TargetClass.FIXTURE
    )
    cases = [
        ("closed", base, 3, Decision.ALLOW),
        (
            "host-swap",
            Binding("run-7", 3, "lab.invalid", "acme.example", MARKER, TargetClass.FIXTURE),
            3,
            Decision.HALT,
        ),
        (
            "marker-drop",
            Binding("run-7", 3, "lab.invalid", "job.lab.invalid", None, TargetClass.FIXTURE),
            3,
            Decision.HALT,
        ),
        ("stale-gen", base, 2, Decision.REJECT),
        (
            "unknown",
            Binding("run-7", 3, "lab.invalid", "job.lab.invalid", MARKER, TargetClass.UNKNOWN),
            3,
            Decision.HALT,
        ),
    ]
    failed = 0
    for name, binding, gen, expected in cases:
        got = decide(binding, gen)
        print(f"{name}: {got.value}")
        failed += got is not expected
    print(f"failed={failed} denominator={len(cases)}")
    raise SystemExit(failed)

if __name__ == "__main__":
    main()

python fixture_gate.py

After the program prints, that acceptance rule is simple enough for you to check by hand. failed must be zero, and the denominator must stay equal to the number of attempted calls in the suite. I do not accept a pass rate computed over runs, because one run can hide three bad attempts. If you add a case, you must add it to that same denominator before you claim a pass.

Where a free runner fits, and where it does not

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I mention MonkeyCode only because a free model seat and a free server option can host this simulator off production routes. I am not claiming a token allotment, a hardware shape, a duration, or a named model. Those product facts were not supplied for this review, so I simply refuse to invent them. If those seats exist for you, use them to execute the file above and to score traces for halt versus silent continuation.

The free server is useful only when it cannot resolve or write to the systems named in a bad fixture. A model endpoint can help label whether a completion reported the mismatch or simply stayed silent. That label remains a classification task, not a permission to issue the next tool call itself. I would not send live credentials to that runner merely to see what happens next there.

Tradeoffs against an explicit denominator

I compare four responses to a bad bind, and none of them is free of cost. Please read the table as a design choice, not as a measured latency or cost study. Which row would you actually ship if a host swap arrived during a long evaluation suite?

Choice	Extra hop	Correctness under host swap	Cost shape	Failure if misused
Halt on unknown	One recheck	No silent live touch	Retries wasted on bad binds	Slower suite
Warn and continue	None	Live touch remains possible	Cheap until an incident	Invariant dies
Rewrite then retry	One more call	Hides the bad binding	Duplicate side effects	Audit trail lies
Reject stale generation	Compare only	No replay onto a new host	Drops in-doubt work	Must rebind explicitly

The denominator for any claim about this gate is attempted tool calls in the injected suite, not wall-clock samples and not model pass rate. I accept the design only when every non-fixture attempt in that suite halts or rejects, with zero allows. A single allow on host swap, marker drop, or unknown fails the review, even if the other cases look clean. I am not publishing a speedup here, because I have not measured one on any workload.

What I would change next

I would split the classifier into its own process so the issuer cannot read a half-updated config map. I would store generation as a compare-and-swap token, not as a field the planner may overwrite after issue. I would add a report queue that is mandatory on halt, and I would keep that queue off the tool path. I would also delete the rewrite helper, because its existence invites the compensation I just rejected.

Those changes stay inside design and evaluation, and they are not a deployment guide or a monitoring rollout. If your next week is about installing agents on hosts, this review will not carry you there. Would a shared mutable config still scare you after the split, or would you trust the process boundary too quickly?

Who should leave this approach alone

Do not use this simulator if your evaluation workers already share a network route with production tool targets. Do not treat a green local run as a certification, an audit, or proof about any hosted model. Do not adopt it if you need incident paging, dashboard setup, or a cluster install guide, because those jobs live outside this review. Skip it also if you cannot tolerate fail-closed halts, since unknown is supposed to stop the run.

Which order should break it?

Which event order still breaks the invariant: a host swap under a stable generation, a dropped marker, or a stale generation issued after rebind? Should that attempt be rejected, replayed, or compensated, and which of those three keeps the live host untouched? I want the answer as a row in the suite, with the same denominator, before anyone fans the worker out. If your answer is compensate, can you show me the attempt that never touched the swapped host?

Make Admission Depth an Invariant Before a Borrowed Eval Plane Steals Retry Budget

Robin — Thu, 08 Oct 2026 06:27:17 +0000

Last Thursday I watched a planner accept a tool intent while a borrowed eval plane was already drowning. The free server had not crashed, yet its queue sat past the depth I would tolerate on the primary path. A retry from that slow plane came back, grabbed the shared retry budget, and delayed compensation for an unrelated intent. Would you call that spare capacity, or would you honestly call it a leaked failure domain?

The common implementation treats a borrowed eval server as infinite spare throughput that cannot hurt the primary saga. That quiet assumption fails the moment both planes share one retry counter and one admission gate. I want a boring invariant here, and I want it enforced before scale makes the bug look like latency. A plane may borrow model access for scoring, but it may not spend another plane's retry budget.

The violating order, before I draw a box

Here is the order I keep replaying when someone says the free lane is just extra workers. The primary accepts intent I1 and then asks the borrowed plane to score fixture revision R3. The borrowed health sample is already older than my max age, but the gate treats silence as capacity. The borrowed retry then decrements the primary budget, so intent I2 cannot compensate when its own worker times out.

Does that queue depth belong to the borrower, or did I just launder its delay into the primary saga? I am not reopening write-key isolation or outbox dedupe in this review, because those are different invariants. This review asks whether admission depth and retry spend stay inside the failure domain that created them. If the answer is no, I would rather shed the score than replay the attempt on the other plane.

Assumptions I will actually defend

I am assuming one admission decision per intent id, and I am assuming health is a pulled sample with an age. I am assuming the borrowed lane is read-only against the primary write path, so a score cannot commit a tool side effect. I am assuming a retry budget is a finite counter, not a promise that some pool will autoscale in time. I am also assuming free model access can throttle or vanish, and that absence must shed rather than look like a pass.

Those assumptions are narrow on purpose, and they will not survive a multi-region payment saga without more machinery. If your system needs cross-region consensus, stop here and do not pretend this fixture is that protocol. Would a single-process simulator still teach the invariant, even though it cannot simulate a real partition? I think yes, if I inject age and depth instead of pretending I measured a cloud.

The lane I am willing to borrow

I am reviewing this borrowed-capacity design as an architecture problem, not as a product tour or a benchmark claim. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator asked me to treat free model access and a free server option as available for a shadow scoring lane. I am not asserting model names, quotas, hardware, duration, or permanence, because those facts were not supplied as verified measurements.

If a free server can score a fixture, that still does not prove it can absorb your primary retry storm. I would park only shadow scoring there, and I would keep compensation, idempotency keys, and user-visible writes on the primary plane. Can a free option stay useful here without quietly becoming a durability dependency for the primary saga? Yes, but only after the admission fence treats missing capacity as a shed rather than a silent pass.

Data flow across two failure domains

sequenceDiagram
  participant P as Primary saga
  participant A as Admission fence
  participant B as Borrowed eval plane
  P->>A: admit(I1, revision R3, borrow=true)
  A->>A: check primary depth, budget, health age
  A->>B: check borrowed depth, budget, health age
  alt borrowed sample stale or depth at limit
    A-->>P: shed, primary budget unchanged
  else both planes fresh and under limit
    A-->>P: admit score only
    B-->>A: score(R3) or timeout
    A-->>P: bind score to R3, or compensate inside B
  end

The primary domain owns the saga, the compensation path, and any write the user can observe. The borrowed domain owns shadow scores, and it may disappear without a durable outbox of its own. The admission controller is the dangerous middle, because a shared counter turns two domains into one blast radius. I would change that middle first, before I tune prompts or add another worker to the borrowed plane.

Numbered steps I would land before scale

1. Pin a fresh admission lease

I record health_age on each plane and I refuse to admit when that age exceeds health_max_age. A stale healthy bit is not evidence of spare capacity, even if the last sample said the queue was empty. Why would I trust a sample that cannot name the moment it was taken by the control plane? I would rather reject the borrow and score locally, or skip the shadow, than guess at capacity I cannot see.

2. Fence depth and retry budget per plane

Each plane gets its own depth_limit and its own retry_budget, and a shed on the borrowed plane must not decrement the primary counter. If borrowed depth is already at the limit, I stop the score request before it becomes a retry. If the borrowed budget is already zero, I compensate or drop inside that plane, and I leave the primary saga alone. Does a shared integer feel simpler until the first unrelated compensation stalls on the primary path?

3. Bind a score only to the revision it observed

A pass for revision R3 must not attach to revision R4 if the fixture moved while the borrowed worker was slow. I keep that bind check beside the fence, because a late score is a different failure than a deep queue. Should the system reject that late score, replay it, or compensate the attempt on some other plane? Reject the bind, do not replay it onto the primary budget, and compensate only if the borrowed plane had already accepted the attempt.

4. Inject the failures before you trust the diagram

I inject stale borrowed health, borrowed depth at the limit, and primary saturation while the borrower looks idle. I also inject a late score that arrives after the fence has already shed the borrow. The acceptance rule is small enough to automate, and it does not need a hosted pool to fail loudly. Across those injected cases, the primary retry budget must stay unchanged whenever the borrow is shed.

A fixture you can run locally

This is a proposal fixture I wrote for the review, not a benchmark against any hosted pool. Save it as admission_fence.py and run python3 admission_fence.py on any machine with a current Python 3. If an assert fires, the invariant failed in the event order that the failing case names. I am labeling this file as an unexecuted proposal fixture, so a later green run is still not a capacity claim.

#!/usr/bin/env python3
# Proposal fixture: fence admission so a borrowed plane cannot spend primary retries.

from dataclasses import dataclass

@dataclass
class Plane:
    name: str
    depth: int = 0
    depth_limit: int = 4
    retry_budget: int = 3
    health_age: int = 0
    health_max_age: int = 2

@dataclass
class Decision:
    admit: bool
    reason: str

def sample_fresh(plane: Plane) -> bool:
    return plane.health_age <= plane.health_max_age

def admit(primary: Plane, borrowed: Plane, borrow: bool) -> Decision:
    if not sample_fresh(primary):
        return Decision(False, 'reject_stale_primary_health')
    if primary.depth >= primary.depth_limit:
        return Decision(False, 'reject_primary_depth')
    if primary.retry_budget <= 0:
        return Decision(False, 'reject_primary_retry_exhausted')
    if borrow:
        if not sample_fresh(borrowed):
            return Decision(False, 'reject_stale_borrowed_health')
        if borrowed.depth >= borrowed.depth_limit:
            return Decision(False, 'shed_borrowed_depth')
        if borrowed.retry_budget <= 0:
            return Decision(False, 'shed_borrowed_retry')
    primary.depth += 1
    return Decision(True, 'admit')

def consume_retry(plane: Plane) -> str:
    if plane.retry_budget <= 0:
        return 'compensate_or_shed'
    plane.retry_budget -= 1
    return 'retry_same_plane'

def test_borrowed_depth_does_not_spend_primary():
    primary = Plane('primary', retry_budget=3)
    borrowed = Plane('borrowed', depth=4, depth_limit=4, retry_budget=0)
    before = primary.retry_budget
    decision = admit(primary, borrowed, borrow=True)
    assert decision.admit is False
    assert decision.reason == 'shed_borrowed_depth'
    assert consume_retry(borrowed) == 'compensate_or_shed'
    assert primary.retry_budget == before

def test_stale_borrowed_health_is_not_capacity():
    primary = Plane('primary')
    borrowed = Plane('borrowed', health_age=9)
    decision = admit(primary, borrowed, borrow=True)
    assert decision.reason == 'reject_stale_borrowed_health'
    assert primary.depth == 0

def test_idle_borrower_cannot_hide_primary_saturation():
    primary = Plane('primary', depth=4, depth_limit=4)
    borrowed = Plane('borrowed', depth=0, retry_budget=3)
    decision = admit(primary, borrowed, borrow=True)
    assert decision.reason == 'reject_primary_depth'
    assert borrowed.retry_budget == 3

def test_fresh_borrow_retries_inside_its_own_budget():
    primary = Plane('primary', retry_budget=2)
    borrowed = Plane('borrowed', retry_budget=1, depth=0)
    decision = admit(primary, borrowed, borrow=True)
    assert decision.admit is True
    assert consume_retry(borrowed) == 'retry_same_plane'
    assert primary.retry_budget == 2

if __name__ == '__main__':
    test_borrowed_depth_does_not_spend_primary()
    test_stale_borrowed_health_is_not_capacity()
    test_idle_borrower_cannot_hide_primary_saturation()
    test_fresh_borrow_retries_inside_its_own_budget()
    print('admission fence properties held')

Tradeoffs I would accept

Choice	Latency effect	Correctness	Cost shape	Failure I accept
Shared retry budget	Fast until the storm	Breaks the invariant	Hides borrowed cost	Unrelated compensation stalls
Fence depth and shed	More rejects on the shadow	Preserves primary budget	Stops useless borrowed calls	Shadow coverage drops
Queue the borrow forever	Primary looks fast	Late scores bind the wrong revision	Unbounded queue	Reject arrives after commit
Skip borrow when health is stale	More local scoring	Safer admission	Local cost rises	Free plane sits underused

I would accept lower shadow coverage in exchange for a primary budget that can still compensate. I would not accept a diagram that shows two planes while the code keeps one shared integer. The denominator for any later comparison should be shed decisions per one hundred injected intents in this fixture. An acceptance rule that cites a hosted tokens-per-second figure would be fiction, so I am not writing one.

What I would change next

I would replace health_age with a signed lease expiry that the control plane refreshes on a known interval. I would persist each retry counter beside the intent record, instead of leaving it in process memory. I would add an explicit revision watermark so a score cannot cross from R3 to R4 without a new admission. I would also split the shadow lane onto its own queue name, so a dashboard cannot page the primary on borrowed saturation.

Would that be more moving parts than a weekend prototype wants to carry into the first review? Yes, and I would still land the fence before I add the extra queue or another scorer. A prototype that shares a budget is not a smaller design, because the missing fence is a hidden coupling. I would rather explain a shed in review than explain a stalled compensation after the fact.

Who should not copy this

Do not use this fixture as a payment saga, a cross-region consensus protocol, or a proof that a free server is durable storage. Do not use it if your scorer can call tools that write, because this review assumed a read-only shadow on purpose. Do not treat the printed line admission fence properties held as a production certification or as measured capacity of any vendor. If you need exactly-once side effects under duplicate delivery, you still need an outbox and a compensation protocol this file does not implement.

The counterexample I want you to keep

Which event order breaks the invariant, and should the system reject, replay, or compensate the borrowed attempt? Take the case where the borrowed plane hits its depth limit after the primary already admitted a different intent. I would reject the new borrow and refuse to replay that score onto the primary retry budget. I would compensate only an attempt that the borrowed plane had already accepted before that shed.

If your gate does anything else, that free lane stops being spare capacity for the primary. If you want a place to park that shadow scoring lane, read the current free-server terms yourself first. Replay this fixture before either plane is allowed to share a retry budget or an admission counter. Bring your own event order, and ask whether that order should reject, replay, or compensate instead.

Treat Free-Server Eval as a Shadow Protocol Before It Shares a Write Key

Robin — Thu, 24 Sep 2026 18:35:42 +0000

Last month I let a free execution server replay an evaluation job that still carried a production write key. The model stream came back late, and the shadow worker applied that payload to the shared store anyway. Production then saw a committed key and skipped a different body that the planner still expected. Have you ever let a cheap canary plane borrow the same commit identity as the live path?

I keep replaying that order because the failure is not a slow model and it is not a flaky server. The failure is an architecture that treats free capacity as a harmless copy of the production plane. If that copy can still write, then it is not a copy anymore, is it really?

The invariant the common path drops

I want one invariant stated in plain words before I draw any boxes on the board. A shadow evaluation plane may observe live inputs, but it must not share commit keys with the serving plane. It also must not share queues or compensation topics, even when that makes diffs harder to read. The common path stamps the same idempotency key onto the free server job because diffs feel easier.

That shortcut is exactly what lets a late shadow completion bind a live write you did not intend. Would you still call it a shadow if its acknowledgement can move production state without a human check? I would not call it a shadow, because a second writer with a friendly name is still a writer.

Assumptions I am willing to defend

I am reviewing a design rather than reporting a load test, so these assumptions need to stay visible. Free model access and a free server option are availability choices for a canary, not a capacity contract. I do not know your quota, hardware, or how long that option lasts, and I will not invent those numbers. The planner can retry after a stall, and delivery to the execution plane is at least once.

Two planes that share a key namespace are one failure domain, even when their dashboards look completely separate. A free tier can reject, stall, or disappear without becoming a reason to spill work onto the live writer. I will treat any missing capacity number as unknown, rather than as permission to fan out blindly. If your measured limit arrives later, you can drop it into the fixture without changing the fence.

Where I place the canary

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am using the MonkeyCode project's free model access and free server option as the shadow plane in this review. That choice lets me exercise the fence without ever pointing the canary at the serving write path. If you remove the product from the story, the same invariant still holds for any shared execution tier.

How I want the data to move

I draw three planes and one fence, and I refuse to let any arrow skip that fence. The control plane admits a shadow job and rewrites its key before any model call starts. The model plane only proposes a payload, and the free server executes that payload against a shadow store. Nothing on that path should be able to append a record to the live commit log.

planner        control plane       model plane        free server       shadow store
   | admit           |                  |                  |                 |
   |---------------->| rewrite key      |                  |                 |
   |                 |----------------->| propose payload  |                 |
   |                 |                  |----------------->| execute         |
   |                 |                  |                  |---------------->| shadow write
   |                 | fence: reject live: prefix          |                 |
live commit log stays untouched unless prefix == live AND admitted_plane == live

violating order when the fence is missing:
1. control admits job-7 with key live:tenant:debit
2. model plane stalls after the proposal is buffered
3. client retries; free server accepts a second worker
4. late completion from worker A commits amount=10
5. worker B commits amount=12 under the same live key

Does that picture still hold if the free server may call the same tool URL as production? It does not hold, because a shared tool URL without a plane header is another shared write key. Have you seen a network address pretend to be an isolation boundary when the handler is shared? I treat that address as a door, not as a domain, until the handler checks the plane.

Failure domains I would not merge

I split failure by who can change state, not by which vendor logo happens to sit on the box. The model plane can stall, truncate, or return a late completion, and that should only invalidate a proposal. The free server can accept twice, die mid-write, or reject for capacity, and that should only touch the shadow store. The control plane can admit a duplicate when it forgets the rewritten key, and that leak is the dangerous one.

If I merge those domains, a capacity reject on the free server starts to look like a model failure. A retry then replays the same body into the live tool, because both errors were typed as timeout. Have you watched a retry policy follow the wrong plane just because the error string looked familiar? I have watched that happen, and the compensation topic then fired for a job the user never confirmed.

The review I rerun before I trust a diagram

I do not want a slide that says the shadow is isolated unless a fixture can fail the build. This simulator is a proposal fixture I can run locally, and it is not a benchmark of any kind. I am not claiming a throughput number, a model name, or a hardware shape I have not measured. I am staying on the design and the fixture, and I am not walking through deploy steps or a monitoring install.

Declare the key namespaces and the capacity bound as inputs, rather than as comments buried in a client.
Admit the shadow job only after the key is rewritten, and record the plane on the admission lease.
Inject a late model completion plus a duplicate delivery aimed at the free server under test.
Assert that no live key changes, and that a second admit with the same shadow key is rejected.
Fail the review if any event log line contains a live prefix after the shadow run has started.

'''Shadow-key fence fixture. Proposal, not a production client.'''
from dataclasses import dataclass

LIVE = 'live'
SHADOW = 'shadow'

@dataclass
class Write:
    key: str
    plane: str
    body: str

class Store:
    def __init__(self):
        self.rows = {}
        self.rejected = []

    def commit(self, write, admitted_plane):
        prefix = write.key.split(':', 1)[0]
        if prefix != admitted_plane or write.plane != admitted_plane:
            self.rejected.append(write)
            return False
        if write.key in self.rows:
            self.rejected.append(write)
            return False
        self.rows[write.key] = write
        return True

def rewrite(step, run):
    return f'{SHADOW}:{run}:{step}'

def late_duplicate_order():
    store = Store()
    key = rewrite('debit', 'run-19')
    first = Write(key, SHADOW, 'amount=10')
    late = Write(key, SHADOW, 'amount=10')
    leaked = Write('live:tenant:debit', LIVE, 'amount=10')
    ok1 = store.commit(first, SHADOW)
    ok2 = store.commit(late, SHADOW)
    ok3 = store.commit(leaked, SHADOW)
    return ok1, ok2, ok3, store

def live_untouched(rows):
    return all(not key.startswith('live:') for key in rows)

if __name__ == '__main__':
    ok1, ok2, ok3, store = late_duplicate_order()
    assert ok1 and not ok2 and not ok3
    assert list(store.rows) == ['shadow:run-19:debit']
    assert live_untouched(store.rows)
    print('fence_held', len(store.rejected))

python shadow_fence.py

I also want a property that does not depend on one scripted order in a single file. Generate permutations of stall, duplicate delivery, and capacity reject, then assert the live prefix never appears. If one permutation breaks that property, the design does not converge, no matter how clean the diagram looked. Would you really promote a design that passes one happy path and then fails the permutation test?

When I read the rejected list, I want every crossed-plane write to show up there, with its original key intact. A silent drop would hide the bug, and a rewrite of the leaked key would launder a live prefix into a shadow prefix. Have you ever fixed a test by normalizing the key before the assertion, and then missed the leak entirely? I keep the raw key in the rejection record so the next reviewer can see which plane tried to cross.

My load model is deliberately tiny, because the question is convergence under reordering, not a peak rate claim. I set capacity to one in-flight shadow job so a second admit must fail closed instead of borrowing serving room. You should replace that bound with your own measured limit, and you should not copy a number I refused to invent. If the free server starts rejecting for capacity, the control plane should queue or shed, not spill into the live plane.

The tradeoff I could not wave away

I could not wave this table away, because each row buys correctness with a different operational cost. I am not claiming a speedup, because I have not run a paired workload on a named model or machine. The denominator I care about is the count of event orders that violate the fence, not tokens per second. Acceptance rule: zero live keys may be written across the injected failure set, including a duplicate after a stall.

Choice	Latency effect	Correctness	Cost posture	What breaks
Shared write key on the free server	Fewer lookups	Shadow can bind live state	Looks cheap until replay	Late completion commits the wrong body
Rewritten shadow key plus a fence	One prefix check	Live prefix is rejected	Canary stays on free access	Diffs need a translation step
No free plane at all	Probes pay the serving path	Strong only if serving is fenced too	Higher for every review run	Failure injection gets skipped

What I would change next

I would stop letting the free server select tools by URL alone, without a plane check beside the key. I would require a plane header that the tool gateway compares with the key prefix before it accepts work. I would drop the request when they disagree, and I would also split compensation topics by plane. A shadow timeout must not enqueue a live undo, and an older run id must not join a newer comparison.

Should the gateway reject, replay, or compensate when the header and the key prefix disagree on a write? I want the gateway to reject, because replay would only amplify a client that is already confused. Compensation assumes a live write that the fence should have blocked before any side effect started. Would replaying a crossed-plane request ever make the invariant stronger than a hard reject at the gateway?

Who should leave this pattern alone

Do not use a free shadow plane as your only correctness proof when serving auth and data differ. Do not point the free server at production tools while you tell yourself that you are only testing prompts. Do not treat unknown free capacity as a queue you can fan out without some explicit admission bound. If a regulator requires an attested environment, a free shared server is the wrong domain for that evidence.

This review also will not tell you which model is smarter, because I am not publishing scores I did not measure. It will only tell you whether a late proposal from the free plane can still move a live key. If your serving path has a different retry budget, a green shadow run is not evidence that production will converge.

The order I want you to break

Which event order breaks the invariant for your system, and should that system reject, replay, or compensate? Take one shadow job, stall the model, deliver the execution twice, and sneak one live prefix into the batch. If the store accepts that prefix, the free server was never a shadow, and it was a second writer you forgot to fence. If you want a bounded place to run that fixture on free model access and a free server, keep the write key fenced.

Treat Tool Intent as a Deduped Outbox Before Planner Retries Double-Commit

Robin — Wed, 23 Sep 2026 16:03:50 +0000

Last Tuesday I sat with a sequence dump that looked successful until I counted the tickets twice. The planner had asked for one incident record, and the board showed two identical rows minutes apart. Have you ever trusted a tool-failed retry line without asking whether the side effect already landed? I had trusted that line, and that is exactly how this architecture review started for me.

The violating order was simple, and it did not need a clever adversary to appear in production. The worker posted the ticket, then the process vanished before the result reached the planner queue. The planner treated silence as failure, emitted the same step again, and a second worker committed. Does a missing ack mean the tool never ran, or only that nobody survived to say so?

The invariant the common retry loop drops

I want one planner step to produce at most one committed side effect, unless named compensation runs. That sentence is the invariant, and everything else in this review exists to protect or refute it. Most agent diagrams draw a happy arrow from model output to tool call to observation, then bolt on retries. Is a timeout a hard failure you may replay, or an unknown that still owns the side effect?

I am reviewing a two-queue tool bus, not narrating a named outage from a company I will not invent. The fixture below is a proposed discrete-event model, not a production service I operate. If a design cannot name the reject, replay, and compensate paths on paper, I will not trust its retry flag in a real agent.

Declared assumptions

I am assuming at-least-once queues, crash-stop workers, and a planner that re-emits a step after timeout. I am assuming external tools are not naturally idempotent, because ticket and mail APIs usually are not. I am assuming the planner may sit on a shared model endpoint, while the worker may die without a graceful drain. I am not assuming exactly-once brokers, sticky sessions, or a private cluster with reserved capacity.

Constraints I would not negotiate

I will not let the model transcript be the source of truth for external writes, because transcripts are not ledgers. I will not let a planner timeout delete an in-flight intent, because silence is not an abort. I will not accept a tool adapter that mints a new external request whenever the queue redelivers a payload. If those three constraints feel expensive, the double-ticket failure is the cheaper-looking alternative that I refuse.

Here is the constraint set I use when I review the bus:

Every planner step must mint a stable intent_id before any worker may touch the outside world.
The outbox, not the model message, is the only record that authorizes a side-effecting call.
A worker may execute an intent only after a compare-and-set from queued into executing.
A late result for a superseded generation must be rejected, not folded into the next observation.

Which of those four would you drop first under schedule pressure, and what event order would punish that drop?

Data flow of the tool bus

I want the planner, the outbox, the worker, and the tool API in separate failure domains. The model may hallucinate a repeated tool call, and the queue may redeliver, but the outbox still owns the write. The result queue carries observations back, and it must key them by intent_id plus generation, not by arrival time. If a result cannot name those keys, I treat it as noise and I do not advance the plan.

sequenceDiagram
    participant P as Planner
    participant O as Intent outbox
    participant W as Tool worker
    participant T as External API
    participant R as Result queue
    P->>O: put(intent_id, gen, tool, args)
    O->>W: deliver at-least-once
    W->>O: CAS queued to executing
    W->>T: call with Idempotency-Key
    Note over W,R: worker may die here
    W->>R: observe(intent_id, gen, result)
    R->>P: bind only if gen still current
    P->>O: timeout does not mint a new intent_id

Happy path, written as a protocol rather than a vibe:

The planner assigns intent_id and generation zero, then inserts the row in queued state.
A worker claims the row with a compare-and-set, then calls the tool with that same key on the wire.
The worker writes the observation to the result queue, then marks the outbox row committed or failed.
The planner consumes the observation only when the generation still matches the live step.

If step two fails the compare-and-set, the second worker must stop. Replay is a delivery fact, not a license to spend again. Compensation is a different protocol, and I will not pretend a second create is a compensate.

Failure domains I actually budget for

I split the design into four domains because they fail on different clocks. The planner domain fails by timeout, duplicate decode, or a model that repeats a tool call with new prose. The outbox domain fails by lost write, lost ack, or a crash between state transitions. The worker domain fails by preemption, slow tool I/O, or a redelivered message after a successful post. The tool domain fails by accepting a body twice when no idempotency header was sent.

The dangerous coupling is the planner treating worker silence as a negative observation. That coupling turns one crash into two commits, and it hides inside retry middleware that looks responsible. Do we reject the late result, replay the same intent, or enqueue a compensate? If the outbox still says executing, I replay nothing and I wait or I escalate.

I also budget for reordered results, because preempted workers come back at the worst time. A generation that the planner already abandoned must not bind the next step. That is not a logging problem. That is a binding rule, and the result consumer has to enforce it.

A minimal simulator you can run

I want a fixture that can print the double-commit without standing up Kafka. The model below is discrete-event Python, and it is intentionally small enough to argue about in review. It is not a performance test, and I am not attaching invented latency numbers to it. Run it locally, then change the event order until the invariant breaks or holds.

# proposed fixture: not a production bus, not a benchmark
from dataclasses import dataclass
from typing import Dict, List, Optional, Tuple

QUEUED, EXECUTING, COMMITTED, FAILED = "queued", "executing", "committed", "failed"

@dataclass
class Intent:
    intent_id: str
    gen: int
    tool: str
    args: dict
    state: str = QUEUED
    side_effects: int = 0

class Outbox:
    def __init__(self) -> None:
        self.rows: Dict[str, Intent] = {}

    def put(self, intent: Intent) -> None:
        if intent.intent_id in self.rows:
            return  # dedupe: same intent_id never mints a second row
        self.rows[intent.intent_id] = intent

    def claim(self, intent_id: str, gen: int) -> bool:
        row = self.rows.get(intent_id)
        if not row or row.gen != gen or row.state != QUEUED:
            return False
        row.state = EXECUTING
        return True

    def complete(self, intent_id: str, gen: int, ok: bool) -> None:
        row = self.rows[intent_id]
        if row.gen != gen:
            return  # stale generation: reject
        row.state = COMMITTED if ok else FAILED

class Tool:
    def __init__(self) -> None:
        self.seen: Dict[str, str] = {}
        self.commits = 0

    def call(self, idem_key: str, body: dict) -> str:
        if idem_key in self.seen:
            return self.seen[idem_key]
        self.commits += 1
        ticket = f"T-{self.commits}"
        self.seen[idem_key] = ticket
        return ticket

def run(events: List[Tuple[str, dict]]) -> Dict[str, int]:
    outbox, tool = Outbox(), Tool()
    planner_timeouts = 0
    rejected_stale = 0
    for name, payload in events:
        if name == "plan":
            outbox.put(Intent(**payload, state=QUEUED))
        elif name == "claim_and_call":
            iid, gen = payload["intent_id"], payload["gen"]
            if not outbox.claim(iid, gen):
                rejected_stale += 1
                continue
            ticket = tool.call(iid, outbox.rows[iid].args)
            outbox.rows[iid].side_effects += 1
            if payload.get("ack", True):
                outbox.complete(iid, gen, True)
                payload["ticket"] = ticket
        elif name == "planner_timeout_replay":
            planner_timeouts += 1
            row = outbox.rows[payload["intent_id"]]
            # illegal: minting a new intent_id on timeout
            if payload.get("mint_new_id"):
                new_id = row.intent_id + "-retry"
                outbox.put(Intent(new_id, 0, row.tool, row.args))
            # legal: redeliver the same intent_id
        elif name == "late_ack":
            iid, gen = payload["intent_id"], payload["gen"]
            if outbox.rows[iid].gen != gen:
                rejected_stale += 1
            else:
                outbox.complete(iid, gen, True)
    return {
        "external_commits": tool.commits,
        "planner_timeouts": planner_timeouts,
        "rejected_stale": rejected_stale,
        "outbox_rows": len(outbox.rows),
    }

BAD = [
    ("plan", {"intent_id": "step-7", "gen": 0, "tool": "create_ticket", "args": {"title": "db blip"}}),
    ("claim_and_call", {"intent_id": "step-7", "gen": 0, "ack": False}),
    ("planner_timeout_replay", {"intent_id": "step-7", "mint_new_id": True}),
    ("claim_and_call", {"intent_id": "step-7-retry", "gen": 0, "ack": True}),
    ("late_ack", {"intent_id": "step-7", "gen": 0}),
]

GOOD = [
    ("plan", {"intent_id": "step-7", "gen": 0, "tool": "create_ticket", "args": {"title": "db blip"}}),
    ("claim_and_call", {"intent_id": "step-7", "gen": 0, "ack": False}),
    ("planner_timeout_replay", {"intent_id": "step-7", "mint_new_id": False}),
    ("claim_and_call", {"intent_id": "step-7", "gen": 0, "ack": True}),  # claim should fail
    ("late_ack", {"intent_id": "step-7", "gen": 0}),
]

if __name__ == "__main__":
    print("bad ", run(BAD))
    print("good", run(GOOD))
    assert run(BAD)["external_commits"] == 2
    assert run(GOOD)["external_commits"] == 1

The acceptance rule is explicit, and I want it in the same file as the events. external_commits must stay one for a single planner step when the worker dies before ack. The bad trace mints a new identifier on timeout, and the tool cannot see that the two calls are the same intent. The good trace redelivers step-7, the second claim fails, and the late ack still binds generation zero.

Injected failures and how I would run them

I do not trust a green happy path when the interesting bugs live in reordering. The commands below assume the fixture is saved as tool_outbox_sim.py in the current directory. They do not measure tokens, dollars, or hosted throughput, because I will not invent those figures.

Run the paired traces and keep the assertion on external_commits as the gate, not a log skim.
Flip mint_new_id to true and confirm the fixture fails, because that is the common planner bug.
Drop the Idempotency-Key path in Tool.call and watch a legal redelivery become a second spend.
Insert a second late_ack with gen=0 after you have advanced gen to one, and demand a reject.

python tool_outbox_sim.py
python -c "from tool_outbox_sim import run, BAD, GOOD; print(run(BAD), run(GOOD))"

If you add a real HTTP tool, send the outbox intent_id as Idempotency-Key and treat any 2xx without that echo as a protocol miss. An empty success body is a separate discussion I will not reopen here. The property I care about is simpler: one live intent, one external commit, or a named compensate row.

Tradeoffs, with a denominator

I am comparing designs per planner step, not per day and not per model call, because those denominators hide duplicates. The table is a review aid. It is not a vendor scorecard.

Design	What it preserves	What it costs	When it still breaks
Retry on planner timeout, new tool payload	Almost nothing	Extra calls that look like diligence	Worker died after commit
At-least-once queue, no outbox row	Delivery, not intent	Duplicate spends under redelivery	Any crash between post and ack
Deduped outbox plus compare-and-set claim	At most one executor	An extra durable write before I/O	Tool ignores the idempotency key
Outbox plus compensating inverse	Liveness after a true stuck execute	You must write the inverse, and it can fail too	Compensate timeout with no idempotency

I would rather pay the extra durable write than explain two tickets to an on-call thread. Throughput can wait until the invariant holds under injected preemption. If your load model cannot name concurrent in-flight intents, you are not measuring the bus. You are measuring hope again.

What I would change next

I would split executing into executing and awaiting_ack, because those are different recovery actions. I would put a bounded wait on awaiting_ack, then move to a compensate intent with its own identifier derived from the original. I would fence result binding with the generation, and I would reject any observation that cannot quote both keys. I would not add more planner retries until those states exist in the outbox schema.

I would also stop letting the model invent tool arguments on replay. Replay must load the stored args from the outbox row, not from a freshly sampled completion. Otherwise the second call is not a retry. It is a different write that only rhymes with the first. Can your current agent runtime even show you the stored args without reading a chat log?

Limitations, and who should skip this

This review does not give you a hosted topology, a queue product, or an incident runbook. The simulator is single-threaded and discrete, so it will not catch disk-ordering bugs or clock skew across regions. Free shared processes are useful for forcing timeouts, and they are the wrong isolation boundary for customer-facing writes. If you need a hard latency SLO, dedicated workers, or regulated audit storage, do not use this fixture as a deployment plan.

Skip the outbox if your tools are truly pure reads and a duplicate call cannot spend money or create records. Skip it if you already have a transactional outbox in the same database as the agent state, and your workers already claim with fencing. Do not skip it if a model timeout currently mints a new tool payload. That path is the double-commit I opened with.

Which event order still breaks the invariant in your bus, and should the system reject, replay, or compensate? If the answer is only "retry the tool," I would send the design back before any planner scales up.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free models that can run this workflow. A free server option is enough to reproduce the setup.

Treat Client Timeout as an In-Doubt Lease Before You Replay Shared Tool Calls

Robin — Tue, 22 Sep 2026 14:21:42 +0000

Last Tuesday I sat with a planner log that looked almost boring until I lined up two clocks. The client aborted a tool call at thirty seconds, then immediately replayed the same payload against a shared slot. The original call finished four seconds later, and the retry still executed against that reservation. Two side effects landed in the same conversation epoch, and nobody had a lease to point at.

That event order is not an incident curiosity, and it is the default protocol most agent backends implement. A client timeout is not an abort, and a shared free slot does not make that physics kinder. I keep reviewing these loops as if HTTP were a transaction, which it never was. So I want the architecture to name the in-doubt lease before anyone hits replay again.

Have you ever watched a retry storm and blamed the model for duplicate tool work? I have, and the model was the least interesting process in the trace.

The invariant the common retry loop fails to preserve

I want one invariant, stated before any diagram or fixture. A side-effecting tool result may bind if and only if three tokens still match: lease_id, idempotency_key, and conversation_epoch. Anything else is an in-doubt write, not a clean miss, and it must not apply.

Why three tokens instead of a single request id on the HTTP client? Because the failure domains are not the same plane, even when they share a TCP connection. The client timer lives in the planner process, the lease lives on the shared slot, and the epoch lives in the conversation store. Collapse those planes and you will bind a late body to the wrong run.

Declared assumptions

I am reviewing a planner that issues tool calls onto a shared remote executor, not a dedicated box you fully own. The executor may reclaim a slot when a lease expires, even if a tool body is still in flight. Side-effecting tools are not locally transactional with the model stream, and duplicate delivery is expected after timeout, reconnect, or planner restart.

I am not assuming named models, published quotas, dedicated hardware, or a permanent free tier. Those would be invented product claims, and they are not required for the race. I am assuming only that a free shared slot can exist long enough to admit work and then disappear under another conversation. If your tools are strictly read-only, this review is heavier than you need.

This fixture is a design proposal. It is not a production measurement, and I am not reporting live traffic.

Data flow: treat admission as a protocol, not an HTTP POST

Here is the sequence I want the architecture to speak, not the sequence most SDKs actually speak when a timeout fires.

The planner creates an idempotency key bound to the current conversation_epoch.
The executor admits a reservation and returns a lease_id with an explicit expiry.
The tool writes a prepare record inside that lease before any external side effect.
Completion is a bind, not a print: match lease, key, and epoch, then commit.
On client timeout, the planner marks the call IN_DOUBT and does not replay yet.
A recover path commits, compensates, or rejects; it never blind-replays the payload.

Does your current client already skip step five? Mine did, for years, because timeouts feel like negatives and negatives feel safe to retry.

sequenceDiagram
    participant Planner
    participant Admit as AdmitGate
    participant Slot as SharedSlot
    participant Store as EpochStore
    Planner->>Admit: reserve(key, epoch)
    Admit-->>Planner: lease_id, expiry
    Planner->>Slot: execute(tool, lease_id)
    Note over Planner: client timer fires; mark IN_DOUBT
    Slot->>Store: bind(lease_id, key, epoch, result)
    alt tokens match
        Store-->>Slot: COMMIT
    else lease or epoch mismatch
        Store-->>Slot: REJECT
    end
    Planner->>Admit: recover(key, epoch)
    Admit-->>Planner: commit | compensate | reject

The HTTP POST is only the transport. If your sequence diagram starts at requests.post and ends at status 200, you have not modeled the bind.

Failure domains

I split the system into four domains so retries stop crossing wires they do not own.

Planner timer domain. A thirty-second client timeout fires while the executor is still inside a valid lease. The planner knows nothing about that lease. The work may still commit, and a replay now is a second admit.

Lease domain. The shared slot expires and is granted to another conversation. A late body can arrive carrying a stale lease_id. Binding it is a consistency bug, not a helpful flush of leftover tokens.

Idempotency domain. The same key is admitted twice because the first admit response was lost on the way back. Two prepare records exist. One must win, and the other must no-op without a second side effect.

Epoch domain. The user started a new turn, or a worker preemption bumped conversation_epoch. A completion from the previous turn must not append, even if the lease still looks alive in a local cache.

Notice what I did not put in this review: dashboards, deploy pipelines, or paging rotations. Those are operations concerns. This article stops at whether the protocol converges under a known event order.

A minimal in-doubt simulator

I wanted an executable counterexample I could step in one file, not a slide about retries. The fixture below models a shared slot, a client timeout that does not abort the server, and a bind gate that either commits or rejects. Label it unexecuted until you run it locally.

from dataclasses import dataclass, field
from typing import Dict, Optional, Literal

Decision = Literal["COMMIT", "REJECT", "IN_DOUBT", "COMPENSATE"]

@dataclass
class Reservation:
    lease_id: str
    key: str
    epoch: int
    expiry: int
    prepared: bool = False
    applied: bool = False

@dataclass
class Slot:
    now: int = 0
    current: Optional[Reservation] = None
    applied_side_effects: int = 0
    binds: Dict[tuple, Decision] = field(default_factory=dict)

    def admit(self, lease_id: str, key: str, epoch: int, ttl: int) -> Reservation:
        if self.current and self.now < self.current.expiry:
            raise RuntimeError("slot still leased")
        self.current = Reservation(lease_id, key, epoch, self.now + ttl)
        return self.current

    def prepare(self, lease_id: str, key: str, epoch: int) -> None:
        r = self._require(lease_id, key, epoch)
        r.prepared = True

    def bind(self, lease_id: str, key: str, epoch: int) -> Decision:
        token = (key, epoch)
        if token in self.binds and self.binds[token] == "COMMIT":
            return "REJECT"  # duplicate bind never re-applies
        r = self.current
        if not r or r.lease_id != lease_id or r.key != key or r.epoch != epoch:
            self.binds[token] = "REJECT"
            return "REJECT"
        if self.now >= r.expiry or not r.prepared:
            self.binds[token] = "REJECT"
            return "REJECT"
        if not r.applied:
            r.applied = True
            self.applied_side_effects += 1
        self.binds[token] = "COMMIT"
        return "COMMIT"

    def recover(self, key: str, epoch: int) -> Decision:
        token = (key, epoch)
        decision = self.binds.get(token)
        if decision == "COMMIT":
            return "COMMIT"
        if decision == "REJECT":
            return "COMPENSATE"
        return "IN_DOUBT"

    def _require(self, lease_id: str, key: str, epoch: int) -> Reservation:
        r = self.current
        if not r or r.lease_id != lease_id or r.key != key or r.epoch != epoch:
            raise RuntimeError("lease mismatch")
        if self.now >= r.expiry:
            raise RuntimeError("lease expired")
        return r


def timeout_then_late_success() -> Slot:
    s = Slot()
    s.admit("L1", "k", 7, ttl=40)
    s.prepare("L1", "k", 7)
    s.now = 30  # client timeout; planner must NOT replay
    assert s.recover("k", 7) == "IN_DOUBT"
    s.now = 34
    assert s.bind("L1", "k", 7) == "COMMIT"
    # Blind replay would try a second admit; the bind gate must no-op.
    assert s.bind("L1", "k", 7) == "REJECT"
    assert s.applied_side_effects == 1
    return s


def lease_recycle_then_late_body() -> Slot:
    s = Slot()
    s.admit("L1", "kA", 1, ttl=10)
    s.prepare("L1", "kA", 1)
    s.now = 11
    s.admit("L2", "kB", 2, ttl=10)
    assert s.bind("L1", "kA", 1) == "REJECT"
    assert s.applied_side_effects == 0
    return s


def lost_admit_ack() -> Slot:
    s = Slot()
    s.admit("L1", "k", 3, ttl=20)
    s.prepare("L1", "k", 3)
    s.now = 5
    # Planner never saw L1; a naive retry would admit again and double-apply.
    try:
        s.admit("L2", "k", 3, ttl=20)
        raise AssertionError("second admit must fail while lease is live")
    except RuntimeError:
        pass
    assert s.bind("L1", "k", 3) == "COMMIT"
    assert s.applied_side_effects == 1
    return s


if __name__ == "__main__":
    timeout_then_late_success()
    lease_recycle_then_late_body()
    lost_admit_ack()
    print("invariant held for the three injected schedules")

How would you run it? Save the file and execute the three schedules before you trust any retry wrapper. The point is not throughput. The point is whether the invariant holds after each injected order.

python in_doubt_lease.py

If you comment out the duplicate-bind guard and allow a second admit after the client timer, you get the original outage in twelve lines. That is the counterexample I want in review, not a narrative about flaky networks.

Injected failures and testable properties

I inject three schedules that naive retry loops treat as identical timeouts.

timeout_then_late_success: the client timer fires; the original tool still commits; replay is attempted.
lease_recycle_then_late_body: the slot is granted to conversation B; A's body arrives late.
lost_admit_ack: admit succeeds on the server; the planner never sees lease_id and wants to retry.

Properties I require after every schedule, with no exceptions for "the model was slow":

applied_side_effects <= 1 per (idempotency_key, conversation_epoch)
no bind occurs when lease_id mismatches the current reservation
a timeout alone never increments applied_side_effects
an IN_DOUBT call ends in commit, compensate, or reject, never in silent replay

If any property fails, the design does not converge. I do not care that the happy path looked fast on an empty slot.

Tradeoff table

Choice	What you keep	What you pay	When it breaks
Blind replay after client timeout	Simple HTTP wrapper	Duplicate side effects	Late success plus retry
Longer client timeout, same bind	Fewer visible errors	Occupied shared slots, still in-doubt	Lease expiry before the timer
Lease + key + epoch bind gate	Converges under recycle	Recover RPC, prepare record	Clock skew beyond lease TTL
Compensate on mismatch	Conversation stays coherent	Needs an inverse for each tool	Tools without a safe inverse

The interesting denominator is not QPS and not token count. Count side-effecting tool calls whose bind decision is made after a client-visible timeout. That is the population the invariant is about. If you report pass rate of the model against that race, you are measuring the wrong plane.

Acceptance rule: for that denominator, applied_side_effects equals 1 on commit and 0 on compensate or reject, with zero cross-epoch binds, across the three injected schedules. Miss any clause and I would not ship the retry loop.

Architecture review: constraints, then what I would change next

Constraints first. A free shared server is an admission-limited executor with preemption. It is not a private worker pool, and it will not hold your conversation because the planner is still waiting. Latency tails are lease tails. Throughput is reservation throughput. Correctness is bind correctness. Cost is how many in-doubt calls you reopen, not how many prompts you can fire into an unbounded queue.

Data flow stays closed: admit, prepare, bind, recover. Anything that writes outside prepare is an uncommitted side effect, and the planner must not advance as if the tool succeeded. Backpressure belongs at admit, not at the HTTP client pool. If the slot is leased, a second planner waits or sheds. It does not invent a parallel universe with the same idempotency key.

If I were changing this system next, I would split prepare from commit for every tool that mutates external state. I would store the reservation beside the conversation epoch, not inside the HTTP session object. I would make recover a first-class RPC, not a for-loop around requests.post. I would refuse to let the planner's timeout close the lease, because the planner does not own that clock.

Would I keep streaming tokens to the UI during IN_DOUBT? Only as uncommitted preview, never as transcript fact. The user can watch a spinner. The store cannot watch a lie.

While I was pressure-testing this reservation protocol, I used MonkeyCode's free model access and free server option as a shared-slot stand-in for the executor, not as a dedicated farm. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The protocol does not depend on that product. If you remove the name, the in-doubt lease still has to exist, and the three schedules still have to hold.

Limitations and who should not use this

Do not use this bind gate as a substitute for an exactly-once log if your tools already run behind a transactional outbox. Do not use it for read-only retrieval, where duplicate calls are cheaper than recover RPCs. Do not use it if you have a crash-stop local executor with no retry and no shared slot. A longer HTTP timeout is not an implementation of this protocol, and I would not review it as one.

Who should skip this article? Anyone looking for deploy recipes, monitoring setup, or incident paging. Anyone trying to score model quality. Anyone hoping a free shared slot behaves like a pinned worker. This is an architecture review of an in-doubt lease, and it stays there.

If you want a shared slot to reproduce the race, the free server option is enough to run the fixture against a real remote executor. Fork the file and break the invariant on purpose before you wrap another retry around tool.call.

Which event order still breaks the invariant on your planner, and should the system reject, replay, or compensate?

Make Dual-Plane Identity an Invariant Before Stale Completions Bind the Gate

Robin — Mon, 21 Sep 2026 12:34:41 +0000

Last Tuesday I watched an eval gate flip green on a batch I had already marked closed. A completion landed from the model plane two minutes after the compute lease died, and the coordinator still wrote the score. Have you ever trusted a late JSON blob just because the schema looked polite? That event order is the counterexample I keep on the whiteboard when people glue free compute and free model calls into one happy box.

I am not writing a deploy diary, and I am not selling a dashboard tour. I am reviewing whether dual-plane identity is a real protocol, or a comment that evaporates under preemption. If the worker is gone and the model answer is still in flight, who is allowed to bind? Stay with that question until we can inject the failure and watch the gate tell the truth.

Declared assumptions

This is an architecture review of a proposed coordinator, not a production postmortem with customer traffic. The workload is a sharded agent-eval batch, and every shard needs both a worker lease and a model completion. I assume leases can vanish without a graceful drain, and I assume completions arrive once, twice, or never. I also assume we would rather leave a hole in the histogram than bind a score to the wrong generation.

The invariant is small enough to tattoo on a sticky note. A score may commit only when compute_lease_id, lease_gen, model_request_id, and prompt_hash all match the live generation that sent the prompt. If any field disagrees, the coordinator must reject or replay. It must not silently compensate by “using whatever came back.” Can a borrowed lab still be useful under that rule? Yes, if you treat the two planes as separate failure domains instead of one warm fuzzy pipeline.

The violating event order

Here is the sequence that common shard-id keys fail to preserve. Read it as a failing test, not as folklore.

Worker W1 on lease L7 sends prompt P with model request R9.
The free server is reclaimed, lease L7 expires, and W2 starts on lease L8 reusing the same shard folder.
The model plane finally returns R9, and the coordinator binds it onto L8 because the shard id still matches.
The gate sees a plausible score for the wrong generation and turns green.

Does that look familiar? It should, because most retry helpers key only on shard id and treat the model as a function, not a plane. They never ask whether the compute generation that issued the prompt is still alive. That is how a late completion becomes a lying mean.

Coordinator -> ComputePlane: grant lease L7 gen=7
ComputePlane -> ModelPlane:  complete(R9, prompt_hash=H)
ComputePlane X              preempt, L7 dead
Coordinator -> ComputePlane: grant lease L8 gen=8 (same shard folder)
ModelPlane  -> Coordinator: late complete R9
Coordinator X               REJECT unless bind_key == (L7,7,R9,H)

Data flow and failure domains

The compute plane owns disk, subprocesses, and the sandbox clock. The model plane owns in-flight tokens, tail latency, and duplicate delivery. The coordinator owns bind decisions, and it must not collapse those two clocks into one monotonic counter. When a design doc says “run the eval on a free server with a free model,” it usually draws one box. I draw two boxes, because the failure classes are not cousins.

Failure domain A is preemption: the worker disappears, and local files may be half written. Failure domain B is late or duplicate completion: the model plane answers after the lease moved. Failure domain C is mixed identity: shard ids get reused, request ids collide, or prompt hashes are computed after a tool side effect. Which of those do you currently log as a first-class event? If the answer is none, the gate is theater, and the histogram is a mood.

Numbered review of the bind protocol

1. Split the run identity before any prompt leaves

I want a closed record before the worker is allowed to call the model. The record is not a vibe. It is a tuple the bind gate can hash without consulting leftover folders.

# proposal / unexecuted fixture — not a production client
from dataclasses import dataclass

@dataclass(frozen=True)
class DualPlaneId:
    batch_id: str
    shard_id: str
    lease_id: str
    lease_gen: int
    model_request_id: str
    prompt_hash: str

    def bind_key(self) -> tuple:
        return (
            self.lease_id,
            self.lease_gen,
            self.model_request_id,
            self.prompt_hash,
        )

Why include lease_gen instead of trusting lease_id alone? Free servers get recycled, and identifiers get reused faster than your mental model. Have you ever logged a recycled worker name that still passed a prefix check? I have in fixtures, and the passing prefix is how L8 inherits L7.

2. Make bind a pure function of the tuple

The coordinator should not merge a late completion into the current lease because the JSON looks healthy. It should answer REJECT, REPLAY, or COMMIT as an explicit enum. Silent merge is the green-gate lie.

from enum import Enum

class BindDecision(Enum):
    COMMIT = "commit"
    REJECT = "reject"
    REPLAY = "replay"


def bind(live: DualPlaneId, incoming: DualPlaneId, payload_ok: bool) -> BindDecision:
    if incoming.bind_key() != live.bind_key():
        return BindDecision.REJECT
    if not payload_ok:
        return BindDecision.REPLAY
    return BindDecision.COMMIT

Is REJECT too harsh for a borrowed-compute lab? Sometimes it is, and that is a tradeoff you should name. It is not an excuse to skip the function and hope the mean converges. Would you rather replay the shard under a new generation, or publish a score that cannot name its parents? I know which one I can defend in a review.

3. Put a watermark in front of the gate

Fan-in must close on committed shards, not on “we received N blobs.” A late R9 must not move the mean after the watermark. I would rather leave a documented hole than let L8 inherit L7’s score. The denominator is not “shards we launched.” The denominator is shards that reached a terminal bind decision.

4. Inject the two failure classes before you trust the lab

Readers should be able to run a discrete-event sketch, not admire a diagram. The sketch below is labeled unexecuted production code on purpose. I am not claiming wall-clock speedups, and I am not claiming a cluster shape. I am claiming that stale binds are countable.

# proposal / deterministic simulator — run locally, no network
import hashlib
import itertools
import random
from collections import Counter

PREEMPT = "preempt"
LATE = "late_complete"
DUP = "duplicate_complete"
OK = "ok"


def prompt_hash(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()[:16]


def simulate(n_shards=2000, seed=7):
    rng = random.Random(seed)
    stale_binds = 0
    decisions = Counter()
    for shard in range(n_shards):
        gen = 1
        live = DualPlaneId(
            batch_id="b1",
            shard_id=f"s{shard}",
            lease_id=f"L{shard}-g{gen}",
            lease_gen=gen,
            model_request_id=f"R{shard}-g{gen}",
            prompt_hash=prompt_hash(f"prompt-{shard}-g{gen}"),
        )
        events = [OK]
        if rng.random() < 0.25:
            events.append(PREEMPT)
        if rng.random() < 0.35:
            events.append(LATE)
        if rng.random() < 0.10:
            events.append(DUP)
        incoming = live
        payload_ok = True
        for ev in events:
            if ev == PREEMPT:
                gen += 1
                live = DualPlaneId(
                    batch_id="b1",
                    shard_id=f"s{shard}",
                    lease_id=f"L{shard}-g{gen}",
                    lease_gen=gen,
                    model_request_id=f"R{shard}-g{gen}",
                    prompt_hash=prompt_hash(f"prompt-{shard}-g{gen}"),
                )
            elif ev == LATE:
                # stale completion still carries gen-1 identity
                incoming = DualPlaneId(
                    batch_id="b1",
                    shard_id=f"s{shard}",
                    lease_id=f"L{shard}-g{gen-1}" if gen > 1 else live.lease_id,
                    lease_gen=max(gen - 1, 1),
                    model_request_id=f"R{shard}-g{max(gen-1, 1)}",
                    prompt_hash=prompt_hash(f"prompt-{shard}-g{max(gen-1, 1)}"),
                )
            elif ev == DUP:
                payload_ok = True
        d = bind(live, incoming, payload_ok)
        decisions[d.value] += 1
        if d is BindDecision.COMMIT and incoming.lease_gen != live.lease_gen:
            stale_binds += 1
    return decisions, stale_binds


if __name__ == "__main__":
    decisions, stale = simulate()
    print(dict(decisions), "stale_binds", stale)
    assert stale == 0, "stale completion bound to a live lease"

Run it as a property check, not as a benchmark ritual.

python dual_plane_bind_sim.py

The acceptance rule needs an explicit denominator. Denominator: shards that reached COMMIT, REJECT, or REPLAY. Numerator: commits whose DualPlaneId still names the generation that sent the prompt. Acceptance: zero stale binds under preempt plus late-complete injection for the simulated N. If you cannot say the denominator out loud, you do not have a gate. You have a vibe with error bars.

Tradeoff table

Choice	Latency	Correctness under preempt	Cost on borrowed planes	What you owe the next on-call
Bind on shard id only	Low	Breaks on folder reuse	Cheap until the gate lies	A green mean you cannot replay
Bind on request id only	Medium	Breaks when ids recycle	Wasted completions on REJECT	Orphan scores after reclaim
Dual-plane tuple + watermark	Higher tail	Holds in the simulator	More replays, fewer false commits	A hole you can name
Quorum of three model calls	Highest	Still needs the compute gen	Burns the free plane fast	A cost story, not an identity story

Notice the last row. Extra model votes do not repair a missing lease generation. Have you ever added another judge because the first one was “flaky,” when the real bug was a stale bind? That is how evaluation architecture turns into incense.

Where a free model plane and a free compute plane actually help

I wanted a lab that could exercise preemption and late completion without pretending I had a reserved fleet. That is a constraint, not a slogan, and it is why the fixture stays single process and deterministic.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I treat MonkeyCode as a place where free model access and a free server option can host this dual-plane rehearsal. I am not freezing model names, quotas, hardware, or duration into this review, because those claims go stale and I will not invent them. If you replay the fixture against that lab, keep the bind function as strict as the simulator. Soften it later only after you have a counterexample that the enum cannot name.

If you want to watch your own coordinator swallow a late R9, take the fixture to a free server and a free model endpoint on MonkeyCode and log the bind enum. That is the only ask. The article still works if you run the simulator on a laptop and never leave localhost.

What I would change next

Replace shard-folder reuse with generation-scoped workspaces, so L8 cannot see L7 bytes.
Add compare-and-swap on the bind record, keyed by the full tuple, not by shard id.
Split retry budgets per plane, because model-plane retries should not extend a dead compute lease.
Property-test the permutation of {preempt, late, duplicate, payload_corrupt} instead of one happy path.
Emit the rejected tuple as a first-class event, so the histogram can show holes without pretending they were zeros.

Would I add a third plane for tools? Not yet. Tool side effects belong inside prompt_hash or they belong in a different protocol. Mixing them into bind without a hash is how domain C sneaks back in.

Limitations, and who should not use this

Do not use this design if you already have a closed consensus protocol across eval workers. Do not use it as a production SLO, and do not treat the simulator as a latency study. The sketch ignores disk corruption, clock jumps, and prompt mutation after the hash. It also ignores billing identity across vendors, which is a different invariant with a different denominator.

Who should not use this approach? Anyone hoping a free plane will mint a statistically closed leaderboard without a bind protocol. Anyone who needs exactly-once side effects in the tool plane. Anyone who will “just average the late scores” because the batch is already late. That last group is the reason I opened with a green gate that should have stayed red.

The counterexample I still want from you

Which event order still breaks the invariant: a duplicate completion with a fresh request id, a preempt that returns during replay, or a prompt_hash computed after a tool write? Should the system reject, replay, or compensate? If you cannot pick one without waving at shard id, the protocol is not closed yet, and the gate is still guessing.

DEV Community: Robin

Architecture Review: Give a Free-Tier Inference Endpoint an Admission Envelope, Not a Retry Storm

Architecture Review: Give a Free-Tier Inference Endpoint an Admission Envelope, Not a Retry Storm

Assumptions (declare them before you measure anything)

Step 1: Measure the envelope instead of guessing it

Step 2: Turn the measurement into a concurrency budget

Step 3: Classify failures, because they demand different actions

Step 4: Make the retry floor a tested property, not a habit

Step 5: Inject the failures before production does

What I would change next

Limitations, and who should not use this

Architecture Review: Keep Eval Provenance on the Control Plane, Not the Borrowed Runtime

Declared assumptions

Constraints, and where they come from

Data flow: who is allowed to name an attempt

The failure domains I actually observed

Build the fixture in five steps

Read the matrix, not the mean

Where a free runtime fits, and where it does not

What I would change next

Tradeoffs I am accepting

Limitations and who should not use this

Make Token Spend an Invariant Before a Preemptible Free Tier Replays Your Eval Batch

Make Token Spend an Invariant Before a Preemptible Free Tier Replays Your Eval Batch

Assumptions I am declaring up front

The event order that broke my accounting

Model: lease, epoch, and a monotonic spend ledger

Step-by-step: building the fence

1. Key every attempt by (run_id, shard_id, epoch)

2. Reserve before the call and commit only after the artifact digest exists

3. Make every checkpoint carry the input digest

4. Gate resumption on lease validity, not on liveness

A minimal fixture you can adapt

Failure injection matrix

Tradeoffs I would accept, and one I would not

Limitations and who should not use this

What I would change next

Make Authorization Fence an Invariant Before a Side Lane Reopens the Attempt

Assumptions I will not smuggle in later

The flow I would actually draw

Numbered steps for one attempt

Failure domains I refuse to collapse

A minimal simulator, labeled as unexecuted

Acceptance rule and the denominator

Tradeoffs I would not hide

What I would change next

Who should not use this

Which order should break it?

Make Witness Generation an Invariant Before a Late Lane Seals Compensation

The invariant a common closer drops

Assumptions I will state before the diagram

Constraints, data flow, and failure domains

Numbered steps for the seal protocol

A simulator that shows the break

Injected failures and the acceptance rule

Tradeoffs I would put in the review

What I would change next

Who should leave this pattern alone

The question I want left on the table

Make Fragment Epoch an Invariant Before a Restarted Scorer Publishes a Grade

Assumptions I will not quietly expand

Constraints, then the data flow

Numbered protocol I want in front of publish

Failure domains the fence does not magically erase

A fixture you can run before you trust the plane

Injected failures, denominator, and the acceptance rule

Tradeoffs I would write on the review

What I would change next

Who should leave this pattern alone

The question I want left open

Canary Fixture Binding With Identity Failure Classes and a Halt Gate

The order that already left the lab

Assumptions I will not quietly expand

Constraints that shape the protocol

Data flow from bind to halt

Failure domains I separate on purpose

Numbered steps to install the halt gate

A simulator you can execute locally

Where a free runner fits, and where it does not

Tradeoffs against an explicit denominator

1. Key every attempt by `(run_id, shard_id, epoch)`