DEV Community: Robin

Treat Free-Server Eval as a Shadow Protocol Before It Shares a Write Key

Robin — Thu, 24 Sep 2026 18:35:42 +0000

Last month I let a free execution server replay an evaluation job that still carried a production write key. The model stream came back late, and the shadow worker applied that payload to the shared store anyway. Production then saw a committed key and skipped a different body that the planner still expected. Have you ever let a cheap canary plane borrow the same commit identity as the live path?

I keep replaying that order because the failure is not a slow model and it is not a flaky server. The failure is an architecture that treats free capacity as a harmless copy of the production plane. If that copy can still write, then it is not a copy anymore, is it really?

The invariant the common path drops

I want one invariant stated in plain words before I draw any boxes on the board. A shadow evaluation plane may observe live inputs, but it must not share commit keys with the serving plane. It also must not share queues or compensation topics, even when that makes diffs harder to read. The common path stamps the same idempotency key onto the free server job because diffs feel easier.

That shortcut is exactly what lets a late shadow completion bind a live write you did not intend. Would you still call it a shadow if its acknowledgement can move production state without a human check? I would not call it a shadow, because a second writer with a friendly name is still a writer.

Assumptions I am willing to defend

I am reviewing a design rather than reporting a load test, so these assumptions need to stay visible. Free model access and a free server option are availability choices for a canary, not a capacity contract. I do not know your quota, hardware, or how long that option lasts, and I will not invent those numbers. The planner can retry after a stall, and delivery to the execution plane is at least once.

Two planes that share a key namespace are one failure domain, even when their dashboards look completely separate. A free tier can reject, stall, or disappear without becoming a reason to spill work onto the live writer. I will treat any missing capacity number as unknown, rather than as permission to fan out blindly. If your measured limit arrives later, you can drop it into the fixture without changing the fence.

Where I place the canary

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am using the MonkeyCode project's free model access and free server option as the shadow plane in this review. That choice lets me exercise the fence without ever pointing the canary at the serving write path. If you remove the product from the story, the same invariant still holds for any shared execution tier.

How I want the data to move

I draw three planes and one fence, and I refuse to let any arrow skip that fence. The control plane admits a shadow job and rewrites its key before any model call starts. The model plane only proposes a payload, and the free server executes that payload against a shadow store. Nothing on that path should be able to append a record to the live commit log.

planner        control plane       model plane        free server       shadow store
   | admit           |                  |                  |                 |
   |---------------->| rewrite key      |                  |                 |
   |                 |----------------->| propose payload  |                 |
   |                 |                  |----------------->| execute         |
   |                 |                  |                  |---------------->| shadow write
   |                 | fence: reject live: prefix          |                 |
live commit log stays untouched unless prefix == live AND admitted_plane == live

violating order when the fence is missing:
1. control admits job-7 with key live:tenant:debit
2. model plane stalls after the proposal is buffered
3. client retries; free server accepts a second worker
4. late completion from worker A commits amount=10
5. worker B commits amount=12 under the same live key

Does that picture still hold if the free server may call the same tool URL as production? It does not hold, because a shared tool URL without a plane header is another shared write key. Have you seen a network address pretend to be an isolation boundary when the handler is shared? I treat that address as a door, not as a domain, until the handler checks the plane.

Failure domains I would not merge

I split failure by who can change state, not by which vendor logo happens to sit on the box. The model plane can stall, truncate, or return a late completion, and that should only invalidate a proposal. The free server can accept twice, die mid-write, or reject for capacity, and that should only touch the shadow store. The control plane can admit a duplicate when it forgets the rewritten key, and that leak is the dangerous one.

If I merge those domains, a capacity reject on the free server starts to look like a model failure. A retry then replays the same body into the live tool, because both errors were typed as timeout. Have you watched a retry policy follow the wrong plane just because the error string looked familiar? I have watched that happen, and the compensation topic then fired for a job the user never confirmed.

The review I rerun before I trust a diagram

I do not want a slide that says the shadow is isolated unless a fixture can fail the build. This simulator is a proposal fixture I can run locally, and it is not a benchmark of any kind. I am not claiming a throughput number, a model name, or a hardware shape I have not measured. I am staying on the design and the fixture, and I am not walking through deploy steps or a monitoring install.

Declare the key namespaces and the capacity bound as inputs, rather than as comments buried in a client.
Admit the shadow job only after the key is rewritten, and record the plane on the admission lease.
Inject a late model completion plus a duplicate delivery aimed at the free server under test.
Assert that no live key changes, and that a second admit with the same shadow key is rejected.
Fail the review if any event log line contains a live prefix after the shadow run has started.

'''Shadow-key fence fixture. Proposal, not a production client.'''
from dataclasses import dataclass

LIVE = 'live'
SHADOW = 'shadow'

@dataclass
class Write:
    key: str
    plane: str
    body: str

class Store:
    def __init__(self):
        self.rows = {}
        self.rejected = []

    def commit(self, write, admitted_plane):
        prefix = write.key.split(':', 1)[0]
        if prefix != admitted_plane or write.plane != admitted_plane:
            self.rejected.append(write)
            return False
        if write.key in self.rows:
            self.rejected.append(write)
            return False
        self.rows[write.key] = write
        return True

def rewrite(step, run):
    return f'{SHADOW}:{run}:{step}'

def late_duplicate_order():
    store = Store()
    key = rewrite('debit', 'run-19')
    first = Write(key, SHADOW, 'amount=10')
    late = Write(key, SHADOW, 'amount=10')
    leaked = Write('live:tenant:debit', LIVE, 'amount=10')
    ok1 = store.commit(first, SHADOW)
    ok2 = store.commit(late, SHADOW)
    ok3 = store.commit(leaked, SHADOW)
    return ok1, ok2, ok3, store

def live_untouched(rows):
    return all(not key.startswith('live:') for key in rows)

if __name__ == '__main__':
    ok1, ok2, ok3, store = late_duplicate_order()
    assert ok1 and not ok2 and not ok3
    assert list(store.rows) == ['shadow:run-19:debit']
    assert live_untouched(store.rows)
    print('fence_held', len(store.rejected))

python shadow_fence.py

I also want a property that does not depend on one scripted order in a single file. Generate permutations of stall, duplicate delivery, and capacity reject, then assert the live prefix never appears. If one permutation breaks that property, the design does not converge, no matter how clean the diagram looked. Would you really promote a design that passes one happy path and then fails the permutation test?

When I read the rejected list, I want every crossed-plane write to show up there, with its original key intact. A silent drop would hide the bug, and a rewrite of the leaked key would launder a live prefix into a shadow prefix. Have you ever fixed a test by normalizing the key before the assertion, and then missed the leak entirely? I keep the raw key in the rejection record so the next reviewer can see which plane tried to cross.

My load model is deliberately tiny, because the question is convergence under reordering, not a peak rate claim. I set capacity to one in-flight shadow job so a second admit must fail closed instead of borrowing serving room. You should replace that bound with your own measured limit, and you should not copy a number I refused to invent. If the free server starts rejecting for capacity, the control plane should queue or shed, not spill into the live plane.

The tradeoff I could not wave away

I could not wave this table away, because each row buys correctness with a different operational cost. I am not claiming a speedup, because I have not run a paired workload on a named model or machine. The denominator I care about is the count of event orders that violate the fence, not tokens per second. Acceptance rule: zero live keys may be written across the injected failure set, including a duplicate after a stall.

Choice	Latency effect	Correctness	Cost posture	What breaks
Shared write key on the free server	Fewer lookups	Shadow can bind live state	Looks cheap until replay	Late completion commits the wrong body
Rewritten shadow key plus a fence	One prefix check	Live prefix is rejected	Canary stays on free access	Diffs need a translation step
No free plane at all	Probes pay the serving path	Strong only if serving is fenced too	Higher for every review run	Failure injection gets skipped

What I would change next

I would stop letting the free server select tools by URL alone, without a plane check beside the key. I would require a plane header that the tool gateway compares with the key prefix before it accepts work. I would drop the request when they disagree, and I would also split compensation topics by plane. A shadow timeout must not enqueue a live undo, and an older run id must not join a newer comparison.

Should the gateway reject, replay, or compensate when the header and the key prefix disagree on a write? I want the gateway to reject, because replay would only amplify a client that is already confused. Compensation assumes a live write that the fence should have blocked before any side effect started. Would replaying a crossed-plane request ever make the invariant stronger than a hard reject at the gateway?

Who should leave this pattern alone

Do not use a free shadow plane as your only correctness proof when serving auth and data differ. Do not point the free server at production tools while you tell yourself that you are only testing prompts. Do not treat unknown free capacity as a queue you can fan out without some explicit admission bound. If a regulator requires an attested environment, a free shared server is the wrong domain for that evidence.

This review also will not tell you which model is smarter, because I am not publishing scores I did not measure. It will only tell you whether a late proposal from the free plane can still move a live key. If your serving path has a different retry budget, a green shadow run is not evidence that production will converge.

The order I want you to break

Which event order breaks the invariant for your system, and should that system reject, replay, or compensate? Take one shadow job, stall the model, deliver the execution twice, and sneak one live prefix into the batch. If the store accepts that prefix, the free server was never a shadow, and it was a second writer you forgot to fence. If you want a bounded place to run that fixture on free model access and a free server, keep the write key fenced.

Treat Tool Intent as a Deduped Outbox Before Planner Retries Double-Commit

Robin — Wed, 23 Sep 2026 16:03:50 +0000

Last Tuesday I sat with a sequence dump that looked successful until I counted the tickets twice. The planner had asked for one incident record, and the board showed two identical rows minutes apart. Have you ever trusted a tool-failed retry line without asking whether the side effect already landed? I had trusted that line, and that is exactly how this architecture review started for me.

The violating order was simple, and it did not need a clever adversary to appear in production. The worker posted the ticket, then the process vanished before the result reached the planner queue. The planner treated silence as failure, emitted the same step again, and a second worker committed. Does a missing ack mean the tool never ran, or only that nobody survived to say so?

The invariant the common retry loop drops

I want one planner step to produce at most one committed side effect, unless named compensation runs. That sentence is the invariant, and everything else in this review exists to protect or refute it. Most agent diagrams draw a happy arrow from model output to tool call to observation, then bolt on retries. Is a timeout a hard failure you may replay, or an unknown that still owns the side effect?

I am reviewing a two-queue tool bus, not narrating a named outage from a company I will not invent. The fixture below is a proposed discrete-event model, not a production service I operate. If a design cannot name the reject, replay, and compensate paths on paper, I will not trust its retry flag in a real agent.

Declared assumptions

I am assuming at-least-once queues, crash-stop workers, and a planner that re-emits a step after timeout. I am assuming external tools are not naturally idempotent, because ticket and mail APIs usually are not. I am assuming the planner may sit on a shared model endpoint, while the worker may die without a graceful drain. I am not assuming exactly-once brokers, sticky sessions, or a private cluster with reserved capacity.

Constraints I would not negotiate

I will not let the model transcript be the source of truth for external writes, because transcripts are not ledgers. I will not let a planner timeout delete an in-flight intent, because silence is not an abort. I will not accept a tool adapter that mints a new external request whenever the queue redelivers a payload. If those three constraints feel expensive, the double-ticket failure is the cheaper-looking alternative that I refuse.

Here is the constraint set I use when I review the bus:

Every planner step must mint a stable intent_id before any worker may touch the outside world.
The outbox, not the model message, is the only record that authorizes a side-effecting call.
A worker may execute an intent only after a compare-and-set from queued into executing.
A late result for a superseded generation must be rejected, not folded into the next observation.

Which of those four would you drop first under schedule pressure, and what event order would punish that drop?

Data flow of the tool bus

I want the planner, the outbox, the worker, and the tool API in separate failure domains. The model may hallucinate a repeated tool call, and the queue may redeliver, but the outbox still owns the write. The result queue carries observations back, and it must key them by intent_id plus generation, not by arrival time. If a result cannot name those keys, I treat it as noise and I do not advance the plan.

sequenceDiagram
    participant P as Planner
    participant O as Intent outbox
    participant W as Tool worker
    participant T as External API
    participant R as Result queue
    P->>O: put(intent_id, gen, tool, args)
    O->>W: deliver at-least-once
    W->>O: CAS queued to executing
    W->>T: call with Idempotency-Key
    Note over W,R: worker may die here
    W->>R: observe(intent_id, gen, result)
    R->>P: bind only if gen still current
    P->>O: timeout does not mint a new intent_id

Happy path, written as a protocol rather than a vibe:

The planner assigns intent_id and generation zero, then inserts the row in queued state.
A worker claims the row with a compare-and-set, then calls the tool with that same key on the wire.
The worker writes the observation to the result queue, then marks the outbox row committed or failed.
The planner consumes the observation only when the generation still matches the live step.

If step two fails the compare-and-set, the second worker must stop. Replay is a delivery fact, not a license to spend again. Compensation is a different protocol, and I will not pretend a second create is a compensate.

Failure domains I actually budget for

I split the design into four domains because they fail on different clocks. The planner domain fails by timeout, duplicate decode, or a model that repeats a tool call with new prose. The outbox domain fails by lost write, lost ack, or a crash between state transitions. The worker domain fails by preemption, slow tool I/O, or a redelivered message after a successful post. The tool domain fails by accepting a body twice when no idempotency header was sent.

The dangerous coupling is the planner treating worker silence as a negative observation. That coupling turns one crash into two commits, and it hides inside retry middleware that looks responsible. Do we reject the late result, replay the same intent, or enqueue a compensate? If the outbox still says executing, I replay nothing and I wait or I escalate.

I also budget for reordered results, because preempted workers come back at the worst time. A generation that the planner already abandoned must not bind the next step. That is not a logging problem. That is a binding rule, and the result consumer has to enforce it.

A minimal simulator you can run

I want a fixture that can print the double-commit without standing up Kafka. The model below is discrete-event Python, and it is intentionally small enough to argue about in review. It is not a performance test, and I am not attaching invented latency numbers to it. Run it locally, then change the event order until the invariant breaks or holds.

# proposed fixture: not a production bus, not a benchmark
from dataclasses import dataclass
from typing import Dict, List, Optional, Tuple

QUEUED, EXECUTING, COMMITTED, FAILED = "queued", "executing", "committed", "failed"

@dataclass
class Intent:
    intent_id: str
    gen: int
    tool: str
    args: dict
    state: str = QUEUED
    side_effects: int = 0

class Outbox:
    def __init__(self) -> None:
        self.rows: Dict[str, Intent] = {}

    def put(self, intent: Intent) -> None:
        if intent.intent_id in self.rows:
            return  # dedupe: same intent_id never mints a second row
        self.rows[intent.intent_id] = intent

    def claim(self, intent_id: str, gen: int) -> bool:
        row = self.rows.get(intent_id)
        if not row or row.gen != gen or row.state != QUEUED:
            return False
        row.state = EXECUTING
        return True

    def complete(self, intent_id: str, gen: int, ok: bool) -> None:
        row = self.rows[intent_id]
        if row.gen != gen:
            return  # stale generation: reject
        row.state = COMMITTED if ok else FAILED

class Tool:
    def __init__(self) -> None:
        self.seen: Dict[str, str] = {}
        self.commits = 0

    def call(self, idem_key: str, body: dict) -> str:
        if idem_key in self.seen:
            return self.seen[idem_key]
        self.commits += 1
        ticket = f"T-{self.commits}"
        self.seen[idem_key] = ticket
        return ticket

def run(events: List[Tuple[str, dict]]) -> Dict[str, int]:
    outbox, tool = Outbox(), Tool()
    planner_timeouts = 0
    rejected_stale = 0
    for name, payload in events:
        if name == "plan":
            outbox.put(Intent(**payload, state=QUEUED))
        elif name == "claim_and_call":
            iid, gen = payload["intent_id"], payload["gen"]
            if not outbox.claim(iid, gen):
                rejected_stale += 1
                continue
            ticket = tool.call(iid, outbox.rows[iid].args)
            outbox.rows[iid].side_effects += 1
            if payload.get("ack", True):
                outbox.complete(iid, gen, True)
                payload["ticket"] = ticket
        elif name == "planner_timeout_replay":
            planner_timeouts += 1
            row = outbox.rows[payload["intent_id"]]
            # illegal: minting a new intent_id on timeout
            if payload.get("mint_new_id"):
                new_id = row.intent_id + "-retry"
                outbox.put(Intent(new_id, 0, row.tool, row.args))
            # legal: redeliver the same intent_id
        elif name == "late_ack":
            iid, gen = payload["intent_id"], payload["gen"]
            if outbox.rows[iid].gen != gen:
                rejected_stale += 1
            else:
                outbox.complete(iid, gen, True)
    return {
        "external_commits": tool.commits,
        "planner_timeouts": planner_timeouts,
        "rejected_stale": rejected_stale,
        "outbox_rows": len(outbox.rows),
    }

BAD = [
    ("plan", {"intent_id": "step-7", "gen": 0, "tool": "create_ticket", "args": {"title": "db blip"}}),
    ("claim_and_call", {"intent_id": "step-7", "gen": 0, "ack": False}),
    ("planner_timeout_replay", {"intent_id": "step-7", "mint_new_id": True}),
    ("claim_and_call", {"intent_id": "step-7-retry", "gen": 0, "ack": True}),
    ("late_ack", {"intent_id": "step-7", "gen": 0}),
]

GOOD = [
    ("plan", {"intent_id": "step-7", "gen": 0, "tool": "create_ticket", "args": {"title": "db blip"}}),
    ("claim_and_call", {"intent_id": "step-7", "gen": 0, "ack": False}),
    ("planner_timeout_replay", {"intent_id": "step-7", "mint_new_id": False}),
    ("claim_and_call", {"intent_id": "step-7", "gen": 0, "ack": True}),  # claim should fail
    ("late_ack", {"intent_id": "step-7", "gen": 0}),
]

if __name__ == "__main__":
    print("bad ", run(BAD))
    print("good", run(GOOD))
    assert run(BAD)["external_commits"] == 2
    assert run(GOOD)["external_commits"] == 1

The acceptance rule is explicit, and I want it in the same file as the events. external_commits must stay one for a single planner step when the worker dies before ack. The bad trace mints a new identifier on timeout, and the tool cannot see that the two calls are the same intent. The good trace redelivers step-7, the second claim fails, and the late ack still binds generation zero.

Injected failures and how I would run them

I do not trust a green happy path when the interesting bugs live in reordering. The commands below assume the fixture is saved as tool_outbox_sim.py in the current directory. They do not measure tokens, dollars, or hosted throughput, because I will not invent those figures.

Run the paired traces and keep the assertion on external_commits as the gate, not a log skim.
Flip mint_new_id to true and confirm the fixture fails, because that is the common planner bug.
Drop the Idempotency-Key path in Tool.call and watch a legal redelivery become a second spend.
Insert a second late_ack with gen=0 after you have advanced gen to one, and demand a reject.

python tool_outbox_sim.py
python -c "from tool_outbox_sim import run, BAD, GOOD; print(run(BAD), run(GOOD))"

If you add a real HTTP tool, send the outbox intent_id as Idempotency-Key and treat any 2xx without that echo as a protocol miss. An empty success body is a separate discussion I will not reopen here. The property I care about is simpler: one live intent, one external commit, or a named compensate row.

Tradeoffs, with a denominator

I am comparing designs per planner step, not per day and not per model call, because those denominators hide duplicates. The table is a review aid. It is not a vendor scorecard.

Design	What it preserves	What it costs	When it still breaks
Retry on planner timeout, new tool payload	Almost nothing	Extra calls that look like diligence	Worker died after commit
At-least-once queue, no outbox row	Delivery, not intent	Duplicate spends under redelivery	Any crash between post and ack
Deduped outbox plus compare-and-set claim	At most one executor	An extra durable write before I/O	Tool ignores the idempotency key
Outbox plus compensating inverse	Liveness after a true stuck execute	You must write the inverse, and it can fail too	Compensate timeout with no idempotency

I would rather pay the extra durable write than explain two tickets to an on-call thread. Throughput can wait until the invariant holds under injected preemption. If your load model cannot name concurrent in-flight intents, you are not measuring the bus. You are measuring hope again.

What I would change next

I would split executing into executing and awaiting_ack, because those are different recovery actions. I would put a bounded wait on awaiting_ack, then move to a compensate intent with its own identifier derived from the original. I would fence result binding with the generation, and I would reject any observation that cannot quote both keys. I would not add more planner retries until those states exist in the outbox schema.

I would also stop letting the model invent tool arguments on replay. Replay must load the stored args from the outbox row, not from a freshly sampled completion. Otherwise the second call is not a retry. It is a different write that only rhymes with the first. Can your current agent runtime even show you the stored args without reading a chat log?

Limitations, and who should skip this

This review does not give you a hosted topology, a queue product, or an incident runbook. The simulator is single-threaded and discrete, so it will not catch disk-ordering bugs or clock skew across regions. Free shared processes are useful for forcing timeouts, and they are the wrong isolation boundary for customer-facing writes. If you need a hard latency SLO, dedicated workers, or regulated audit storage, do not use this fixture as a deployment plan.

Skip the outbox if your tools are truly pure reads and a duplicate call cannot spend money or create records. Skip it if you already have a transactional outbox in the same database as the agent state, and your workers already claim with fencing. Do not skip it if a model timeout currently mints a new tool payload. That path is the double-commit I opened with.

Which event order still breaks the invariant in your bus, and should the system reject, replay, or compensate? If the answer is only "retry the tool," I would send the design back before any planner scales up.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free models that can run this workflow. A free server option is enough to reproduce the setup.

Treat Client Timeout as an In-Doubt Lease Before You Replay Shared Tool Calls

Robin — Tue, 22 Sep 2026 14:21:42 +0000

Last Tuesday I sat with a planner log that looked almost boring until I lined up two clocks. The client aborted a tool call at thirty seconds, then immediately replayed the same payload against a shared slot. The original call finished four seconds later, and the retry still executed against that reservation. Two side effects landed in the same conversation epoch, and nobody had a lease to point at.

That event order is not an incident curiosity, and it is the default protocol most agent backends implement. A client timeout is not an abort, and a shared free slot does not make that physics kinder. I keep reviewing these loops as if HTTP were a transaction, which it never was. So I want the architecture to name the in-doubt lease before anyone hits replay again.

Have you ever watched a retry storm and blamed the model for duplicate tool work? I have, and the model was the least interesting process in the trace.

The invariant the common retry loop fails to preserve

I want one invariant, stated before any diagram or fixture. A side-effecting tool result may bind if and only if three tokens still match: lease_id, idempotency_key, and conversation_epoch. Anything else is an in-doubt write, not a clean miss, and it must not apply.

Why three tokens instead of a single request id on the HTTP client? Because the failure domains are not the same plane, even when they share a TCP connection. The client timer lives in the planner process, the lease lives on the shared slot, and the epoch lives in the conversation store. Collapse those planes and you will bind a late body to the wrong run.

Declared assumptions

I am reviewing a planner that issues tool calls onto a shared remote executor, not a dedicated box you fully own. The executor may reclaim a slot when a lease expires, even if a tool body is still in flight. Side-effecting tools are not locally transactional with the model stream, and duplicate delivery is expected after timeout, reconnect, or planner restart.

I am not assuming named models, published quotas, dedicated hardware, or a permanent free tier. Those would be invented product claims, and they are not required for the race. I am assuming only that a free shared slot can exist long enough to admit work and then disappear under another conversation. If your tools are strictly read-only, this review is heavier than you need.

This fixture is a design proposal. It is not a production measurement, and I am not reporting live traffic.

Data flow: treat admission as a protocol, not an HTTP POST

Here is the sequence I want the architecture to speak, not the sequence most SDKs actually speak when a timeout fires.

The planner creates an idempotency key bound to the current conversation_epoch.
The executor admits a reservation and returns a lease_id with an explicit expiry.
The tool writes a prepare record inside that lease before any external side effect.
Completion is a bind, not a print: match lease, key, and epoch, then commit.
On client timeout, the planner marks the call IN_DOUBT and does not replay yet.
A recover path commits, compensates, or rejects; it never blind-replays the payload.

Does your current client already skip step five? Mine did, for years, because timeouts feel like negatives and negatives feel safe to retry.

sequenceDiagram
    participant Planner
    participant Admit as AdmitGate
    participant Slot as SharedSlot
    participant Store as EpochStore
    Planner->>Admit: reserve(key, epoch)
    Admit-->>Planner: lease_id, expiry
    Planner->>Slot: execute(tool, lease_id)
    Note over Planner: client timer fires; mark IN_DOUBT
    Slot->>Store: bind(lease_id, key, epoch, result)
    alt tokens match
        Store-->>Slot: COMMIT
    else lease or epoch mismatch
        Store-->>Slot: REJECT
    end
    Planner->>Admit: recover(key, epoch)
    Admit-->>Planner: commit | compensate | reject

The HTTP POST is only the transport. If your sequence diagram starts at requests.post and ends at status 200, you have not modeled the bind.

Failure domains

I split the system into four domains so retries stop crossing wires they do not own.

Planner timer domain. A thirty-second client timeout fires while the executor is still inside a valid lease. The planner knows nothing about that lease. The work may still commit, and a replay now is a second admit.

Lease domain. The shared slot expires and is granted to another conversation. A late body can arrive carrying a stale lease_id. Binding it is a consistency bug, not a helpful flush of leftover tokens.

Idempotency domain. The same key is admitted twice because the first admit response was lost on the way back. Two prepare records exist. One must win, and the other must no-op without a second side effect.

Epoch domain. The user started a new turn, or a worker preemption bumped conversation_epoch. A completion from the previous turn must not append, even if the lease still looks alive in a local cache.

Notice what I did not put in this review: dashboards, deploy pipelines, or paging rotations. Those are operations concerns. This article stops at whether the protocol converges under a known event order.

A minimal in-doubt simulator

I wanted an executable counterexample I could step in one file, not a slide about retries. The fixture below models a shared slot, a client timeout that does not abort the server, and a bind gate that either commits or rejects. Label it unexecuted until you run it locally.

from dataclasses import dataclass, field
from typing import Dict, Optional, Literal

Decision = Literal["COMMIT", "REJECT", "IN_DOUBT", "COMPENSATE"]

@dataclass
class Reservation:
    lease_id: str
    key: str
    epoch: int
    expiry: int
    prepared: bool = False
    applied: bool = False

@dataclass
class Slot:
    now: int = 0
    current: Optional[Reservation] = None
    applied_side_effects: int = 0
    binds: Dict[tuple, Decision] = field(default_factory=dict)

    def admit(self, lease_id: str, key: str, epoch: int, ttl: int) -> Reservation:
        if self.current and self.now < self.current.expiry:
            raise RuntimeError("slot still leased")
        self.current = Reservation(lease_id, key, epoch, self.now + ttl)
        return self.current

    def prepare(self, lease_id: str, key: str, epoch: int) -> None:
        r = self._require(lease_id, key, epoch)
        r.prepared = True

    def bind(self, lease_id: str, key: str, epoch: int) -> Decision:
        token = (key, epoch)
        if token in self.binds and self.binds[token] == "COMMIT":
            return "REJECT"  # duplicate bind never re-applies
        r = self.current
        if not r or r.lease_id != lease_id or r.key != key or r.epoch != epoch:
            self.binds[token] = "REJECT"
            return "REJECT"
        if self.now >= r.expiry or not r.prepared:
            self.binds[token] = "REJECT"
            return "REJECT"
        if not r.applied:
            r.applied = True
            self.applied_side_effects += 1
        self.binds[token] = "COMMIT"
        return "COMMIT"

    def recover(self, key: str, epoch: int) -> Decision:
        token = (key, epoch)
        decision = self.binds.get(token)
        if decision == "COMMIT":
            return "COMMIT"
        if decision == "REJECT":
            return "COMPENSATE"
        return "IN_DOUBT"

    def _require(self, lease_id: str, key: str, epoch: int) -> Reservation:
        r = self.current
        if not r or r.lease_id != lease_id or r.key != key or r.epoch != epoch:
            raise RuntimeError("lease mismatch")
        if self.now >= r.expiry:
            raise RuntimeError("lease expired")
        return r


def timeout_then_late_success() -> Slot:
    s = Slot()
    s.admit("L1", "k", 7, ttl=40)
    s.prepare("L1", "k", 7)
    s.now = 30  # client timeout; planner must NOT replay
    assert s.recover("k", 7) == "IN_DOUBT"
    s.now = 34
    assert s.bind("L1", "k", 7) == "COMMIT"
    # Blind replay would try a second admit; the bind gate must no-op.
    assert s.bind("L1", "k", 7) == "REJECT"
    assert s.applied_side_effects == 1
    return s


def lease_recycle_then_late_body() -> Slot:
    s = Slot()
    s.admit("L1", "kA", 1, ttl=10)
    s.prepare("L1", "kA", 1)
    s.now = 11
    s.admit("L2", "kB", 2, ttl=10)
    assert s.bind("L1", "kA", 1) == "REJECT"
    assert s.applied_side_effects == 0
    return s


def lost_admit_ack() -> Slot:
    s = Slot()
    s.admit("L1", "k", 3, ttl=20)
    s.prepare("L1", "k", 3)
    s.now = 5
    # Planner never saw L1; a naive retry would admit again and double-apply.
    try:
        s.admit("L2", "k", 3, ttl=20)
        raise AssertionError("second admit must fail while lease is live")
    except RuntimeError:
        pass
    assert s.bind("L1", "k", 3) == "COMMIT"
    assert s.applied_side_effects == 1
    return s


if __name__ == "__main__":
    timeout_then_late_success()
    lease_recycle_then_late_body()
    lost_admit_ack()
    print("invariant held for the three injected schedules")

How would you run it? Save the file and execute the three schedules before you trust any retry wrapper. The point is not throughput. The point is whether the invariant holds after each injected order.

python in_doubt_lease.py

If you comment out the duplicate-bind guard and allow a second admit after the client timer, you get the original outage in twelve lines. That is the counterexample I want in review, not a narrative about flaky networks.

Injected failures and testable properties

I inject three schedules that naive retry loops treat as identical timeouts.

timeout_then_late_success: the client timer fires; the original tool still commits; replay is attempted.
lease_recycle_then_late_body: the slot is granted to conversation B; A's body arrives late.
lost_admit_ack: admit succeeds on the server; the planner never sees lease_id and wants to retry.

Properties I require after every schedule, with no exceptions for "the model was slow":

applied_side_effects <= 1 per (idempotency_key, conversation_epoch)
no bind occurs when lease_id mismatches the current reservation
a timeout alone never increments applied_side_effects
an IN_DOUBT call ends in commit, compensate, or reject, never in silent replay

If any property fails, the design does not converge. I do not care that the happy path looked fast on an empty slot.

Tradeoff table

Choice	What you keep	What you pay	When it breaks
Blind replay after client timeout	Simple HTTP wrapper	Duplicate side effects	Late success plus retry
Longer client timeout, same bind	Fewer visible errors	Occupied shared slots, still in-doubt	Lease expiry before the timer
Lease + key + epoch bind gate	Converges under recycle	Recover RPC, prepare record	Clock skew beyond lease TTL
Compensate on mismatch	Conversation stays coherent	Needs an inverse for each tool	Tools without a safe inverse

The interesting denominator is not QPS and not token count. Count side-effecting tool calls whose bind decision is made after a client-visible timeout. That is the population the invariant is about. If you report pass rate of the model against that race, you are measuring the wrong plane.

Acceptance rule: for that denominator, applied_side_effects equals 1 on commit and 0 on compensate or reject, with zero cross-epoch binds, across the three injected schedules. Miss any clause and I would not ship the retry loop.

Architecture review: constraints, then what I would change next

Constraints first. A free shared server is an admission-limited executor with preemption. It is not a private worker pool, and it will not hold your conversation because the planner is still waiting. Latency tails are lease tails. Throughput is reservation throughput. Correctness is bind correctness. Cost is how many in-doubt calls you reopen, not how many prompts you can fire into an unbounded queue.

Data flow stays closed: admit, prepare, bind, recover. Anything that writes outside prepare is an uncommitted side effect, and the planner must not advance as if the tool succeeded. Backpressure belongs at admit, not at the HTTP client pool. If the slot is leased, a second planner waits or sheds. It does not invent a parallel universe with the same idempotency key.

If I were changing this system next, I would split prepare from commit for every tool that mutates external state. I would store the reservation beside the conversation epoch, not inside the HTTP session object. I would make recover a first-class RPC, not a for-loop around requests.post. I would refuse to let the planner's timeout close the lease, because the planner does not own that clock.

Would I keep streaming tokens to the UI during IN_DOUBT? Only as uncommitted preview, never as transcript fact. The user can watch a spinner. The store cannot watch a lie.

While I was pressure-testing this reservation protocol, I used MonkeyCode's free model access and free server option as a shared-slot stand-in for the executor, not as a dedicated farm. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The protocol does not depend on that product. If you remove the name, the in-doubt lease still has to exist, and the three schedules still have to hold.

Limitations and who should not use this

Do not use this bind gate as a substitute for an exactly-once log if your tools already run behind a transactional outbox. Do not use it for read-only retrieval, where duplicate calls are cheaper than recover RPCs. Do not use it if you have a crash-stop local executor with no retry and no shared slot. A longer HTTP timeout is not an implementation of this protocol, and I would not review it as one.

Who should skip this article? Anyone looking for deploy recipes, monitoring setup, or incident paging. Anyone trying to score model quality. Anyone hoping a free shared slot behaves like a pinned worker. This is an architecture review of an in-doubt lease, and it stays there.

If you want a shared slot to reproduce the race, the free server option is enough to run the fixture against a real remote executor. Fork the file and break the invariant on purpose before you wrap another retry around tool.call.

Which event order still breaks the invariant on your planner, and should the system reject, replay, or compensate?

Make Dual-Plane Identity an Invariant Before Stale Completions Bind the Gate

Robin — Mon, 21 Sep 2026 12:34:41 +0000

Last Tuesday I watched an eval gate flip green on a batch I had already marked closed. A completion landed from the model plane two minutes after the compute lease died, and the coordinator still wrote the score. Have you ever trusted a late JSON blob just because the schema looked polite? That event order is the counterexample I keep on the whiteboard when people glue free compute and free model calls into one happy box.

I am not writing a deploy diary, and I am not selling a dashboard tour. I am reviewing whether dual-plane identity is a real protocol, or a comment that evaporates under preemption. If the worker is gone and the model answer is still in flight, who is allowed to bind? Stay with that question until we can inject the failure and watch the gate tell the truth.

Declared assumptions

This is an architecture review of a proposed coordinator, not a production postmortem with customer traffic. The workload is a sharded agent-eval batch, and every shard needs both a worker lease and a model completion. I assume leases can vanish without a graceful drain, and I assume completions arrive once, twice, or never. I also assume we would rather leave a hole in the histogram than bind a score to the wrong generation.

The invariant is small enough to tattoo on a sticky note. A score may commit only when compute_lease_id, lease_gen, model_request_id, and prompt_hash all match the live generation that sent the prompt. If any field disagrees, the coordinator must reject or replay. It must not silently compensate by “using whatever came back.” Can a borrowed lab still be useful under that rule? Yes, if you treat the two planes as separate failure domains instead of one warm fuzzy pipeline.

The violating event order

Here is the sequence that common shard-id keys fail to preserve. Read it as a failing test, not as folklore.

Worker W1 on lease L7 sends prompt P with model request R9.
The free server is reclaimed, lease L7 expires, and W2 starts on lease L8 reusing the same shard folder.
The model plane finally returns R9, and the coordinator binds it onto L8 because the shard id still matches.
The gate sees a plausible score for the wrong generation and turns green.

Does that look familiar? It should, because most retry helpers key only on shard id and treat the model as a function, not a plane. They never ask whether the compute generation that issued the prompt is still alive. That is how a late completion becomes a lying mean.

Coordinator -> ComputePlane: grant lease L7 gen=7
ComputePlane -> ModelPlane:  complete(R9, prompt_hash=H)
ComputePlane X              preempt, L7 dead
Coordinator -> ComputePlane: grant lease L8 gen=8 (same shard folder)
ModelPlane  -> Coordinator: late complete R9
Coordinator X               REJECT unless bind_key == (L7,7,R9,H)

Data flow and failure domains

The compute plane owns disk, subprocesses, and the sandbox clock. The model plane owns in-flight tokens, tail latency, and duplicate delivery. The coordinator owns bind decisions, and it must not collapse those two clocks into one monotonic counter. When a design doc says “run the eval on a free server with a free model,” it usually draws one box. I draw two boxes, because the failure classes are not cousins.

Failure domain A is preemption: the worker disappears, and local files may be half written. Failure domain B is late or duplicate completion: the model plane answers after the lease moved. Failure domain C is mixed identity: shard ids get reused, request ids collide, or prompt hashes are computed after a tool side effect. Which of those do you currently log as a first-class event? If the answer is none, the gate is theater, and the histogram is a mood.

Numbered review of the bind protocol

1. Split the run identity before any prompt leaves

I want a closed record before the worker is allowed to call the model. The record is not a vibe. It is a tuple the bind gate can hash without consulting leftover folders.

# proposal / unexecuted fixture — not a production client
from dataclasses import dataclass

@dataclass(frozen=True)
class DualPlaneId:
    batch_id: str
    shard_id: str
    lease_id: str
    lease_gen: int
    model_request_id: str
    prompt_hash: str

    def bind_key(self) -> tuple:
        return (
            self.lease_id,
            self.lease_gen,
            self.model_request_id,
            self.prompt_hash,
        )

Why include lease_gen instead of trusting lease_id alone? Free servers get recycled, and identifiers get reused faster than your mental model. Have you ever logged a recycled worker name that still passed a prefix check? I have in fixtures, and the passing prefix is how L8 inherits L7.

2. Make bind a pure function of the tuple

The coordinator should not merge a late completion into the current lease because the JSON looks healthy. It should answer REJECT, REPLAY, or COMMIT as an explicit enum. Silent merge is the green-gate lie.

from enum import Enum

class BindDecision(Enum):
    COMMIT = "commit"
    REJECT = "reject"
    REPLAY = "replay"


def bind(live: DualPlaneId, incoming: DualPlaneId, payload_ok: bool) -> BindDecision:
    if incoming.bind_key() != live.bind_key():
        return BindDecision.REJECT
    if not payload_ok:
        return BindDecision.REPLAY
    return BindDecision.COMMIT

Is REJECT too harsh for a borrowed-compute lab? Sometimes it is, and that is a tradeoff you should name. It is not an excuse to skip the function and hope the mean converges. Would you rather replay the shard under a new generation, or publish a score that cannot name its parents? I know which one I can defend in a review.

3. Put a watermark in front of the gate

Fan-in must close on committed shards, not on “we received N blobs.” A late R9 must not move the mean after the watermark. I would rather leave a documented hole than let L8 inherit L7’s score. The denominator is not “shards we launched.” The denominator is shards that reached a terminal bind decision.

4. Inject the two failure classes before you trust the lab

Readers should be able to run a discrete-event sketch, not admire a diagram. The sketch below is labeled unexecuted production code on purpose. I am not claiming wall-clock speedups, and I am not claiming a cluster shape. I am claiming that stale binds are countable.

# proposal / deterministic simulator — run locally, no network
import hashlib
import itertools
import random
from collections import Counter

PREEMPT = "preempt"
LATE = "late_complete"
DUP = "duplicate_complete"
OK = "ok"


def prompt_hash(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()[:16]


def simulate(n_shards=2000, seed=7):
    rng = random.Random(seed)
    stale_binds = 0
    decisions = Counter()
    for shard in range(n_shards):
        gen = 1
        live = DualPlaneId(
            batch_id="b1",
            shard_id=f"s{shard}",
            lease_id=f"L{shard}-g{gen}",
            lease_gen=gen,
            model_request_id=f"R{shard}-g{gen}",
            prompt_hash=prompt_hash(f"prompt-{shard}-g{gen}"),
        )
        events = [OK]
        if rng.random() < 0.25:
            events.append(PREEMPT)
        if rng.random() < 0.35:
            events.append(LATE)
        if rng.random() < 0.10:
            events.append(DUP)
        incoming = live
        payload_ok = True
        for ev in events:
            if ev == PREEMPT:
                gen += 1
                live = DualPlaneId(
                    batch_id="b1",
                    shard_id=f"s{shard}",
                    lease_id=f"L{shard}-g{gen}",
                    lease_gen=gen,
                    model_request_id=f"R{shard}-g{gen}",
                    prompt_hash=prompt_hash(f"prompt-{shard}-g{gen}"),
                )
            elif ev == LATE:
                # stale completion still carries gen-1 identity
                incoming = DualPlaneId(
                    batch_id="b1",
                    shard_id=f"s{shard}",
                    lease_id=f"L{shard}-g{gen-1}" if gen > 1 else live.lease_id,
                    lease_gen=max(gen - 1, 1),
                    model_request_id=f"R{shard}-g{max(gen-1, 1)}",
                    prompt_hash=prompt_hash(f"prompt-{shard}-g{max(gen-1, 1)}"),
                )
            elif ev == DUP:
                payload_ok = True
        d = bind(live, incoming, payload_ok)
        decisions[d.value] += 1
        if d is BindDecision.COMMIT and incoming.lease_gen != live.lease_gen:
            stale_binds += 1
    return decisions, stale_binds


if __name__ == "__main__":
    decisions, stale = simulate()
    print(dict(decisions), "stale_binds", stale)
    assert stale == 0, "stale completion bound to a live lease"

Run it as a property check, not as a benchmark ritual.

python dual_plane_bind_sim.py

The acceptance rule needs an explicit denominator. Denominator: shards that reached COMMIT, REJECT, or REPLAY. Numerator: commits whose DualPlaneId still names the generation that sent the prompt. Acceptance: zero stale binds under preempt plus late-complete injection for the simulated N. If you cannot say the denominator out loud, you do not have a gate. You have a vibe with error bars.

Tradeoff table

Choice	Latency	Correctness under preempt	Cost on borrowed planes	What you owe the next on-call
Bind on shard id only	Low	Breaks on folder reuse	Cheap until the gate lies	A green mean you cannot replay
Bind on request id only	Medium	Breaks when ids recycle	Wasted completions on REJECT	Orphan scores after reclaim
Dual-plane tuple + watermark	Higher tail	Holds in the simulator	More replays, fewer false commits	A hole you can name
Quorum of three model calls	Highest	Still needs the compute gen	Burns the free plane fast	A cost story, not an identity story

Notice the last row. Extra model votes do not repair a missing lease generation. Have you ever added another judge because the first one was “flaky,” when the real bug was a stale bind? That is how evaluation architecture turns into incense.

Where a free model plane and a free compute plane actually help

I wanted a lab that could exercise preemption and late completion without pretending I had a reserved fleet. That is a constraint, not a slogan, and it is why the fixture stays single process and deterministic.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I treat MonkeyCode as a place where free model access and a free server option can host this dual-plane rehearsal. I am not freezing model names, quotas, hardware, or duration into this review, because those claims go stale and I will not invent them. If you replay the fixture against that lab, keep the bind function as strict as the simulator. Soften it later only after you have a counterexample that the enum cannot name.

If you want to watch your own coordinator swallow a late R9, take the fixture to a free server and a free model endpoint on MonkeyCode and log the bind enum. That is the only ask. The article still works if you run the simulator on a laptop and never leave localhost.

What I would change next

Replace shard-folder reuse with generation-scoped workspaces, so L8 cannot see L7 bytes.
Add compare-and-swap on the bind record, keyed by the full tuple, not by shard id.
Split retry budgets per plane, because model-plane retries should not extend a dead compute lease.
Property-test the permutation of {preempt, late, duplicate, payload_corrupt} instead of one happy path.
Emit the rejected tuple as a first-class event, so the histogram can show holes without pretending they were zeros.

Would I add a third plane for tools? Not yet. Tool side effects belong inside prompt_hash or they belong in a different protocol. Mixing them into bind without a hash is how domain C sneaks back in.

Limitations, and who should not use this

Do not use this design if you already have a closed consensus protocol across eval workers. Do not use it as a production SLO, and do not treat the simulator as a latency study. The sketch ignores disk corruption, clock jumps, and prompt mutation after the hash. It also ignores billing identity across vendors, which is a different invariant with a different denominator.

Who should not use this approach? Anyone hoping a free plane will mint a statistically closed leaderboard without a bind protocol. Anyone who needs exactly-once side effects in the tool plane. Anyone who will “just average the late scores” because the batch is already late. That last group is the reason I opened with a green gate that should have stayed red.

The counterexample I still want from you

Which event order still breaks the invariant: a duplicate completion with a fresh request id, a preempt that returns during replay, or a prompt_hash computed after a tool write? Should the system reject, replay, or compensate? If you cannot pick one without waving at shard id, the protocol is not closed yet, and the gate is still guessing.

Make Pass Rate a Quorumed Histogram Before Late Workers Flip the Gate

Robin — Sun, 20 Sep 2026 10:38:51 +0000

Last Tuesday I traced a canary that flipped after a recycled worker delivered one late score. The dashboard showed a 0.81 pass rate, then 0.74, then 0.81 again, with no model change. Have you ever shipped a planner because a dashboard mean looked stable while stragglers were still rewriting bins? I had already marked the gate green before I noticed the histogram was not closed.

The violating order was almost boring once I laid the events on a whiteboard. Worker W1 scored run r7 against digest dA and the coordinator published a live mean. Worker W2, recycled after a preemption, scored a stale replica of r7 against digest dB and the gate crossed the threshold backward. Why do we still store pass rate as a mutable scalar when the underlying set of scores is still open?

This piece is an architecture review of that control plane, not a production war story with invented SLOs. I treat the numbers in the fixture as labels inside a simulator, and I ask you to break them. If the invariant does not survive injected event orders, the dashboard is theater.

Declared assumptions

I am reviewing a batch evaluator for long-running agent workflows, not an online recommender with sliding windows. Each run has a content-addressed artifact, and workers may restart on shared or free compute without warning. The ship gate reads a single pass rate, and retries are at-least-once. I assume clocks are not trusted across machines, so wall time cannot close a batch. I also assume duplicate delivery is normal, because queues retry after ack loss. If your evaluator never preempts and never retries, this protocol is heavier than you need. Do you actually have that luxury on shared inference?

Constraints I will not hand-wave

Latency, throughput, correctness, and cost pull in four different directions on this path. A live mean is cheap to render and easy to explain to a reviewer who wants a single number. Waiting for every worker is correct until one worker never returns, which is the common case on preemptible capacity. Quorum without a digest check is fast and still wrong, because two scores can share a run id and describe different bytes. Cost stays bounded only if retries cannot steal slots from fresh runs. Which constraint do you actually want to be load-bearing when the batch is half dead?

I refuse to treat “the model got worse” as the first hypothesis. The first hypothesis is that the merge protocol accepted an event it should have fenced. Evaluation architecture fails in the control plane long before it fails in the decoder.

Data flow and failure domains

Here is the sequence I want the coordinator to enforce. Notice that the gate never reads a scalar that workers can still mutate.

sequenceDiagram
    participant Q as Eval queue
    participant W as Worker
    participant C as Coordinator
    participant H as Histogram log
    participant G as Ship gate

    Q->>W: lease run r7, epoch e, digest dA
    W->>C: score(r7, dA, e, bin=pass)
    C->>H: append if digest+epoch legal
    Note over C,H: watermark closed_count, not wall clock
    H->>G: derive pass_rate from closed bins
    W-->>C: late score(r7, dB, e_stale)
    C-->>H: reject stale epoch / digest mismatch

I split the system into four failure domains on purpose. Domain A is the queue, which can redeliver, delay, or drop an ack. Domain B is the worker, which can preempt, restart with a new epoch, or finish a stale replica. Domain C is the coordinator log, which must append monotonically and never rewrite a published bin. Domain D is the gate, which must refuse to read an open histogram. If you collapse C and D into one mutable counter, a single late write moves the business decision. Is that really an evaluation system, or a shared integer?

The invariant the common implementation drops

Invariant. A ship/no-ship pass rate is legal only when it is derived from a closed histogram of scores whose (run_id, artifact_digest) pairs are unique, whose worker epochs are still fenced in, and whose accepted count meets a declared quorum denominator. Late, duplicate, or digest-mismatched events must not move any bin that the gate can read. The denominator is closed_accepted_count, not “runs we hoped to finish.” If you divide by intended count while stragglers still arrive, you are mixing two different experiments.

The common implementation stores pass_rate = passes / n on a row and updates it in place. That row is not a protocol. It is a race.

Numbered protocol I would actually implement

Fence the worker, not the hostname. Issue (run_id, epoch, digest) as a single lease. A restarted process must bump epoch before it can speak. Hostnames lie after recycle, and I will not key correctness on them.
Address the artifact, not the prompt text. Hash the frozen workspace bytes before the worker starts. If the digest changes, it is a different run, even when the run id looks familiar. Would you let two judges vote on different files and still call it one sample?
Append bins, never mutate a mean. The log stores counts per bin. The mean is a pure function of a closed snapshot. Derived numbers can be cached, but the cache is not the source of truth.
Close on watermark plus quorum, never on wall clock. Declare N intended runs, Q minimum accepted scores, and a max in-flight lease set. When accepted >= Q and no legal epoch remains in flight, the histogram closes. Wall time is a timeout for leases, not a close signal for science.
Classify every rejected event. Stale epoch is a reject. Digest mismatch is a reject and a new run proposal. Duplicate (run_id, digest, epoch) is an idempotent ack. After close, everything is a reject. Compensation, if any, allocates a fresh run_id rather than rewriting history.
Put backpressure in the same contract. If retry storms fill the lease set, refuse new admissions until in-flight epochs drain. Otherwise retries starve fresh eval and your denominator silently shrinks. Have you measured how many of last week’s “model regressions” were actually queue occupancy?

Minimal simulator you can run locally

This fixture is a proposal, not a production service. I ran it as a single-process event pump so the merge rules are visible. You should inject orders, not admire the happy path.

from collections import defaultdict
from dataclasses import dataclass

PASS, FAIL = "pass", "fail"

@dataclass(frozen=True)
class Score:
    run_id: str
    digest: str
    epoch: int
    bin: str

class HistogramLog:
    def __init__(self, quorum: int):
        self.quorum = quorum
        self.fenced_epoch = {}
        self.accepted = {}  # run_id -> (digest, epoch, bin)
        self.bins = defaultdict(int)
        self.closed = False
        self.rejects = []

    def fence(self, run_id: str, epoch: int) -> None:
        prev = self.fenced_epoch.get(run_id, -1)
        if epoch <= prev:
            raise ValueError("epoch must increase to fence a restart")
        self.fenced_epoch[run_id] = epoch

    def apply(self, s: Score) -> str:
        if self.closed:
            self.rejects.append(("after_close", s))
            return "reject_closed"
        if self.fenced_epoch.get(s.run_id, -1) != s.epoch:
            self.rejects.append(("stale_epoch", s))
            return "reject_epoch"
        prior = self.accepted.get(s.run_id)
        if prior:
            digest, epoch, bin_ = prior
            if (digest, epoch, bin_) == (s.digest, s.epoch, s.bin):
                return "idempotent"
            self.rejects.append(("digest_or_dup_conflict", s))
            return "reject_conflict"
        self.accepted[s.run_id] = (s.digest, s.epoch, s.bin)
        self.bins[s.bin] += 1
        if len(self.accepted) >= self.quorum:
            self.closed = True
        return "accept"

    def pass_rate(self):
        n = len(self.accepted)
        if not self.closed or n == 0:
            raise RuntimeError("gate cannot read an open histogram")
        return self.bins[PASS] / n

def replay(events, quorum=3):
    log = HistogramLog(quorum=quorum)
    actions = []
    for kind, payload in events:
        if kind == "fence":
            log.fence(*payload)
            actions.append("fence")
        else:
            actions.append(log.apply(payload))
    return actions, log

A counterexample that should fail the naive scalar, and pass this log, looks like the following event order. Read it as a test, not as decoration.

events = [
    ("fence", ("r7", 1)),
    ("score", Score("r7", "dA", 1, PASS)),
    ("fence", ("r8", 1)),
    ("score", Score("r8", "dA", 1, PASS)),
    ("fence", ("r9", 1)),
    ("score", Score("r9", "dA", 1, FAIL)),
    # recycled worker tries to rewrite r7 with different bytes
    ("score", Score("r7", "dB", 1, FAIL)),
    # restart with a new epoch, still too late after close
    ("fence", ("r7", 2)),
    ("score", Score("r7", "dB", 2, FAIL)),
]

actions, log = replay(events, quorum=3)
assert actions[-2] == "reject_conflict"
assert actions[-1] == "reject_closed"
assert log.closed and abs(log.pass_rate() - 2 / 3) < 1e-9
print(actions, dict(log.bins), log.pass_rate())

Run it with a stock interpreter. If pass_rate() is callable before close, your gate is already lying. If a digest mismatch can decrement a bin, you do not have a histogram, you have a tug of war. I want the fixture to make that argument boringly mechanical.

Failure analysis I actually care about

Preemption is not a rare ops event on shared capacity; it is a first-class input. A recycled worker that keeps its old epoch is a split brain, because two processes can both believe they own r7. Duplicate delivery after a successful append must be idempotent, or your counts inflate and the denominator becomes fan-out. Digest mismatch after a partial write is a new experiment, and folding it into the old run id launders contamination into the mean. Close-then-late is the flip I opened with, and it is the one dashboards love, because the line moves after someone already screenshotted it. Which of those four do you currently classify, and which do you still file as “model variance”?

I also inject a fifth class: retry storms that occupy every lease. Backpressure has to live beside the histogram, or correctness wins a tiny closed set while freshness dies. A perfectly fenced batch of twelve retries is not a canary. It is a congested queue wearing a lab coat.

Tradeoff table

Merge policy	Latency to a number	Correctness under preemption	Cost under retries	What the gate can claim
Mutable scalar mean	Lowest	Poor; late writes flip the gate	Unbounded retries rewrite the same row	A screenshot, not a contract
Wait for every worker	Highest	Good until one worker vanishes	Idle capacity while stragglers sit	Completeness you will not get
Quorum without digest	Medium	False consensus on mixed bytes	Medium	A majority of the wrong artifact
Quorumed histogram plus epoch fence	Medium, bounded by Q	Rejects stale epochs and mixed bytes	Bounded if leases backpressure	A closed denominator you can defend

I would pick the last row for ship gates and keep the live scalar, if you must, as a non-binding progress widget. Mixing those two audiences on one chart is how 0.81 becomes 0.74 becomes a postmortem. Are you designing for operators who need a pulse, or for a gate that must not blink?

Where free shared compute actually belongs

I needed a scratch place to execute the fixture and a cheap model endpoint to generate dummy traces, without standing up a reserved cluster. I used MonkeyCode’s free model access and free server option for that scratch path. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not asserting model names, quotas, hardware, duration, or permanence; those two availability options are the only product claims I am willing to stand on here. Shared free workers make preemption more likely, which is exactly why a mutable mean is the wrong gate. If you only ever run eval on pinned capacity, you can still use the log, but you will feel the protocol as overhead rather than as survival.

If you try the same fixture there, keep the product at the edge of the method. The invariant lives in the merge rules, not in the vendor logo. One pass through replay() on your laptop is enough to see the reject classes. A remote server is optional, and it should not become the argument.

Limitations, and who should not use this

This protocol is wrong for streaming product analytics that must fold late events into a sliding window by design. It is also not a substitute for power analysis, confidence intervals, or paired comparisons against a frozen baseline. Quorum Q can hide bias if the missing workers are not missing at random, and I did not model that selection effect here. Do not use a closed histogram as permission to skip artifact hashing. Do not use free shared servers for eval that cannot tolerate preemption even with fencing, such as hardware-in-the-loop jobs that cannot restart cleanly. If your audience needs incident dashboards and pager policies, that operational layer belongs elsewhere; this review stops at the merge contract.

The simulator is single process. It does not prove linearizability under a real log. Treat it as a property sketch you can promote into a replicated append-only store later.

What I would change next

I would stop exporting pass rate as a time series that updates in place. I would export closed snapshots with a snapshot id, a digest set, and an explicit denominator. I would add a property test that generates shuffled event orders instead of the one counterexample above. I would also split retry admission from score merge, because they fail independently and currently share one mental model. After that, I would canary the gate with three failure classes only: stale epoch, digest mismatch, and after-close delivery. If those three cannot move the snapshot, I will start arguing about models again. Until then, the model is not the suspect.

Readers can validate the conclusion without trusting me. Run the fixture. Shuffle the event list. Add a worker that reuses run_id with a new digest before close. If your current evaluator still returns a number, you found the bug the dashboard will not name.

Which event order still moves your pass rate after close, and should the system reject, replay, or compensate?

Make Judge Binding a Closed Contract Before Pool Rebalance Flips Pass/Fail

Robin — Sat, 19 Sep 2026 08:57:46 +0000

I start this architecture review with a canary that went red while candidate traces stayed byte-identical. The rubric file had not moved, and the planner graph was untouched. The judge prose had drifted overnight anyway. Have you ever signed a pass/fail bit that was authored by a critic you never bound?

That event order is the bug, not a mysterious model regression. The candidate finished under binding B1, the pool rebalanced, and the scorer ran as B2. The log still treated the row as one closed eval. I want that mix to be a protocol violation, not a flaky-model ticket.

The violating order

Here is the counterexample I keep on the whiteboard during a control-plane review. It is short, ugly, and enough to dissolve a release gate.

Enqueue records run_id=R, candidate_binding=C1, and judge_binding=J1.
The candidate stream completes, and the tool log is sealed for R.
The pool rebalances, and the next completion is routed to J2.
A score for R is appended with no fingerprint check against J1.

Did the system reject, replay, or compensate? In the common implementation it just commits. Why do we debug the model when the contract already dissolved?

Invariant

A persisted eval outcome is valid only when four fields close together: run_id, candidate_binding, judge_binding, and score_digest. If any field is rewritten after bind, the outcome is not the same object. I do not care that both critics are good enough for a demo. I care that the bit we ship is the bit we contracted.

Call the invariant closed judge binding. Once bind() returns, scoring traffic for that run_id must carry the same binding token, or the write is refused. Anything else is a different experiment wearing the original name.

Assumptions

I am reviewing a shared-pool eval service, not a single-process notebook. These assumptions stay in force for the rest of the design, and you should reject the conclusion if yours differ.

Candidate and judge may be scheduled on different workers, and those workers do not share memory.
The model pool can rebalance, preempt, or replace a replica between any two RPCs.
Score rows outlive the worker that wrote them, and downstream gates read those rows as truth.
Bindings are opaque tokens with an equality test; I am not assuming a stable vendor model name.
Readers will run the simulator below as the acceptance test, not a production SLO board.

If your judge is a deterministic program with no model pool, you can skip this contract. If you intentionally ensemble several critics, you need a different object: a quorum, not a singleton bind.

Constraints I will not relax

Latency, throughput, correctness, and cost pull in four directions here. I will not pretend they agree, and I will not hide the conflict behind a retry loop.

The control plane must keep enqueue cheap, because eval traffic is bursty around a release. It must not hide a pool swap inside a 200 from the scorer. It must not bill a second full candidate run unless replay was explicit. And it must survive shared-pool preemption without corrupting the pass bit. Which of those would you drop first? I will not drop correctness of the score row.

Data flow

The ledger, not the HTTP client, owns equality. Workers are allowed to die. The score log is not allowed to forget which critic signed the bit.

sequenceDiagram
    participant P as Planner
    participant B as BindingLedger
    participant C as CandidatePool
    participant J as JudgePool
    participant L as ScoreLog

    P->>B: bind(run, C1, J1)
    B-->>P: token T
    P->>C: complete(run, T)
    C-->>P: transcript
    Note over J: rebalance replaces J1 with J2
    P->>J: score(run, T, transcript)
    J-->>P: score plus observed_binding
    P->>B: compare(T, observed_binding)
    alt mismatch
        P->>L: reject or replay
    else match
        P->>L: commit(run, T, score)
    end

Notice the compare sits on the planner side of the log. A judge replica that cannot echo identity is not a scorer. It is an untrusted completion source, and I would fail that path closed.

Failure domains

I split the system into four domains so a rebalance cannot impersonate a rubric change. If you draw only one box labeled eval, you will keep filing model bugs forever.

Binding ledger. This is the source of truth for tokens. If it is down, enqueue must fail closed. A stale cache of bindings is a silent mix, which is worse than a hard error.

Candidate pool. Preemption here should fence the transcript, not the judge. A restarted candidate with the same token may replay work. It may not mint a new binding under the old run_id.

Judge pool. This is the dangerous domain on a shared pool. Rebalance, warm-start, and sticky-session loss all present as the scorer returned. I treat an identity mismatch as a transport error, not as a slightly different grade.

Score log. Append-only rows need the token in the primary key. A late score without a token is garbage, even when the numeric grade looks plausible. Would you merge that row because the dashboard went green?

Treat routing as a binding protocol

I want the implementation to look like a protocol, not a retry wrapper. Stickiness can reduce mismatch rate, but it is not a lock. Have you been using affinity as if it were a compare-and-set?

Here is the closed-contract path I would require before the service scales.

Mint. On enqueue, write (run_id, candidate_binding, judge_binding, epoch) in one compare-and-set.
Stamp. Every candidate and judge RPC carries the token; responses echo the observed identity.
Compare. Before append, the planner checks observed == bound. Inequality is not retried against a new replica.
Decide. Mismatch chooses reject, replay-with-rebind, or compensate; it does not fall through to commit.
Seal. A committed row stores the token beside the score digest so later audits can re-verify equality.

The missing piece in most harnesses is step three. People retry the scorer, get a warmer replica, and call that resilience. I call it a contract rewrite.

Minimal simulator

The fixture below is a proposal you can run locally. It is not a production service, and I am not claiming cluster measurements against it. Run it until the empty-log assertion bores you.

from dataclasses import dataclass


@dataclass(frozen=True)
class Binding:
    run_id: str
    candidate: str
    judge: str
    epoch: int


class ScoreLog:
    def __init__(self):
        self.rows = {}

    def commit(self, b: Binding, digest: str) -> None:
        key = (b.run_id, b.judge, b.epoch)
        if key in self.rows:
            raise ValueError('duplicate score')
        self.rows[key] = digest


class JudgeBinder:
    def __init__(self):
        self.bound = {}
        self.log = ScoreLog()
        self.pool_judge = 'J1'

    def bind(self, run_id: str, candidate: str, judge: str, epoch: int) -> Binding:
        b = Binding(run_id, candidate, judge, epoch)
        if run_id in self.bound:
            raise ValueError('already bound')
        self.bound[run_id] = b
        return b

    def rebalance(self, new_judge: str) -> None:
        self.pool_judge = new_judge

    def score(self, run_id: str, digest: str) -> str:
        b = self.bound[run_id]
        observed = Binding(b.run_id, b.candidate, self.pool_judge, b.epoch)
        if observed != b:
            return 'reject'
        self.log.commit(b, digest)
        return 'commit'


def test_rebalance_must_not_commit():
    h = JudgeBinder()
    h.bind('R', 'C1', 'J1', epoch=1)
    h.rebalance('J2')
    assert h.score('R', 'pass') == 'reject'
    assert h.log.rows == {}


def test_stable_pool_commits_once():
    h = JudgeBinder()
    h.bind('R', 'C1', 'J1', epoch=1)
    assert h.score('R', 'pass') == 'commit'
    try:
        h.score('R', 'pass')
        raise AssertionError('duplicate must fail')
    except ValueError:
        pass


if __name__ == '__main__':
    test_rebalance_must_not_commit()
    test_stable_pool_commits_once()
    print('ok')

Run it with python judge_binder.py. The interesting assertion is the empty log after rebalance. If your harness still writes a row, the invariant is already dead, and prompt edits will not revive it.

Injected failures I actually care about

I do not inject the model got worse. I inject order, because order is what a shared pool actually breaks. If your test suite only perturbs temperature, you are measuring the wrong failure class.

Rebalance after candidate completion, before score. Must reject or replay-with-rebind.
Double score with the same token. Second append must fail on the composite key.
Rebind with a new epoch, then score. Only the new epoch may commit; the old token is tombstoned.
Score that echoes a different candidate binding. Reject even if the judge matches, because mixed candidate identity is a different run.
Lost ack after a successful commit. Replay may read the row; it may not create a second digest.

If you cannot name which of those five your service currently mishandles, you do not have an eval architecture. You have a script that sometimes prints green.

Tradeoffs

Choice	What you keep	What you pay	When I would pick it
Reject on mismatch	Closed contract, cheap enqueue	More red canaries	Default for release gates
Replay candidate under a new bind	Eventual score	Double candidate cost, longer tail	Nightly research sweeps
Dual-judge quorum	Tolerance to one bad replica	Extra critic spend, tie-break rules	Safety-critical graders
Sticky routing only	Lower mismatch rate	Silent mixes when stickiness drops	Never, as the sole control
Compensate with a human review queue	Explicit uncertainty	Slow gates, review load	Tiny gold sets

Sticky routing is the tempting row. It looks free until the pool rebalances on a quiet night. I will take reject as the default, with replay as an explicit operator action. Quorum is a different protocol, and I will not smuggle it in through retries.

The denominator for any canary is committed rows that still match their bind token, not jobs that returned 200. An acceptance rule I would ship is simple: mismatch rate must be zero in the score log, and retries live in a separate replay counter. If you mix those counters, you will congratulate yourself for availability you did not keep.

Where a spare shared pool actually helps

I keep this simulator local until the protocol is boring. When I want the same fixture sitting next to a live shared pool, I need a spare worker that can die without taking the ledger with it. That is a scheduling constraint, not a reason to skip the compare.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, which is a reasonable place to park this harness while you inject rebalance and preemption. If you try it, run the five failures above before you trust a dashboard color; the protocol still owns the pass bit.

What I would change next

The binder above is a single-process sketch, and I would not ship it as the ledger. The next revision I want is a compare-and-set on a real log, with the token in the row key. I would also stamp epoch onto the candidate transcript so a late judge cannot score a preempted draft. I would not add another status page before that key exists. Would a prettier graph have caught J1 versus J2? Not if the row schema collapsed them.

I would also stop treating empty judge identity as a default. An unbound score path is how pool swaps sneak in. Fail closed at mint, or do not call it an eval service. The planner should refuse to advance when echo identity is missing, even if the numeric grade looks confident.

Limitations, and who should not use this

This contract assumes you persist scores and later treat them as gates. Ad-hoc chat experiments do not need it, and forcing a ledger there is ceremony. Multi-critic ensembles that want disagreement need a quorum object with declared members, not a singleton bind you keep swapping. If your pool cannot echo an observed identity, you cannot implement compare, and you should not fake it with a local config file.

I also will not claim this removes variance inside one bound model. It only stops you from attributing a routing mix to the model. Do not use reject-by-default if a missed nightly score is more expensive than a mixed critic. In that world, pick explicit replay, and pay for it in the budget line, not in corrupted history.

Validate the conclusion

You do not need my word. You need an order, a log, and a property that stays true after injection.

Bind a run and seal a transcript.
Change the pool identity without minting a new epoch.
Attempt score and watch the ledger.
Confirm the score log stayed empty, or a replay row was written under a new token.
Re-run with a stable pool and confirm a single commit.

If step 4 writes a row under the old token, the design does not converge. Fix the compare, not the prompt. Which event order still breaks closed judge binding in your tree, and should the system reject, replay, or compensate?

Make Transcript Append a Dual-Key Compare Before Preempted Workers Mix Eval Runs

Robin — Fri, 18 Sep 2026 06:59:52 +0000

Here is a violating event order I keep on the whiteboard during architecture reviews of agent eval harnesses. Run R1 starts a tool call on shared worker W, then a preemption notice arrives before the tool result is fenced. Worker W later resumes under run R2, and the late tool event from R1 is appended to R2's transcript as if causality still held. Does your scoreboard treat that as a successful tool use, or as a protocol violation?

Most harnesses I review treat resume as a convenience restart, and they silently mix two runs. The invariant the common implementation fails to preserve is not exactly-once tool execution in the abstract. Every transcript event must carry a closed envelope of run_id and lease_generation before any append is legal. Why do teams miss this during design review, when resume looks local and the failure is cross-run?

If either field disagrees, the event is poison for scoring, not a retry candidate you can reshape. This is a proposed architecture review with a labeled simulator, not a customer postmortem or metric dump. I will not invent production numbers, because the point is the event order, not a dashboard.

1. Declared assumptions

I am reviewing a shared eval pool, not a sticky accelerator reserved for a single candidate run. The assumptions below are load-bearing, and they change the protocol you should accept. If they do not match your world, the handshake I want is the wrong cost.

Workers are preemptible, including scavenged or free servers that can vanish in the middle of a step.
Tool results are delivered at least once, and they can arrive after the planner has already moved on.
Clocks are not a source of truth across workers, so wall time cannot fence causal order.
The scoreboard is the system of record for pass or fail, and mixed transcripts are invalid measurements.
Model identity may rotate between leases on a free pool, so a resume is not the same closed world.

If your eval never shares a worker and never resumes, this review is the wrong document for your constraints. Skip it and keep a simpler sticky-session model. Are you actually measuring one closed world, or just concatenating whatever the process still had in memory?

2. Data flow and sequence model

I want the happy path and the violating path in the same picture, because reviews that only draw success hide the invariant. The objects are a planner, a worker lease manager, a tool bus, and a scoreboard. Ask yourself which process is allowed to append, and under which key.

sequenceDiagram
    participant P as Planner
    participant L as LeaseManager
    participant W as Worker
    participant T as ToolBus
    participant S as Scoreboard

    P->>L: acquire(run=R1)
    L-->>W: lease(R1, gen=7)
    W->>T: tool_call(env={R1,7})
    Note over W: preemption
    L->>W: fence(R1, gen=7)
    P->>L: acquire(run=R2)
    L-->>W: lease(R2, gen=8)
    T-->>W: late_result(env={R1,7})
    alt naive resume
        W->>S: append(R2, late_result)
    else dual-key compare
        W->>S: reject(stale envelope)
    end

The naive branch is the counterexample that most just-retry-the-worker designs will accept without blinking. The dual-key branch is the protocol I want the harness to speak under preemption. If your sequence diagram has no reject arrow, you have not modeled resume. What does your current diagram do with that late result?

3. Failure domains

I split the system into four domains because a single worker-crashed ticket hides the real coupling. Domain A is the lease manager, which is the only writer of generation. Domain B is the worker runtime, which holds an uncommitted transcript buffer. Domain C is the tool bus, which is at-least-once and unordered relative to leases.

Domain D is the scoreboard, which must refuse mixed envelopes or the eval is not a measurement. Cross-talk happens at the seams, not inside a neatly drawn box. A preemption in B without a fence visible to C leaves a poison message in flight. Do you isolate those seams in tests, or only in architecture slides?

4. The resume handshake, in numbered steps

I would not add retries and call the review finished. I would treat resume as a compare-and-append protocol with an explicit reject. The steps below are the minimum I would demand before anyone scales the shared pool.

Mint a lease as the pair (run_id, generation), and never reuse generation for that run_id under any retry story.
Stamp every tool call and every partial transcript chunk with that pair before the bytes leave the worker.
On preemption, publish a fence for the old pair and drop the local uncommitted buffer instead of flushing it.
On resume, require a handshake: the worker presents the pair it still believes, and the manager returns live or stale.
Append to the scoreboard only with a dual-key compare on (run_id, generation), never with process liveness alone.
Dead-letter stale envelopes; do not replay them into a new lease, and do not compensate by guessing intent.
Close the eval only when the scoreboard has a join of fenced leases, not when the worker process is merely alive.

If step four is a log line instead of a decision, you do not have a handshake. You have hope dressed up as recovery. Would you accept that hope in a payment ledger, or only in an eval because the score feels softer?

5. Minimal simulator, labeled as a proposed fixture

This fixture is proposed and unexecuted in your environment until you run it yourself. It is small on purpose, because I want the invariant to fail in a few dozen steps. I do not want a cluster bring-up to hide a one-line compare.

from dataclasses import dataclass, field
from typing import List, Optional, Tuple

Envelope = Tuple[str, int]  # (run_id, generation)

@dataclass
class Lease:
    run_id: str
    generation: int
    live: bool = True

@dataclass
class Scoreboard:
    transcripts: dict = field(default_factory=dict)
    rejects: List[Envelope] = field(default_factory=list)

    def append(self, live: Lease, env: Envelope, payload: str) -> str:
        if not live.live or env != (live.run_id, live.generation):
            self.rejects.append(env)
            return "reject"
        self.transcripts.setdefault(live.run_id, []).append(payload)
        return "accept"

@dataclass
class Worker:
    lease: Optional[Lease] = None
    buf: List[tuple] = field(default_factory=list)

    def attach(self, lease: Lease) -> None:
        self.lease = lease
        self.buf.clear()

    def tool_call(self, payload: str) -> Envelope:
        assert self.lease and self.lease.live
        env = (self.lease.run_id, self.lease.generation)
        self.buf.append((env, payload))
        return env

    def resume_handshake(self, presented: Envelope, live: Lease) -> str:
        if presented != (live.run_id, live.generation):
            self.buf.clear()
            return "stale"
        self.lease = live
        return "live"

def simulate_crosstalk(dual_key: bool) -> dict:
    sb = Scoreboard()
    w = Worker()
    r1 = Lease("R1", 7)
    w.attach(r1)
    env = w.tool_call("tool:search")
    r1.live = False
    r2 = Lease("R2", 8)
    presented = env  # poisoned resume still holding R1
    decision = w.resume_handshake(presented, r2)
    if not dual_key:
        # naive path lies about the envelope and appends under R2
        result = sb.append(r2, ("R2", 8), "tool:search")
    else:
        result = sb.append(r2, env, "tool:search")
    mixed = any(
        "tool:search" in xs
        for rid, xs in sb.transcripts.items()
        if rid != "R1"
    )
    return {
        "decision": decision,
        "append": result,
        "mixed": mixed,
        "rejects": sb.rejects,
    }

Run the two branches as a property check, not as a demo screenshot someone might ignore. I care about the boolean mixed, not about a throughput number nobody measured on this fixture. Save it as test_resume_envelope.py and keep the assertions boring.

def test_naive_mixes_runs():
    out = simulate_crosstalk(dual_key=False)
    assert out["mixed"] is True

def test_dual_key_rejects_poison():
    out = simulate_crosstalk(dual_key=True)
    assert out["mixed"] is False
    assert out["append"] == "reject"
    assert out["rejects"] == [("R1", 7)]

python -m pytest test_resume_envelope.py -q

If the naive test does not fail on your harness, your harness is already lying about measurement. That is the entire point of the fixture. Can your current resume path fail this in under a second, or only after a human reads logs?

6. Injected failures and testable properties

I would inject three schedules before anyone argues about model quality on a shared pool. The first is preemption after tool_call and before the result envelope returns. The second is a late envelope after the generation has already bumped. The third is a double resume where two workers believe they hold generation eight.

Properties I would gate on, with an explicit denominator, look like this. P1 says every accepted append has envelope.run_id equal to the live lease run. P2 says every accepted append has envelope.generation equal to the live lease generation. P3 says mixed-transcript count is zero per one thousand injected preemption schedules.

P4 says stale envelopes are rejected or dead-lettered, never compensated by copying payload into a new run. The denominator is injected preemption schedules, not wall-clock hours and not tokens consumed. Acceptance is P3 equal to zero under the three schedules above. If you cannot state the denominator, you are not canarying a protocol.

7. Tradeoff table

I want the cost in the same row as the failure class, because correctness without a load model is just a preference. Sticky workers look clean until the free pool reclaims the box. Dual-key compare looks chatty until you price a polluted score.

Design	Correctness under preemption	Extra latency	Density on a shared pool	What I would not pretend it solves
Naive process resume	Mixes runs; scores stop being measurements	Lowest	Highest apparent	Cross-talk and closed-world eval
Sticky worker per run	Holds if the worker never dies	Medium	Poor when slots vanish	Preemption, which is in the contract
Dual-key compare-and-append	Rejects poison; eval stays closed-world	One handshake on resume	Recovers the slot	Semantic splice of an old prefix
Isolated machine per eval	Strong isolation	Highest	Worst	Short unit evals that never resume

I would pick dual-key compare for shared pools, and sticky workers only when preemption is outside the contract. Full isolation is a different product, not a resume protocol. Which row are you actually operating, when the worker image is free and therefore reclaimable?

8. Where a free shared pool actually matters

Shared, preemptible workers are not a hypothetical topology for this review, because spare capacity is how many eval pools actually run. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option recreate that shared shape, which is enough to host this fixture. I am not attaching a model name, a quota, a hardware SKU, or a duration, because the handshake does not need them.

If you already have an isolated lab, run the same fixture there and ignore any vendor path entirely. If you want a scratch box, the free server option is sufficient to host the simulator process. The interesting work remains the dual-key compare, not the signup form.

9. What I would change next

Generation integers are a weak vector clock, and I would not stop there after the gate turns green. I would stamp each tool envelope with a parent hash of the last committed transcript prefix. A replay then cannot splice a semantically old payload into a live generation that merely shares an integer.

I would also split the scoreboard append into a two-phase intent, because dual writers still exist between the worker buffer and the log. Would I do that before the dual-key gate is green? No. First close the mixed-transcript hole, then raise the bar. The next review should start from a forged prefix, not from another process restart.

10. Limitations, and who should not use this

Do not adopt this handshake if every eval is a single process with no resume and no shared cache. You will pay a round trip for a failure class you cannot hit. Do not use the fixture as evidence of model quality, because it measures harness causality, not answers. Do not treat reject as compensate: copying a stale tool payload into a new run is how you launder poison.

And do not ask this review for dashboards, paging playbooks, or deploy scripts. Those are a different failure domain than the protocol, and they will not make a mixed transcript become a valid measurement. If your world is one laptop and one process, keep the sticky model and save the handshake for later.

Which event order still breaks the invariant after dual-key compare lands? A tool result that carries the live pair but a prefix from a previous world, because generation never saw the bytes. Should the system reject that append, replay the tool under a new generation, or compensate from the dead-letter? I would reject, then replay only with a fresh envelope, because anything else pretends the eval still measured one closed world.

Make Sandbox Epoch a Fencing Token Before Free Compute Vanishes Mid-Write

Robin — Thu, 17 Sep 2026 05:03:10 +0000

Here is the event order I use as a paper counterexample when someone treats a free sandbox as durable storage. The planner sends apply_patch, the sandbox answers 200, and the saga steps forward as if the write were fenced. Then the replica is reclaimed, a retry lands on cold disk, and the file is gone. Did the tool succeed? The HTTP log says yes. The filesystem says no.

Would you let the planner commit that step? I would not, not on a replica that can vanish without a fencing token. Free compute is useful, but it is a preemptable replica, not a disk you own. This article is an architecture review of that constraint, not a production war story and not a benchmark.

The invariant the common sketch fails

The common sketch stores tool_status = success as soon as the sandbox returns a 200. That sketch assumes the replica that accepted the write is the replica that will still exist on the next tick. Free servers break that assumption in the most ordinary way: they go away.

Invariant. A planner must not commit a mutating tool step until a write receipt joins a live sandbox epoch, and that epoch is still the fencing token for the replica. If the epoch has moved, the system must reject or compensate. Replay without a new fence is how you mint a second write.

That is a protocol question, not a model-quality question. If your eval harness treats a vanished sandbox as a flaky model, you will debug the wrong layer.

Declared assumptions

I am reviewing a candidate design on paper, with these assumptions held fixed.

The planner is the only component allowed to commit saga state.
The sandbox is a remote workspace that may be preempted between any two messages.
Mutating tools (write_file, apply_patch, run_migration) require a fenced write; read-only tools do not.
Duplicate delivery is possible after a timeout, an ack loss, or a reconnect.
We do not assume dedicated hardware, reserved capacity, or a measured SLA.
The fixture below is a proposed simulator, not an executed production test.

If your sandbox is a reserved VM with a disk you control, this review is heavier than you need. If your sandbox is a free replica, this is the failure class you should canary first.

Constraints and data flow

The design has three objects, and only one of them is allowed to tell the truth about commits.

Planner log. Append-only saga steps. A step is pending, fenced, committed, or compensated.
Epoch lease. A monotonic integer owned by the current sandbox generation. Heartbeats keep it live.
Replica disk. Ephemeral. Writes are visible only while the epoch that accepted them is still live.

Data flow is narrow on purpose. The planner mints a step id, the sandbox returns {epoch, receipt}, and the planner commits only on a join of those two values. No other signal, including a 200 with an empty body, is allowed to move the saga.

planner --(step_id, tool, epoch?)--> sandbox
sandbox --(epoch, receipt | preempted)--> planner
planner --(commit | reject | compensate)--> saga log

Notice what is missing. There is no path from “the model sounded confident” into the commit bit. There is also no path from “the HTTP client did not throw” into the commit bit. Those are observations, not fences.

Failure domains

I split the system into four domains because they fail independently, and mixing them hides the counterexample.

Domain	What it owns	How it fails
Planner process	Saga log, step ids	Crash after receipt, before commit
Network	In-flight apply and heartbeat	Loss, reorder, duplicate
Sandbox replica	Epoch, ephemeral disk	Preempt, cold start, silent drop
Model loop	Tool arguments	Retry with the same step id

The interesting collision is replica preemption plus a late 200. The planner sees success from a generation that no longer exists. If you commit there, you have advanced past a write that the next replica cannot see. That is not eventual consistency. That is a lost update with a green dashboard.

Can the model loop save you? No. A retry that does not carry the new epoch will either no-op on a dead generation or double-apply on a live one. The fence has to sit under the tool, not inside the prompt.

Sequence model

I keep a tiny state machine on paper. If you cannot draw the join, you cannot test it.

sequenceDiagram
    participant P as Planner
    participant L as Epoch lease
    participant S as Sandbox
    P->>L: acquire(step_id)
    L-->>P: epoch=7
    P->>S: apply_patch(step_id, epoch=7)
    Note over S: replica preempted, epoch becomes 8
    S-->>P: 200 receipt epoch=7 (late)
    P->>L: join(receipt.epoch, live_epoch)
    L-->>P: mismatch, reject
    P->>P: compensate or re-acquire

The late 200 is the violating event. A common implementation stores success on that 200 and never asks the lease. The invariant dies on that single message.

Build the fencing protocol in numbered steps

This is the workflow I want a reviewer to implement against a local simulator first. Remote wiring comes after the properties pass.

Mint a step id in the planner, not in the sandbox. The replica does not own identity, because it can die.
Acquire an epoch before any mutating tool. Treat the epoch as a fencing token, not as a cache key.
Send the epoch with the write. A sandbox that sees a stale epoch must refuse the body, even if the path is valid.
Heartbeat the epoch on a shorter period than your preemption budget. A missed heartbeat is preemption, not a slow model.
Join receipt and live epoch before committed. A 200 with a stale epoch is a reject, not a retry hint.
Compensate on mismatch when the tool had side effects outside the replica. Replay only after a fresh acquire.
Record the denominator. Count mutating_steps, not wall time, when you say the protocol held.

If step four feels operational, ask the question a different way. Are you measuring liveness of the replica, or are you hoping the next tool call will notice the disk is empty? Hope is not a fence.

Minimal simulator

The fixture is deliberately small. It is meant to be copied into a unit test, not wrapped in a framework. Label it as unexecuted until you run it on your machine.

from dataclasses import dataclass

@dataclass
class Replica:
    epoch: int = 0
    live: bool = False
    disk: dict = None

    def __post_init__(self):
        self.disk = {}

    def acquire(self):
        self.epoch += 1
        self.live = True
        self.disk = {}
        return self.epoch

    def preempt(self):
        self.live = False
        self.disk = {}
        # Epoch advances so a late receipt cannot fence a new generation.
        self.epoch += 1

    def write(self, step_id, epoch, path, body):
        if not self.live or epoch != self.epoch:
            return {"ok": False, "reason": "stale_epoch", "epoch": self.epoch}
        self.disk[path] = body
        return {"ok": True, "receipt": f"{step_id}:{epoch}", "epoch": epoch}


class Planner:
    def __init__(self, replica):
        self.replica = replica
        self.log = []  # (step_id, state, epoch)

    def mutating_step(self, step_id, path, body, preempt_before_join=False):
        epoch = self.replica.acquire()
        self.log.append((step_id, "pending", epoch))
        ack = self.replica.write(step_id, epoch, path, body)
        if preempt_before_join:
            self.replica.preempt()
            # Late 200 still in hand. Join must fail.
        live = self.replica.live and ack.get("epoch") == epoch and ack.get("ok")
        if not live:
            self.log.append((step_id, "rejected", epoch))
            return "reject"
        self.log.append((step_id, "committed", epoch))
        return "commit"


def test_late_receipt_cannot_commit_after_preempt():
    p = Planner(Replica())
    assert p.mutating_step("s1", "/app.py", "x", preempt_before_join=True) == "reject"
    assert ("s1", "committed", 1) not in p.log


def test_live_epoch_may_commit():
    p = Planner(Replica())
    assert p.mutating_step("s2", "/app.py", "x") == "commit"
    assert p.replica.disk["/app.py"] == "x"

Run it as a module and keep the assertion names as the acceptance rule. I care about one denominator: mutating steps that reached committed without a live matching epoch. That count must stay zero.

python -m pytest -q sandbox_epoch_fence.py

Inject three failure classes before you call the protocol done: preemption before write, preemption after write but before join, and duplicate apply with a stale epoch. If any of those three can mint a committed row, the design is not ready for a remote replica.

Tradeoffs

Choice	What you gain	What you pay
Commit on HTTP 200	Low latency, simple client	Lost writes after preemption
Epoch fence + join	Rejects late receipts	Extra round trip, more rejects
Always replay the tool	Recovers empty disks	Duplicate side effects
Compensate on mismatch	Honest saga state	You must write compensations
Dedicated replica	Fewer preemptions	Cost, and you still need a fence

I would not pick the first row for any mutating tool, even on a quiet afternoon. Latency you cannot attribute to a live epoch is not latency you can spend. Throughput without a denominator is a story, not a model.

Where a free remote server actually fits

You still need a replica that is allowed to vanish, because that is the failure class the simulator is canarying. A reserved box will under-inject preemption and give you a false pass. This is the one place I would use a free remote sandbox on purpose.

MonkeyCode is an open-source project with free model access and a free server option, which matches that preemptable-replica role. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming a quota, a model list, a hardware SKU, or a duration. I am claiming a failure domain: a workspace you do not own can disappear between apply and join.

Wire the same epoch header you used locally. If the remote process dies, treat the next connect as preempt(), not as a retry of the old generation. If you cannot attach an epoch to the workspace, you do not yet have a fence. You have a hope that the next prompt will notice.

If you try that path, keep promotion out of the acceptance rule. The rule is still zero committed steps with a stale epoch, measured over mutating steps, with preemption injected on purpose.

Limitations, and who should not use this

This review is for planners that mutate a remote workspace and then continue a saga. It is not a general reliability sermon.

Do not use a free replica as the system of record for billing, medical state, or anything you cannot compensate. Do not skip the fence because the model is “usually right.” Do not treat this simulator as a load test. It has no arrival process, no tail latency, and no claim about tokens.

Skip this design if every tool is read-only, or if the sandbox is a container your orchestrator fences with its own generation number. In that case you already have an epoch. You should still join it. You should not invent a second one.

What I would change next

The current fence is a single integer. That is enough to kill the late-200 counterexample, and it is not enough to prove a prefix after a replica revives with leftover disk. Next I would pair epoch with a log index, so a revived sandbox can say “I still have writes 1..k under epoch 7” instead of wiping on every acquire.

I would also split reject from compensate in the planner. Reject is the right answer when the write never became visible. Compensate is the right answer when a side effect may have escaped the replica, such as an email or a payment intent. Blending those two into a blind replay is how you get the duplicate-apply bug the fence was meant to stop.

Finally, I would property-test event order instead of listing three cases. Generate shuffles of {acquire, write, preempt, late_receipt, retry} and assert the invariant on every shuffle. Three unit tests are a tutorial. The shuffle is the review.

Acceptance rule

Declare the load as N mutating steps with preemption injected on at least one third of them. The protocol holds only if committed steps with a mismatched epoch equal zero, and if every rejected step is either retried under a new epoch or compensated. If you cannot state that sentence with a denominator, you are not ready to scale the planner.

Which event order still breaks the invariant in your sketch, and should the system reject, replay, or compensate? If the answer is “the 200 was green, so commit,” the replica is already gone, and the saga has moved without a fence.

Make Remote Eval Workspace a Hash-Fenced Snapshot Before Late Scores Stick

Robin — Wed, 16 Sep 2026 03:29:06 +0000

I was reviewing a planner design last Tuesday when a sequence trace showed a green score landing on a rewritten step. A remote worker had started pytest against generation seven, stalled hard, then posted success after generation eight already existed. Does that failure look like a weak model to you, or does it look like a missing snapshot protocol around the checkout? I am writing this as an architecture review of constraints, data flow, and failure domains, not as an operations runbook.

Most pipelines farm coding evals to a remote box and then store the first JSON blob that resembles a test summary. That pattern assumes the worker still holds the workspace the planner intended to score, which is a lie under preemption. I want a tighter invariant than job completion: a verdict may attach only when its tree hash still equals the step's current workspace generation.

The violating event order

I keep returning to one counterexample because it is small enough to simulate and still wrecks a scoreboard.

The planner writes workspace generation seven and records tree hash H7 before dispatch.
Worker W1 leases that job and starts the test command against the H7 tree.
The planner times out, compensates, and writes generation eight with a new hash H8.
W1 finally wakes and uploads a pass record keyed only by job id, and the store overwrites the live step.

Which field in that JSON payload actually names the workspace tree you believe you scored just now? If you answer "the job identifier," you already lost, because identifiers survive generation bumps without describing contents. The invariant the common store fails to preserve is verdict.tree_hash == step.workspace_hash at attach time, otherwise the write must be rejected.

Declared assumptions

I assume a planner that owns one mutable checkout per eval step and farms tests onto an untrusted remote worker process. I assume that worker may be preempted, retried, or reused, and that wall clocks across hosts are not a shared truth. I assume the planner can hash the workspace before dispatch and that the score store can refuse a compare-and-swap. I do not assume exactly-once delivery, sticky leases, or a dedicated machine per generation.

Why am I this loud about assumptions before showing a diagram? Because a remote host is a separate failure domain even when you launched it from the same repo. If you skip the hash check, you are trusting scheduler luck to preserve correctness.

Data flow and failure domains

sequenceDiagram
    participant P as Planner
    participant S as Score store
    participant W as Remote worker
    P->>S: record gen=7 hash=H7
    P->>W: dispatch job, gen, hash
    Note over P: timeout, compensate
    P->>S: record gen=8 hash=H8
    W->>S: late pass for gen=7
    S-->>W: reject_stale
    P->>W: dispatch gen=8
    W->>S: pass for H8
    S-->>P: attached

I split three failure domains on purpose so the review does not collapse into a single happy path. The planner can crash after bumping generation and leave a live worker holding a dead tree. The score store can accept a write the planner would have rejected if it still sat on the request path. The remote worker can sleep for minutes and then finish work against a checkout that no longer exists.

Ask yourself which domain your current design treats as source of truth for pass and fail. If the worker is canonical, a preempted process can rewind the step after you already compensated. If the planner is canonical but the key is only a job id, a retry still collides on the same slot.

Constraints I refuse to relax

Latency of a coding eval is dominated by dependency install and test runtime, not by comparing a thirty-byte hash. Throughput is dominated by how many workspaces you can isolate, not by how quickly you acknowledge a JSON body. Correctness means a green score never attaches to a rewritten tree, even when the late worker was honest. Cost may include retrying compute, but it must not include double-counting two generations as one truth.

I would rather drop a late pass than publish a score for a snapshot the current step does not contain. Would you rather explain a rejected verdict to a dashboard, or explain why a shipped step never saw the tests you filed?

Build the fence in five numbered steps

Follow this path if you want a protocol you can property-test instead of a slide about idempotent jobs.

Step 1. Snapshot the tree on the planner

Hash the workspace in the planner process before dispatch, not on the worker after it mutates files. I include tracked contents plus a generation integer so retries cannot impersonate an older snapshot by accident. The worker receives the expected hash as an opaque token that it must echo on completion.

Step 2. Bind the lease to generation and hash together

A job record should look like step identifier, generation, tree hash, and lease deadline as one tuple. Completing the job requires echoing every field, not just presenting a still-valid job id. This is a protocol check, and it is not a retry backoff policy in disguise.

Step 3. Attach through compare-and-swap only

The store attaches a verdict only when current generation equals the verdict generation and both hashes match exactly. A late worker receives rejected_stale instead of a silent overwrite of the live step. Should you page someone for that rejection? I would not, because the fence did the job.

Step 4. Compensate by bumping generation, not by recycling the slot

When the planner times out, it increments generation and writes a new hash rather than deleting the old job identity. Old workers may still finish, and that outcome is acceptable when the store refuses their payloads. Reusing the same slot is how a green rewind sneaks into the scoreboard.

Step 5. Inject the late-wakeup path inside the harness

Your evaluation harness is incomplete until it can pause worker one, bump generation, and then deliver the stale payload. If that path exists only in production traffic, you do not have an evaluation system yet. You have a scheduler that has been lucky.

Minimal simulator

The fixture below is a proposed state machine for the counterexample, not a production score service with measured throughput. I want the late pass to bounce, and I want the on-time fail to attach to generation eight.

from dataclasses import dataclass
from typing import Optional, List, Tuple

@dataclass
class Step:
    generation: int
    tree_hash: str
    verdict: Optional[str] = None

@dataclass
class Verdict:
    generation: int
    tree_hash: str
    result: str

def attach(step: Step, v: Verdict) -> str:
    if v.generation != step.generation:
        return "reject_generation"
    if v.tree_hash != step.tree_hash:
        return "reject_hash"
    if step.verdict is not None:
        return "reject_duplicate"
    step.verdict = v.result
    return "attached"

def simulate_late_wakeup() -> Tuple[List[str], Optional[str]]:
    step = Step(generation=7, tree_hash="H7")
    late = Verdict(generation=7, tree_hash="H7", result="pass")
    step.generation = 8
    step.tree_hash = "H8"
    events = [attach(step, late)]
    on_time = Verdict(generation=8, tree_hash="H8", result="fail")
    events.append(attach(step, on_time))
    return events, step.verdict

def simulate_hash_drift() -> str:
    step = Step(generation=8, tree_hash="H8")
    drifted = Verdict(generation=8, tree_hash="H7", result="pass")
    return attach(step, drifted)

if __name__ == "__main__":
    events, verdict = simulate_late_wakeup()
    assert events == ["reject_generation", "attached"]
    assert verdict == "fail"
    assert simulate_hash_drift() == "reject_hash"
    print("invariant held", events, verdict)

Run it with python fence_sim.py from any working directory that can import the stdlib. If attach returns attached for the late payload, the invariant is already dead and you should not scale the worker pool. I also want a second command that prints the drifted-hash path so reviewers cannot claim only timeouts were tested.

python - <<'PY'
from fence_sim import simulate_hash_drift
print(simulate_hash_drift())
PY

Failure injection

I inject four failure classes because each one belongs to a different domain in the diagram.

Preempt and resume: the worker pauses after tests and resumes after a generation bump.
Duplicate delivery: the same matching verdict arrives twice against the live hash.
Hash drift: the worker mutates the tree and still reports the original generation integer.
Clock lie: the planner expires the lease while the worker still believes it owns the job.

The property I will accept is narrow and testable. After any injected class, step.verdict either matches the current hash or stays empty, and it never equals a previous hash. Can your queue consumer state that property as code, or does it live only as a comment beside the ACK?

Tradeoff table

Choice	Latency	Throughput	Correctness	Cost
First JSON wins	Lowest	Highest	Breaks on late wakeup	Cheap until a rewind ships
Job-id idempotency only	Low	High	Collides across generations	Hidden retry waste
Hash plus generation CAS	One extra compare	Bounded by isolated checkouts	Preserves snapshot scores	Retry compute, not truth
Full replica commit fence	Higher	Lower	Strongest and heaviest	Overkill for single-writer steps

I pick hash plus generation for a single-writer checkout, and I would not stretch it over a multi-writer monorepo without a real commit fence. The explicit denominator is scored steps whose workspace hash still matches at attach time, not jobs that returned HTTP 200. Acceptance rule: in one hundred injected late-wakeup runs, zero stale passes attach, and one attach means the design does not converge.

What I would change next

I would stop treating the remote worker as a scorer and start treating it as an untrusted producer of candidate artifacts. The planner should pull a tarball, rehash the tree, and only then commit a tiny local verifier, even when heavy tests already ran remotely. That split keeps compute in the cheap domain and keeps commit inside the planner's failure domain.

I would also version the verdict schema so a missing tree hash is a protocol violation instead of a defaulted pass. An empty 200 is not success, and pytest JSON should not get a weaker contract than tool calls already required. Next after that, I would record rejected stale verdicts as first-class events so eval dashboards stop calling fences "worker errors."

Where a spare remote worker fits

You cannot honestly review this design if both processes share one laptop scheduler and one memory space. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that offers free model access and a free server option, which is enough to stand up a disposable worker that can stall and post a late payload.

I would park the score store on the planner side and use that free server only as worker W1 in the sequence. I would use free model access to draft additional property cases against attach(), not to decide whether a stale hash may stick. The point is cheap preemption rehearsal, and it is not a durability guarantee.

If that domain split helps you, run the fixture against a worker you do not control and watch which event order your store currently swallows.

Limitations and who should skip this

Skip this fence if your eval already runs in a hermetic single-process harness with no remote mutation at all. You would be adding protocol surface for a failure domain you do not actually have. Skip it if you cannot hash the workspace, because the echoed token becomes theater and the compare-and-swap protects nothing.

Do not treat free remote capacity as a durability contract, a quota promise, or a substitute for a commit log. This is a snapshot protocol for single-writer coding evaluations, and it is not a saga, a quorum, or an incident playbook. People chasing model quality without a score store should fix the harness first.

Validate the conclusion

Clone the simulator and add a fifth event where the worker reports generation eight with tree hash H7. Should the system reject that row, replay the job, or compensate in place? I want a reject, then a planner-owned replay on a fresh snapshot, and I never want a silent compensate that reuses the stale hash.

Which event order in your pipeline still writes a pass without looking at the tree, and will you reject, replay, or compensate when it shows up?

Make Token Spend a Chunk-Level Ledger Before a Stream Cut Rewinds the Step

Robin — Tue, 15 Sep 2026 03:07:23 +0000

At 02:40 my nightly eval loop stopped mid-sentence, and at 09:00 I had two numbers that could not both be right.

The worker reported 6,400 tokens spent for the run; the gateway reported 11,900. Which one do you believe when the pass rate also dropped and nobody changed the prompts? My loop was not wrong about the model, it was wrong about accounting granularity, and free-tier workers make that mistake loud.

This post is an architecture review of one narrow question: when a stream is cut mid-step, where do you record the spend, and what invariant decides whether the step may be replayed? I will declare assumptions, model the data flow, enumerate failure domains, and build a small offline harness you can run before you trust any loop on a free tier.

The constraints I assume (and you should write yours down)

Assumptions. (1) The worker process can be killed without warning; there is no graceful drain. (2) The model gateway bills per delivered chunk, not per completed step. (3) A stream can end three ways: terminator, transport error, or silence followed by eviction. (4) Only durably flushed bytes are reusable after a restart. (5) The token budget is finite and shared across steps, so one orphaned step changes what the next step can afford. (6) I am not assuming exactly-once billing from anyone.

In my current setup the worker runs on MonkeyCode's free server option and the model calls go through its free model access. Those availability claims are operator-supplied, as is the advertised free allotment (my working figure is on the order of ten million tokens). Terms like this move, so verify the current ones on the project page before you size a budget against them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Could a step-level counter ever be enough? Only if streams never get cut, and preemptible free-tier workers are exactly the environment where that assumption fails.

Data flow, and the three places it can break

 control plane        worker (free-tier server)        gateway (free model access)
      |                        |                                  |
      |-- admit(step,est) ---->|                                  |
      |                        |----- stream request ------------>|
      |                        |<---- chunk k (billed) -----------|
      |                        |  flush? yes -> durable bytes     |
      |                        |  flush? no  -> orphan bytes      |
      |                        |                                  |
      |                        |   X eviction / 429 / reset       |
      |<--- cut(attempt) ------|                                  |
      |  reconcile, then decide: replay (new attempt) | abandon   |

The control plane never sees the chunks, so it can only reason about spend through whatever the worker managed to persist. That is the whole problem in one sentence: the billing authority is the gateway, the durability authority is the worker, and the admission authority is the control plane. Three authorities, no shared clock, and a worker that can vanish between the second and the third.

Failure domains

Failure class	Domain	Symptom	Naive fix	Invariant-preserving move
Eviction mid-stream	free-tier worker	chunks stop, no terminator	retry the same step id	mark attempt cut, recompute orphan spend, admit a fresh attempt
Rate limit or 5xx mid-stream	gateway	partial artifact, billed chunks	reuse the attempt id	new attempt id, never recycled
Redelivery from a stale queue	control plane	second dispatch after commit	trust a cache TTL	durable commit marker keyed by (step, attempt)
Out-of-order arrival	coordinator	DONE before the last chunk	trust arrival order	require a per-attempt durable flush watermark

Notice that only the first row is really about the free tier; the other three are the same bugs you would have on paid hardware, just quieter because you would not be watching the budget as closely.

Step 1: write the spend invariant before writing the loop

I use four rules, and I write them into the repository README so a reviewer can point at them.

S1 — Spend identity. spent == durable + orphan at every checkpoint, where orphan = billed tokens attached to bytes that are not durable.
S2 — No replay after commit. A step with a durable commit marker is never admitted again, even if a duplicate dispatch arrives.
S3 — Commit requires durable bytes. A terminator without a matching durable flush set does not commit; it becomes retryable or abandoned.
S4 — Admission is budget-aware. A new attempt is admitted only if spent + estimate <= budget; otherwise it is abandoned explicitly rather than started and starved.

S1 is the one people skip, because per-step counters satisfy it trivially until a cut happens, and then they satisfy it falsely.

Step 2: build an offline harness that reproduces the cut

I do not want my first encounter with this ordering to be at 02:40, so I rehearse it deterministically. The harness below makes no network calls: you hand it an event script, it maintains per-attempt accounting, and it tells you which invariant broke.

#!/usr/bin/env python3
"""chunk_ledger.py - offline rehearsal for chunk-level spend accounting."""
from __future__ import annotations
import argparse
from dataclasses import dataclass, field
from enum import Enum


class Kind(Enum):
    CHUNK = "chunk"
    CUT = "cut"      # stream ended without a terminator
    DONE = "done"    # terminator received, artifact is usable


@dataclass(frozen=True)
class Ev:
    kind: Kind
    key: tuple[str, int]      # (step, attempt)
    tokens: int = 0
    persisted: bool = False   # did the worker flush this chunk durably?


@dataclass
class Attempt:
    billed: int = 0
    durable: int = 0
    state: str = "open"       # open -> committed | cut | abandoned


@dataclass
class Ledger:
    budget: int
    attempts: dict[tuple[str, int], Attempt] = field(default_factory=dict)
    violations: list[str] = field(default_factory=list)

    @property
    def spent(self) -> int:
        return sum(a.billed for a in self.attempts.values())

    @property
    def durable(self) -> int:
        return sum(a.durable for a in self.attempts.values())

    @property
    def orphan(self) -> int:
        return self.spent - self.durable

    def admit(self, key: tuple[str, int], estimate: int) -> bool:
        step, _ = key
        if any(k[0] == step and a.state == "committed"
               for k, a in self.attempts.items()):
            self.violations.append(f"S2: {key} started after {step} committed")
            return False
        if self.spent + estimate > self.budget:
            self.violations.append(
                f"S4: {key} needs {estimate}, remaining {self.budget - self.spent}")
            return False
        return True

    def apply(self, ev: Ev) -> Attempt:
        a = self.attempts.setdefault(ev.key, Attempt())
        if ev.kind is Kind.CHUNK:
            a.billed += ev.tokens
            if ev.persisted:
                a.durable += ev.tokens
        elif ev.kind is Kind.DONE:
            if a.durable != a.billed:
                a.state = "cut"
                self.violations.append(
                    f"S3: {ev.key} done, durable {a.durable} != billed {a.billed}")
            else:
                a.state = "committed"
        elif ev.kind is Kind.CUT:
            a.state = "cut"
        return a

    def check(self) -> list[str]:
        errs = list(self.violations)
        for key, a in self.attempts.items():
            if a.durable > a.billed:
                errs.append(f"S1: {key} durable exceeds billed")
            if a.state == "committed" and a.durable != a.billed:
                errs.append(f"S1: committed {key} carries orphan bytes")
        return errs

    def report(self) -> None:
        print(f"{'attempt':<10}{'state':<11}{'billed':>7}{'durable':>9}{'orphan':>8}")
        for (step, n), a in self.attempts.items():
            print(f"{step}#{n:<8}{a.state:<11}{a.billed:>7}{a.durable:>9}"
                  f"{a.billed - a.durable:>8}")
        ratio = 0.0 if self.spent == 0 else self.orphan / self.spent
        print(f"spent={self.spent} durable={self.durable} orphan={self.orphan} "
              f"budget={self.budget} orphan_ratio={ratio:.2f}")


def cut_then_replay() -> list[Ev]:
    return [
        Ev(Kind.CHUNK, ("A", 1), 40, True),
        Ev(Kind.CHUNK, ("A", 1), 40, True),
        Ev(Kind.DONE, ("A", 1)),
        # attempt 1 dies: last chunk billed, never flushed
        Ev(Kind.CHUNK, ("B", 1), 60, True),
        Ev(Kind.CHUNK, ("B", 1), 60, False),
        Ev(Kind.CUT, ("B", 1)),
        # replay on a fresh attempt
        Ev(Kind.CHUNK, ("B", 2), 60, True),
        Ev(Kind.CHUNK, ("B", 2), 60, True),
        Ev(Kind.DONE, ("B", 2)),
    ]


def replay_after_commit() -> list[Ev]:
    return [
        Ev(Kind.CHUNK, ("A", 1), 40, True),
        Ev(Kind.CHUNK, ("A", 1), 40, True),
        Ev(Kind.DONE, ("A", 1)),
        Ev(Kind.CHUNK, ("A", 2), 40, True),   # stale queue redelivery
        Ev(Kind.DONE, ("A", 2)),
    ]


SCENARIOS = {
    "cut_then_replay": cut_then_replay,
    "replay_after_commit": replay_after_commit,
}


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--scenario", choices=sorted(SCENARIOS), default="cut_then_replay")
    ap.add_argument("--budget", type=int, default=1000)
    ap.add_argument("--estimate", type=int, default=200)
    args = ap.parse_args()

    led = Ledger(budget=args.budget)
    for ev in SCENARIOS[args.scenario]():
        if ev.key not in led.attempts and not led.admit(ev.key, args.estimate):
            continue
        led.apply(ev)

    led.report()
    errs = led.check()
    for e in errs:
        print("VIOLATION:", e)
    return 1 if errs else 0


if __name__ == "__main__":
    raise SystemExit(main())

One honest caveat: in this harness S1 holds by construction because totals are derived from per-attempt rows. The rules that actually bite are S2 and S3, and the orphan ratio, which is the number I want on a dashboard before I let a loop run unattended.

Step 3: inject the two orderings that matter

python3 chunk_ledger.py --scenario cut_then_replay --budget 1000 --estimate 200; echo "exit=$?"
python3 chunk_ledger.py --scenario replay_after_commit --budget 1000 --estimate 200; echo "exit=$?"

The first run exits 0 and prints one cut attempt carrying 60 orphan tokens against 200 durable ones. The second run exits 1 with S2: ('A', 2) started after A committed, which is the counterexample I want in CI: a duplicate dispatch from a stale queue must be rejected by the ledger, not absorbed because the cache had already expired.

Note that the retry in scenario one is admitted because spent + estimate still fits; shrink --budget to 250 and watch S4 fire instead, which is the correct outcome on a finite free allotment.

Step 4: turn the checks into an acceptance rule

Define the denominator before you argue about results. Mine is eight cut positions across an eight-chunk step, crossed with two retry policies (immediate replay, and replay gated on a durable commit marker), for 16 scenario runs. Acceptance is: zero S2 and S3 violations, and orphan_ratio <= 0.25 under gated replay. If a policy cannot hold both, I change the policy, not the threshold.

Tradeoffs, stated plainly

Option	Spend accuracy under cuts	Added latency	New failure surface	Wins when
Per-step counter	drifts silently	none	wrong retry decisions	you are still prototyping
Reserve per step	overshoots on cut	none	stranded reservations	streams almost never cut
Chunk-level ledger	bounded by orphan ratio	one durable write per chunk	write amplification	long steps on preemptible workers
Gateway-side reconciliation	highest	batch delay	needs a billing export	you have provider billing APIs

Who should not use this approach

If your gateway bills per completed step with exactly-once semantics, the ledger buys you little and costs you a write per chunk. If your steps are two seconds long, the orphan window is too small to justify the bookkeeping. And if you have no durable store other than the worker's disk, do not pretend a ledger exists: you have a counter, and it will drift the first time a free-tier worker is evicted while holding your accounting state.

What I would change next

The next revision is an admission control that estimates step cost from the last N attempts of the same step type, so S4 fires before a doomed attempt starts rather than after it burns the tail of the budget. That estimate is itself a small model, and it deserves its own failure injection.

So here is the question I would put to you: which event order breaks your invariant — the cut before the flush, the terminator before the durable set, or the redelivery after the commit — and when it does, should your loop reject the work, replay it as a fresh attempt, or compensate the effect it already caused? If you want a cheap place to rehearse those three, the free server option and free model access are worth a look; bring a workload that actually cuts, and check the current terms before you size anything against them.

Make Admission a Generation-Counted Lease Before Shared Inference Completions Duplicate a Planner Step

Robin — Sun, 13 Sep 2026 19:17:17 +0000

Last Tuesday I reconstructed a fan-out eval where twelve planner loops entered one shared inference queue together. The first timeout did not cancel the original call; it issued a second call without a new lease. When both completions returned out of order, the planner applied two tool plans for a single step. Have you ever blamed the model for a duplicated tool call, when the queue just amplified a retry?

The invariant I want is narrow, and most agent loops fail it the moment retries exist. For any eval-run identifier, in-flight inference must stay inside a declared outstanding bound, and a completion may apply only while it still holds a live lease generation. Without that fence, a shared backend does not give you throughput. It gives you a retry amplifier dressed up as an agent loop.

Assumptions I will not hide

I am going to declare assumptions before I sketch the protocol, because otherwise the simulator quietly lies. Inference is a remote queue I do not own, so completion order is not submit order. Timeouts are local clocks, not server-side cancellation, which means the original call can still finish. Retries are at-least-once, so the backend may execute both the original request and the retry. Tool side effects sit outside this article; I only fence planner-visible model completions. Does your loop already assume any of those away?

I also assume the lab can use free model access and a free server as a backend I do not schedule. That is a constraint, not a capacity promise. I am not claiming exclusive GPUs, named models, or a durable quota. I am claiming a shared queue that will reorder work under load, which is exactly the failure class this lease is for.

Constraints, data flow, and failure domains

The constraint that actually bites is hidden head-of-line blocking on a queue you cannot inspect. Parallel planner loops treat “send another completion request” as progress, while the queue treats every retry as more work ahead of the original. Latency then looks like model slowness, so the loop shortens its timeout, which makes the stampede worse. Why would a free shared backend behave any differently?

Data flow is small if you draw the fence in the right place. An eval-run asks an admission desk for a lease that carries a generation integer. The worker may emit one inference request tagged with that generation, then it must wait or release. Completions re-enter the desk, not the planner, and the desk either commits the text into the step log or rejects it as a stale generation. Only a committed step may mint the next lease. That is the entire control plane.

I split failure domains along three lines, because mixing them makes the postmortem unreadable. Domain A is the client timeout clock, which can fire while the server still holds the original request. Domain B is the shared inference queue, which can delay, duplicate, or reorder completions. Domain C is the planner log, which must not apply text that lost its lease. If you debug domain C with domain A metrics, you will tune timeouts forever and never stop the duplicate step.

 eval-run
    |
    v
 admission desk (bound N, generation g)
    |
    |  lease(g)          release / reject
    v                       ^
 worker ----request(g)----> shared inference queue
    ^                       |
    |                       v
    +----- completion(g') --+
              |
              v
        planner step log  (apply iff g' == live g)

The violating event order

Here is the counterexample I keep on the whiteboard, because it is shorter than a principles slide. Worker W holds lease generation 4 and sends request R1. The local timeout fires at T+8s and W sends R2 with the same generation, or worse, with no generation at all. R2 completes first with a tool plan; the planner applies it and mints generation 5. R1 then completes with a different tool plan and the planner applies that too, because nothing checked the generation. Which of those two plans is “the” step?

A common implementation preserves none of the invariant. It uses a semaphore in the happy path, then bypasses the semaphore on retry because “we already paid for the wait.” It treats HTTP 200 as authority, even when the body belongs to a lease that expired. It never records the generation on the completion record, so a property test has nothing to assert. Is that an agent architecture, or an if-statement with unbounded outstanding I/O?

Protocol: numbered steps I would actually implement

I want the admission desk to be a tiny state machine, not a framework. Each eval-run owns one desk. The desk stores bound, in_flight, generation, and a map of live lease ids. Everything else is a comment waiting to drift.

On step start, the worker calls acquire(). If in_flight == bound, it waits or fails closed; it does not sneak a retry around the counter.
acquire() increments generation, creates lease_id, stores both, and increments in_flight. The request header carries both values.
The worker arms a local timer. On timer fire it calls expire(lease_id), which increments generation again and keeps in_flight until the outstanding call is accounted.
A completion is applied only when completion.generation == desk.generation and lease_id is still live. Otherwise the desk rejects and records stale_completion.
release() decrements in_flight exactly once per acquired lease, including expired leases whose late body just arrived. Double release is a bug, not a cleanup.
The planner may mint the next tool call only after a committed apply. Rejected text never becomes memory.

Would I put this desk inside the model client, or in front of the planner? I would put it in front of the planner. The client already lies about cancellation. The planner is the thing that must not see duplicate steps.

Sequence model

The sequence below is the legal path. Notice that timeout does not create a second legal apply. It creates a new generation, which turns the late body into a reject.

sequenceDiagram
    participant W as Worker
    participant D as Admission desk
    participant Q as Shared inference queue
    participant L as Planner log
    W->>D: acquire()
    D-->>W: lease g=4
    W->>Q: request(g=4)
    Note over W: local timeout
    W->>D: expire(g=4)
    D-->>W: live generation now 5
    Q-->>W: completion(g=4, text A)
    W->>D: apply(g=4, text A)
    D-->>L: REJECT stale generation
    W->>D: acquire()
    D-->>W: lease g=6
    W->>Q: request(g=6)
    Q-->>W: completion(g=6, text B)
    W->>D: apply(g=6, text B)
    D-->>L: COMMIT text B
    W->>D: release()

If your current trace cannot distinguish that reject from a commit, the invariant is already untestable. Can you point at one field in the log that proves generation 4 died before text A arrived?

Minimal simulator

The fixture below is a proposed discrete-event lab, not a production measurement and not a benchmark. I would run it first in process, then against a shared backend I do not own. MonkeyCode’s free model access and free server option are one such backend for that lab, because the whole point is a queue I cannot pause. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

from dataclasses import dataclass, field
from typing import List, Optional, Tuple
import random

@dataclass
class Desk:
    bound: int
    in_flight: int = 0
    generation: int = 0
    live: dict = field(default_factory=dict)
    log: List[Tuple[str, int, str]] = field(default_factory=list)

    def acquire(self) -> Optional[int]:
        if self.in_flight >= self.bound:
            self.log.append(("reject_acquire", self.generation, "bound"))
            return None
        self.generation += 1
        self.in_flight += 1
        self.live[self.generation] = True
        self.log.append(("acquire", self.generation, "ok"))
        return self.generation

    def expire(self, g: int) -> None:
        if g in self.live:
            self.live.pop(g, None)
            self.generation += 1  # fence late bodies
            self.log.append(("expire", g, f"live->{self.generation}"))

    def apply(self, g: int, text: str) -> bool:
        if self.live.get(g) and g == max(self.live, default=g):
            self.log.append(("commit", g, text))
            return True
        self.log.append(("reject_stale", g, text))
        return False

    def release(self, g: int) -> None:
        if g in self.live:
            self.live.pop(g, None)
        if self.in_flight:
            self.in_flight -= 1
        self.log.append(("release", g, str(self.in_flight)))

@dataclass
class Event:
    t: float
    kind: str
    g: int
    text: str = ""

def simulate(seed: int = 7, bound: int = 1, timeout: float = 8.0) -> Desk:
    rng = random.Random(seed)
    desk = Desk(bound=bound)
    q: List[Event] = []
    now = 0.0
    g = desk.acquire()
    assert g is not None
    service = rng.uniform(1.0, 20.0)
    q.append(Event(now + timeout, "timeout", g))
    q.append(Event(now + service, "complete", g, text=f"plan@{g}"))
    applied = 0
    while q:
        q.sort(key=lambda e: e.t)
        ev = q.pop(0)
        now = ev.t
        if ev.kind == "timeout":
            desk.expire(ev.g)
            g2 = desk.acquire()
            if g2 is None:
                continue
            service2 = rng.uniform(1.0, 12.0)
            q.append(Event(now + service2, "complete", g2, text=f"plan@{g2}"))
        elif ev.kind == "complete":
            if desk.apply(ev.g, ev.text):
                applied += 1
            desk.release(ev.g)
    desk.log.append(("applied_count", applied, "denominator=injected_timeout_retry"))
    return desk

if __name__ == "__main__":
    d = simulate()
    commits = [e for e in d.log if e[0] == "commit"]
    stales = [e for e in d.log if e[0] == "reject_stale"]
    print("commits", commits)
    print("stale_rejects", stales)
    print("in_flight_end", d.in_flight)
    assert d.in_flight == 0
    assert len(commits) <= 1

Run it as a command, not as a slide. If commits ever exceeds one after a timeout, the desk lost the generation fence. If in_flight is nonzero at the end, a release path leaked. That is the whole unit of validation for this article.

python admission_lease_sim.py

Injected failures and testable properties

I would inject four failure classes before I trusted the desk on a shared queue. First, a late original completion after expire, which must reject. Second, a retry that cannot acquire because bound=1, which must fail closed instead of sending. Third, two completions with the same generation, which must commit at most once. Fourth, a release that arrives twice, which must not drive in_flight negative. Which of those four does your client test today?

Properties I would freeze in the harness, with an explicit denominator.

P1: in_flight is always in [0, bound] after every event.
P2: commit count per eval-run step is 0 or 1.
P3: every commit.generation was live at apply time.
P4: every timeout produces either a reject_stale for the old generation or no second send.
Denominator: duplicate commits per 100 injected timeout-then-late-completion pairs.
Acceptance rule: denominator result equals 0, and P1–P4 hold on every seed in a fixed set {1..200}.

If you cannot name the denominator, you do not have an eval. You have a dashboard.

Tradeoff table

Design	What it preserves	What it spends	When it breaks
Unbounded retry	Apparent liveness	Queue depth, duplicate applies	Any timeout on a slow shared backend
Client semaphore, retry bypass	Happy-path bound	The invariant on the failure path	The exact event order in the opener
Generation-counted lease, fail closed	At-most-one apply per step	Extra reject / extra wait	Need more than `bound` true parallelism
Gateway admission with server cancel	Bound plus fewer late bodies	Control-plane coupling to the vendor	Backend that cannot cancel
Coalesce identical prompts	Load	Correctness if prompts are not actually identical	Tool-using planners with step-local state

I would take the generation-counted lease on the planner side first, because I can ship it without vendor help. I would not take unbounded retry because it invents throughput that the queue cannot honor. Would I later push admission into the gateway? Yes, that is the next change, not the first one.

What I would change next

After the client desk holds the invariant, I would move cancellation into the request itself. A lease expire should send a cancel token the backend actually honors, so domain A and domain B stop drifting. I would also persist the desk to the eval-run log, so a crashed worker does not resurrect generation 4. I would not start with autoscaling stories, because scale without a bound just multiplies the stampede.

I would also keep model quality out of this control loop. Schema failures and empty bodies are protocol errors, but they are a different fence. This article only answers whether a completion is still allowed to exist. Mixing quality scores into admission is how people hide retry bugs inside “the model was bad today.”

Limitations, and who should not use this

This desk is the wrong tool if you already have a hard reservation system with server-side cancel and exactly-once apply. It is also the wrong tool for interactive single-user chats that never fan out and never retry. Do not use it as a billing system, a fairness scheduler, or a promise that free shared inference is production capacity. The simulator does not measure tokens, dollars, or latency percentiles; it measures whether stale generations can still commit.

If your planner must apply speculative partial streams, this design will reject work you wanted to keep. That is intentional. Partial text is uncommitted state, and I will not let it mint a tool call. If you need streaming UX, buffer tokens off to the side and commit only the final generation. Can your product live with that lag, or are you selling a live token firehose?

The free model access and free server option are useful here only as a shared queue I do not control. They do not replace a load model, and they do not make P1–P4 true by themselves. If you cannot run the fixture locally first, do not drag a remote queue into the story yet.

Closing counterexample

Picture this order: acquire g=4, send R1, expire g=4, acquire fails because bound=1, then R1 completes with a perfect JSON tool plan. Should the system reject that plan, replay a new acquire, or compensate by applying it anyway because the text looks valid? I would reject, then replay acquire only when in_flight allows it, and I would never compensate by trusting a late body. Which event order in your traces still applies that late plan, and which of those three verbs does your log actually implement?

Make Eval Completion a Join Barrier Before Duplicate Tool Spend Rewinds Pass/Fail

Robin — Sat, 12 Sep 2026 17:17:31 +0000

I keep a lab fixture because shared eval loops lie in a very specific way. The dashboard flips a flaky agent case from red to green after the shared budget is already gone. The planner joined two tool results, stored pass, and then a slower worker delivered a duplicate retry. Does that retry still spend the token window, or do we pretend the case is already closed?

That event order is the counterexample I use whenever shared eval compute looks cheap enough to skip a protocol. I am not reviewing a prompt. I am reviewing whether completion is a join, or just the first HTTP 200 that made the notebook feel done.

The invariant the common loop fails to preserve

I want one property, and I want it testable without a production cluster. An eval case may commit pass or fail only after every tool attempt in its fan-out has joined or been rejected against a still-open admission window. Late deliveries after that window closes must not mutate the recorded verdict, and they must not spend budget that belongs to the next case. If your harness treats a worker 200 as completion, you already dropped the invariant. Have you ever marked a case green while a retry was still in flight on another core?

MCP-style tool fan-out makes the join set obvious, which is why I care about it now. The tools are not the architecture. The architecture is whether completion is closed before the next case is admitted.

Declared assumptions

I am reviewing a lab architecture, not a production planner, and every fixture below is labeled until you run it. Assumption one: tool workers are at-least-once, so duplicates are normal rather than exceptional. Assumption two: the scarce resource is a shared token budget on one eval host, not a private GPU map. Assumption three: verdicts live in an append-only log, and readers treat the latest committed verdict as truth. Assumption four: I will not invent wall-clock SLOs, model names, or load numbers I did not collect.

If any assumption is false in your shop, the join protocol is the wrong fence. Would you still commit on the first 200 if assumption one is the only honest one?

Constraints, data flow, and where the join lives

I treat the eval host as a tiny distributed system because the workers already are. The control plane admits a case, leases a token window, fans out tool calls, and must not commit until the join set is closed. The data plane stays dull on purpose: each attempt carries case_id, attempt_id, and window_id, and the join table keys on that triple. Why would you key only on case_id when retries exist? That is how most notebooks rewind pass/fail without noticing.

Here is the lab flow I want, drawn as a sequence rather than a slide.

sequenceDiagram
    participant Admit as Admission
    participant Log as VerdictLog
    participant W1 as ToolWorkerA
    participant W2 as ToolWorkerB
    Admit->>Log: open window(case, budget)
    Admit->>W1: call tool (case, attempt=1, window)
    Admit->>W2: call tool (case, attempt=2, window)
    W1-->>Admit: result attempt=1
    Admit->>Log: join(attempt=1)
    Note over W2,Admit: retry or late duplicate
    W2-->>Admit: result attempt=2 after close?
    Admit->>Log: reject if window closed
    Admit->>Log: commit verdict iff join set complete and window open

Notice the reject path is part of completion, not an error handler you bolt on later. If the late result lands after close, the system should reject, not replay, and not compensate by flipping the stored verdict. Compensation belongs to the product workflow under test, not to the harness that claims to measure it. Do you see how easy it is to mix those two planes?

Failure domains I actually care about

I split failures into four domains because they do not share a fix. Domain A is admission: the window opens with a budget that cannot cover the fan-out, so the case should never start. Domain B is in-flight duplication: two workers execute the same attempt id, or two attempt ids for one logical tool. Domain C is window expiry: remaining tokens hit zero while the join set is still open. Domain D is log rewind: a late writer appends a second verdict for a case that already committed.

Which domain is your green dashboard actually hiding? Shared free compute makes Domain C louder, because noisy neighbors steal tokens without stealing your process. That is a constraint on the join protocol, not an operations tutorial. If you cannot observe remaining budget as a first-class signal, you cannot close the window honestly. I would rather fail the case closed than let a late duplicate spend the next case's window.

Minimal state machine

I keep five states, and I refuse hidden sixth states in worker-local memory. NEW means admitted but no window. OPEN means a window id is live and the join set is incomplete. JOINED means every expected attempt has a terminal event inside the window. COMMITTED means a verdict was appended and further tool events must be rejected. ABORTED means the window closed early and the verdict is fail or inconclusive, never a silent pass.

Can a case move from COMMITTED back to OPEN? Not in this protocol, and that is the whole point.

NEW -> OPEN -> JOINED -> COMMITTED
              \-> ABORTED
OPEN -> ABORTED
COMMITTED -> (reject late events, stay COMMITTED)

A numbered lab you can actually run

I want you to validate the claim with a fixture, not with a slide about backpressure. Follow these steps on one machine.

Save the simulator below as eval_join.py and read the assertions before you edit anything.
Run python3 eval_join.py and confirm the late duplicate is rejected after commit.
Run python3 eval_join.py --inject duplicate_after_close and watch the next case keep its budget.
Run python3 eval_join.py --inject budget_exhausted_mid_join and confirm the verdict is ABORTED, not pass.
Change the join key to case_id only, rerun, and record which property breaks first.

If step five does not fail a property, your test is too weak. What event order did you forget?

Simulator (labeled fixture, not a production service)

#!/usr/bin/env python3
"""Eval join-barrier simulator. Fixture only; not a production planner."""
from __future__ import annotations

import argparse
from dataclasses import dataclass, field
from typing import Dict, List, Optional, Set


@dataclass
class Event:
    case_id: str
    attempt_id: str
    window_id: str
    tokens: int
    kind: str  # result | duplicate | late


@dataclass
class Case:
    case_id: str
    expected: Set[str]
    window_id: Optional[str] = None
    state: str = "NEW"
    joined: Set[str] = field(default_factory=set)
    spent: int = 0
    verdict: Optional[str] = None


class EvalJoin:
    def __init__(self, budget: int) -> None:
        self.budget = budget
        self.remaining = budget
        self.cases: Dict[str, Case] = {}
        self.log: List[str] = []
        self.rejected: List[Event] = []

    def admit(self, case: Case, window_cost: int) -> None:
        if self.remaining < window_cost:
            case.state = "ABORTED"
            case.verdict = "inconclusive"
            self.cases[case.case_id] = case
            self.log.append(f"abort_admit {case.case_id}")
            return
        case.window_id = f"w-{case.case_id}"
        case.state = "OPEN"
        self.remaining -= window_cost
        case.spent += window_cost
        self.cases[case.case_id] = case
        self.log.append(f"open {case.case_id} {case.window_id}")

    def on_tool(self, ev: Event) -> None:
        case = self.cases[ev.case_id]
        if case.state in {"COMMITTED", "ABORTED"}:
            self.rejected.append(ev)
            self.log.append(f"reject_late {ev.case_id} {ev.attempt_id}")
            return
        if ev.window_id != case.window_id:
            self.rejected.append(ev)
            self.log.append(f"reject_window {ev.case_id} {ev.attempt_id}")
            return
        if ev.attempt_id in case.joined:
            self.rejected.append(ev)
            self.log.append(f"reject_dup {ev.case_id} {ev.attempt_id}")
            return
        if ev.attempt_id not in case.expected:
            self.rejected.append(ev)
            self.log.append(f"reject_unknown {ev.case_id} {ev.attempt_id}")
            return
        case.joined.add(ev.attempt_id)
        self.log.append(f"join {ev.case_id} {ev.attempt_id}")
        if case.joined == case.expected:
            case.state = "JOINED"
            self.commit(case, "pass")

    def expire(self, case_id: str) -> None:
        case = self.cases[case_id]
        if case.state == "OPEN":
            case.state = "ABORTED"
            case.verdict = "inconclusive"
            self.log.append(f"abort_expire {case_id}")

    def commit(self, case: Case, verdict: str) -> None:
        if case.state not in {"JOINED", "ABORTED"}:
            raise AssertionError("commit without join or abort")
        if case.state == "JOINED":
            case.state = "COMMITTED"
            case.verdict = verdict
            self.log.append(f"commit {case.case_id} {verdict}")


def scenario(inject: str) -> EvalJoin:
    harness = EvalJoin(budget=100)
    c1 = Case("case-1", expected={"a1", "a2"})
    c2 = Case("case-2", expected={"b1"})
    harness.admit(c1, window_cost=40)
    harness.on_tool(Event("case-1", "a1", "w-case-1", 10, "result"))
    if inject == "budget_exhausted_mid_join":
        harness.remaining = 0
        harness.expire("case-1")
        harness.on_tool(Event("case-1", "a2", "w-case-1", 10, "late"))
        harness.admit(c2, window_cost=40)
        return harness
    harness.on_tool(Event("case-1", "a2", "w-case-1", 10, "result"))
    late = Event("case-1", "a2", "w-case-1", 10, "duplicate")
    if inject == "duplicate_after_close":
        harness.on_tool(late)
        harness.admit(c2, window_cost=40)
        return harness
    harness.on_tool(late)
    return harness


def check(harness: EvalJoin, inject: str) -> None:
    c1 = harness.cases["case-1"]
    if inject == "duplicate_after_close":
        assert c1.state == "COMMITTED" and c1.verdict == "pass"
        assert any(e.kind == "duplicate" for e in harness.rejected)
        assert harness.cases["case-2"].state == "OPEN"
        assert harness.remaining == 20
    elif inject == "budget_exhausted_mid_join":
        assert c1.state == "ABORTED" and c1.verdict == "inconclusive"
        assert c1.verdict != "pass"
        assert harness.cases["case-2"].state == "ABORTED"
    else:
        assert c1.state == "COMMITTED"
        assert any("reject_dup" in line for line in harness.log)


if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("--inject", default="duplicate_after_close")
    args = p.parse_args()
    h = scenario(args.inject)
    check(h, args.inject)
    print("ok", args.inject, h.log, "rejected", len(h.rejected))

Run it like this.

python3 eval_join.py --inject duplicate_after_close
python3 eval_join.py --inject budget_exhausted_mid_join

The first command should keep case-2's window intact after rejecting the duplicate. The second command should refuse to call a mid-join case pass once the shared budget is gone. If either assertion fails on your machine, the protocol is not implemented. Do not argue with the dashboard.

Testable properties and injected failures

I care about three properties, and I want them as predicates over the log. Property P1: after commit, no later join line exists for that case_id. Property P2: a rejected duplicate does not decrease remaining a second time. Property P3: pass never appears on an ABORTED case. Which property does your current harness skip because the happy path is green?

Injected failures I would keep in CI are duplicate after close, unknown attempt id, mismatched window id, admit when remaining is below window cost, and expire while one attempt is missing. I would not inject model-quality noise here, because that confuses planner bugs with harness bugs. The denominator for acceptance is one hundred seeded permutations of those five failures, and the rule is zero violations of P1 through P3. Anything weaker is a demo, not a gate.

Tradeoffs

Choice	Latency	Throughput	Correctness	Cost
Commit on first 200	Lowest	Highest	Breaks P1 on late retry	Cheap until the rewind
Join barrier, reject late	Wait for the slowest attempt	Caps in-flight cases by budget	Preserves P1–P3	Extra bookkeeping
Replay late events into a new window	Unbounded tail	Burns the next case	Rewinds verdicts	Hidden token spend
Compensate by flipping pass/fail	Looks responsive	Pollutes metrics	Mixes product and harness	Debug cost dominates

I pick the join barrier for eval, and I keep compensation inside the agent under test. Replay is the wrong default on a shared token pool. Would you rather wait for a missing attempt, or publish a green number you cannot replay?

Where a free shared server actually fits

I needed optional lab capacity that can replay this join log against a real tokenizer path without turning the review into a capacity paper. MonkeyCode is an open-source project that currently offers free model access, stated by the operator as on the order of ten million tokens, plus a free server option I would treat as a lab rather than a multi-tenant SLO. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I would park the fixture there to replay rejects, not to claim any model is accurate, and I would still gate every case on the window instead of a worker 200.

If you want a scratch box for the simulator, that free server is enough to store the log and reject late events. I would not point a production planner at it. The protocol is the artifact; the server is only scaffolding around the same fixture.

Limitations, and who should not use this

This protocol does not give you model quality, prompt regression insight, or a tracing story. It will abort aggressively when the shared budget is tight, which is correct for a lab and hostile for an interactive product. Do not use it if you need exactly-once side effects in customer systems, because the workers are still at-least-once. Do not use it if eval cases share mutable world state without their own idempotency keys. Do not use it as an incident runbook; I am not covering dashboards, deploys, or on-call here.

I also did not measure tokens per case on a live model, and I will not invent a speedup. The acceptance rule is the three properties, not a throughput multiplier. If you cannot run the fixture, you cannot claim the architecture converges.

What I would change next

I would persist the join table in an append-only file before I trusted a second process. I would bind window_id to a hashed (case_id, expected_attempts, budget_epoch) so a restarted harness cannot reopen a committed window. I would add a property test that permutes event arrival instead of the two scripted injections above. I would still reject late events rather than compensating inside the harness. The next design review is whether inconclusive should block a release gate or merely quarantine the case.

Counterexample question

Which event order breaks the invariant: a duplicate a2 after commit, an a2 with a stale window_id, or an admit of case-2 while case-1 is still OPEN and remaining is already zero? Should the system reject, replay, or compensate? If your answer is replay because the tool is idempotent, you are optimizing the product path and abandoning the eval path. Run the fixture, inject that order, and only then decide whether the join barrier is too strict.