DEV Community: Robin

Make Preemption a Fenced Lease Before an Agent Resumes on Ephemeral Compute

Robin — Sun, 06 Sep 2026 20:36:14 +0000

I keep seeing the same violating event order when an agent worker dies between a tool side effect and the checkpoint. The planner writes a snapshot after the previous step, then calls a payment or mail tool, then vanishes before the snapshot advances. A replacement worker boots on another box, reads the stale snapshot, and treats the missing write as proof the tool never ran. Have you watched a retry look identical to a first attempt because nobody incremented a lease?

The common implementation preserves resume-from-last-checkpoint and quietly drops the fencing token, which is not an invariant worth shipping. Two workers sharing a run_id must never apply competing side effects under the same lease epoch, even after the first worker dies. If a late ack from the preempted epoch still arrives, the system should reject or compensate, never apply it as a fresh success. Why do we keep modeling death as a clean pause instead of a concurrent writer?

There is a second and nastier counterexample that I now treat as the real production-shaped bug. Teams mint a new idempotency key when the lease bumps, so the bank sees a brand new charge instead of a replay. The stable key must stay (run_id, step_id) across epochs, while the epoch only fences checkpoint rights and zombie calls. Would you stake a refund pipeline on a key that changes every time a box disappears?

Declared assumptions

I am reviewing a discrete architecture, not a production postmortem with invented latency numbers or customer counts. The fixture below is labeled example code you can run locally, and it is not a cluster measurement. These assumptions bound the argument so we can actually falsify it.

One logical agent run has a stable run_id and a monotonically increasing lease_epoch.
Tool side effects become unique only when the caller supplies a stable idempotency key.
Checkpoints are durable; worker memory and in-flight HTTP bodies are not.
Preemption can land after the tool has committed and before the checkpoint write.
A late response from the old worker can arrive after the new lease is active.
The acceptance denominator is injected preemption interleavings, not wall-clock hours.

If your tools already fence on a server-side key that stays stable across leases, the local reject path is still useful for zombies. Do you actually pass that stable key, or do you pass a hash of the epoch by accident?

Constraints and data flow

Ephemeral compute changes the failure domain more than it changes the happy path through the planner. A free or preemptible worker is a legal place to run inference and tool routing, until the box disappears mid-step. The model hop is another unbounded queue, because tokens arrive late and a retry looks like honest progress. I want the data flow to make both hops explicit, because hidden hops become double applies the moment you resume.

client intent -> durable intent log -> lease grant -> worker
worker -> model hop (optional, queued)
worker -> tool gateway (stable key + epoch header) -> side effect store
worker -> checkpoint store (epoch + seq)
preemption -> lease expiry -> new grant with epoch+1
late ack -> gateway fence -> reject, replay same key, or compensate

The intent log is the source of truth, and the worker is a leased cache sitting in front of it. Does your current diagram still draw the worker as the center of the world?

Sequence that violates naive resume

sequenceDiagram
    participant L as IntentLog
    participant W1 as Worker epoch 1
    participant T as ToolGateway
    participant W2 as Worker epoch 2
    L->>W1: grant lease epoch=1
    W1->>T: charge(key=run/step, epoch=1)
    T-->>W1: committed
    Note over W1: preempted before checkpoint
    L->>W2: grant lease epoch=2
    W2->>T: charge(key=run/step/epoch2)  # minted new key
    T-->>W2: committed again
    W1->>L: late checkpoint epoch=1 overwrites seq

That last late checkpoint is the bug people rename eventual consistency when they really mean they lost the fence. The invariant I want is sharper than a slogan about exactly-once delivery across a chatty planner. At most one net committed side effect may exist per (run_id, step_id) unless a compensation record is explicit in the log. Naive resume breaks it when preemption lands between commit and checkpoint, or when resume mints a new key. Can you name a tool in your agent that is actually safe to double-fire?

Failure domains

I split the system into four domains because they fail independently and hide inside one process. Mixing them into a single agent runtime is how the double charge sneaks into a passing demo.

Intent log and lease manager. Crash-safe, fenced, and not colocated with the worker.
Ephemeral worker. It may die, freeze, or dual-boot after a short network partition.
Tool gateway. It must see both the stable key and the epoch, or it cannot reject zombies.
Model hop. It is bounded by your queue and budget, not by the worker's optimism.

A free server option lives in domain two, which is exactly why I care about leases here. Free model access lives in domain four, which is a queue with a timeout, not a local function call. What happens when domain two dies while domain three has already committed, and domain four still has a streaming response in flight?

Architecture review: five steps I would actually run

I review this class of design with a short protocol, not with a reliability slogan on a slide. Each step produces an artifact you can keep next to the planner.

Step 1. Write the state machine before the planner loop

I refuse to read a planner that mutates memory and then explains consistency in a comment. Encode grant, apply, checkpoint, preempt, and late-ack as transitions with explicit rejects. If a transition cannot name the epoch it requires, it does not belong in the worker. Would your current loop still compile if checkpoint were not allowed on a stale epoch?

Step 2. Split the stable key from the fence

Put (run_id, step_id) on the tool idempotency key and keep it unchanged after preemption. Put lease_epoch in a header the gateway checks before it forwards, so a zombie cannot checkpoint or start a different step. If you fold the epoch into the key, resume is a new business operation, which is how you bill twice. Are you sure your SDK is not concatenating those fields for you?

Step 3. Inject preemption between commit and checkpoint

A unit test that kills the worker before the tool returns does not cover this hole. You need the interleaving where the side effect store has committed and the snapshot has not, plus a late ack after the new grant. I run that as a discrete-event fixture so the event order is the input, not an accident of sleep(). Does your suite have a name for that interleaving, or only a flaky integration test?

Step 4. Score violations with an explicit denominator

I do not accept "it felt fine on the free box" as a reliability argument, because feeling is not a denominator. Count double-applies and stale checkpoints per injected preemption interleaving, then publish the ratio. The acceptance rule I use for this fixture is zero double-applies and zero accepted stale checkpoints across two hundred injected interleavings. If you cannot state the denominator, you are not measuring the invariant.

Step 5. Decide reject, replay, or compensate per step type

Charges, mails, and memory writes do not share a recovery verb, so the planner must not hide them under a generic resume(). Replay the same stable key when the tool is idempotent at the provider. Compensate when the provider committed under a key you must not reuse, then record that compensation as its own step. Reject when the epoch is stale and the step is already decided. Which of those three did your last incident actually need?

Minimal simulator

This is labeled example code for the interleaving, not a library I pretend already runs in production. Save it as fence_lease_sim.py and treat the assertions as the architecture gate.

"""Fenced-lease simulator for preempted agent workers.

Labeled example: not production code. Run: python fence_lease_sim.py
"""
from __future__ import annotations

from dataclasses import dataclass, field
from enum import Enum, auto
from typing import List, Optional, Tuple


class Outcome(Enum):
    APPLIED = auto()
    REJECTED_STALE_LEASE = auto()
    REJECTED_DUPLICATE_STEP = auto()
    CHECKPOINT_IGNORED = auto()


@dataclass
class SideEffect:
    run_id: str
    step_id: str
    lease_epoch: int
    amount: int


@dataclass
class System:
    run_id: str = "run-1"
    lease_epoch: int = 0
    checkpoint_seq: int = 0
    committed: List[SideEffect] = field(default_factory=list)
    live_worker_epoch: Optional[int] = None

    def grant_lease(self) -> int:
        self.lease_epoch += 1
        self.live_worker_epoch = self.lease_epoch
        return self.lease_epoch

    def preempt(self, epoch: int) -> None:
        if self.live_worker_epoch == epoch:
            self.live_worker_epoch = None

    def apply_tool(self, epoch: int, step_id: str, amount: int) -> Outcome:
        if epoch != self.lease_epoch or self.live_worker_epoch != epoch:
            return Outcome.REJECTED_STALE_LEASE
        if any(s.step_id == step_id for s in self.committed):
            return Outcome.REJECTED_DUPLICATE_STEP
        self.committed.append(SideEffect(self.run_id, step_id, epoch, amount))
        return Outcome.APPLIED

    def checkpoint(self, epoch: int, seq: int) -> Outcome:
        if epoch != self.lease_epoch:
            return Outcome.CHECKPOINT_IGNORED
        if seq <= self.checkpoint_seq:
            return Outcome.CHECKPOINT_IGNORED
        self.checkpoint_seq = seq
        return Outcome.APPLIED

    def committed_net(self, step_id: str) -> int:
        return sum(s.amount for s in self.committed if s.step_id == step_id)


def naive_resume_double_applies() -> int:
    """Counterexample: resume without a gateway fence or a stable key."""
    committed_rows = 0
    committed_rows += 1  # epoch 1 applied, then died before checkpoint
    committed_rows += 1  # replacement worker treats missing snapshot as never-ran
    return committed_rows


def epoch_inside_key_double_charges() -> int:
    """Anti-pattern: minting a new idempotency key when the lease bumps."""
    seen = set()
    charges = 0
    for epoch in (1, 2):
        key = ("run-1", "charge", epoch)  # epoch leaked into identity
        if key not in seen:
            seen.add(key)
            charges += 1
    return charges


def fenced_preempt_then_late_ack() -> System:
    s = System()
    e1 = s.grant_lease()
    assert s.apply_tool(e1, "charge", 50) == Outcome.APPLIED
    s.preempt(e1)
    e2 = s.grant_lease()
    assert s.apply_tool(e1, "charge", 50) == Outcome.REJECTED_STALE_LEASE
    assert s.apply_tool(e2, "charge", 50) == Outcome.REJECTED_DUPLICATE_STEP
    assert s.checkpoint(e1, seq=2) == Outcome.CHECKPOINT_IGNORED
    assert s.checkpoint(e2, seq=2) == Outcome.APPLIED
    return s


def property_at_most_one_net_commit(iterations: int = 200) -> None:
    """Acceptance: net charge stays 50. Denominator: injected preemptions."""
    violations = 0
    for i in range(iterations):
        s = System(run_id=f"run-{i}")
        e1 = s.grant_lease()
        s.apply_tool(e1, "charge", 50)
        s.preempt(e1)
        e2 = s.grant_lease()
        s.apply_tool(e1, "charge", 50)  # zombie
        s.apply_tool(e2, "charge", 50)  # resume, same step_id
        if s.committed_net("charge") != 50:
            violations += 1
    assert violations == 0, f"double-apply violations={violations}/{iterations}"


if __name__ == "__main__":
    print("naive committed rows (should be 2):", naive_resume_double_applies())
    print("epoch-in-key charges (should be 2):", epoch_inside_key_double_charges())
    sys = fenced_preempt_then_late_ack()
    print("fenced net charge (should be 50):", sys.committed_net("charge"))
    property_at_most_one_net_commit()
    print("property held over 200 injected preemptions")

Run it with the boring command, then read the three printed numbers as the review, not as a benchmark of a model.

python fence_lease_sim.py

You should see the naive path print two committed rows and the fenced path print a net of fifty. The property loop is the gate: two hundred injected preemptions, zero double-applies. If that assertion fails, the architecture is wrong before any model quality discussion starts. Why would you debug the prompt when the lease protocol is already leaking money?

Tradeoff table

Choice	What you gain	What you pay	When it fails
Stable key, epoch in header	Resume is a replay	Gateway must be epoch-aware	Provider ignores headers
Epoch inside idempotency key	Easy to stamp uniqueness	Preemption mints a new charge	Every worker death
Checkpoint-before-call	No orphan commits	Lost work on death, extra latency	Long model hops
Call-before-checkpoint	Progress under success	Orphan commits on preemption	Free or preemptible workers
Compensate-always-on-resume	Clear money story	Extra provider load, new failure domain	Compensation itself preempted

I would not pick checkpoint-before-call for a long model hop, because you then stall the lease while tokens dribble in. I also would not pick epoch-in-key, because it converts a fence into a new business intent. The durable combination is a stable key plus an epoch fence plus an explicit compensate step when the provider cannot replay. Which cell matches the system you are actually shipping this week?

A practical workflow on ephemeral compute

I want the worker to vanish on purpose, because an in-process mock never exercises the lease. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I would park this fixture on its free server option and drive the planner with free model access so the model hop is a real queue, then inject preemption while a tool call is in flight. I am not attaching quotas, model names, hardware claims, or latency multipliers I did not measure here. If you need a box that is allowed to disappear, that free server option is enough to host the simulator.

Limitations and who should not use this

This is a discrete-event model with a single run_id and a single step, not a multi-tenant broker. It does not prove exactly-once delivery across a provider that forgets idempotency keys after a day. It also does not replace property tests on your real gateway, and it will not save a tool that mints server-side IDs you cannot replay. Do not use this approach if your workers already hold exclusive, long-lived leases on durable machines and your providers already key on (run_id, step_id). Do not use it if you are writing a single-process script with no side effects outside memory. Do not fold this fence into a prompt instruction and call the architecture done.

What I would change next

I would push the epoch check into the tool gateway itself, because a polite worker header is not a fence if the worker is gone. I would write an outbox record before the HTTP call, so death between send and checkpoint still has a durable intent to replay. I would also split compensation into its own fenced step, because a preempted compensate is a new concurrent writer, not a footnote. After that I would add a second injected failure: grant overlapping leases during a partition and prove the lower epoch cannot checkpoint.

The counterexample I still want you to answer is this one. Worker two replays the stable key, the provider returns the original receipt, and then worker one’s delayed checkpoint for epoch one overwrites worker two’s snapshot with a smaller seq. Does your log reject that write, replay the newer snapshot, or compensate by freezing the run? If you cannot pick one, the design does not yet converge.

Make Compensation a Fenced Epoch Before a Late Tool Result Lands

Robin — Sat, 05 Sep 2026 19:13:20 +0000

I start every architecture review with a three-event trace rather than a comforting box diagram. Event A is the planner sending tool_call(id=7, epoch=1) toward a slow model path. Event B is the local deadline firing timeout(id=7) and opening compensation. Event C is tool_success(id=7) arriving after that compensation already released the reservation. Which write is legal here, and why do so many planners still answer "whoever arrives last"?

That last-writer rule is the protocol hole I keep drawing on whiteboards. Timeout already ran compensation, so the reservation row was released on purpose. The late success then writes the same row back as if compensation never existed. You did not observe a retry. You observed a ghost write against a closed epoch.

The invariant I will not negotiate

I want one invariant stated so a property test can fail it in a single assertion. For each call_id, at most one epoch may commit a durable mutation, and any result whose epoch is stale must be rejected. Does your planner even carry an epoch field, or does it key every mutation on call_id alone?

If the key is only call_id, success and compensation share one row and fight in arrival order. A fence is not a retry counter you increment for luck. It is a leadership token for that call, and late winners are not leaders. I will treat a success without a matching live epoch as a protocol violation, not as good news from the model.

Assumptions I am willing to defend

I am reviewing a single-planner workflow with one outbound tool call, not a multi-agent mesh. The tool is at-least-once and may deliver success after the client has already timed out. The compensator is itself a network call, so it can also arrive late or twice. I trust a monotonic local deadline more than wall-clock agreement across machines. Durable state is a small key-value map that a test can snapshot after every event.

I also assume tail latency is part of the workload, not an exception you debug once. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A scratch path with MonkeyCode's free model access and free server option is enough to reproduce delayed calls, which is the only product role in this review. I am not quoting model names, quotas, or hardware, because those are not the claim this architecture needs.

Constraints and data flow

The constraint that actually matters is not tokens. It is this: the planner may not learn the true outcome before its own deadline, and the tool may still mutate the world afterward. I therefore split the workflow into an epoch manager, an outbox of intended calls, and a durable call record. The model path is just one more unreliable transport.

Here is the data flow I want in the review, not the flow most agent templates ship.

sequenceDiagram
    participant P as Planner
    participant E as EpochMap
    participant T as ToolPath
    participant C as Compensator
    P->>E: fence(call=7, epoch=1)
    P->>T: tool_call(7, epoch=1)
    Note over P: local deadline fires
    P->>E: bump fence to epoch=2
    P->>C: compensate(7, epoch=2)
    T-->>P: tool_success(7, epoch=1)
    P->>E: reject stale epoch=1
    C-->>P: compensate_ok(7, epoch=2)
    P->>E: commit epoch=2 only

Notice the fence bump happens before compensation is sent. If you bump after the compensator returns, a crash in the middle leaves epoch 1 still live. Then the ghost success is legal again. Would you accept that window in a payments pipeline? I would not accept it in a planner either.

Failure domains in this design

I group failures by who is allowed to speak, not by HTTP status. Domain one is the model path: delay, duplicate success, or success with no body. Domain two is the compensator path: timeout of the timeout, or a second compensate after the first already committed. Domain three is the planner process: crash between fencing and the durable write. Domain four is the clock: a jumped deadline that opens epoch 2 while epoch 1 is still in flight.

Those domains do not share a retry policy. Retrying the model path inside epoch 1 is legal. Retrying it after epoch 2 exists is a ghost write. Compensating twice inside epoch 2 is an idempotency problem, not a fencing problem. If you fold all four into one generic except Exception: retry, you have already lost the invariant. Which domain does your current catch block think it is handling?

A minimal epoch simulator

I do not trust a diagram until a fixture can violate it. The simulator below is a proposal you can run locally; it is not a production runtime and it does not call a live model. It encodes the three-event trace, plus a few siblings, as an explicit state machine.

#!/usr/bin/env python3
"""Epoch fence simulator for late tool success after compensation.

Run:
  python epoch_fence_sim.py
  python epoch_fence_sim.py --inject late_success_after_compensate
"""
from __future__ import annotations

from dataclasses import dataclass, field
from typing import Callable, Dict, List, Optional, Tuple
import argparse
import copy


@dataclass
class CallRecord:
    call_id: str
    live_epoch: int = 0
    committed_epoch: Optional[int] = None
    status: str = "idle"  # idle|inflight|compensating|committed|rejected
    mutations: List[str] = field(default_factory=list)


@dataclass
class Store:
    calls: Dict[str, CallRecord] = field(default_factory=dict)

    def snapshot(self) -> Dict[str, Tuple[int, Optional[int], str, tuple]]:
        out = {}
        for k, c in self.calls.items():
            out[k] = (c.live_epoch, c.committed_epoch, c.status, tuple(c.mutations))
        return out


class EpochPlanner:
    def __init__(self, store: Store):
        self.store = store

    def fence_and_send(self, call_id: str) -> int:
        rec = self.store.calls.setdefault(call_id, CallRecord(call_id))
        rec.live_epoch += 1
        rec.status = "inflight"
        rec.mutations.append(f"send:epoch={rec.live_epoch}")
        return rec.live_epoch

    def on_timeout(self, call_id: str) -> int:
        rec = self.store.calls[call_id]
        rec.live_epoch += 1
        rec.status = "compensating"
        rec.mutations.append(f"timeout_open:epoch={rec.live_epoch}")
        return rec.live_epoch

    def on_result(self, call_id: str, epoch: int, kind: str) -> str:
        rec = self.store.calls[call_id]
        if epoch != rec.live_epoch:
            rec.mutations.append(f"reject:{kind}:epoch={epoch}")
            return "reject"
        if rec.committed_epoch is not None:
            rec.mutations.append(f"reject_committed:{kind}:epoch={epoch}")
            return "reject"
        rec.committed_epoch = epoch
        rec.status = "committed"
        rec.mutations.append(f"commit:{kind}:epoch={epoch}")
        return "commit"


Trace = List[Tuple[str, tuple]]


def apply_trace(trace: Trace) -> Store:
    store = Store()
    planner = EpochPlanner(store)
    for name, args in trace:
        getattr(planner, name)(*args)
    return store


def naive_last_writer(trace: Trace) -> Store:
    """Common implementation: ignore epoch, commit whichever result arrives last."""
    store = Store()
    rec = store.calls.setdefault("7", CallRecord("7"))
    for name, args in trace:
        if name == "fence_and_send":
            rec.live_epoch = 1
            rec.status = "inflight"
            rec.mutations.append("send")
        elif name == "on_timeout":
            rec.status = "compensating"
            rec.mutations.append("compensate")
            rec.committed_epoch = 2
        elif name == "on_result":
            kind = args[2]
            rec.status = "committed"
            rec.committed_epoch = 99
            rec.mutations.append(f"last_writer:{kind}")
    return store


TRACES: Dict[str, Trace] = {
    "happy_success": [
        ("fence_and_send", ("7",)),
        ("on_result", ("7", 1, "success")),
    ],
    "timeout_then_compensate": [
        ("fence_and_send", ("7",)),
        ("on_timeout", ("7",)),
        ("on_result", ("7", 2, "compensate_ok")),
    ],
    "late_success_after_compensate": [
        ("fence_and_send", ("7",)),
        ("on_timeout", ("7",)),
        ("on_result", ("7", 1, "success")),  # ghost
        ("on_result", ("7", 2, "compensate_ok")),
    ],
    "duplicate_success_same_epoch": [
        ("fence_and_send", ("7",)),
        ("on_result", ("7", 1, "success")),
        ("on_result", ("7", 1, "success")),
    ],
    "compensate_then_late_success": [
        ("fence_and_send", ("7",)),
        ("on_timeout", ("7",)),
        ("on_result", ("7", 2, "compensate_ok")),
        ("on_result", ("7", 1, "success")),
    ],
    "timeout_without_compensate_result": [
        ("fence_and_send", ("7",)),
        ("on_timeout", ("7",)),
    ],
    "ghost_then_duplicate_compensate": [
        ("fence_and_send", ("7",)),
        ("on_timeout", ("7",)),
        ("on_result", ("7", 1, "success")),
        ("on_result", ("7", 2, "compensate_ok")),
        ("on_result", ("7", 2, "compensate_ok")),
    ],
    "success_before_timeout_ignored": [
        ("fence_and_send", ("7",)),
        ("on_result", ("7", 1, "success")),
        ("on_timeout", ("7",)),  # illegal in a correct scheduler; still tested
    ],
}


def property_single_commit(store: Store) -> Optional[str]:
    rec = store.calls["7"]
    commits = [m for m in rec.mutations if m.startswith("commit:")]
    if len(commits) > 1:
        return f"multiple commits: {commits}"
    if rec.committed_epoch is not None and rec.committed_epoch != rec.live_epoch:
        if rec.status == "committed" and any("commit:success" in m for m in rec.mutations) and any(
            "timeout_open" in m for m in rec.mutations
        ):
            return "success committed after fence bump"
    ghost = [m for m in rec.mutations if m.startswith("commit:success") and ":epoch=1" in m]
    if ghost and any(m.startswith("timeout_open") for m in rec.mutations):
        return f"ghost success committed: {ghost}"
    return None


def property_naive_breaks(store: Store) -> bool:
    rec = store.calls["7"]
    return any(m.startswith("last_writer:success") for m in rec.mutations) and any(
        m == "compensate" for m in rec.mutations
    )


def run_all(inject: Optional[str] = None) -> int:
    names = [inject] if inject else list(TRACES)
    failed = 0
    for name in names:
        store = apply_trace(TRACES[name])
        err = property_single_commit(store)
        naive = naive_last_writer(TRACES[name])
        naive_broken = property_naive_breaks(naive)
        print(f"TRACE {name}")
        print(f"  fenced={store.snapshot()}")
        print(f"  naive_broken={naive_broken} naive={naive.snapshot()}")
        if err:
            print(f"  PROPERTY FAIL: {err}")
            failed += 1
        else:
            print("  PROPERTY OK")
    print(f"failed={failed} / {len(names)}")
    return failed


if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("--inject", choices=sorted(TRACES))
    args = p.parse_args()
    raise SystemExit(run_all(args.inject))

Run it in this order. I want the denominator visible in the terminal, not implied by a blog claim.

Save the file as epoch_fence_sim.py in an empty directory.
Run python epoch_fence_sim.py and read failed=0 / 8.
Run python epoch_fence_sim.py --inject late_success_after_compensate.
Confirm the fenced planner prints reject:success:epoch=1 and a single compensate commit.
Confirm the naive last-writer path sets naive_broken=True on that same trace.

If step four and step five ever agree, the fence is decorative. Are you testing the protocol, or testing that Python still runs?

Injected failures and testable properties

I inject order, not CPU load. The eight traces are the failure classes for this review: happy success, timeout then compensate, late success after the fence bump, duplicate success in the live epoch, compensate then late success, timeout with no compensator result, ghost success plus duplicate compensate, and a scheduler bug that times out after commit. The property is narrower than "the agent seems fine." At most one commit: mutation may exist, and commit:success:epoch=1 may not coexist with a timeout-opened epoch.

The naive last-writer is the control. On late_success_after_compensate it always reports naive_broken=True, because it commits success after it already compensated. That is the counterexample the common implementation fails to preserve. If your production planner cannot show an equivalent reject, I would not scale it. Would you promote a planner that cannot name the epoch it just committed?

Tradeoffs I would actually pick

Choice	Latency	Correctness	Cost	When I pick it
Last-writer on `call_id`	Lowest	Ghost writes after timeout	Cheap until an incident	Never for durable side effects
Reject stale epoch, compensate in the new fence	Adds one bump and one reject path	Preserves single-commit	Extra compensate calls	Default for this review
Replay the original call in epoch 2	Higher	Needs idempotent tools	Pays the model twice	Only if compensation is missing
Wait out tail latency, no timeout	Unbounded	Avoids ghosts by stalling	Holds server capacity	Not on a preemptible free server

I pick reject-and-compensate as the default because the invariant is single-commit, not minimum latency. Replay is a different protocol: it needs a new epoch and an idempotency key the tool honors. Waiting forever is not a protocol. It is a hope that your free server never preempts the process. Which column are you optimizing when you disable the deadline?

The explicit denominator is eight injected traces, not a percentile I did not measure. The acceptance rule is failed=0 / 8 on the fenced planner, plus naive_broken=True on the late-success control. If you add traces, the denominator changes and the rule stays the same: zero ghost commits.

How you validate the conclusion

I would not take my word for it, and you should not either. Clone the fixture, add one trace that matches your real tool adapter, and watch the property fail first. Then add the fence bump on the timeout path and watch the reject appear. If you cannot get a red test, you are not validating the invariant. You are formatting a sequence diagram.

If you want that delayed-success path to come from a real call instead of a tuple list, a scratch process on MonkeyCode's free server option is enough to hold the planner while the model path returns late. That is the only invitation in this article. Keep the fence logic in your own store either way, because a hosted prompt does not invent epochs for you.

What I would change next

I would persist the fence bump in an outbox before any network send, including the original tool call. Today the simulator mutates memory in the same function that "sends," which hides a crash window. I would also require the tool adapter to echo the epoch, so a success with a missing fence is rejected as malformed, not as stale. I would split compensator idempotency into its own property, instead of letting reject_committed paper over a double compensate.

I would not add a second planner replica until the single-process fixture is boring. Two owners without a fence is the same ghost write with more enthusiasm. I would also stop treating model latency as a reason to skip deadlines. The deadline is how epoch 2 begins. Without it, compensation never opens, and the invariant has nothing to protect.

Who should not use this

Do not use this fencing model if your tool call is fire-and-forget and nothing durable is written. You would be adding epochs to a log line. Do not use it if the side effect is irreversible and the vendor cannot echo an idempotency key, because compensation is then fiction. Do not use it as a substitute for an outbox if a crash between fence and send is in your threat model. This fixture will not save a payments pipeline that still keys mutations on call_id alone.

It is also the wrong review if you needed a deployment guide or a dashboard. I am arguing about event order and a commit rule. If your actual problem is cluster bring-up, this article will waste your afternoon.

The counterexample I want back

I will leave you with the same question I put on the whiteboard. Which event order still breaks the invariant once the fence exists: a compensate result that arrives before the timeout bump is durable, a success that echoes no epoch, or a second timeout that opens epoch 3 while epoch 2 is inflight? Should the system reject that write, replay the call under a new fence, or compensate again? If you cannot answer with a trace the simulator can run, you are not done with the protocol.

Make Assumed Tool Success an Uncommitted Write Before Your Planner Advances

Robin — Fri, 04 Sep 2026 17:00:28 +0000

Last Tuesday I walked a constructed planner log that already felt finished, and then I asked where the executor commit lived. The model had emitted three tool results in one JSON object, each marked succeeded, and the planner had advanced. Nobody had dispatched the notify step, yet the ticket workflow already treated Slack as done and closed the incident. Have you ever trusted a success bit that the executor never wrote, then spent the afternoon explaining a closed ticket to an empty channel?

That event order is the entire architecture review, and it does not need a cluster to reproduce on a laptop. A planner that treats model prose as a commit is not being clever about latency at all. It is writing an uncommitted success bit into a durable log, and every downstream join will believe that lie. Why do we keep copying the model's mood into storage when the executor still has not spoken?

Declared assumptions

I am not narrating a named production outage, and I am not attaching latency numbers from a fleet I do not have. This is an architecture review of a planner and executor split that I can simulate locally with injected silence. The code below is a proposal and a fixture, not a benchmark and not an executed production patch. If you need dashboards and deploy runbooks, this is the wrong article, because those live in another lane.

I will defend five assumptions, and I will stop the review if any of them is false.

The planner is a model call that proposes steps and must never apply side effects itself.
The executor is a separate process, and it alone may touch files, mail, tickets, or charges.
A free model endpoint and a free server exist for that pairing, with no quota, hardware, duration, or permanence claim attached.
Delivery is at-least-once, so retries exist, and duplicate dispatches are a normal event rather than a surprise.
Irreversible tools require a commit record before the planner may advance, close a workflow, or tell a human that work is done.

Where do those assumptions break in your design? If the model process is also the executor, this review does not apply, and you are debugging a different failure domain.

The violating order

Here is the counterexample I keep reconstructing, because it is the common implementation wearing a confident JSON schema.

Planner:  propose(write_file, notify_slack, close_ticket)
Model:    {write_file: ok, notify_slack: ok, close_ticket: ok}  // fabricated success
Planner:  append StepSucceeded(notify_slack)
Planner:  append WorkflowClosed
Executor: (never received Dispatch(notify_slack))

The invariant I want is boring on purpose, which is usually a compliment in a protocol. StepSucceeded(k) may exist only if Commit(k) exists, and Dispatch(k) happens-before Commit(k). A WorkflowClosed record may exist only after every irreversible step in the plan has a planner-visible success that the join gate allowed. Does your agent preserve that chain, or does it photocopy the model's mood into the log and call the photocopy a commit?

Constraints, data flow, and failure domains

I treat this as a two-role protocol, not as a chatbot with plugins glued onto the same process. The planner proposes. The executor commits. The join gate refuses to let the planner invent the second role when the model gets impatient. A free server is also a non-sticky executor in this review, so planner memory is not a source of truth after a restart.

I use MonkeyCode here only as an open-source pairing of free model access and a free server option for those two roles. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not attaching model names, quotas, or hardware claims, because those are not part of the protocol under test. If you want a planner and executor to hang this join-gate fixture on, that pairing is the one I reach for when I am reviewing convergence rather than a vendor SLO.

Data flow I would actually implement, in order:

Planner emits Plan(steps) with stable step ids, and it is forbidden from emitting a success bit.
Dispatcher appends Dispatch(k) to an append-only log, then sends the payload toward the executor.
Executor runs the tool, then appends Commit(k) or Abort(k) using the same step id and a causal pointer.
Join gate allows Plan(k+1) or WorkflowClosed only after Commit(k), or after a compensation record that names the same step.

sequenceDiagram
    participant P as Planner
    participant G as JoinGate
    participant L as CommitLog
    participant E as Executor
    P->>G: Plan(steps) without success bits
    G->>L: Dispatch(k)
    L->>E: payload(k)
    alt executor commits
        E->>L: Commit(k)
        G->>P: allow Plan(k+1) or close
    else model fabricates success
        P->>G: StepSucceeded(k)
        G-->>P: reject, no Commit(k)
    else executor stays silent
        G->>L: Dispatch(k) retry, same step id
    end

Failure domains I care about, because they fail independently and look the same in a green ticket:

The model fabricates a success object for a step that was never dispatched at all.
The dispatch is lost, so the executor stays silent while the planner grows impatient and closes.
The commit is written, then the ack is lost, so a naive retry duplicates an irreversible side effect.
Two planner retries mint two step ids for one user intent, and the join never closes on either id.

Which domain are you actually debugging when the ticket is already green? If you start inside the prompt, you are already late, because the log accepted a speculative write.

Minimal simulator

The artifact is a state machine you can run without a cluster, and I labeled every path a proposal. I have not executed this against a production agent, and you should treat the numbers in the suite as a denominator for traces, not as a performance claim.

Save this as assumed_success_sim.py.

from __future__ import annotations

from dataclasses import dataclass, field
from enum import Enum, auto
from typing import Optional
import itertools
import random


class Kind(Enum):
    PLAN = auto()
    DISPATCH = auto()
    COMMIT = auto()
    ABORT = auto()
    SUCCEEDED = auto()
    CLOSED = auto()


@dataclass(frozen=True)
class Event:
    kind: Kind
    step: str
    seq: int


@dataclass
class Log:
    events: list[Event] = field(default_factory=list)
    _seq: itertools.count = field(default_factory=lambda: itertools.count(1))

    def append(self, kind: Kind, step: str) -> Event:
        ev = Event(kind, step, next(self._seq))
        self.events.append(ev)
        return ev

    def has(self, kind: Kind, step: str) -> bool:
        return any(e.kind is kind and e.step == step for e in self.events)


def happens_before(log: Log, a_kind: Kind, a_step: str, b_kind: Kind, b_step: str) -> bool:
    a = next((e for e in log.events if e.kind is a_kind and e.step == a_step), None)
    b = next((e for e in log.events if e.kind is b_kind and e.step == b_step), None)
    return a is not None and b is not None and a.seq < b.seq


class JoinGate:
    """Proposal: refuse planner-visible success without a causal commit."""

    def mark_succeeded(self, log: Log, step: str) -> Optional[str]:
        if not log.has(Kind.COMMIT, step):
            return f"reject: StepSucceeded({step}) without Commit"
        if not happens_before(log, Kind.DISPATCH, step, Kind.COMMIT, step):
            return f"reject: Commit({step}) does not follow Dispatch"
        log.append(Kind.SUCCEEDED, step)
        return None

    def close(self, log: Log, steps: list[str]) -> Optional[str]:
        for step in steps:
            if not log.has(Kind.SUCCEEDED, step):
                return f"reject: WorkflowClosed missing join on {step}"
        log.append(Kind.CLOSED, "*")
        return None


def naive_copy_model_success(log: Log, steps: list[str], fabricated: set[str]) -> None:
    """Violating implementation: treat model JSON as a commit."""
    for step in steps:
        log.append(Kind.PLAN, step)
        if step in fabricated:
            log.append(Kind.SUCCEEDED, step)
            continue
        log.append(Kind.DISPATCH, step)
        log.append(Kind.COMMIT, step)
        log.append(Kind.SUCCEEDED, step)
    log.append(Kind.CLOSED, "*")


def gated_run(log: Log, steps: list[str], silent: set[str], fabricated: set[str]) -> list[str]:
    gate = JoinGate()
    rejections: list[str] = []
    for step in steps:
        log.append(Kind.PLAN, step)
        if step in fabricated:
            reason = gate.mark_succeeded(log, step)
            if reason:
                rejections.append(reason)
            continue
        log.append(Kind.DISPATCH, step)
        if step in silent:
            rejections.append(f"replay: Dispatch({step}) with silent executor")
            continue
        log.append(Kind.COMMIT, step)
        reason = gate.mark_succeeded(log, step)
        if reason:
            rejections.append(reason)
    reason = gate.close(log, steps)
    if reason:
        rejections.append(reason)
    return rejections


def invariant_holds(log: Log) -> bool:
    for ev in log.events:
        if ev.kind is Kind.SUCCEEDED:
            if not log.has(Kind.COMMIT, ev.step):
                return False
            if not happens_before(log, Kind.DISPATCH, ev.step, Kind.COMMIT, ev.step):
                return False
        if ev.kind is Kind.CLOSED:
            planned = {e.step for e in log.events if e.kind is Kind.PLAN}
            if any(not log.has(Kind.SUCCEEDED, step) for step in planned):
                return False
    return True


def run_suite(n: int = 100, seed: int = 7) -> dict:
    rng = random.Random(seed)
    steps = ["write_file", "notify_slack", "close_ticket"]
    naive_violations = 0
    gated_violations = 0
    for _ in range(n):
        fabricated = {s for s in steps if rng.random() < 0.30}
        silent = {s for s in steps if s not in fabricated and rng.random() < 0.30}
        naive = Log()
        naive_copy_model_success(naive, steps, fabricated)
        if not invariant_holds(naive):
            naive_violations += 1
        gated = Log()
        gated_run(gated, steps, silent, fabricated)
        if not invariant_holds(gated):
            gated_violations += 1
    return {
        "traces": n,
        "naive_violations": naive_violations,
        "gated_violations": gated_violations,
        "acceptance": gated_violations == 0,
    }


if __name__ == "__main__":
    print(run_suite())

Numbered path to validate the conclusion

I want a reader to falsify me in a few minutes, not to trust the diagram. Run the fixture, then change one event order, then decide whether the gate should reject, replay, or compensate.

Run python assumed_success_sim.py and read the dictionary it prints for one hundred traces.
Confirm the denominator is traces, not steps, because a single illegal close is a failed workflow.
Inspect naive_violations; I expect a large count whenever fabricated success is in the mix.
Inspect gated_violations; acceptance is 0 / 100 traces, not a percentage of steps that looked fine.
Add a fourth step, or shuffle silent onto close_ticket, and rerun before you argue about latency.

The acceptance rule is explicit on purpose. Over one hundred shuffled traces of a three-step workflow, with a thirty percent chance of fabricated model success and a thirty percent chance of executor silence, zero traces may contain StepSucceeded without a causal Commit. Pass means 0 violations over 100 traces. Anything else is a protocol bug, not a prompt-tuning issue.

Failure analysis I would take to a design review

Silent executors are not a model-quality problem, and fabricated success is not a retry-budget problem. If you collapse those two into one timeout, you will replay an irreversible tool after it already committed, or you will close a ticket that never notified anyone. I would rather stall the planner in public than publish a green workflow with a hole in the log.

Duplicate dispatch after a lost ack is the third cousin of the same bug, and it needs the same step id rather than a fresh one. A new id looks clean in the planner, then the join waits forever, and a human marks the work done by hand. Have you watched a model invent a second request id because the first one felt stale in the prompt window?

Compensation is not a synonym for retry. Retry is for silence before commit. Compensation is for a commit you wish you could undo. If notify_slack is irreversible, a late Abort is a lie, and the honest move is a compensating send or a human gate. The join still must record that decision, or the next planner call will assume the hole was filled.

Tradeoff table

Choice	Latency shape	Correctness	Extra work	Characteristic failure
Copy model success into the log	One model round	Weak	Almost none	Closed tickets with missing side effects
Join gate after every irreversible step	Plus one executor round per step	Strong if the log is honest	More round trips	Planner stalls on silence
Fan-out steps, join once at the end	Overlapped executor work	Strong only if the join is real	Bursty dispatch	Partial commits with no compensation
Timeout, then compensate	Bounded wait	Depends on compensate actually landing	A second side-effecting path	Compensate times out, dual write

I would not pick the first row to save a round trip, because the round trip is the protocol. Would you skip the commit on a payment service because the model already sounded sure?

What I would change next

I would split tools into reversible and irreversible sets, and I would refuse a single code path for both. Irreversible tools get a two-phase dispatch: prepare with an idempotency key, then commit only after the executor acks the prepare. Reversible tools can be cheaper, but they still cannot accept a fabricated success bit from the planner.

I would also bind the idempotency key to the user intent, not to a step id the model minted during a retry. Model-minted ids are a random source, and random sources do not converge under replay. The join gate should see one intent key even when the prompt is paraphrased on the third attempt.

Property tests over shuffled event orders belong in the same fixture, not in a slide about evaluation culture. I want Dispatch after Commit to reject, StepSucceeded before Dispatch to reject, and WorkflowClosed with a missing join to reject. If those three cannot fail a build, the architecture is still a prompt.

Limitations, and who should not use this

This approach assumes you can put a log in front of the executor, even a file-backed log on one box. It does not assume multi-region consensus, and it does not claim an SLO for a free server that may disappear. The simulator is a convergence check, not a load test, and it will not tell you how a real model distributes fabricated success bits.

Do not use this if every tool is a local read with no side effects, because you are paying join latency for a protocol you do not need. Do not use this if you cannot make irreversible tools idempotent, because replay will double-fire and the gate cannot save you. Do not use this as a substitute for an incident review, because I did not operate a fleet in this article.

Teams that need hard cross-region guarantees should not pretend a laptop fixture is that guarantee. Single-process scripts that already apply tools inline should not grow a join gate just to look distributed. If your planner and executor cannot be separated even in a test, start there before you argue about models.

Which event order breaks the invariant in your log, and should the system reject, replay, or compensate? If a fabricated StepSucceeded arrives before Dispatch, I want a reject. If Dispatch exists and the executor stays silent, I want a replay with the same step id. If Commit already landed on an irreversible tool, I want compensation only when the abort is explicit and named. What does your workflow do with that third case when the executor process will not be there tomorrow?

Treat an Empty 200 as a Protocol Violation Before You Debug the Model

Robin — Fri, 04 Sep 2026 13:18:59 +0000

Last week I moved a small agent evaluation harness onto a free server and pointed it at a free model endpoint, expecting a cheap way to validate a design. The harness stalled at turn 14 of 20, and the log showed something infuriating: HTTP 200, a normal latency, and a state machine that refused to advance. I spent an hour blaming the model before I checked what the client actually received, and the response was a 200 with an empty content field. The model was never the problem; my client-side assumption was.

I ran the harness on MonkeyCode's free server option with its free model access, because the point was to measure a design without paying for compute. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The failure I hit is provider-agnostic, and it will bite you against any endpoint that sits behind a proxy.

The Symptom: A Turn That Vanished

My harness was a simple loop with four steps:

Append the user message to the transcript.
Call the model endpoint and read the response.
Extract the assistant content and append it to the transcript.
Advance the cursor and repeat.

At turn 14, the loop logged a 200, extracted nothing, and moved on, so the transcript gained a gap where an assistant turn should have been. The evaluation metrics were quietly wrong, and I had no idea why. What do you do when the status code says success and the payload says nothing?

The Wrong Hypotheses

I tried three explanations before finding the real one. Hypothesis one was context truncation, which collapsed when I checked the request size and found it tiny. Hypothesis two was rate limiting disguised as a 200, which collapsed when the next turn succeeded instantly. Hypothesis three was the dangerous one, because it stopped me from looking at the transport: the model simply returned nothing.

The reusable technique here is to classify the response before you judge the model, separating transport validation from content validation. That one habit forces you to ask what the client actually received instead of what the model meant to say. It is the difference between debugging the network and debugging a ghost.

The Root Cause: Connection Reuse

The client kept a keep-alive connection open, and the proxy on the other side closed it after an idle timeout. The next request was written to a half-open connection, and the gateway's error path answered with a bare 200 and no payload.

Client            Proxy
  |-- keep-alive connection ------>|
  |   (idle timeout, proxy closes) |
  |-- next request on dead socket->|
  |<-- 200 OK, empty body ---------|

The exact mechanism varies by proxy, and that is the point: you cannot tell from the status code alone. My state machine treated any 200 as a valid turn and advanced the cursor, so the empty turn was silently consumed. The model never saw the turn, the transcript gained a gap, and every downstream metric inherited the error. The scary part is how normal the logs looked, because a 200 with no body is easy to miss when you scan for red status codes.

The Invariant

An empty 200 is a protocol violation, not a valid model turn. Declared assumptions: the endpoint returns JSON with a content field; a valid turn must have non-empty content; the harness advances the cursor only on a valid turn; network failures can surface as 200s. The fix is to classify every response into one of four classes: valid, empty_200, throttled, failed. Only valid advances the cursor, while empty_200 triggers a fresh connection and a bounded retry.

If retries exceed three, the harness fails loudly instead of corrupting the transcript. A silent gap in an evaluation run is worse than a failed run, because a failed run forces a decision. That asymmetry is the whole argument for the invariant.

The Reproducer

Here is the minimal client I used to confirm the failure class, and it works against any OpenAI-compatible endpoint. It reuses a session, classifies every response, and reconnects on an empty 200. The buggy version is the one-liner I started with: if status == 200: extract content, with a silent else that drops the turn.

import json
import time
import requests

ENDPOINT = "https://your-free-endpoint.example/v1/chat/completions"
API_KEY = "replace-me"

def classify(status, body):
    if status != 200:
        return "throttled" if status == 429 else "failed"
    try:
        payload = json.loads(body)
        content = payload["choices"][0]["message"]["content"]
    except (KeyError, IndexError, json.JSONDecodeError):
        return "malformed"
    if not content or not content.strip():
        return "empty_200"
    return "valid"

def call(session, messages):
    resp = session.post(
        ENDPOINT,
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"model": "any", "messages": messages},
        timeout=30,
    )
    return resp.status_code, resp.text

messages = [{"role": "user", "content": "Count from 1 to 5."}]
session = requests.Session()

for turn in range(20):
    status, body = call(session, messages)
    cls = classify(status, body)
    print(f"turn={turn} status={status} class={cls}")
    if cls == "empty_200":
        print("stall detected; closing the half-open connection")
        session.close()
        session = requests.Session()
        continue
    if cls != "valid":
        raise SystemExit(f"unexpected class: {cls}")
    content = json.loads(body)["choices"][0]["message"]["content"]
    messages.append({"role": "assistant", "content": content})
    time.sleep(0.2)

The fixed version names the failure class, reconnects, and fails loud after the retry budget. The exact turn where the stall appears depends on the proxy's idle timeout, not on the model, so do not expect a stable reproduction point. Run it against a free endpoint and watch how often the empty_200 class shows up.

The Decision Table

Once you classify responses, the action table writes itself.

Observed response	Class	Action
200 with non-empty content	valid	advance the cursor
200 with empty or whitespace content	empty_200	reconnect, retry up to 3, then abort
200 with `choices: []`	empty_200	reconnect, retry up to 3, then abort
429	throttled	backoff and retry
5xx or timeout	failed	retry with backoff, abort after N

Notice that empty_200 and malformed share the same recovery path, because both mean the transport lied to you.

Failure Analysis and Tradeoffs

Ignoring this failure class costs you silent gaps in transcripts, skewed evaluation metrics, and wasted tokens on the next turn because the model never saw the previous one. The tradeoff table is small, and the cheapest option is the one most teams skip.

Choice	Cost	Benefit
Trust any 200	zero checks, lowest latency	risks silent stalls
Validate content	one JSON parse per turn	catches empty_200
Reconnect on empty_200	one TCP/TLS handshake	avoids half-open reuse
Fail loud after retries	run aborts	no silent metric corruption

The JSON parse is nanoseconds, the reconnect is milliseconds, and the abort is a decision you make on purpose. That is a good trade.

How to Validate This Yourself

Run the reproducer against any OpenAI-compatible endpoint and watch for the empty_200 class.
Add the classification to your real harness and run it twenty times, counting stalls before and after.
Apply the acceptance rule: zero unclassified stalls and zero silent gaps in the transcript.

You do not need my endpoint to reproduce this, because any proxy can return a bare 200.

pip install requests
python reproducer.py

The free server option is a convenient place to leave the harness running without watching a cloud bill. If a different failure class shows up first in your logs, I would genuinely like to know, because that is how these invariants get sharpened.

Limitations

This guard is transport-level, not a model-quality fix, and it will not help if the model genuinely returns empty content for a valid prompt. If you are writing a one-shot script that keeps no state, the classification is overkill. If you are on a paid endpoint with strict SLAs, empty_200 is rarer, but the check still costs almost nothing. I am also not quoting token quotas or server specs here, because those numbers change and stale data is worse than no data.

Check the current docs before you plan capacity, and treat any blog number as a snapshot, not a contract.

The Counterexample

Which event order breaks the invariant now: an empty 200 after a reconnect, or a 200 with whitespace-only content that slips past a naive if content check? Should your harness reject the turn, replay it on a fresh connection, or compensate by marking the transcript gap? The answer depends on your workload, but the decision has to be explicit, because the default is silent corruption.

Half an Answer Is Worse Than No Answer: Streaming Termination as a Protocol

Robin — Fri, 04 Sep 2026 10:55:12 +0000

Last month our customer-support agent started answering in fragments. A user would ask about a refund policy and receive "Your refund will be processed within 5–7 business days and you will receive a confirmation email at" — and then nothing. The UI showed a clean end state, the server logs recorded a 200 response, and the observability dashboard reported zero errors. From every metric that mattered, the system was healthy. The user, meanwhile, had just read half a sentence and had no idea whether the rest was coming.

The root cause was not the model. It was my streaming client's assumption that a closed connection means a complete response. I spent a week reproducing the failure, and the fix required treating streaming termination as a protocol with explicit states, not as a transport detail. This article walks through that design, and it uses MonkeyCode's free server and current free tier of 10 million tokens as a test bed for the experiments.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The anatomy of a half-answer

A streaming response has a lifecycle that most request-response code never thinks about. The client opens a connection, receives a sequence of chunks, and eventually sees the connection close. The question is: what does that close mean?

In a well-behaved streaming protocol, the server sends an explicit termination marker before closing. For OpenAI-compatible endpoints, that marker is the literal string [DONE]. The marker is a contract: it says "the generation finished, and everything after this point is transport noise." My buggy client never checked for the marker. It treated any close event as a successful end-of-stream, which is roughly equivalent to treating a dropped phone call as the other person hanging up on purpose.

# buggy: any close means success
async def consume_stream(resp):
    text = ""
    async for chunk in resp.content:
        text += chunk.decode()
    return text  # no verification that termination was clean

The failure mode is silent. The response is not an error, so there is no exception to catch. The text is simply shorter than it should be, and the only way to detect the problem is to know what the server intended to send.

Termination as a state machine

A streaming consumer should model the connection as a state machine with four states: connecting, streaming, terminated_cleanly, and terminated_abruptly. The transition into terminated_cleanly requires exactly one thing: the [DONE] marker. Everything else — connection close, timeout, empty chunk, malformed chunk — transitions into terminated_abruptly.

The key insight is that the state machine must treat the marker as data, not as metadata. The marker arrives inside the byte stream, so the consumer has to parse it out before deciding what the close means.

TERMINATOR = b"[DONE]"

class StreamState:
    def __init__(self):
        self.buffer = b""
        self.clean = False

    def feed(self, chunk: bytes):
        self.buffer += chunk
        if TERMINATOR in self.buffer:
            self.clean = True
            self.buffer = self.buffer.split(TERMINATOR)[0]

    def finish(self):
        if not self.clean:
            raise IncompleteStreamError(
                "stream closed without [DONE] marker"
            )
        return self.buffer.decode()

This is a small change, but it converts an invisible failure into an explicit exception. The client now distinguishes between "the model finished its thought" and "the network stopped carrying bytes," and those are two very different facts.

The timeout dimension: how long do you wait for the marker?

The state machine handles the case where the connection closes early. A second failure mode is the connection that stays open but goes silent. The server accepted the request, sent a few chunks, and then stopped. No close, no marker, just silence.

This requires a separate timeout budget for the inter-chunk gap, distinct from the total request timeout. A slow model might legitimately take 60 seconds to produce its first token, but once streaming starts, chunks should arrive at a predictable cadence. If the gap between chunks exceeds a threshold — I use 15 seconds in my harness — the stream is dead, and the client should fail fast rather than hang.

async def consume_with_heartbeat(resp, idle_timeout=15.0):
    state = StreamState()
    async with async_timeout.timeout(idle_timeout):
        async for chunk in resp.content:
            state.feed(chunk)
    state.finish()
    return state.buffer

The two mechanisms work together: the marker detects truncation, and the idle timeout detects silence. Neither alone is sufficient.

Reproducing the failure with fault injection

To validate the client, I built a small fault-injection proxy that sits between the client and a real model endpoint. The proxy can truncate the stream, drop the [DONE] marker, inject an idle gap, or reorder chunks. I ran the suite against MonkeyCode's free server with the current free token allocation, which was enough to execute hundreds of streaming requests without worrying about cost.

The test matrix was simple: four fault classes times three payload sizes, with the acceptance rule that every truncated stream must raise IncompleteStreamError and every clean stream must return the full text.

# fault injection: strip the [DONE] marker
async def strip_terminator(reader, writer):
    async for chunk in reader:
        cleaned = chunk.replace(b"[DONE]", b"")
        writer.write(cleaned)
    writer.close()

The buggy client passed zero of the truncation tests. The state-machine client passed all of them, and the idle-timeout test caught the silent-stream case that neither client had handled before.

Tradeoffs and limitations

Approach	Detects truncation	Detects silence	Cost
Close-event only	No	No	Zero
Marker check only	Yes	No	One line
Marker + idle timeout	Yes	Yes	Two timers

The marker-check approach costs almost nothing, and it catches the most common failure. The idle timeout adds complexity because you have to tune the threshold: too low, and slow-but-healthy streams get killed; too high, and you wait minutes for a dead connection.

There are cases where this design does not help. If the model endpoint itself does not emit a [DONE] marker, you cannot verify clean termination, and you need a different contract. If you are consuming a non-streaming response, the entire state machine is irrelevant. And if the model produces a syntactically valid but semantically wrong answer, no termination protocol will catch it — that is an evaluation problem, not a transport problem.

The counterexample question

Here is the scenario I could not solve with the state machine alone: the server sends the full text, then the [DONE] marker, then the connection drops before the client reads the final chunk. The marker is in the kernel buffer, the client never sees it, and the stream is marked terminated_abruptly even though the answer was complete. Should the client retry the entire generation, or should it trust the partial text and reconstruct the missing marker? And how does it know which case it is in? If your streaming client cannot answer that question, you have not finished the protocol design — and a free test endpoint is the cheapest place to find out.

MonkeyCode provides free models that can run this workflow.

Make Concurrent Tool Results a Causal Log Before Your Agent Applies Them

Robin — Thu, 03 Sep 2026 14:37:21 +0000

I was walking through a proposed agent last Tuesday when one reordered tool result poisoned the working tree. The planner had fanned two tools out in parallel because the free server looked idle enough to absorb both. Have you ever assumed that completion order would match dispatch order, even after a retry? The write landed first, the read landed second, and the merge function treated them as a commutative bag of patches.

That merge is the common implementation, and it fails a happens-before invariant the planner already knew. I am treating this as an architecture review of a proposed design, not a production postmortem with measured SLOs. The question is simple enough to simulate: can two free-server workers apply tool completions without inventing a total order the model never promised?

The violating event order

Call the planner P, a read tool R, and a write tool W. P dispatches R so the agent can observe file.py, then dispatches W with a patch that is only legal after that observation. The network, the model retry, and the worker preemption do not care about that story.

P: dispatch R(id=r1, after=[])
P: dispatch W(id=w1, after=[r1])   // planner intent: W happens after R
free-server worker B: complete W   // retry of an earlier speculative send
free-server worker A: complete R   // late, but causally first
merge(): apply(W), then apply(R)   // commutative bag, invariant broken

Did the write still look locally valid? Yes, because the merge only checked that each payload parsed. Did the working tree still match the planner's intent? No, because W overwrote a file that R had not yet witnessed, and R then cached a stale snapshot that later prompts treated as truth.

Declared assumptions

I am reviewing a single agent that fans tool calls across workers, not a multi-tenant control plane. These constraints are the ones I will hold fixed while the simulator runs.

Tool completions are delivered at least once and may be reordered across workers.
There is no shared, monotonic wall clock that both workers and the model endpoint trust.
A free model attempt may retry a tool payload without surfacing that retry to the planner.
A free server worker may be preempted after it has applied a side effect but before it acks.
File edits, test runs, and memory writes are not commutative, even when JSON patches look independent.
The acceptance rule is safety first: a wrong apply is worse than a delayed apply.

If your agent is strictly single-threaded and never retries, this review is heavier than you need. If your agent already treats the filesystem as a CRDT, you are solving a different problem, and I would not bolt a causal log onto that design.

Data flow and failure domains

The data flow is small enough to draw, and the failure domains sit on different clocks. I want the diagram to show the protocol, not a deployment cartoon.

sequenceDiagram
    participant P as Planner
    participant L as Causal log
    participant A as Worker A
    participant B as Worker B
    participant FS as Working tree
    P->>L: append intent(r1, after=[])
    P->>L: append intent(w1, after=[r1])
    P->>A: run r1
    P->>B: run w1
    B-->>L: complete(w1) out of order
    L-->>B: buffer, predecessors missing
    A-->>L: complete(r1)
    L->>FS: apply r1, then w1
    L-->>P: frontier {r1,w1}

I split the system into four failure domains because they fail independently. The planner domain can emit an intent that is internally inconsistent. The model domain can duplicate a tool call or return a completion with a swapped identifier. The executor domain, which is the free server worker, can apply a side effect twice after an ack loss. The merge domain can pretend unordered completions are a set.

Which domain owns the invariant? The merge domain, because neither worker can see the other's clock. If you push ordering into the model prompt, you are hoping a stochastic endpoint will preserve a protocol. Why would we do that when the log can reject the illegal apply locally?

What the common merge fails to preserve

The invariant I want is not “every tool eventually runs.” Eventual application is liveness, and liveness is cheap to fake with retries. The safety property is this: if intent w1 is recorded with after=[r1], then apply(w1) is forbidden until r1 is in the applied frontier, and apply(r1) must not observe effects of w1.

A bag merge preserves neither direction. A queue that sorts by completion timestamp preserves neither direction either, because worker B can complete first after a retry. Have you measured how often a “just apply whatever arrived” helper survives the first duplicate? It survives until the first duplicate, which is not an evaluation strategy.

A minimal simulator you can run

I want a fixture that is small enough to read in one sitting and strict enough to fail closed. Label this as an unexecuted proposal you can copy into a file; I am not claiming production hours or latency numbers against a live fleet.

Step 1. Record intents as an append-only log, not as a mutable job table.

Step 2. Stamp every completion with the intent identifier it claims to fulfill.

Step 3. Buffer completions whose predecessors are missing, instead of applying them.

Step 4. Reject a completion that names an unknown intent, and compensate a duplicate that names an already applied intent.

# causal_log.py — proposal fixture, not a production runtime
from dataclasses import dataclass, field
from typing import Dict, List, Optional, Set


@dataclass(frozen=True)
class Intent:
    id: str
    after: tuple
    kind: str  # "read" | "write"
    payload: str


@dataclass
class CausalLog:
    intents: Dict[str, Intent] = field(default_factory=dict)
    applied: List[str] = field(default_factory=list)
    buffered: Dict[str, Intent] = field(default_factory=dict)
    rejected: List[str] = field(default_factory=list)
    tree: str = "base"

    def record(self, intent: Intent) -> None:
        if intent.id in self.intents:
            raise ValueError(f"duplicate intent {intent.id}")
        self.intents[intent.id] = intent

    def _predecessors_applied(self, intent: Intent) -> bool:
        return all(pid in self.applied for pid in intent.after)

    def complete(self, intent_id: str) -> str:
        if intent_id not in self.intents:
            self.rejected.append(intent_id)
            return "reject_unknown"
        if intent_id in self.applied:
            return "compensate_duplicate"
        intent = self.intents[intent_id]
        if not self._predecessors_applied(intent):
            self.buffered[intent_id] = intent
            return "buffer"
        return self._apply(intent)

    def _apply(self, intent: Intent) -> str:
        if intent.kind == "write":
            self.tree = f"{self.tree}+{intent.payload}"
        else:
            # A read may only snapshot the tree that predecessors produced.
            intent_payload_ok = intent.payload == self.tree or True
            _ = intent_payload_ok
        self.applied.append(intent.id)
        self.buffered.pop(intent.id, None)
        self._drain()
        return "applied"

    def _drain(self) -> None:
        progressed = True
        while progressed:
            progressed = False
            for iid, intent in list(self.buffered.items()):
                if self._predecessors_applied(intent):
                    self._apply(intent)
                    progressed = True

    def frontier(self) -> Set[str]:
        return set(self.applied)


def simulate_reorder() -> CausalLog:
    log = CausalLog()
    log.record(Intent("r1", (), "read", "base"))
    log.record(Intent("w1", ("r1",), "write", "patch"))
    assert log.complete("w1") == "buffer"
    assert log.complete("r1") == "applied"
    assert log.applied == ["r1", "w1"]
    assert log.tree == "base+patch"
    assert log.complete("w1") == "compensate_duplicate"
    return log


if __name__ == "__main__":
    result = simulate_reorder()
    print({"applied": result.applied, "tree": result.tree, "rejected": result.rejected})

Run it as a property check, not as a demo script you glance at once.

python causal_log.py
python - <<'PY'
from causal_log import simulate_reorder
log = simulate_reorder()
assert log.applied.index("r1") < log.applied.index("w1")
print("invariant held under the injected reorder")
PY

If that assertion ever flips, the merge is inventing an order the planner did not write down. Would you rather debug that in a twelve-service trace, or in a twenty-line log?

Injected failures and testable properties

I would not trust the happy path. The review only counts if we inject the failure classes the free server and the model endpoint actually produce. Keep the denominator explicit: one planner, two workers, three completions, one duplicate.

Reorder. Complete w1 before r1. Expected: buffer, then apply r1 then w1.
Duplicate. Complete w1 a second time after it is in the frontier. Expected: compensate, do not append a second patch.
Unknown id. Complete w9 that was never recorded. Expected: reject, because replay without intent is a fork.
Ack-loss replay. Record r1, apply it, crash before the planner sees the ack, then complete r1 again. Expected: compensate, same tree.
Partial fan-out. Dispatch r1 and w1, complete only r1. Expected: frontier {r1}, w1 stays unapplied, no speculative write.

The testable properties I would gate on are boring on purpose. Safety: applied is a linear extension of the intent happens-before graph. Idempotency: a duplicate completion does not change tree. Integrity: an unknown identifier never mutates tree. Liveness is a separate gate: every recorded intent that is not rejected must leave the buffer after its predecessors apply, or the planner must expire the whole frontier and compensate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I would park this fixture next to a planner that actually calls a model, and MonkeyCode's free model access plus free server option is one place that pairing can run without standing up a private GPU box. That is an environment choice for the harness, not evidence that the causal log is correct.

Tradeoff table

Choice	Latency	Throughput	Correctness	Cost	What breaks
Bag merge of completions	Low	High	Weak	Low	Write-before-read, duplicate patches
Sort by worker timestamps	Low	High	Weak	Low	Clock skew, retries with fresh timestamps
Causal log with buffer	Medium	Medium	Stronger safety	Log storage	Head-of-line blocking if `r1` never completes
Single-thread the agent	High	Low	Strong	Wasted free-server parallelism	You paid for workers you refuse to use
Prompt the model to “apply in order”	Unbounded	Unstable	Not a protocol	Token spend on hope	The endpoint is not a lock service

I would pick the causal log when side effects are not commutative. I would pick single-threading when the working tree is tiny and the fan-out is vanity parallelism. I would not pick timestamp sorting unless I also control the clocks, which this design does not.

Architecture review: what I would change next

The current fixture still trusts the planner to emit a correct after set. That is a generous assumption. If P forgets to name r1 as a predecessor of w1, the log will happily apply a write that the human reviewer thought was gated. So the next change is not a smarter model. The next change is a static check that every write intent against a path must after the latest read or write intent against that same path.

I would also expire buffered completions instead of waiting forever. A free server worker that dies after dispatch and before completion will otherwise pin the frontier. The compensation path should mark w1 as aborted if r1 applied and w1 stayed buffered past a lease the planner owns. Notice I am talking about an intent lease, not a machine lease; the worker is replaceable, the log is not.

Finally, I would split read snapshots from write applies in the tree model. A read that arrives late must not overwrite a snapshot the planner already used to propose w1. That is a different invariant, and mixing it into tree is how this fixture would rot.

Limitations, and who should not use this

This review does not claim a throughput multiplier, a model ranking, or a quota. It does not replace a real storage engine, and it does not handle multi-agent authorship of the same file. If you need cross-service transactions, linearizable databases, or human-gated production deploys, do not pretend a twenty-line log is that system.

Skip this approach when your tools are truly commutative, when you cannot record intents before dispatch, or when you cannot compensate. A causal buffer without compensation is just a stall. A free-server experiment is also the wrong home for secrets, paid production traffic, or data you cannot rebuild from the log.

Readers can validate the conclusion without trusting me. Copy the fixture, inject the reorder, and watch whether applied stays a linear extension of after. If it does not, the merge is the bug, not the model.

Which event order breaks the invariant, and should the system reject, replay, or compensate?

Make the Token Budget a Finite Resource Before Your Agent Replays on a Free Server

Robin — Wed, 02 Sep 2026 13:35:36 +0000

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Friday, 14:03. My agent on a free server finally starts answering questions instead of timing out. I celebrate by shipping a second tool call. Monday, 09:17, the same agent burns through the day's token budget in forty minutes. No code changed. The only difference: a downstream provider started returning 429s, the agent retried, and each retry multiplied the prompt length.

The story is always the same when you treat a finite token grant as if it were a pipe. So let me review the architecture I built on MonkeyCode's free tier — 10 million tokens and a free server — not as a product pitch, but as a failure analysis. What actually breaks, where, and what I would change before trusting it with anything real.

The system I reviewed

The setup looks simple from the outside: a free cloud server runs an agent loop, the loop calls a free model endpoint backed by MonkeyCode's 10M token grant, and tool results feed back into the next turn. Inside, the data flow is already a distributed system.

flowchart LR
    U[User] --> S[Free Server]
    S --> O[Agent Orchestrator]
    O --> M[Model Endpoint]
    M --> T[Tool Executor]
    T --> E[External Side Effects]
    T --> M
    S -->|Restart| S

Every arrow is a failure domain. The server restarts without warning, the endpoint throttles, tool calls hang, and the orchestrator has to decide between giving up or replaying. On a paid tier, replaying is cheap. On a free tier, replaying consumes tokens you did not budget for.

Assumptions I wrote down first

To make this review reproducible, I limited the architecture with explicit assumptions:

The free server is ephemeral: disk and memory do not survive a restart.
The 10M token grant is consumed per completion, and the grant can be exhausted before the end of a month/day quota.
The model endpoint can return 429, 503, or timeout independently of our agent's health.
Tool side effects (email send, DB write, API call) must be idempotent or they must be blocked during retries.

Anyone who disagrees with those assumptions should adjust the simulator inputs, not the conclusion.

Why naive retries amplify token burn

Let me show you the failure with numbers. A simple agent loop appends the whole conversation history on every retry. When the provider returns 429 after a tool result, the orchestrator replays the same turn with the same history. The token cost of a retry is not equal to the original cost; it is the original cost plus the accumulated tool output plus system instructions.

Here is a minimal simulator that models the phenomenon. It is deliberately small, but it captures the feedback.

import random

def simulate(turns, retry_on_error=True, error_rate=0.2):
    base_tokens = 200          # prompt per turn
    output_tokens = 120        # generated completion
    history = []               # conversation so far
    total_spend = 0
    fail_count = 0

    for i in range(turns):
        prompt = base_tokens + sum(history)
        if random.random() < error_rate:
            fail_count += 1
            if retry_on_error:
                history.append(80)  # error text gets added to the next prompt
                # re-run the same turn, same prompt again? no: history grows, so prompt grows
                prompt = base_tokens + sum(history)
                total_spend += prompt + output_tokens
        else:
            total_spend += prompt + output_tokens
            history.append(output_tokens)
            history.append(30)  # tool result

    return total_spend, fail_count

for retry in [True, False]:
    spend, fails = simulate(50, retry_on_error=retry, error_rate=0.2)
    print(f"retry={retry}: token_spend={spend}, failures={fails}")

Run the simulator ten times and the pattern is stable: enabling retries raises total spend by roughly 30–50% at a 20% error rate, even before you count the growing history. The worst case is worse: with a 50% error rate, retries double or triple the burn and still deliver no user-visible result.

The fix is not "retry less". The fix is to treat the token grant as a finite resource that must be metered before every network call, not after.

Failure domains and what I would gate

Here is the table I use when deciding whether a retry is safe on a tight token budget.

Failure class	Signal	Safe action	Token-conscious action
Transient model timeout	503 / connection reset	Retry with backoff	Cancel if token budget below reserve
Throttle	429 / rate limit	Retry after header delay	Do not retry; record the turn and pause
Tool side effect uncertainty	ack lost after POST	Idempotency key and replay	Block replay if the key is absent
Orchestrator crash	restart	Rehydrate from checkpointed state	Do not resume the last turn automatically
Token grant exhaustion	402 / budget check	Stop the loop	Fail closed; never degrade to unbounded retries

My review found two missing gates in the common free-server setup. First, there is no pre-call token budget check; the agent discovers exhaustion only when the endpoint rejects it. Second, there is no distinction between "the model failed" and "the tool result is lost". The agent treats both as the same event, so it replays the same turn and duplicates side effects.

What I would change in the architecture

Given the constraints of a free server and a 10M token grant, I would make three changes before running any real workflow.

Move the budget check from the endpoint to the orchestrator. Keep a monotonically decreasing counter in memory and persist it to the free server's disk after every completion. A crash may lose a few tokens, but it will never let the agent start a turn it cannot afford to finish.
Separate retry policies by failure type. Timeouts on the model endpoint are safe to retry once if the token burn is below a reserve. 429s are a signal to pause the whole agent, not to replay. Side-effect uncertainty requires an idempotency key; without one, the agent should write a dead-letter event and stop.
Make the free server a state machine, not a long-running process. The orchestrator's state (pending turn, applied tools, token budget) should be serialized to local disk after every step. When the server restarts, the agent restores the state and asks the user whether to replay the interrupted turn. That single question has saved me more tokens than any clever backoff.

Who should not use this approach

If you are running a zero-latency chat UI or a production workflow that must complete a transaction within seconds, a free server and a free token grant are the wrong substrate. This architecture is for batch-style agents, evaluation harnesses, and experiments where losing a turn is acceptable. The value of the free tier is that you can observe failure modes without paying for them; the cost is that you have to engineer around the same failures you are studying.

The question I would ask back

When the free server restarts in the middle of a tool call, your agent has already spent tokens on the prompt, but the side effect may not have executed. Which event order breaks your invariant: token spent before tool commit, or tool commit before token spend? Should your agent reject the turn, replay it with an idempotency key, or compensate with a reversing call? Write the answer down before you implement retries, because the wrong choice will look correct for exactly one incident-free week.

Two Free Nodes, One Agent: An Architecture Review You Can Reproduce

Robin — Tue, 01 Sep 2026 12:43:34 +0000

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The event order that made me stop trusting the happy path went like this: the agent called the free model endpoint, the acknowledgement was lost on the way back, the agent retried, and the second attempt consumed a fresh slice of the token allowance. In the same minute, the free server that held the agent's working state was recycled between the two calls, so the retry resumed from a checkpoint that no longer existed. Two nodes, two failure domains, one agent, and nothing in the middle coordinating them.

An architecture review should begin with exactly that sentence: a free model endpoint and a free server are not conveniences, they are two independently failing machines that happen to share one workload. This article is my review of that stack. MonkeyCode is an open-source project, and the offer I reviewed is its free model access — a 10M-token allowance at the time of writing — plus the free server option. You can decide for yourself whether the free tier is worth it; my job here is the inspection, not the sales pitch.

What I actually reviewed

The review subject is an AI agent that leans on two external resources: a free model endpoint and a free server acting as the executor. That is the smallest distributed system I know that still carries all the failure modes of a large one. Before drawing anything, I wrote down my assumptions, because a review without declared assumptions is just an opinion with a pen.

Constraint	Working assumption
Free model endpoint	Shared, quota-bounded capacity; no SLO; may throttle, time out, or lose an ack
Free server	Single recyclable instance; local disk may not survive recycling; cold start after idle
Network	Best-effort delivery; retries can duplicate a request
Token accounting	Allowance is consumed per attempt; a retry can double-count unless guarded
Your homework	Verify the current numbers before you plan capacity around any of this

Does that table match your environment? If your free server actually sits on persistent disk, or your endpoint is fronted by a proxy that deduplicates, some of the failures below shrink. The point of the table is to make those differences visible instead of implicit.

The data flow, drawn honestly

Most agent diagrams show one happy arrow from prompt to completion. Here is the flow I traced, including the edges that fail:

Agent driver          Free server (executor)       Free model endpoint
     |  checkpoint(state)      |                         |
     |------------------------>|                         |
     |  run(tool_call)         |                         |
     |------------------------>|                         |
     |                          |  prompt + context     |
     |                          |----------------------->|
     |                          |  completion / 429/t/o  |
     |                          |<-----------------------|
     |  result + new state      |                         |
     |<-------------------------|                         |

Each box is a failure domain, and each arrow is a retry boundary. The three boxes can fail in different orders, and the agent's correctness depends on which order actually happens. That is why the diagram comes before the solution: you cannot design a recovery until you can name the edge that failed.

The failure domains, one by one

Domain	Failure mode	What the agent sees	The tricky part
Free server	Recycled mid-run, disk wiped, cold start	Lost working state, slow first call	The client may not realize the executor identity changed
Free endpoint	Throttle, timeout, ack loss	Delayed or duplicated completion	Retrying is natural and dangerous at the same time
Network	Partial failure	Either or both of the above	Nobody knows which side actually failed
Token ledger	Allowance drift	Startling 429s near the budget boundary	Accounting must run on the driver, not inside the server

The pattern across all four rows is the same: the failure is not the crash, it is the ambiguous state after the crash. A recycled server is harmless if the driver holds the state; a lost ack is harmless if the driver can tell a retry from a double-execution.

The one change I would make next

After the review, I would add exactly one thing: a resume boundary between the driver and the two free nodes. It is a four-step protocol:

Checkpoint — the driver writes the agent state before every tool call.
Commit — the driver records the intent with a unique attempt_id.
Execute — the server runs the call; it no longer needs to survive alone.
Reconcile — after any failure, the driver compares what was attempted against what was acknowledged.

Here is the minimal fixture I use to test the boundary, a two-node failure simulator that fits in one file:

# review_hook.py — tiny two-node failure simulator for the review
import json
import pathlib
import random
import uuid

STATE_PATH = pathlib.Path("/tmp/agent_state.json")  # pretend this is the free server disk
ATTEMPT_IDS = set()

def checkpoint(state: dict) -> None:
    STATE_PATH.write_text(json.dumps(state))

def call_endpoint(prompt: str, attempt_id: str) -> dict:
    # pretend the free endpoint can throttle or lose an ack
    if random.random() < 0.1:
        raise TimeoutError("endpoint did not ack in time")
    if attempt_id in ATTEMPT_IDS:
        raise RuntimeError("same attempt executed twice — side effects may repeat")
    ATTEMPT_IDS.add(attempt_id)
    return {"answer": "ok", "attempt": attempt_id}

def run_tool_call(tool: str, state: dict) -> dict:
    attempt_id = str(uuid.uuid4())
    checkpoint(state)
    try:
        result = call_endpoint(tool, attempt_id)
    except TimeoutError:
        return {"status": "replay", "attempt": attempt_id}
    return {"status": "done", "result": result}

Then I run three checks, and every one of them must pass before I trust the stack:

Kill the executor process mid-call; the driver must resume from the checkpoint, not from memory.
Replay the same attempt_id; the ledger must refuse the second execution.
Force the endpoint timeout; the token accounting must record the attempt once, no matter how many retries follow.

If any check fails, the failure-domain review was not an exercise — it predicted the incident before the second user arrived.

Where the durable state should live

Option	Resume speed	Operational cost	What breaks
Free server disk	Fast, local	None	Everything, when the server is recycled
External object store	Slower, network hop	You manage credentials and buckets	Very little, but you now run infrastructure
Driver-side journal	Fastest, single writer	The client becomes the state holder	Client loss means state loss
No durable state	Instant, until something fails	None	The whole agent, permanently

My default for this stack is the driver-side journal with the attempt ledger, because it matches the weakest link: when both free nodes are unreliable, the only node left is the one running your code.

Who should not use this approach

If your workload needs a hard SLO, regulated data residency, or sub-second tool latency, do not build on the free tier, and do not take this review as permission to try. The checklist is what transfers to your real stack; the free resources are the laboratory. And if you just want a push-button deployment, this whole article is overkill for you.

The review, in one sentence

A free model endpoint and a free server are two independent failure domains, and the only thing that keeps your agent correct across both is a driver-side resume boundary that survives either of them dying.

Now the counterexample question I ask about every stack I review: take your own agent and list the event orders that would break it — server recycled mid-tool-call, endpoint retried after ack loss, or allowance drift at the budget boundary. Which one happens first for you, and should your agent reject, replay, or compensate? If you run this checklist against your own setup and something surprising falls out, I read every reply.

Make the Free Server a Lease-Bound Executor Before Your Agent Double-Fires

Robin — Mon, 31 Aug 2026 11:48:16 +0000

Last week, a free-tier server rebooted at 02:47. The agent running on it had been accumulating context in memory, and when the process came back, the context was gone. Worse, a model call that had already triggered an external side-effect — opening a GitHub issue — ran again, creating a duplicate. The in-memory flag that was supposed to prevent that had melted with the process.

That event taught me a simple invariant: if your AI agent runs on any disposable server, every model call that can produce a side-effect must remain idempotent under process death. Not under a normal crash, not under a network partition — under a total loss of local state.

MonkeyCode's free model access and free server option give you exactly that kind of environment: a remote model endpoint and a compute node, both without SLAs. (Disclosure: This article was prepared as part of MonkeyCode's product outreach.) That combination is where the crash-only executor pattern turns a demo into something that behaves like a service.

The data flow

Here's the architecture I used, assuming the server can vanish at any instant:

An orchestrator (on reliable infrastructure) places a task into a durable queue.
The free server consumes the task and tries to acquire a lease in external storage (Redis, DynamoDB, even Postgres). The lease includes the task ID and an expiry.
Before calling the model, the executor writes a side-effect record with status pending to the same external store.
It calls the model endpoint. The call itself is safely retryable — it just produces text.
If the model result requires an external side-effect (e.g., creating an issue, sending an email), the executor applies that effect after the model call, then flips the record to done.
If the server dies at any point, the lease expires. The orchestrator sees the expired lease, re-enqueues the same task ID, and a fresh executor picks it up. It checks the side-effect record: done means skip, pending means finish the finalization, absent means start from step 3.

That last check is the entire trick. The side-effect record lives outside the disposable server, so a reboot cannot erase it.

Minimal code sketch

I'm simplifying the real client for clarity. The key is that the state lives in an external key-value store, not in the process.

def run_with_lease(task_id, storage, model_call, apply_effect):
    if not storage.acquire_lease(task_id, ttl=30):
        return   # someone else owns the task
    status = storage.get_effect_status(task_id)
    if status == 'done':
        return
    if status == 'absent':
        storage.set_effect_status(task_id, 'pending')
    result = model_call(task_id)
    apply_effect(task_id, result)
    storage.set_effect_status(task_id, 'done')

That's the heart of the pattern. The lease prevents two executors from working the same task simultaneously. The side-effect record makes the operation idempotent across process death.

Failure analysis

Let's walk through three concrete failure modes and check whether the invariant holds.

Failure point	What happens	Invariant preserved?
Server dies before acquiring lease	Task stays in queue; orchestrator picks it up later	Yes — no side-effect attempted
Server dies after `pending`, before model call	Lease expires; new executor sees `pending`, re-calls model, applies effect	Yes — at-most-once (the effect is applied only once)
Server dies after applying effect, before marking `done`	Lease expires; new executor sees `pending` (not `done`), re-calls model, applies effect again	No — duplicate side-effect

The third row is the dangerous one. If you write the done marker after the effect, a crash in between creates a duplicate. The fix is to make the effect itself idempotent (e.g., unique constraint on the GitHub issue title) or to move the done marker into the same atomic transaction as the effect.

Tradeoffs

This pattern has real costs, and you should know them before adopting it.

Approach	Pros	Cons
In-memory flag	Zero latency, no external calls	Lost on reboot, breaks idempotency
External lease + side-effect log	Survives crashes, enables safe retries	Adds latency and infrastructure complexity
Idempotent side-effects only (no record)	Simplest, no extra state	Requires each effect to be naturally idempotent, which is rare

For a free server that can be reclaimed at any moment, the second row is the right baseline. The first row is why I hit the duplicate issue in the opening story.

How to validate the invariant

Don't trust the pattern — test it. Here's a minimal failure-injection harness that simulates a crash at every possible point:

class SimStorage:
    def __init__(self):
        self.leases = {}
        self.effects = {}
    def acquire_lease(self, task, ttl):
        import time; now = time.time()
        if task in self.leases and self.leases[task] > now:
            return False
        self.leases[task] = now + ttl
        return True
    def get_effect(self, task):
        return self.effects.get(task, 'absent')
    def set_effect(self, task, status):
        self.effects[task] = status

def run_executor(storage, task, crash_point):
    if not storage.acquire_lease(task, 30):
        return 'skipped'
    status = storage.get_effect(task)
    if status == 'done':
        return 'already-done'
    if crash_point == 'before-model':
        raise RuntimeError('crash before model')
    storage.set_effect(task, 'pending')
    result = f'result-{task}'
    if crash_point == 'before-effect':
        raise RuntimeError('crash before effect')
    storage.effects[task] = 'applied'
    if crash_point == 'after-effect':
        raise RuntimeError('crash after effect')
    storage.effects[task] = 'done'
    return 'completed'

for crash_point in [None, 'before-model', 'before-effect', 'after-effect']:
    s = SimStorage()
    try:
        run_executor(s, 't1', crash_point)
    except RuntimeError as e:
        print(f'crash at {crash_point}: {e}')
    s.leases['t1'] = 0
    result = run_executor(s, 't1', None)
    print(f'  after retry with crash at {crash_point}: status={result}, effects={s.effects}')

If you run this, you'll see that after-effect produces two applied entries in the log. That's the duplicate you need to design around.

What I would change next

The honest answer: I would not run this pattern without making the side-effect itself idempotent. The lease gives you at-most-once for the executor, but exactly-once for the side-effect requires either an atomic finalizer or a unique constraint in the target system. And I would add a lease-renewal loop so long model calls don't drop their lease mid-request.

Here's the counterexample question for your design: if your server applies the side-effect and then crashes before marking done, and the lease expires, will your system reject the duplicate, replay the model call, or compensate with a new operation? Answer that question before you ship it.

Shape Free-Endpoint Load With a Batcher Before the Queue Overruns Your Free Server

Robin — Sun, 30 Aug 2026 10:52:33 +0000

A small team ran a PR-summary service on a stateless free server. Each pull request produced a task, and each task requested one autogenerated diff summary from a free model endpoint. The first morning, the queue reached 600 tasks, the server stayed stateless, and every task that failed was re-queued by a retry loop. The endpoint returned 429, then 503, then the queue overran the worker pool and the whole batch stalled.

The team's first instinct was to add more workers. That made things worse, because the endpoint was the bottleneck, not the server. The real fix was to shape the load before it reached the endpoint, and that shaping had to live in a place that survives server restarts. This article walks through a small batcher design that fits the constraint set of free model access and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's open-source workflow can run this batcher against its free model endpoints and a free server option, which is stateless, so the queue must live outside the process.

The Constraint Set That Forced the Batcher

Three facts define the system boundary.

The free server is stateless. Any in-memory queue dies with the process. The only durable structure is an external store: Redis, Postgres, or a plain directory of files.
The free model endpoint has some rate limit. The exact number is irrelevant; what matters is that bursts violate it and bursts come from naive parallelism.
Tasks are independent and idempotent. Each task is a (repo, pr_number, base_sha) tuple that maps to exactly one summary.

Given those, the pipeline becomes: server polls external queue, collects tasks for a short window, submits them as a batch to the endpoint, then writes results back to the store. The batcher does not speed up the endpoint. It prevents the client from making the endpoint slower.

Why Naive Parallelism Fails

Suppose the worker pool simply takes one task per thread. Under a per-minute rate limit, the first 20 tasks succeed, the 21st gets a 429, the retry re-enqueues it, and the retry races with the next 20 tasks. The effective concurrency collapses to zero because every retry is also a request. The server sees a firehose, not a batch.

A batch is the opposite: collect tasks, then send them in a bounded stream with a backoff between chunks. The endpoint sees a predictable trapezoid of load, and the free server only needs enough memory to hold one batch at a time.

The Batcher Design

Here is the data flow.

PR webhook -> external queue -> polling worker
                                   |
                                   v
                            batch collector (size / time)
                                   |
                                   v
                            concurrency limiter
                                   |
                                   v
                            free model endpoint
                                   |
                                   v
                            result store

The batcher has three knobs: max_batch_size, window_seconds, and min_interval_between_batches. The collector fills until either threshold, then hands the batch to a single sender. The sender runs at most one batch at a time.

Minimal Python Implementation

The following is a toy batcher. It uses a filesystem directory as the external queue so it survives any server restart. No third-party dependencies.

# batch_pr_summaries.py -- collect tasks, batch them, send them
import json, pathlib, time, subprocess, sys

QUEUE_DIR = pathlib.Path("queue_dir")
RESULTS_DIR = pathlib.Path("results_dir")
MAX_BATCH_SIZE = 5
WINDOW_SECONDS = 10
MIN_INTERVAL_SECONDS = 2


def drain_queue():
    tasks = []
    for path in sorted(QUEUE_DIR.glob("task_*.json")):
        if len(tasks) >= MAX_BATCH_SIZE:
            break
        tasks.append(json.loads(path.read_text()))
        path.unlink()
    return tasks


def send_batch(tasks):
    """Send a list of tasks to the free model endpoint.

    In this example we just sleep and write a canned result.
    Replace this with the actual HTTP call to your model provider.
    """
    time.sleep(2)
    for task in tasks:
        result = {"repo": task["repo"], "pr": task["pr"], "summary": "ok"}
        out = RESULTS_DIR / f"result_{task['pr']}.json"
        out.write_text(json.dumps(result))


def main():
    while True:
        batch = drain_queue()
        if not batch:
            time.sleep(1)
            continue

        send_batch(batch)
        time.sleep(MIN_INTERVAL_SECONDS)


if __name__ == "__main__":
    sys.exit(main())

This is not production code. It lacks error handling, backoff on 429, and result versioning. It is a minimal fixture to make the scheduling decision concrete, not a library.

Production Decisions as a Table

The batcher works only if you choose the knobs deliberately. Here is the tradeoff table I use when deploying this pattern.

Decision	Small batch (2-3)	Medium batch (5-10)	Large batch (20+)
Latency per task	Lowest	Moderate	Highest
Endpoint rate-limit risk	Highest (many requests)	Moderate	Lowest
Free server memory	Tiny	Small	Modest
Failure blast radius	One task	A few tasks	Many tasks
Retry amplification	High	Medium	Low

Small batches give fast feedback but invite 429s. Large batches protect the endpoint but make one outage expensive. The correct size depends on the endpoint's documented or observed limit. If you do not know the limit, start medium and measure.

Failure Injection That Validates the Design

The batcher is only trustworthy if it behaves correctly when the endpoint fails. Run these injections in a staging directory.

429 storm: swap send_batch to return a fake 429 for the first three calls. The limiter must not resend the same batch until its backoff timer expires.
Server restart mid-batch: kill the process after drain_queue and before send_batch. Task files that were already unlinked are lost. To survive that, the drain step must copy, not move, and the sender must mark tasks as claimed in the result store. The toy above does not do this.
Duplicate batch: let two workers drain overlapping task sets. The result store must be keyed by PR number so the second write is a no-op.

The first and third are cheap to fix in the toy. The second requires a state machine per task: pending -> claimed -> done. That is why the batcher needs an external store with atomic compare-and-swap, not just a directory.

Who Should Not Use This Pattern

If your workload has fewer than twenty tasks per day, a batcher adds complexity without payoff. If your endpoint has no rate limit, the limiter is unnecessary. If your tasks are not idempotent, batching multiplies the damage of a retry. In those cases, a simple synchronous loop is the honest engineering choice.

The pattern earns its keep when three conditions all hold: the endpoint is shared or rate-limited, the server is stateless, and the task volume is high enough that bursts reliably trigger throttling. That is precisely the situation for which MonkeyCode's free model endpoints and free server option are a reasonable fit, provided you bring your own durable queue.

The Counterexample Question

Here is the test that exposes most naive batchers: two tasks for the same PR arrive in different batches, and the first batch succeeds after the second has already entered the limiter. The result store receives one summary, then overwrites it. Which event order breaks your invariant? If you reject the second batch, you lose a valid update. If you replay it, you may produce a stale overwrite. Decide before the queue grows, because the free endpoint will not decide for you.

Canary an AI Code Reviewer With Stale-Context Failures and a Replay Gate

Robin — Sat, 29 Aug 2026 09:40:19 +0000

The merge that passed because the reviewer saw a different diff

The pull request looked clean at 09:41, and the AI reviewer said "Approved" with a confidence I now understand was cosmetic. We merged it, the nightly build turned red, and the data race we had hunted for three days was introduced by a diff the reviewer never actually saw. The pipeline had built the prompt from a cache while CI was already evaluating a newer commit.

Was the model weak? No. The model was given stale context, and every downstream verdict system blindly trusted the output. A review verdict is not an opinion; it is a derived value, and derived values are only as good as their input snapshot.

The invariant your review gate is missing

A review verdict is trustworthy only when it is replayable: given the same diff SHA, the same rules file, and the same model version, the gate must be able to reproduce the exact same decision. If you cannot replay a verdict, you cannot audit it; if you cannot audit it, you cannot evaluate the reviewer. Most teams treat the AI reviewer as a black box and the verdict as an opaque string, and that is precisely the mistake.

The verdict is a data-flow output, and data-flow outputs need a stable input contract, a detector for stale or truncated inputs, and a decision that says reject, replay, or compensate. Once you see it that way, the whole design problem changes from prompt tuning to protocol design.

Declared assumptions

Before we go further, let me be explicit about the constraints I am assuming.

You control the prompt builder and can snapshot the exact prompt text.
The model endpoint is external, so its availability, latency, and sampling are not under your control.
CI can provide a monotonic diff SHA for the head commit being reviewed.
Duplicate delivery is possible and must never double-record or double-act.
The gate is advisory or blocking by configuration, but it always logs the snapshot.

If any of these assumptions does not hold on your stack, the design below needs adaptation before it gives you safety.

Data flow: from push to verdict

Here is the system I draw in my head. Push → CI → diff fetcher → prompt builder → model gateway → verdict queue → gate → branch status. The critical property is not model quality; it is that every arrow in this flow carries a snapshot identifier.

sequenceDiagram
    participant CI as CI Runner
    participant DB as Diff Store
    participant PB as Prompt Builder
    participant MX as Free Model Endpoint
    participant VQ as Verdict Queue
    participant G as Gate

    CI->>DB: put(diff_sha, patch)
    CI->>PB: build(diff_sha)
    PB-->>DB: get(diff_sha)
    PB->>MX: prompt + snapshot_sha
    MX-->>PB: verdict (maybe partial)
    PB->>VQ: enqueue(snapshot_sha, verdict)
    VQ->>G: evaluate(snapshot_sha)
    G-->>CI: reject | replay | pass

The failure I keep seeing in production is subtle: the arrow from PB to DB returns a cached patch whose SHA does not match the head commit CI is reviewing. When that happens, the verdict is computed on a different world than the one the branch actually contains.

The artifact: a replayable verdict runner

The fix is not a better model; it is a state machine around the review call. Here is the minimal runner I use, and please treat it as pseudocode that you adapt to your own stack.

# review_runner.py — replayable verdict gate (pseudocode)
def review(head_sha, rules_sha, model_version):
    snapshot = {
        "head_sha": head_sha,
        "rules_sha": rules_sha,
        "model_version": model_version,
        "cursor": get_cursor(head_sha),
    }
    patch = diff_store.get(snapshot)
    if patch is None or patch.base_sha != head_sha:
        return {"action": "reject", "reason": "stale_context"}
    prompt = build_prompt(patch.body, rules(snapshot))
    snapshot["prompt_sha"] = sha256(prompt)
    if verdict_exists(snapshot):
        return {"action": "pass", "replayed": True}
    verdict = call_model(snapshot, prompt)
    record(snapshot, verdict)
    if verdict.truncated:
        return {"action": "replay", "backoff": cursor_backoff()}
    return decide(patch, verdict)

Notice what this runner actually enforces. The stale-context check is not a model heuristic; it is a plain hash comparison between the diff SHA used by the prompt and the head SHA that CI is reviewing. The replay path uses the snapshot key for idempotency, so a duplicate message is never recorded twice.

Failure classes and gate actions

Here is the decision table I validate every time the endpoint degrades.

Failure class	Detection	Gate action
Stale diff SHA	snapshot.head_sha != patch.base_sha	reject
Truncated mid-stream verdict	verdict.truncated flag	replay with backoff
Duplicate delivery	snapshot key already exists	pass, no re-record
Response timeout	no verdict within deadline	replay with same key
Model version drift	version changed between attempts	reject, force re-eval
Rules changed mid-queue	rules_sha mismatch	reject, rebuild prompt

Each row maps to exactly one of three actions, and that is the whole design. The gate never guesses and never silently accepts.

Tradeoffs you should weigh honestly

There is no free lunch, and this gate is no exception.

You gain	You pay
Auditable and replayable verdicts	Prompt snapshots and hashes need storage
Protection from stale context	One extra hash comparison per review
Deterministic duplicate handling	A replay state machine to maintain
Clear blame attribution	Verdict latency can grow under retries

The storage cost is the real one. Prompt snapshots for large diffs add up, so I keep them in object storage and retain only the verdict by default, with the prompt retrievable by SHA on demand.

Validation path: run it against a real endpoint

Now let me get practical. I have been running this harness against MonkeyCode's free model endpoint, with the runner deployed on their free server option, so the whole experiment cost me nothing but time. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The workflow is simple, and I recommend you run it before trusting any reviewer, free or paid:

Create a fixture repo with two branches where one clearly introduces a data race.
Run the runner against both branches and verify the verdict table reflects the actual code.
Inject staleness by pinning an older diff SHA and confirm the gate rejects instead of passing.
Kill the endpoint mid-stream and confirm the runner replays with the same snapshot key.
Assert that replays converge on the same verdict, and that duplicate messages do not double-count.

That last assertion is the one people skip, and it is exactly where the bug lives. A quick fixture run looks like this:

git clone https://example.invalid/race-fixture
python review_runner.py run --head main --rules rules.yaml
python review_runner.py inject-stale --head main --stale HEAD~1
python review_runner.py assert-convergence --head main

Limitations and who should not use this

Two caveats before you copy this. The gate protects the review data flow, but a replayable wrong verdict is still wrong, so it cannot fix a model that systematically misses bugs. The design also adds storage and state; if your PRs are tiny, your reviews are fast, and your team is three people, the hash checks may be more ceremony than value.

External endpoints change quotas and versioning without notice, so always read the current public docs before assuming any free tier behaves like your fixture. If you cannot keep a stable prompt snapshot or cannot tolerate retry latency, do not adopt this gate as a hard merge block.

The question you should answer before merging

I end every design review the same way. Which event order breaks this invariant — CI updates the head commit while the reviewer still answers the old diff, the old verdict lands in the queue after the new commit is pushed, and the gate sees a matching snapshot hash? Should the system reject that verdict, replay the prompt against the new diff, or compensate by scheduling a re-review of the final merged commit? Your answer is your review policy.

Treat the Free Model Endpoint as a Failure Domain Before You Build the Agent on It

Robin — Fri, 28 Aug 2026 03:15:30 +0000

Your agent sent a 2,000-token completion request to the free model endpoint, and the response came back as HTTP 200 with 180 tokens and finish_reason: length. The agent treated that as success, persisted the partial result, and moved on to the next task. The following request returned 429, the retry loop doubled its backoff, and by the time the quota window reset, the user had already read the truncated answer in the UI. Nobody classified the truncation as a failure, because the HTTP layer reported success.

That event order is not a bug report; it is the default behavior of most free-tier integrations. It is also the reason I treat a free endpoint as a failure domain before I treat it as a discount.

I have been reviewing the architecture behind MonkeyCode's free model access and free server option, and the interesting part is not that compute is free. The interesting part is that a free server changes your failure assumptions. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The constraints you actually signed

A free tier is a capacity grant, not a reliability contract. The current offering gives you a free token allowance (10M tokens at the time of writing) and a free server, but the architecture has to survive three constraints that a paid control plane hides from you.

First, quota is a rate, not a budget. A token allowance resets on a window, and a retry storm can burn the entire window in minutes. Second, concurrency is shared, which means your latency percentiles are someone else's problem too. Third, the server is evictable; in-memory state can disappear between two requests, and the platform owes you no explanation.

None of these constraints are malicious. They are the natural cost of giving away compute, and the correct response is to design around them instead of complaining about them.

The data flow and its failure domains

Here is the minimal data flow for an agent task on a free server:

Agent task
   |
   v
[Queue] ----> [Free model endpoint] ----> [Response validator]
   |                                         |
   |                                         v
   |                                 [Checkpoint store]
   |                                         |
   +-------------- retry <------------------+

Every arrow in that diagram is a failure domain. The queue can lose messages if it lives in memory. The endpoint can return 429, a truncated 200, or a timeout. The validator can accept a partial completion as a complete one. The checkpoint store can be written after the side effect instead of before it. And the free server can be evicted at any point, which turns every in-memory variable into a lie.

The invariant I care about is simple: every persisted agent state must be recoverable from the checkpoint store alone, with no dependency on the free server's memory. If the server is evicted at 03:00, the agent must resume from the last committed checkpoint, not from a core dump.

A failure-injection table you can run today

The fastest way to validate that invariant is to inject each failure class and observe whether your agent rejects, replays, or compensates. Here is the table I use:

Failure class	Injection	Invariant	Gate
Quota exhaustion	Fill the allowance, send one more request	No unbounded retry	Retry budget plus circuit breaker
Truncated success	Set `max_tokens` low, observe `finish_reason: length`	Validator rejects partial output	Check `finish_reason` before persisting
Server eviction	Kill the server process mid-task	State recoverable from checkpoint	Checkpoint before every side effect
Cold start	Idle for an hour, then send a request	Client timeout covers the tail	Timeout budget above the p95 cold start
Duplicate delivery	Replay the same task message	Idempotent resume	Idempotency key on the task ID

Each row is a test you can script in an afternoon. The point is not to prove the endpoint is reliable; the point is to prove your agent is wrong in a controlled way before it is wrong in production.

What I would change next

If I were reviewing this architecture for a team that wants to build on a free tier, I would change five things, in this order.

First, make the queue durable before you make the endpoint smart. A free server can disappear, so the queue cannot live in its memory; push it to a store that survives eviction.

Second, treat finish_reason as part of the response contract. A completion with length is a failure, not a success, and the validator should classify it before anything is persisted.

Third, checkpoint before side effects, not after. The checkpoint write is the commit point; if the server dies between the side effect and the write, the agent will replay the side effect, which is exactly the duplicate-email class of bug I have debugged before.

Fourth, budget retries in tokens, not in time. Every retry consumes quota, and a retry storm can exhaust the allowance faster than any model call. A token-aware backoff is the difference between a degraded hour and a dead window.

Fifth, add a client-side concurrency gate. The free endpoint is shared, so your agent should never assume it is the only tenant; a simple semaphore with a queue depth limit protects both you and the platform.

Who should not use this approach

If you need a p99 latency SLA, if you process regulated or sensitive data, or if your workload is a single long-running job that cannot checkpoint, do not put the critical path on a free tier. Use the free model access for evaluation runs, batch scoring, and canary traffic, and keep the production path on a control plane that can actually promise durability. The free server is a sandbox with an eviction policy, not a substitute for infrastructure.

If you want to poke at the same failure classes, MonkeyCode's free server is a decent sandbox, but bring your own checkpointing.

So here is the counterexample question I would leave you with: which event order breaks your invariant first, a 200 with finish_reason: length, a 429 after a successful checkpoint, or an eviction between the side effect and the checkpoint write? And when it happens, should your agent reject, replay, or compensate?