Make Preemption a Fenced Lease Before an Agent Resumes on Ephemeral Compute

I keep seeing the same violating event order when an agent worker dies between a tool side effect and the checkpoint. The planner writes a snapshot after the previous step, then calls a payment or mail tool, then vanishes before the snapshot advances. A replacement worker boots on another box, reads the stale snapshot, and treats the missing write as proof the tool never ran. Have you watched a retry look identical to a first attempt because nobody incremented a lease?

The common implementation preserves resume-from-last-checkpoint and quietly drops the fencing token, which is not an invariant worth shipping. Two workers sharing a run_id must never apply competing side effects under the same lease epoch, even after the first worker dies. If a late ack from the preempted epoch still arrives, the system should reject or compensate, never apply it as a fresh success. Why do we keep modeling death as a clean pause instead of a concurrent writer?

There is a second and nastier counterexample that I now treat as the real production-shaped bug. Teams mint a new idempotency key when the lease bumps, so the bank sees a brand new charge instead of a replay. The stable key must stay (run_id, step_id) across epochs, while the epoch only fences checkpoint rights and zombie calls. Would you stake a refund pipeline on a key that changes every time a box disappears?

Declared assumptions

I am reviewing a discrete architecture, not a production postmortem with invented latency numbers or customer counts. The fixture below is labeled example code you can run locally, and it is not a cluster measurement. These assumptions bound the argument so we can actually falsify it.

1. One logical agent run has a stable run_id and a monotonically increasing lease_epoch.
2. Tool side effects become unique only when the caller supplies a stable idempotency key.
3. Checkpoints are durable; worker memory and in-flight HTTP bodies are not.
4. Preemption can land after the tool has committed and before the checkpoint write.
5. A late response from the old worker can arrive after the new lease is active.
6. The acceptance denominator is injected preemption interleavings, not wall-clock hours.

If your tools already fence on a server-side key that stays stable across leases, the local reject path is still useful for zombies. Do you actually pass that stable key, or do you pass a hash of the epoch by accident?

Constraints and data flow

Ephemeral compute changes the failure domain more than it changes the happy path through the planner. A free or preemptible worker is a legal place to run inference and tool routing, until the box disappears mid-step. The model hop is another unbounded queue, because tokens arrive late and a retry looks like honest progress. I want the data flow to make both hops explicit, because hidden hops become double applies the moment you resume.

client intent -> durable intent log -> lease grant -> worker
worker -> model hop (optional, queued)
worker -> tool gateway (stable key + epoch header) -> side effect store
worker -> checkpoint store (epoch + seq)
preemption -> lease expiry -> new grant with epoch+1
late ack -> gateway fence -> reject, replay same key, or compensate

The intent log is the source of truth, and the worker is a leased cache sitting in front of it. Does your current diagram still draw the worker as the center of the world?

Sequence that violates naive resume

sequenceDiagram
    participant L as IntentLog
    participant W1 as Worker epoch 1
    participant T as ToolGateway
    participant W2 as Worker epoch 2
    L->>W1: grant lease epoch=1
    W1->>T: charge(key=run/step, epoch=1)
    T-->>W1: committed
    Note over W1: preempted before checkpoint
    L->>W2: grant lease epoch=2
    W2->>T: charge(key=run/step/epoch2)  # minted new key
    T-->>W2: committed again
    W1->>L: late checkpoint epoch=1 overwrites seq

That last late checkpoint is the bug people rename eventual consistency when they really mean they lost the fence. The invariant I want is sharper than a slogan about exactly-once delivery across a chatty planner. At most one net committed side effect may exist per (run_id, step_id) unless a compensation record is explicit in the log. Naive resume breaks it when preemption lands between commit and checkpoint, or when resume mints a new key. Can you name a tool in your agent that is actually safe to double-fire?

Failure domains

I split the system into four domains because they fail independently and hide inside one process. Mixing them into a single agent runtime is how the double charge sneaks into a passing demo.

1. Intent log and lease manager. Crash-safe, fenced, and not colocated with the worker.
2. Ephemeral worker. It may die, freeze, or dual-boot after a short network partition.
3. Tool gateway. It must see both the stable key and the epoch, or it cannot reject zombies.
4. Model hop. It is bounded by your queue and budget, not by the worker's optimism.

A free server option lives in domain two, which is exactly why I care about leases here. Free model access lives in domain four, which is a queue with a timeout, not a local function call. What happens when domain two dies while domain three has already committed, and domain four still has a streaming response in flight?

Architecture review: five steps I would actually run

I review this class of design with a short protocol, not with a reliability slogan on a slide. Each step produces an artifact you can keep next to the planner.

Step 1. Write the state machine before the planner loop

I refuse to read a planner that mutates memory and then explains consistency in a comment. Encode grant, apply, checkpoint, preempt, and late-ack as transitions with explicit rejects. If a transition cannot name the epoch it requires, it does not belong in the worker. Would your current loop still compile if checkpoint were not allowed on a stale epoch?

Step 2. Split the stable key from the fence

Put (run_id, step_id) on the tool idempotency key and keep it unchanged after preemption. Put lease_epoch in a header the gateway checks before it forwards, so a zombie cannot checkpoint or start a different step. If you fold the epoch into the key, resume is a new business operation, which is how you bill twice. Are you sure your SDK is not concatenating those fields for you?

Step 3. Inject preemption between commit and checkpoint

A unit test that kills the worker before the tool returns does not cover this hole. You need the interleaving where the side effect store has committed and the snapshot has not, plus a late ack after the new grant. I run that as a discrete-event fixture so the event order is the input, not an accident of sleep(). Does your suite have a name for that interleaving, or only a flaky integration test?

Step 4. Score violations with an explicit denominator

I do not accept "it felt fine on the free box" as a reliability argument, because feeling is not a denominator. Count double-applies and stale checkpoints per injected preemption interleaving, then publish the ratio. The acceptance rule I use for this fixture is zero double-applies and zero accepted stale checkpoints across two hundred injected interleavings. If you cannot state the denominator, you are not measuring the invariant.

Step 5. Decide reject, replay, or compensate per step type

Charges, mails, and memory writes do not share a recovery verb, so the planner must not hide them under a generic resume(). Replay the same stable key when the tool is idempotent at the provider. Compensate when the provider committed under a key you must not reuse, then record that compensation as its own step. Reject when the epoch is stale and the step is already decided. Which of those three did your last incident actually need?

Minimal simulator

This is labeled example code for the interleaving, not a library I pretend already runs in production. Save it as fence_lease_sim.py and treat the assertions as the architecture gate.

"""Fenced-lease simulator for preempted agent workers.

Labeled example: not production code. Run: python fence_lease_sim.py
"""
from __future__ import annotations

from dataclasses import dataclass, field
from enum import Enum, auto
from typing import List, Optional, Tuple


class Outcome(Enum):
    APPLIED = auto()
    REJECTED_STALE_LEASE = auto()
    REJECTED_DUPLICATE_STEP = auto()
    CHECKPOINT_IGNORED = auto()


@dataclass
class SideEffect:
    run_id: str
    step_id: str
    lease_epoch: int
    amount: int


@dataclass
class System:
    run_id: str = "run-1"
    lease_epoch: int = 0
    checkpoint_seq: int = 0
    committed: List[SideEffect] = field(default_factory=list)
    live_worker_epoch: Optional[int] = None

    def grant_lease(self) -> int:
        self.lease_epoch += 1
        self.live_worker_epoch = self.lease_epoch
        return self.lease_epoch

    def preempt(self, epoch: int) -> None:
        if self.live_worker_epoch == epoch:
            self.live_worker_epoch = None

    def apply_tool(self, epoch: int, step_id: str, amount: int) -> Outcome:
        if epoch != self.lease_epoch or self.live_worker_epoch != epoch:
            return Outcome.REJECTED_STALE_LEASE
        if any(s.step_id == step_id for s in self.committed):
            return Outcome.REJECTED_DUPLICATE_STEP
        self.committed.append(SideEffect(self.run_id, step_id, epoch, amount))
        return Outcome.APPLIED

    def checkpoint(self, epoch: int, seq: int) -> Outcome:
        if epoch != self.lease_epoch:
            return Outcome.CHECKPOINT_IGNORED
        if seq <= self.checkpoint_seq:
            return Outcome.CHECKPOINT_IGNORED
        self.checkpoint_seq = seq
        return Outcome.APPLIED

    def committed_net(self, step_id: str) -> int:
        return sum(s.amount for s in self.committed if s.step_id == step_id)


def naive_resume_double_applies() -> int:
    """Counterexample: resume without a gateway fence or a stable key."""
    committed_rows = 0
    committed_rows += 1  # epoch 1 applied, then died before checkpoint
    committed_rows += 1  # replacement worker treats missing snapshot as never-ran
    return committed_rows


def epoch_inside_key_double_charges() -> int:
    """Anti-pattern: minting a new idempotency key when the lease bumps."""
    seen = set()
    charges = 0
    for epoch in (1, 2):
        key = ("run-1", "charge", epoch)  # epoch leaked into identity
        if key not in seen:
            seen.add(key)
            charges += 1
    return charges


def fenced_preempt_then_late_ack() -> System:
    s = System()
    e1 = s.grant_lease()
    assert s.apply_tool(e1, "charge", 50) == Outcome.APPLIED
    s.preempt(e1)
    e2 = s.grant_lease()
    assert s.apply_tool(e1, "charge", 50) == Outcome.REJECTED_STALE_LEASE
    assert s.apply_tool(e2, "charge", 50) == Outcome.REJECTED_DUPLICATE_STEP
    assert s.checkpoint(e1, seq=2) == Outcome.CHECKPOINT_IGNORED
    assert s.checkpoint(e2, seq=2) == Outcome.APPLIED
    return s


def property_at_most_one_net_commit(iterations: int = 200) -> None:
    """Acceptance: net charge stays 50. Denominator: injected preemptions."""
    violations = 0
    for i in range(iterations):
        s = System(run_id=f"run-{i}")
        e1 = s.grant_lease()
        s.apply_tool(e1, "charge", 50)
        s.preempt(e1)
        e2 = s.grant_lease()
        s.apply_tool(e1, "charge", 50)  # zombie
        s.apply_tool(e2, "charge", 50)  # resume, same step_id
        if s.committed_net("charge") != 50:
            violations += 1
    assert violations == 0, f"double-apply violations={violations}/{iterations}"


if __name__ == "__main__":
    print("naive committed rows (should be 2):", naive_resume_double_applies())
    print("epoch-in-key charges (should be 2):", epoch_inside_key_double_charges())
    sys = fenced_preempt_then_late_ack()
    print("fenced net charge (should be 50):", sys.committed_net("charge"))
    property_at_most_one_net_commit()
    print("property held over 200 injected preemptions")

Run it with the boring command, then read the three printed numbers as the review, not as a benchmark of a model.

python fence_lease_sim.py

You should see the naive path print two committed rows and the fenced path print a net of fifty. The property loop is the gate: two hundred injected preemptions, zero double-applies. If that assertion fails, the architecture is wrong before any model quality discussion starts. Why would you debug the prompt when the lease protocol is already leaking money?

Tradeoff table

Choice	What you gain	What you pay	When it fails
Stable key, epoch in header	Resume is a replay	Gateway must be epoch-aware	Provider ignores headers
Epoch inside idempotency key	Easy to stamp uniqueness	Preemption mints a new charge	Every worker death
Checkpoint-before-call	No orphan commits	Lost work on death, extra latency	Long model hops
Call-before-checkpoint	Progress under success	Orphan commits on preemption	Free or preemptible workers
Compensate-always-on-resume	Clear money story	Extra provider load, new failure domain	Compensation itself preempted

I would not pick checkpoint-before-call for a long model hop, because you then stall the lease while tokens dribble in. I also would not pick epoch-in-key, because it converts a fence into a new business intent. The durable combination is a stable key plus an epoch fence plus an explicit compensate step when the provider cannot replay. Which cell matches the system you are actually shipping this week?

A practical workflow on ephemeral compute

I want the worker to vanish on purpose, because an in-process mock never exercises the lease. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I would park this fixture on its free server option and drive the planner with free model access so the model hop is a real queue, then inject preemption while a tool call is in flight. I am not attaching quotas, model names, hardware claims, or latency multipliers I did not measure here. If you need a box that is allowed to disappear, that free server option is enough to host the simulator.

Limitations and who should not use this

This is a discrete-event model with a single run_id and a single step, not a multi-tenant broker. It does not prove exactly-once delivery across a provider that forgets idempotency keys after a day. It also does not replace property tests on your real gateway, and it will not save a tool that mints server-side IDs you cannot replay. Do not use this approach if your workers already hold exclusive, long-lived leases on durable machines and your providers already key on (run_id, step_id). Do not use it if you are writing a single-process script with no side effects outside memory. Do not fold this fence into a prompt instruction and call the architecture done.

What I would change next

I would push the epoch check into the tool gateway itself, because a polite worker header is not a fence if the worker is gone. I would write an outbox record before the HTTP call, so death between send and checkpoint still has a durable intent to replay. I would also split compensation into its own fenced step, because a preempted compensate is a new concurrent writer, not a footnote. After that I would add a second injected failure: grant overlapping leases during a partition and prove the lower epoch cannot checkpoint.

The counterexample I still want you to answer is this one. Worker two replays the stable key, the provider returns the original receipt, and then worker one’s delayed checkpoint for epoch one overwrites worker two’s snapshot with a smaller seq. Does your log reject that write, replay the newer snapshot, or compensate by freezing the run? If you cannot pick one, the design does not yet converge.