Make Compensation a Fenced Epoch Before a Late Tool Result Lands

I start every architecture review with a three-event trace rather than a comforting box diagram. Event A is the planner sending tool_call(id=7, epoch=1) toward a slow model path. Event B is the local deadline firing timeout(id=7) and opening compensation. Event C is tool_success(id=7) arriving after that compensation already released the reservation. Which write is legal here, and why do so many planners still answer "whoever arrives last"?

That last-writer rule is the protocol hole I keep drawing on whiteboards. Timeout already ran compensation, so the reservation row was released on purpose. The late success then writes the same row back as if compensation never existed. You did not observe a retry. You observed a ghost write against a closed epoch.

The invariant I will not negotiate

I want one invariant stated so a property test can fail it in a single assertion. For each call_id, at most one epoch may commit a durable mutation, and any result whose epoch is stale must be rejected. Does your planner even carry an epoch field, or does it key every mutation on call_id alone?

If the key is only call_id, success and compensation share one row and fight in arrival order. A fence is not a retry counter you increment for luck. It is a leadership token for that call, and late winners are not leaders. I will treat a success without a matching live epoch as a protocol violation, not as good news from the model.

Assumptions I am willing to defend

I am reviewing a single-planner workflow with one outbound tool call, not a multi-agent mesh. The tool is at-least-once and may deliver success after the client has already timed out. The compensator is itself a network call, so it can also arrive late or twice. I trust a monotonic local deadline more than wall-clock agreement across machines. Durable state is a small key-value map that a test can snapshot after every event.

I also assume tail latency is part of the workload, not an exception you debug once. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A scratch path with MonkeyCode's free model access and free server option is enough to reproduce delayed calls, which is the only product role in this review. I am not quoting model names, quotas, or hardware, because those are not the claim this architecture needs.

Constraints and data flow

The constraint that actually matters is not tokens. It is this: the planner may not learn the true outcome before its own deadline, and the tool may still mutate the world afterward. I therefore split the workflow into an epoch manager, an outbox of intended calls, and a durable call record. The model path is just one more unreliable transport.

Here is the data flow I want in the review, not the flow most agent templates ship.

sequenceDiagram
    participant P as Planner
    participant E as EpochMap
    participant T as ToolPath
    participant C as Compensator
    P->>E: fence(call=7, epoch=1)
    P->>T: tool_call(7, epoch=1)
    Note over P: local deadline fires
    P->>E: bump fence to epoch=2
    P->>C: compensate(7, epoch=2)
    T-->>P: tool_success(7, epoch=1)
    P->>E: reject stale epoch=1
    C-->>P: compensate_ok(7, epoch=2)
    P->>E: commit epoch=2 only

Notice the fence bump happens before compensation is sent. If you bump after the compensator returns, a crash in the middle leaves epoch 1 still live. Then the ghost success is legal again. Would you accept that window in a payments pipeline? I would not accept it in a planner either.

Failure domains in this design

I group failures by who is allowed to speak, not by HTTP status. Domain one is the model path: delay, duplicate success, or success with no body. Domain two is the compensator path: timeout of the timeout, or a second compensate after the first already committed. Domain three is the planner process: crash between fencing and the durable write. Domain four is the clock: a jumped deadline that opens epoch 2 while epoch 1 is still in flight.

Those domains do not share a retry policy. Retrying the model path inside epoch 1 is legal. Retrying it after epoch 2 exists is a ghost write. Compensating twice inside epoch 2 is an idempotency problem, not a fencing problem. If you fold all four into one generic except Exception: retry, you have already lost the invariant. Which domain does your current catch block think it is handling?

A minimal epoch simulator

I do not trust a diagram until a fixture can violate it. The simulator below is a proposal you can run locally; it is not a production runtime and it does not call a live model. It encodes the three-event trace, plus a few siblings, as an explicit state machine.

#!/usr/bin/env python3
"""Epoch fence simulator for late tool success after compensation.

Run:
  python epoch_fence_sim.py
  python epoch_fence_sim.py --inject late_success_after_compensate
"""
from __future__ import annotations

from dataclasses import dataclass, field
from typing import Callable, Dict, List, Optional, Tuple
import argparse
import copy


@dataclass
class CallRecord:
    call_id: str
    live_epoch: int = 0
    committed_epoch: Optional[int] = None
    status: str = "idle"  # idle|inflight|compensating|committed|rejected
    mutations: List[str] = field(default_factory=list)


@dataclass
class Store:
    calls: Dict[str, CallRecord] = field(default_factory=dict)

    def snapshot(self) -> Dict[str, Tuple[int, Optional[int], str, tuple]]:
        out = {}
        for k, c in self.calls.items():
            out[k] = (c.live_epoch, c.committed_epoch, c.status, tuple(c.mutations))
        return out


class EpochPlanner:
    def __init__(self, store: Store):
        self.store = store

    def fence_and_send(self, call_id: str) -> int:
        rec = self.store.calls.setdefault(call_id, CallRecord(call_id))
        rec.live_epoch += 1
        rec.status = "inflight"
        rec.mutations.append(f"send:epoch={rec.live_epoch}")
        return rec.live_epoch

    def on_timeout(self, call_id: str) -> int:
        rec = self.store.calls[call_id]
        rec.live_epoch += 1
        rec.status = "compensating"
        rec.mutations.append(f"timeout_open:epoch={rec.live_epoch}")
        return rec.live_epoch

    def on_result(self, call_id: str, epoch: int, kind: str) -> str:
        rec = self.store.calls[call_id]
        if epoch != rec.live_epoch:
            rec.mutations.append(f"reject:{kind}:epoch={epoch}")
            return "reject"
        if rec.committed_epoch is not None:
            rec.mutations.append(f"reject_committed:{kind}:epoch={epoch}")
            return "reject"
        rec.committed_epoch = epoch
        rec.status = "committed"
        rec.mutations.append(f"commit:{kind}:epoch={epoch}")
        return "commit"


Trace = List[Tuple[str, tuple]]


def apply_trace(trace: Trace) -> Store:
    store = Store()
    planner = EpochPlanner(store)
    for name, args in trace:
        getattr(planner, name)(*args)
    return store


def naive_last_writer(trace: Trace) -> Store:
    """Common implementation: ignore epoch, commit whichever result arrives last."""
    store = Store()
    rec = store.calls.setdefault("7", CallRecord("7"))
    for name, args in trace:
        if name == "fence_and_send":
            rec.live_epoch = 1
            rec.status = "inflight"
            rec.mutations.append("send")
        elif name == "on_timeout":
            rec.status = "compensating"
            rec.mutations.append("compensate")
            rec.committed_epoch = 2
        elif name == "on_result":
            kind = args[2]
            rec.status = "committed"
            rec.committed_epoch = 99
            rec.mutations.append(f"last_writer:{kind}")
    return store


TRACES: Dict[str, Trace] = {
    "happy_success": [
        ("fence_and_send", ("7",)),
        ("on_result", ("7", 1, "success")),
    ],
    "timeout_then_compensate": [
        ("fence_and_send", ("7",)),
        ("on_timeout", ("7",)),
        ("on_result", ("7", 2, "compensate_ok")),
    ],
    "late_success_after_compensate": [
        ("fence_and_send", ("7",)),
        ("on_timeout", ("7",)),
        ("on_result", ("7", 1, "success")),  # ghost
        ("on_result", ("7", 2, "compensate_ok")),
    ],
    "duplicate_success_same_epoch": [
        ("fence_and_send", ("7",)),
        ("on_result", ("7", 1, "success")),
        ("on_result", ("7", 1, "success")),
    ],
    "compensate_then_late_success": [
        ("fence_and_send", ("7",)),
        ("on_timeout", ("7",)),
        ("on_result", ("7", 2, "compensate_ok")),
        ("on_result", ("7", 1, "success")),
    ],
    "timeout_without_compensate_result": [
        ("fence_and_send", ("7",)),
        ("on_timeout", ("7",)),
    ],
    "ghost_then_duplicate_compensate": [
        ("fence_and_send", ("7",)),
        ("on_timeout", ("7",)),
        ("on_result", ("7", 1, "success")),
        ("on_result", ("7", 2, "compensate_ok")),
        ("on_result", ("7", 2, "compensate_ok")),
    ],
    "success_before_timeout_ignored": [
        ("fence_and_send", ("7",)),
        ("on_result", ("7", 1, "success")),
        ("on_timeout", ("7",)),  # illegal in a correct scheduler; still tested
    ],
}


def property_single_commit(store: Store) -> Optional[str]:
    rec = store.calls["7"]
    commits = [m for m in rec.mutations if m.startswith("commit:")]
    if len(commits) > 1:
        return f"multiple commits: {commits}"
    if rec.committed_epoch is not None and rec.committed_epoch != rec.live_epoch:
        if rec.status == "committed" and any("commit:success" in m for m in rec.mutations) and any(
            "timeout_open" in m for m in rec.mutations
        ):
            return "success committed after fence bump"
    ghost = [m for m in rec.mutations if m.startswith("commit:success") and ":epoch=1" in m]
    if ghost and any(m.startswith("timeout_open") for m in rec.mutations):
        return f"ghost success committed: {ghost}"
    return None


def property_naive_breaks(store: Store) -> bool:
    rec = store.calls["7"]
    return any(m.startswith("last_writer:success") for m in rec.mutations) and any(
        m == "compensate" for m in rec.mutations
    )


def run_all(inject: Optional[str] = None) -> int:
    names = [inject] if inject else list(TRACES)
    failed = 0
    for name in names:
        store = apply_trace(TRACES[name])
        err = property_single_commit(store)
        naive = naive_last_writer(TRACES[name])
        naive_broken = property_naive_breaks(naive)
        print(f"TRACE {name}")
        print(f"  fenced={store.snapshot()}")
        print(f"  naive_broken={naive_broken} naive={naive.snapshot()}")
        if err:
            print(f"  PROPERTY FAIL: {err}")
            failed += 1
        else:
            print("  PROPERTY OK")
    print(f"failed={failed} / {len(names)}")
    return failed


if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("--inject", choices=sorted(TRACES))
    args = p.parse_args()
    raise SystemExit(run_all(args.inject))

Run it in this order. I want the denominator visible in the terminal, not implied by a blog claim.

1. Save the file as epoch_fence_sim.py in an empty directory.
2. Run python epoch_fence_sim.py and read failed=0 / 8.
3. Run python epoch_fence_sim.py --inject late_success_after_compensate.
4. Confirm the fenced planner prints reject:success:epoch=1 and a single compensate commit.
5. Confirm the naive last-writer path sets naive_broken=True on that same trace.

If step four and step five ever agree, the fence is decorative. Are you testing the protocol, or testing that Python still runs?

Injected failures and testable properties

I inject order, not CPU load. The eight traces are the failure classes for this review: happy success, timeout then compensate, late success after the fence bump, duplicate success in the live epoch, compensate then late success, timeout with no compensator result, ghost success plus duplicate compensate, and a scheduler bug that times out after commit. The property is narrower than "the agent seems fine." At most one commit: mutation may exist, and commit:success:epoch=1 may not coexist with a timeout-opened epoch.

The naive last-writer is the control. On late_success_after_compensate it always reports naive_broken=True, because it commits success after it already compensated. That is the counterexample the common implementation fails to preserve. If your production planner cannot show an equivalent reject, I would not scale it. Would you promote a planner that cannot name the epoch it just committed?

Tradeoffs I would actually pick

Choice	Latency	Correctness	Cost	When I pick it
Last-writer on call_id	Lowest	Ghost writes after timeout	Cheap until an incident	Never for durable side effects
Reject stale epoch, compensate in the new fence	Adds one bump and one reject path	Preserves single-commit	Extra compensate calls	Default for this review
Replay the original call in epoch 2	Higher	Needs idempotent tools	Pays the model twice	Only if compensation is missing
Wait out tail latency, no timeout	Unbounded	Avoids ghosts by stalling	Holds server capacity	Not on a preemptible free server

I pick reject-and-compensate as the default because the invariant is single-commit, not minimum latency. Replay is a different protocol: it needs a new epoch and an idempotency key the tool honors. Waiting forever is not a protocol. It is a hope that your free server never preempts the process. Which column are you optimizing when you disable the deadline?

The explicit denominator is eight injected traces, not a percentile I did not measure. The acceptance rule is failed=0 / 8 on the fenced planner, plus naive_broken=True on the late-success control. If you add traces, the denominator changes and the rule stays the same: zero ghost commits.

How you validate the conclusion

I would not take my word for it, and you should not either. Clone the fixture, add one trace that matches your real tool adapter, and watch the property fail first. Then add the fence bump on the timeout path and watch the reject appear. If you cannot get a red test, you are not validating the invariant. You are formatting a sequence diagram.

If you want that delayed-success path to come from a real call instead of a tuple list, a scratch process on MonkeyCode's free server option is enough to hold the planner while the model path returns late. That is the only invitation in this article. Keep the fence logic in your own store either way, because a hosted prompt does not invent epochs for you.

What I would change next

I would persist the fence bump in an outbox before any network send, including the original tool call. Today the simulator mutates memory in the same function that "sends," which hides a crash window. I would also require the tool adapter to echo the epoch, so a success with a missing fence is rejected as malformed, not as stale. I would split compensator idempotency into its own property, instead of letting reject_committed paper over a double compensate.

I would not add a second planner replica until the single-process fixture is boring. Two owners without a fence is the same ghost write with more enthusiasm. I would also stop treating model latency as a reason to skip deadlines. The deadline is how epoch 2 begins. Without it, compensation never opens, and the invariant has nothing to protect.

Who should not use this

Do not use this fencing model if your tool call is fire-and-forget and nothing durable is written. You would be adding epochs to a log line. Do not use it if the side effect is irreversible and the vendor cannot echo an idempotency key, because compensation is then fiction. Do not use it as a substitute for an outbox if a crash between fence and send is in your threat model. This fixture will not save a payments pipeline that still keys mutations on call_id alone.

It is also the wrong review if you needed a deployment guide or a dashboard. I am arguing about event order and a commit rule. If your actual problem is cluster bring-up, this article will waste your afternoon.

The counterexample I want back

I will leave you with the same question I put on the whiteboard. Which event order still breaks the invariant once the fence exists: a compensate result that arrives before the timeout bump is durable, a success that echoes no epoch, or a second timeout that opens epoch 3 while epoch 2 is inflight? Should the system reject that write, replay the call under a new fence, or compensate again? If you cannot answer with a trace the simulator can run, you are not done with the protocol.