Make Shared Catalog Version a Watermark Before a Late Loop Tick Rewinds

Last Tuesday I sat with a loop that looked almost local. An agent asked a remote planner for the next tick, merged a shared tool catalog, and wrote the result into a knowledge store that other agents treated as truth. The happy-path logs were boring, which is usually when I start hunting event-order bugs. What happens when tick 12 is still in flight, tick 13 commits a newer catalog, and tick 12 finally returns with an older tool list?

Does your shared knowledge rewind, or do you reject that late ballot? I am reviewing the commit protocol around the prompt, not the prompt itself. The common implementation treats each HTTP 200 as loop progress, and that is the invariant it fails to preserve.

The invariant the if-statement loop drops

I want one property to survive reordering, retries, and a slow remote planner. Shared catalog version is a watermark: a tick may mutate the catalog only when its ballot still owns the commit watermark. A late, duplicated, or stale planner response is not progress. It is a ballot that arrived after another writer already moved the log forward.

That sounds obvious until the planner lives on someone else's host. Why would a free remote tick source preserve your local sequence numbers? It will not, unless you make the watermark an explicit protocol object.

Declared assumptions

I am labeling this as a design review with an unexecuted simulator, not a production postmortem. The model below assumes the following constraints, and it is wrong the moment any of them is false.

1. One logical agent owns one catalog log, but several in-flight ticks may exist because the planner is remote.
2. Catalog writes are the only side effect that other agents can observe; tool calls are fenced elsewhere.
3. The remote planner is at-least-once and unordered; responses can duplicate, stall, or arrive inverted.
4. Wall-clock time is not a watermark. Sequence numbers and catalog versions are.
5. Readers tolerate lag, but they must never observe a catalog version that moves backward.

If your store already offers linearizable compare-and-swap, you still need the ballot. CAS without a watermark still accepts a stale body that happens to win a race.

Constraints I would actually design against

The interesting constraint is not tokens. It is that the planner and the catalog live in different failure domains, and the loop still pretends they are one process. Latency of a remote tick is unbounded relative to local merge time, which means in-flight ballots are the normal case, not the outage case.

Throughput is limited by how many uncommitted ticks you allow, not by how fast the model streams tokens. Correctness is the monotonic catalog watermark. Cost is the rejected late ballots you are willing to replay instead of applying.

I keep asking the same question in reviews: what is the denominator? Here it is committed catalog versions that never rewind, divided by planner responses that the loop accepted. If that ratio can drop because a late 200 overwrote a newer catalog, the architecture is already lying.

Data flow, drawn as a protocol

sequenceDiagram
    participant Loop as Agent loop
    participant WM as Watermark store
    participant Planner as Remote planner
    participant Cat as Shared catalog log

    Loop->>WM: reserve ballot b, watermark w
    Loop->>Planner: tick(b, w, catalog_head)
    Note over Planner: delay, retry, or reorder
    Planner-->>Loop: ballot b' body (maybe stale)
    Loop->>WM: accept only if b==head and w==current
    alt watermark still owned
        Loop->>Cat: append version w+1
        Loop->>WM: commit watermark w+1
    else late or duplicate ballot
        Loop->>WM: reject, maybe replay
        Note over Cat: catalog must not rewind
    end

The catalog is a log, not a mutable JSON blob. The watermark store is a single integer plus the in-flight ballot id. The remote planner is an untrusted proposer. That split is the whole architecture.

Failure domains, named on purpose

I split the system into four domains because a single “agent is down” story hides the bug. Domain A is the loop process, which can restart and retry with the same ballot. Domain B is the watermark store, which must not lose the head without a fence. Domain C is the remote planner, which can stall, duplicate, or return a body computed against an older head. Domain D is the catalog log that other agents read.

Cross-domain writes without a ballot are how shared knowledge walks backward. A timeout in C is not a local if-statement miss. It is a proposer that may still commit after you have already moved on. Have you actually drawn that boundary on your own loop, or did the framework hide it inside await complete()?

A watermark protocol you can test

I would implement the protocol as a tiny state machine before I let a remote host near the catalog. The steps below are a review checklist, not a framework.

1. Persist watermark and inflight_ballot before you send the planner request.
2. Put both values in the request body so a retry is identifiable, not clever.
3. Accept a response only when ballot == inflight_ballot and observed_watermark == watermark.
4. Append the catalog record with version = watermark + 1, then advance the watermark.
5. On mismatch, reject and decide reject-versus-replay without mutating the log.
6. Never take updated_at from the planner host as proof of freshness.

That is the whole commit path. If your current code writes the catalog inside the HTTP callback, you are missing steps 3 and 5.

Minimal simulator, labeled unexecuted

The fixture below is a proposal I would run locally. It is not a benchmark, and I am not claiming production numbers from it.

from dataclasses import dataclass, field
from typing import Optional, List, Tuple

@dataclass
class Ballot:
    ballot_id: int
    observed_wm: int
    body_version: int  # catalog version the planner thought it saw

@dataclass
class Catalog:
    versions: List[int] = field(default_factory=lambda: [0])

    @property
    def head(self) -> int:
        return self.versions[-1]

class WatermarkLoop:
    def __init__(self) -> None:
        self.watermark = 0
        self.inflight: Optional[int] = None
        self.next_ballot = 1
        self.catalog = Catalog()
        self.rejects = 0
        self.commits = 0

    def reserve(self) -> Ballot:
        assert self.inflight is None, "single inflight ballot for this review"
        b = self.next_ballot
        self.next_ballot += 1
        self.inflight = b
        return Ballot(b, self.watermark, self.catalog.head)

    def apply(self, ballot: Ballot, proposed_head: int) -> str:
        if self.inflight != ballot.ballot_id:
            self.rejects += 1
            return "reject_duplicate_or_unknown"
        if ballot.observed_wm != self.watermark:
            self.inflight = None
            self.rejects += 1
            return "reject_stale_watermark"
        if proposed_head < self.catalog.head:
            self.inflight = None
            self.rejects += 1
            return "reject_rewind"
        self.catalog.versions.append(proposed_head)
        self.watermark += 1
        self.inflight = None
        self.commits += 1
        return "commit"

def invert_late_tick() -> Tuple[str, str, List[int]]:
    loop = WatermarkLoop()
    b12 = loop.reserve()
    # tick 12 stays in flight; we simulate a crash-retry owner by clearing inflight
    loop.inflight = None
    b13 = loop.reserve()
    r13 = loop.apply(b13, proposed_head=b13.body_version + 1)
    r12 = loop.apply(b12, proposed_head=b12.body_version + 1)
    return r13, r12, loop.catalog.versions

Run it as a property check, not as a demo screenshot.

python - <<'PY'
from watermark_loop import invert_late_tick, WatermarkLoop, Ballot

r13, r12, versions = invert_late_tick()
assert r13 == "commit", r13
assert r12 in {"reject_duplicate_or_unknown", "reject_stale_watermark", "reject_rewind"}, r12
assert versions == sorted(versions), versions
print("late tick did not rewind", versions, r13, r12)
PY

If that assertion ever fails, the catalog rewound. That is the counterexample I opened with, now executable.

Injected failures I would actually gate on

I do not trust a green happy path. I would inject four classes and require the same property.

1. Inverted arrival: reserve b12, reserve b13 after a simulated loss of inflight, apply b13 then b12.
2. Duplicate delivery: apply the same ballot twice and require the second call to reject.
3. Stale body: keep the watermark, but propose head - 1 and require reject_rewind.
4. Retry after commit: replay b13 after a successful commit and require unknown-ballot rejection.

The acceptance rule is boring on purpose. For N injected planner responses, catalog versions must be strictly nondecreasing, and commits / accepted_responses equals 1. Rejects are allowed. Rewinds are not. The denominator is accepted responses, not issued HTTP calls, because calls are not commits.

Tradeoffs, written as a table

Design	Latency	Throughput	Correctness under reorder	Cost of a late tick
Write catalog in the HTTP callback	Lowest	Highest until the first rewind	Fails	Silent corruption
Last-write-wins on updated_at	Low	High	Fails when clocks and stalls disagree	Shared knowledge walks backward
Single inflight ballot + watermark	Adds a reserve round	Caps concurrency at one tick	Holds for this model	Reject and replay
Pipelined ballots with fencing tokens	Higher bookkeeping	More in-flight ticks	Holds if each ballot carries a fence	More rejected work

I would start with the single inflight ballot. Pipelining is the next architecture change, not the first one. Why add pipeline complexity before you can reject a single late tick?

Where a free remote tick source actually fits

This is the point where a remote planner stops being a product slide and becomes a failure injector you chose. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I treat MonkeyCode here as an open project with free model access and a free server option, which is useful only as a real remote proposer in the data flow above.

I would not put that host in the catalog's failure domain. I would put it in domain C, behind the watermark, and I would keep catalog commits local. The free server is then a source of delay, duplication, and reordering you do not have to fake well. It is not a source of truth for version numbers.

If you already have a loop, point only the planner call at that free remote option and keep the simulator's apply() on your side. Do not let the remote body write the catalog directly. That is the entire integration advice I am willing to give without inventing quotas, hardware, or model names I cannot verify.

What I would change next

After the single-ballot gate turns green, I would change three things, in this order. First, allow pipelined ballots with an explicit fence per inflight id, because one-at-a-time will cap throughput on a high-latency planner. Second, store catalog records as hashes of the tool list so a reject can replay without guessing. Third, add a reader-side assertion that head never decreases across process restarts.

I would not add a dashboard before those three. Monitoring setup is a different job. The architecture question is still whether a late ballot can rewind shared knowledge.

Limitations, and who should not use this

This review assumes a single catalog log per agent and a watermark store that does not lie. If you need multi-agent writers on one catalog, you need consensus or a sequencer, not this ballot. If your planner side effects include billing, email, or anything outside the log, this protocol is incomplete and you should stop at reserve.

Do not use the simulator as evidence of model quality. It never scores answers. Do not use a free remote host as your watermark store. Do not treat this as a license to skip linearizable storage. And do not copy the happy-path if-statement from an agent tutorial and call it a loop protocol.

The approach is for senior backend and systems people who already suspect their shared MCP-style catalogs are eventually wrong. If you are still wiring the first tool call, you will overfit to this machine.

Validate the conclusion before you scale the loop

Run the inverted-arrival fixture against your own event log. Count rewinds with a real denominator: catalog heads observed by a second reader process. If you cannot name the event order that should reject, you do not have an invariant yet.

Which event order still breaks the watermark if tick 12 and tick 14 both retry after tick 13 committed, and should the system reject, replay, or compensate without touching the catalog head?