Treat Client Timeout as an In-Doubt Lease Before You Replay Shared Tool Calls

Last Tuesday I sat with a planner log that looked almost boring until I lined up two clocks. The client aborted a tool call at thirty seconds, then immediately replayed the same payload against a shared slot. The original call finished four seconds later, and the retry still executed against that reservation. Two side effects landed in the same conversation epoch, and nobody had a lease to point at.

That event order is not an incident curiosity, and it is the default protocol most agent backends implement. A client timeout is not an abort, and a shared free slot does not make that physics kinder. I keep reviewing these loops as if HTTP were a transaction, which it never was. So I want the architecture to name the in-doubt lease before anyone hits replay again.

Have you ever watched a retry storm and blamed the model for duplicate tool work? I have, and the model was the least interesting process in the trace.

The invariant the common retry loop fails to preserve

I want one invariant, stated before any diagram or fixture. A side-effecting tool result may bind if and only if three tokens still match: lease_id, idempotency_key, and conversation_epoch. Anything else is an in-doubt write, not a clean miss, and it must not apply.

Why three tokens instead of a single request id on the HTTP client? Because the failure domains are not the same plane, even when they share a TCP connection. The client timer lives in the planner process, the lease lives on the shared slot, and the epoch lives in the conversation store. Collapse those planes and you will bind a late body to the wrong run.

Declared assumptions

I am reviewing a planner that issues tool calls onto a shared remote executor, not a dedicated box you fully own. The executor may reclaim a slot when a lease expires, even if a tool body is still in flight. Side-effecting tools are not locally transactional with the model stream, and duplicate delivery is expected after timeout, reconnect, or planner restart.

I am not assuming named models, published quotas, dedicated hardware, or a permanent free tier. Those would be invented product claims, and they are not required for the race. I am assuming only that a free shared slot can exist long enough to admit work and then disappear under another conversation. If your tools are strictly read-only, this review is heavier than you need.

This fixture is a design proposal. It is not a production measurement, and I am not reporting live traffic.

Data flow: treat admission as a protocol, not an HTTP POST

Here is the sequence I want the architecture to speak, not the sequence most SDKs actually speak when a timeout fires.

1. The planner creates an idempotency key bound to the current conversation_epoch.
2. The executor admits a reservation and returns a lease_id with an explicit expiry.
3. The tool writes a prepare record inside that lease before any external side effect.
4. Completion is a bind, not a print: match lease, key, and epoch, then commit.
5. On client timeout, the planner marks the call IN_DOUBT and does not replay yet.
6. A recover path commits, compensates, or rejects; it never blind-replays the payload.

Does your current client already skip step five? Mine did, for years, because timeouts feel like negatives and negatives feel safe to retry.

sequenceDiagram
    participant Planner
    participant Admit as AdmitGate
    participant Slot as SharedSlot
    participant Store as EpochStore
    Planner->>Admit: reserve(key, epoch)
    Admit-->>Planner: lease_id, expiry
    Planner->>Slot: execute(tool, lease_id)
    Note over Planner: client timer fires; mark IN_DOUBT
    Slot->>Store: bind(lease_id, key, epoch, result)
    alt tokens match
        Store-->>Slot: COMMIT
    else lease or epoch mismatch
        Store-->>Slot: REJECT
    end
    Planner->>Admit: recover(key, epoch)
    Admit-->>Planner: commit | compensate | reject

The HTTP POST is only the transport. If your sequence diagram starts at requests.post and ends at status 200, you have not modeled the bind.

Failure domains

I split the system into four domains so retries stop crossing wires they do not own.

Planner timer domain. A thirty-second client timeout fires while the executor is still inside a valid lease. The planner knows nothing about that lease. The work may still commit, and a replay now is a second admit.

Lease domain. The shared slot expires and is granted to another conversation. A late body can arrive carrying a stale lease_id. Binding it is a consistency bug, not a helpful flush of leftover tokens.

Idempotency domain. The same key is admitted twice because the first admit response was lost on the way back. Two prepare records exist. One must win, and the other must no-op without a second side effect.

Epoch domain. The user started a new turn, or a worker preemption bumped conversation_epoch. A completion from the previous turn must not append, even if the lease still looks alive in a local cache.

Notice what I did not put in this review: dashboards, deploy pipelines, or paging rotations. Those are operations concerns. This article stops at whether the protocol converges under a known event order.

A minimal in-doubt simulator

I wanted an executable counterexample I could step in one file, not a slide about retries. The fixture below models a shared slot, a client timeout that does not abort the server, and a bind gate that either commits or rejects. Label it unexecuted until you run it locally.

from dataclasses import dataclass, field
from typing import Dict, Optional, Literal

Decision = Literal["COMMIT", "REJECT", "IN_DOUBT", "COMPENSATE"]

@dataclass
class Reservation:
    lease_id: str
    key: str
    epoch: int
    expiry: int
    prepared: bool = False
    applied: bool = False

@dataclass
class Slot:
    now: int = 0
    current: Optional[Reservation] = None
    applied_side_effects: int = 0
    binds: Dict[tuple, Decision] = field(default_factory=dict)

    def admit(self, lease_id: str, key: str, epoch: int, ttl: int) -> Reservation:
        if self.current and self.now < self.current.expiry:
            raise RuntimeError("slot still leased")
        self.current = Reservation(lease_id, key, epoch, self.now + ttl)
        return self.current

    def prepare(self, lease_id: str, key: str, epoch: int) -> None:
        r = self._require(lease_id, key, epoch)
        r.prepared = True

    def bind(self, lease_id: str, key: str, epoch: int) -> Decision:
        token = (key, epoch)
        if token in self.binds and self.binds[token] == "COMMIT":
            return "REJECT"  # duplicate bind never re-applies
        r = self.current
        if not r or r.lease_id != lease_id or r.key != key or r.epoch != epoch:
            self.binds[token] = "REJECT"
            return "REJECT"
        if self.now >= r.expiry or not r.prepared:
            self.binds[token] = "REJECT"
            return "REJECT"
        if not r.applied:
            r.applied = True
            self.applied_side_effects += 1
        self.binds[token] = "COMMIT"
        return "COMMIT"

    def recover(self, key: str, epoch: int) -> Decision:
        token = (key, epoch)
        decision = self.binds.get(token)
        if decision == "COMMIT":
            return "COMMIT"
        if decision == "REJECT":
            return "COMPENSATE"
        return "IN_DOUBT"

    def _require(self, lease_id: str, key: str, epoch: int) -> Reservation:
        r = self.current
        if not r or r.lease_id != lease_id or r.key != key or r.epoch != epoch:
            raise RuntimeError("lease mismatch")
        if self.now >= r.expiry:
            raise RuntimeError("lease expired")
        return r


def timeout_then_late_success() -> Slot:
    s = Slot()
    s.admit("L1", "k", 7, ttl=40)
    s.prepare("L1", "k", 7)
    s.now = 30  # client timeout; planner must NOT replay
    assert s.recover("k", 7) == "IN_DOUBT"
    s.now = 34
    assert s.bind("L1", "k", 7) == "COMMIT"
    # Blind replay would try a second admit; the bind gate must no-op.
    assert s.bind("L1", "k", 7) == "REJECT"
    assert s.applied_side_effects == 1
    return s


def lease_recycle_then_late_body() -> Slot:
    s = Slot()
    s.admit("L1", "kA", 1, ttl=10)
    s.prepare("L1", "kA", 1)
    s.now = 11
    s.admit("L2", "kB", 2, ttl=10)
    assert s.bind("L1", "kA", 1) == "REJECT"
    assert s.applied_side_effects == 0
    return s


def lost_admit_ack() -> Slot:
    s = Slot()
    s.admit("L1", "k", 3, ttl=20)
    s.prepare("L1", "k", 3)
    s.now = 5
    # Planner never saw L1; a naive retry would admit again and double-apply.
    try:
        s.admit("L2", "k", 3, ttl=20)
        raise AssertionError("second admit must fail while lease is live")
    except RuntimeError:
        pass
    assert s.bind("L1", "k", 3) == "COMMIT"
    assert s.applied_side_effects == 1
    return s


if __name__ == "__main__":
    timeout_then_late_success()
    lease_recycle_then_late_body()
    lost_admit_ack()
    print("invariant held for the three injected schedules")

How would you run it? Save the file and execute the three schedules before you trust any retry wrapper. The point is not throughput. The point is whether the invariant holds after each injected order.

python in_doubt_lease.py

If you comment out the duplicate-bind guard and allow a second admit after the client timer, you get the original outage in twelve lines. That is the counterexample I want in review, not a narrative about flaky networks.

Injected failures and testable properties

I inject three schedules that naive retry loops treat as identical timeouts.

1. timeout_then_late_success: the client timer fires; the original tool still commits; replay is attempted.
2. lease_recycle_then_late_body: the slot is granted to conversation B; A's body arrives late.
3. lost_admit_ack: admit succeeds on the server; the planner never sees lease_id and wants to retry.

Properties I require after every schedule, with no exceptions for "the model was slow":

• applied_side_effects <= 1 per (idempotency_key, conversation_epoch)
• no bind occurs when lease_id mismatches the current reservation
• a timeout alone never increments applied_side_effects
• an IN_DOUBT call ends in commit, compensate, or reject, never in silent replay

If any property fails, the design does not converge. I do not care that the happy path looked fast on an empty slot.

Tradeoff table

Choice	What you keep	What you pay	When it breaks
Blind replay after client timeout	Simple HTTP wrapper	Duplicate side effects	Late success plus retry
Longer client timeout, same bind	Fewer visible errors	Occupied shared slots, still in-doubt	Lease expiry before the timer
Lease + key + epoch bind gate	Converges under recycle	Recover RPC, prepare record	Clock skew beyond lease TTL
Compensate on mismatch	Conversation stays coherent	Needs an inverse for each tool	Tools without a safe inverse

The interesting denominator is not QPS and not token count. Count side-effecting tool calls whose bind decision is made after a client-visible timeout. That is the population the invariant is about. If you report pass rate of the model against that race, you are measuring the wrong plane.

Acceptance rule: for that denominator, applied_side_effects equals 1 on commit and 0 on compensate or reject, with zero cross-epoch binds, across the three injected schedules. Miss any clause and I would not ship the retry loop.

Architecture review: constraints, then what I would change next

Constraints first. A free shared server is an admission-limited executor with preemption. It is not a private worker pool, and it will not hold your conversation because the planner is still waiting. Latency tails are lease tails. Throughput is reservation throughput. Correctness is bind correctness. Cost is how many in-doubt calls you reopen, not how many prompts you can fire into an unbounded queue.

Data flow stays closed: admit, prepare, bind, recover. Anything that writes outside prepare is an uncommitted side effect, and the planner must not advance as if the tool succeeded. Backpressure belongs at admit, not at the HTTP client pool. If the slot is leased, a second planner waits or sheds. It does not invent a parallel universe with the same idempotency key.

If I were changing this system next, I would split prepare from commit for every tool that mutates external state. I would store the reservation beside the conversation epoch, not inside the HTTP session object. I would make recover a first-class RPC, not a for-loop around requests.post. I would refuse to let the planner's timeout close the lease, because the planner does not own that clock.

Would I keep streaming tokens to the UI during IN_DOUBT? Only as uncommitted preview, never as transcript fact. The user can watch a spinner. The store cannot watch a lie.

While I was pressure-testing this reservation protocol, I used MonkeyCode's free model access and free server option as a shared-slot stand-in for the executor, not as a dedicated farm. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The protocol does not depend on that product. If you remove the name, the in-doubt lease still has to exist, and the three schedules still have to hold.

Limitations and who should not use this

Do not use this bind gate as a substitute for an exactly-once log if your tools already run behind a transactional outbox. Do not use it for read-only retrieval, where duplicate calls are cheaper than recover RPCs. Do not use it if you have a crash-stop local executor with no retry and no shared slot. A longer HTTP timeout is not an implementation of this protocol, and I would not review it as one.

Who should skip this article? Anyone looking for deploy recipes, monitoring setup, or incident paging. Anyone trying to score model quality. Anyone hoping a free shared slot behaves like a pinned worker. This is an architecture review of an in-doubt lease, and it stays there.

If you want a shared slot to reproduce the race, the free server option is enough to run the fixture against a real remote executor. Fork the file and break the invariant on purpose before you wrap another retry around tool.call.

Which event order still breaks the invariant on your planner, and should the system reject, replay, or compensate?