Make Timeout-Retry Safety an Invariant Before You Route Across Free Model Endpoints

A counterexample first. Your agent calls a model endpoint, the request times out at the client, and your retry logic fires the same "apply this patch, then mark the task done" request again. What actually happened upstream: the first request succeeded — the model produced its output, your tool executor applied the patch — but the response died in the network. Your retry now applies the patch a second time. If the patch is append migration row, you have two rows. If it is git commit, you have two commits. The system did not fail loudly; it converged to a wrong state silently.

This is the event order that matters:

client              endpoint            executor
  |--- request R1 --->|                    |
  |                   |--- apply patch --->|  (side effect S1 commits)
  |<-- response lost  |                    |
  |--- retry R1' ---->|                    |
  |                   |--- apply patch --->|  (side effect S2 commits)
  |<--- 200 OK -------|                    |

The common implementation — "retry on timeout with exponential backoff" — preserves availability and violates the invariant you actually need:

Invariant: For any logical operation op with idempotency key k, the set of committed side effects attributable to k has cardinality at most one, regardless of how many times a request carrying k is delivered.

This matters more, not less, when you route work across cheap or free model endpoints, because heterogeneous endpoints have heterogeneous timeout behavior, and every distinct timeout profile multiplies the number of retry-after-uncertain-outcome events your system produces.

Declared assumptions

• Requests can be lost, delayed, or duplicated; responses can be lost after the server commits. No exactly-once delivery exists, so the invariant must live in the effect layer, not the transport.
• You control a thin executor between the model call and the side effect (a tool runner, a job worker). If you do not, this design does not apply.
• Side effects are attributable to a single logical operation. Fan-out effects need one key per effect.

The protocol, minimally

The executor keeps a durable record keyed by k with a state machine:

RECEIVED --claim--> IN_PROGRESS --commit--> COMMITTED
                    IN_PROGRESS --crash---> (re-claimed only via fencing check)

A second delivery of k in state COMMITTED returns the recorded result without re-executing. A delivery in IN_PROGRESS must fence: it checks a monotonically increasing claim token and refuses to execute against a stale claim. That fencing step is what most hand-rolled idempotency code omits, and it is exactly what a crash between claim and commit requires.

# Minimal property-test harness (stdlib only). Labeled: this is a design
# fixture, not production code. Run: python3 idempotency_sim.py
import itertools, random

class Executor:
    def __init__(self):
        self.store = {}          # k -> (state, claim_token, result)
        self.effects = []        # committed side effects (the ground truth)
        self.token = 0

    def execute(self, k, effect, crash_before_commit=False):
        state = self.store.get(k)
        if state and state[0] == "COMMITTED":
            return state[2]                      # replay, no new effect
        self.token += 1
        claim = self.token
        self.store[k] = ("IN_PROGRESS", claim, None)
        if crash_before_commit:
            # simulate crash: a later duplicate must fence against `claim`
            return self._recover(k, claim, effect)
        self.effects.append(effect)
        self.store[k] = ("COMMITTED", claim, effect)
        return effect

    def _recover(self, k, stale_claim, effect):
        # Re-claim with a new token; stale claim cannot commit.
        self.token += 1
        if self.token <= stale_claim:
            raise AssertionError("fencing violation")
        self.effects.append(effect)
        self.store[k] = ("COMMITTED", self.token, effect)
        return effect

def effects_for(ex, k):
    # In this fixture each effect records its key: count attributable effects.
    return sum(1 for e in ex.effects if e[0] == k)

random.seed(7)
for trial in range(2000):
    ex = Executor()
    k = f"op-{trial}"
    # adversarial schedule: 1-5 deliveries, random crashes, random dupes
    for _ in range(random.randint(1, 5)):
        ex.execute(k, (k, "commit"), crash_before_commit=random.random() < 0.4)
    assert effects_for(ex, k) == 1, f"invariant broken on trial {trial}"
print("invariant held across 2000 adversarial schedules")

The property under test is the invariant itself: under randomized duplicate delivery and crash points, the committed-effect count per key is exactly one. If you remove the fencing branch, the harness finds a violating schedule within a few dozen trials — that is the counterexample generator doing its job.

Failure classes to inject before you trust it

#	Injected failure	What it catches
1	Response loss after commit	Naive retry re-executes the effect
2	Duplicate delivery, no crash	Missing dedupe on COMMITTED
3	Crash between claim and commit	Missing fencing token; zombie claim double-commits
4	Same key, different payload	Key derivation that ignores the payload silently replays the wrong result
5	Timeout shorter than p99 execution	Retries pile up while the first attempt is still legitimately running

Failure class 5 is the one free-tier endpoints make acute: queueing variance is higher, so a client timeout tuned for a fast paid endpoint will manufacture spurious retries all day.

Where free model access fits this workflow

Fault-injection harnesses are token-hungry: you want thousands of adversarial runs, not three. This is where I ran the evaluation loop against MonkeyCode, which offers free model access and a free server option — enough to run the router-plus-executor against a real model endpoint with artificial latency and dropped responses, rather than only against a stub. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness above is fully self-contained and does not depend on any specific provider; the live-endpoint variant just swaps the stub execute for an HTTP call behind a fault-injecting proxy.

If you want to reproduce the live version: put a tiny proxy in front of the endpoint that (a) injects 200–2000 ms latency, (b) drops 10% of responses after forwarding the request, and (c) logs every delivery. Point your router at it, run class-1 and class-3 faults, and assert the same cardinality invariant on the executor's effect log. If you are evaluating MonkeyCode for this, the free tier is a reasonable place to run exactly that loop — the acceptance criterion below is the whole test.

Tradeoff table

Design choice	Latency	Correctness	Cost	When it wins
Retry on timeout, no dedupe	Lowest happy-path	Violates invariant on response loss	Lowest	Pure read-only calls only
Idempotency key + committed-state replay	+1 store read per call	Holds for classes 1, 2	Small durable store	Default for agent tool calls
Key + fencing token (above)	+1 store write on recovery	Holds for classes 1–4	Store with compare-and-swap	Any effect you cannot undo
Distributed transaction / 2PC	Highest	Strongest	Highest, blocks on failure	Rarely justified at this layer

Explicit denominator: the latency cost is one durable read per tool call — measurable in single-digit milliseconds against a local store — against a failure that silently corrupts state. At agent-loop scale (hundreds of tool calls per task), that is a cheap insurance premium, not a bottleneck. Measure it against your own effect latency before believing either number.

Acceptance rule

Ship the routing change only if: under fault classes 1–4 injected at 10% rates over at least 1,000 logical operations, the executor's committed-effect log contains exactly one effect per idempotency key, and duplicate deliveries return the recorded result without re-execution. Any violation blocks the rollout; there is no "mostly idempotent."

Limitations and who should not use this

• If your side effects are not attributable to a single key (bulk fan-out, cross-system sagas), you need per-effect keys or a compensation protocol, not this record.
• If the endpoint itself performs the side effect opaquely and you cannot interpose an executor, client-side keys do nothing — the dedupe must live where the effect commits.
• Free tiers are appropriate for evaluation harnesses and canary traffic, not as the correctness foundation: quotas, latency variance, and availability of any free endpoint can change, so the invariant must hold assuming the endpoint misbehaves. Do not build the production path such that it only works when the cheap endpoint is healthy.

The counterexample question

Look at your own executor and ask: which event order breaks the invariant — response loss after commit, crash between claim and commit, or duplicate delivery with a mutated payload? Once you find it, decide explicitly: should the system reject the duplicate (return the recorded result), replay it (re-execute only if no effect committed), or compensate (undo and redo)? If you cannot point to the branch of code that answers that question, the answer today is "whatever the network decides."