Last Tuesday I reconstructed a fan-out eval where twelve planner loops entered one shared inference queue together. The first timeout did not cancel the original call; it issued a second call without a new lease. When both completions returned out of order, the planner applied two tool plans for a single step. Have you ever blamed the model for a duplicated tool call, when the queue just amplified a retry?
The invariant I want is narrow, and most agent loops fail it the moment retries exist. For any eval-run identifier, in-flight inference must stay inside a declared outstanding bound, and a completion may apply only while it still holds a live lease generation. Without that fence, a shared backend does not give you throughput. It gives you a retry amplifier dressed up as an agent loop.
Assumptions I will not hide
I am going to declare assumptions before I sketch the protocol, because otherwise the simulator quietly lies. Inference is a remote queue I do not own, so completion order is not submit order. Timeouts are local clocks, not server-side cancellation, which means the original call can still finish. Retries are at-least-once, so the backend may execute both the original request and the retry. Tool side effects sit outside this article; I only fence planner-visible model completions. Does your loop already assume any of those away?
I also assume the lab can use free model access and a free server as a backend I do not schedule. That is a constraint, not a capacity promise. I am not claiming exclusive GPUs, named models, or a durable quota. I am claiming a shared queue that will reorder work under load, which is exactly the failure class this lease is for.
Constraints, data flow, and failure domains
The constraint that actually bites is hidden head-of-line blocking on a queue you cannot inspect. Parallel planner loops treat “send another completion request” as progress, while the queue treats every retry as more work ahead of the original. Latency then looks like model slowness, so the loop shortens its timeout, which makes the stampede worse. Why would a free shared backend behave any differently?
Data flow is small if you draw the fence in the right place. An eval-run asks an admission desk for a lease that carries a generation integer. The worker may emit one inference request tagged with that generation, then it must wait or release. Completions re-enter the desk, not the planner, and the desk either commits the text into the step log or rejects it as a stale generation. Only a committed step may mint the next lease. That is the entire control plane.
I split failure domains along three lines, because mixing them makes the postmortem unreadable. Domain A is the client timeout clock, which can fire while the server still holds the original request. Domain B is the shared inference queue, which can delay, duplicate, or reorder completions. Domain C is the planner log, which must not apply text that lost its lease. If you debug domain C with domain A metrics, you will tune timeouts forever and never stop the duplicate step.
eval-run
|
v
admission desk (bound N, generation g)
|
| lease(g) release / reject
v ^
worker ----request(g)----> shared inference queue
^ |
| v
+----- completion(g') --+
|
v
planner step log (apply iff g' == live g)
The violating event order
Here is the counterexample I keep on the whiteboard, because it is shorter than a principles slide. Worker W holds lease generation 4 and sends request R1. The local timeout fires at T+8s and W sends R2 with the same generation, or worse, with no generation at all. R2 completes first with a tool plan; the planner applies it and mints generation 5. R1 then completes with a different tool plan and the planner applies that too, because nothing checked the generation. Which of those two plans is “the” step?
A common implementation preserves none of the invariant. It uses a semaphore in the happy path, then bypasses the semaphore on retry because “we already paid for the wait.” It treats HTTP 200 as authority, even when the body belongs to a lease that expired. It never records the generation on the completion record, so a property test has nothing to assert. Is that an agent architecture, or an if-statement with unbounded outstanding I/O?
Protocol: numbered steps I would actually implement
I want the admission desk to be a tiny state machine, not a framework. Each eval-run owns one desk. The desk stores bound, in_flight, generation, and a map of live lease ids. Everything else is a comment waiting to drift.
- On step start, the worker calls
acquire(). Ifin_flight == bound, it waits or fails closed; it does not sneak a retry around the counter. -
acquire()incrementsgeneration, createslease_id, stores both, and incrementsin_flight. The request header carries both values. - The worker arms a local timer. On timer fire it calls
expire(lease_id), which incrementsgenerationagain and keepsin_flightuntil the outstanding call is accounted. - A completion is applied only when
completion.generation == desk.generationandlease_idis still live. Otherwise the desk rejects and recordsstale_completion. -
release()decrementsin_flightexactly once per acquired lease, including expired leases whose late body just arrived. Double release is a bug, not a cleanup. - The planner may mint the next tool call only after a committed apply. Rejected text never becomes memory.
Would I put this desk inside the model client, or in front of the planner? I would put it in front of the planner. The client already lies about cancellation. The planner is the thing that must not see duplicate steps.
Sequence model
The sequence below is the legal path. Notice that timeout does not create a second legal apply. It creates a new generation, which turns the late body into a reject.
sequenceDiagram
participant W as Worker
participant D as Admission desk
participant Q as Shared inference queue
participant L as Planner log
W->>D: acquire()
D-->>W: lease g=4
W->>Q: request(g=4)
Note over W: local timeout
W->>D: expire(g=4)
D-->>W: live generation now 5
Q-->>W: completion(g=4, text A)
W->>D: apply(g=4, text A)
D-->>L: REJECT stale generation
W->>D: acquire()
D-->>W: lease g=6
W->>Q: request(g=6)
Q-->>W: completion(g=6, text B)
W->>D: apply(g=6, text B)
D-->>L: COMMIT text B
W->>D: release()If your current trace cannot distinguish that reject from a commit, the invariant is already untestable. Can you point at one field in the log that proves generation 4 died before text A arrived?
Minimal simulator
The fixture below is a proposed discrete-event lab, not a production measurement and not a benchmark. I would run it first in process, then against a shared backend I do not own. MonkeyCode’s free model access and free server option are one such backend for that lab, because the whole point is a queue I cannot pause. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
from dataclasses import dataclass, field
from typing import List, Optional, Tuple
import random
@dataclass
class Desk:
bound: int
in_flight: int = 0
generation: int = 0
live: dict = field(default_factory=dict)
log: List[Tuple[str, int, str]] = field(default_factory=list)
def acquire(self) -> Optional[int]:
if self.in_flight >= self.bound:
self.log.append(("reject_acquire", self.generation, "bound"))
return None
self.generation += 1
self.in_flight += 1
self.live[self.generation] = True
self.log.append(("acquire", self.generation, "ok"))
return self.generation
def expire(self, g: int) -> None:
if g in self.live:
self.live.pop(g, None)
self.generation += 1 # fence late bodies
self.log.append(("expire", g, f"live->{self.generation}"))
def apply(self, g: int, text: str) -> bool:
if self.live.get(g) and g == max(self.live, default=g):
self.log.append(("commit", g, text))
return True
self.log.append(("reject_stale", g, text))
return False
def release(self, g: int) -> None:
if g in self.live:
self.live.pop(g, None)
if self.in_flight:
self.in_flight -= 1
self.log.append(("release", g, str(self.in_flight)))
@dataclass
class Event:
t: float
kind: str
g: int
text: str = ""
def simulate(seed: int = 7, bound: int = 1, timeout: float = 8.0) -> Desk:
rng = random.Random(seed)
desk = Desk(bound=bound)
q: List[Event] = []
now = 0.0
g = desk.acquire()
assert g is not None
service = rng.uniform(1.0, 20.0)
q.append(Event(now + timeout, "timeout", g))
q.append(Event(now + service, "complete", g, text=f"plan@{g}"))
applied = 0
while q:
q.sort(key=lambda e: e.t)
ev = q.pop(0)
now = ev.t
if ev.kind == "timeout":
desk.expire(ev.g)
g2 = desk.acquire()
if g2 is None:
continue
service2 = rng.uniform(1.0, 12.0)
q.append(Event(now + service2, "complete", g2, text=f"plan@{g2}"))
elif ev.kind == "complete":
if desk.apply(ev.g, ev.text):
applied += 1
desk.release(ev.g)
desk.log.append(("applied_count", applied, "denominator=injected_timeout_retry"))
return desk
if __name__ == "__main__":
d = simulate()
commits = [e for e in d.log if e[0] == "commit"]
stales = [e for e in d.log if e[0] == "reject_stale"]
print("commits", commits)
print("stale_rejects", stales)
print("in_flight_end", d.in_flight)
assert d.in_flight == 0
assert len(commits) <= 1
Run it as a command, not as a slide. If commits ever exceeds one after a timeout, the desk lost the generation fence. If in_flight is nonzero at the end, a release path leaked. That is the whole unit of validation for this article.
python admission_lease_sim.py
Injected failures and testable properties
I would inject four failure classes before I trusted the desk on a shared queue. First, a late original completion after expire, which must reject. Second, a retry that cannot acquire because bound=1, which must fail closed instead of sending. Third, two completions with the same generation, which must commit at most once. Fourth, a release that arrives twice, which must not drive in_flight negative. Which of those four does your client test today?
Properties I would freeze in the harness, with an explicit denominator.
- P1:
in_flightis always in[0, bound]after every event. - P2:
commitcount per eval-run step is0or1. - P3: every
commit.generationwas live at apply time. - P4: every
timeoutproduces either areject_stalefor the old generation or no second send. - Denominator: duplicate commits per 100 injected timeout-then-late-completion pairs.
- Acceptance rule: denominator result equals
0, and P1–P4 hold on every seed in a fixed set{1..200}.
If you cannot name the denominator, you do not have an eval. You have a dashboard.
Tradeoff table
| Design | What it preserves | What it spends | When it breaks |
|---|---|---|---|
| Unbounded retry | Apparent liveness | Queue depth, duplicate applies | Any timeout on a slow shared backend |
| Client semaphore, retry bypass | Happy-path bound | The invariant on the failure path | The exact event order in the opener |
| Generation-counted lease, fail closed | At-most-one apply per step | Extra reject / extra wait | Need more than bound true parallelism |
| Gateway admission with server cancel | Bound plus fewer late bodies | Control-plane coupling to the vendor | Backend that cannot cancel |
| Coalesce identical prompts | Load | Correctness if prompts are not actually identical | Tool-using planners with step-local state |
I would take the generation-counted lease on the planner side first, because I can ship it without vendor help. I would not take unbounded retry because it invents throughput that the queue cannot honor. Would I later push admission into the gateway? Yes, that is the next change, not the first one.
What I would change next
After the client desk holds the invariant, I would move cancellation into the request itself. A lease expire should send a cancel token the backend actually honors, so domain A and domain B stop drifting. I would also persist the desk to the eval-run log, so a crashed worker does not resurrect generation 4. I would not start with autoscaling stories, because scale without a bound just multiplies the stampede.
I would also keep model quality out of this control loop. Schema failures and empty bodies are protocol errors, but they are a different fence. This article only answers whether a completion is still allowed to exist. Mixing quality scores into admission is how people hide retry bugs inside “the model was bad today.”
Limitations, and who should not use this
This desk is the wrong tool if you already have a hard reservation system with server-side cancel and exactly-once apply. It is also the wrong tool for interactive single-user chats that never fan out and never retry. Do not use it as a billing system, a fairness scheduler, or a promise that free shared inference is production capacity. The simulator does not measure tokens, dollars, or latency percentiles; it measures whether stale generations can still commit.
If your planner must apply speculative partial streams, this design will reject work you wanted to keep. That is intentional. Partial text is uncommitted state, and I will not let it mint a tool call. If you need streaming UX, buffer tokens off to the side and commit only the final generation. Can your product live with that lag, or are you selling a live token firehose?
The free model access and free server option are useful here only as a shared queue I do not control. They do not replace a load model, and they do not make P1–P4 true by themselves. If you cannot run the fixture locally first, do not drag a remote queue into the story yet.
Closing counterexample
Picture this order: acquire g=4, send R1, expire g=4, acquire fails because bound=1, then R1 completes with a perfect JSON tool plan. Should the system reject that plan, replay a new acquire, or compensate by applying it anyway because the text looks valid? I would reject, then replay acquire only when in_flight allows it, and I would never compensate by trusting a late body. Which event order in your traces still applies that late plan, and which of those three verbs does your log actually implement?
Top comments (0)