Architecture Review: Give a Free-Tier Inference Endpoint an Admission Envelope, Not a Retry Storm

Consider the event order below, which any client with blind retries can produce against a shared inference endpoint. Nothing in it is exotic, and that is exactly the problem.

t0  worker-1 sends request A            (service time unknown, deadline 30s)
t1  worker-1 gives up at 30s            (A is still executing on the server)
t2  worker-1 retries as A'              (no idempotency key, no observation)
t3  worker-2 sends B                    (believes capacity is free)
t4  endpoint returns 429 for A' and B   (server is at its limit, not broken)
t5  both workers sleep ~1s, then retry  (synchronized herd)
t6  endpoint returns 429 again          (queue never drains)

The endpoint was never down. My client simply had no theory of its capacity, so every timeout became a new admission decision made by a process that knew nothing. The invariant I want is narrow and testable: a request may enter the system only if its projected wait fits inside its deadline, and a retry may never outrun the observed service rate. The typical implementation preserves neither half.

This is an architecture review of that boundary — the constraint set, the data flow, the failure domains, and the parts I would change next.

Assumptions (declare them before you measure anything)

• A1 Service time is roughly stationary over a probe window of ten minutes; if it is not, your envelope is a snapshot, not a property.
• A2 The endpoint publishes no concurrency limit, so the limit must be inferred from observed rejections and latency.
• A3 Free-tier capacity is shared and can shrink without notice; treat any measured ceiling as an upper bound that decays.
• A4 Work is replay-safe, or every request carries an idempotency key the downstream respects.
• A5 The deadline belongs to the caller. The server has no idea what your request was for.

If A1 or A4 is false, stop here and fix that first; admission control on top of non-replayable work just produces faster corruption.

Step 1: Measure the envelope instead of guessing it

Free model access and a free server option are useful precisely because they let you run this probe without a procurement ticket. The offer I am describing is MonkeyCode's free model access plus a free server option; the project advertises a free token allowance (the operator's published figure at the time of writing was on the order of ten million tokens), and both terms can change, so read the console rather than this paragraph. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Here is the probe. It sweeps concurrency, records the status mix, and refuses to average away the tail.

# probe.py — characterize the endpoint; do not trust a single concurrency level
import asyncio, json, os, statistics, time
import httpx

ENDPOINT = os.environ["MC_ENDPOINT"]
KEY      = os.environ["MC_KEY"]

async def one(client, sem, i, out):
    async with sem:
        t0 = time.perf_counter()
        try:
            r = await client.post(
                ENDPOINT, timeout=90,
                headers={"Authorization": f"Bearer {KEY}"},
                json={"messages": [{"role": "user", "content": "reply with: ok"}],
                      "max_tokens": 8},
            )
            out.append({"i": i, "status": r.status_code,
                        "ms": (time.perf_counter() - t0) * 1000,
                        "retry_after": r.headers.get("retry-after")})
        except Exception as e:
            out.append({"i": i, "status": type(e).__name__,
                        "ms": (time.perf_counter() - t0) * 1000, "retry_after": None})

async def sweep(levels=(1, 2, 4, 8, 16), per_level=40):
    async with httpx.AsyncClient() as client:
        for level in levels:
            sem, out = asyncio.Semaphore(level), []
            await asyncio.gather(*(one(client, sem, i, out) for i in range(per_level)))
            ok = [o["ms"] for o in out if o["status"] == 200]
            mix = {}
            for o in out:
                mix[str(o["status"])] = mix.get(str(o["status"]), 0) + 1
            print(json.dumps({
                "level": level, "n": len(out), "ok": len(ok),
                "p50_ms": round(statistics.median(ok), 1) if ok else None,
                "p95_ms": round(sorted(ok)[int(len(ok) * 0.95) - 1], 1) if len(ok) > 20 else None,
                "status_mix": mix,
            }))

asyncio.run(sweep())

Run it. The number you want is not the mean; it is the level at which the rejection mix stops being zero while p50 stays flat. That level is your measured service rate mu, in requests per second, and everything downstream depends on it.

Step 2: Turn the measurement into a concurrency budget

Little's law gives you the only budget that matters: L = lambda * W. At saturation lambda = mu, so the in-flight budget is throughput times the latency you are willing to pay for.

class Envelope:
    """Concurrency budget derived from a measured service rate."""
    def __init__(self, mu_rps: float, deadline_s: float, safety: float = 0.6):
        self.mu, self.deadline = mu_rps, deadline_s
        self.cap = max(1, int(mu_rps * deadline_s * safety))

    def projected_wait(self, inflight: int) -> float:
        return inflight / self.mu          # seconds, saturated server

    def admit(self, inflight: int, service_p50: float, remaining: float):
        if inflight >= self.cap:
            return False, "cap"
        if self.projected_wait(inflight) + service_p50 > remaining:
            return False, "deadline"
        return True, "ok"

The safety factor is not timidity; it is your only defense against A3, because a shared free tier can lose capacity between two probe runs. I set it at 0.6 and I would go lower before I went higher.

Step 3: Classify failures, because they demand different actions

A single except Exception: retry() collapses five distinct failure domains into one response, and four of those responses are wrong.

Observed	Failure domain	Correct action
429 with Retry-After	Shedding by the server	Wait at least that long, then re-admit through the gate
429 without header	Shedding, no guidance	Back off on the measured mu, never on the retry count alone
Timeout / ReadTimeout	Unknown outcome on the server	Replay only with an idempotency key
5xx	Server-side fault	Bounded retry with jitter, decrement budget
Truncated stream	Partial success	Discard, never stitch; the artifact is unusable

That last row is the one people miss. A stream that ends early looks like a successful request with short output, and downstream code happily persists half an answer.

Step 4: Make the retry floor a tested property, not a habit

The classic bug is multiplicative jitter that lands below the floor the server asked for. Write the delay function so the floor is structural.

import random

class RetryFloor:
    def __init__(self, base=1.0, cap=30.0):
        self.base, self.cap = base, cap

    def delay(self, attempt: int, retry_after=None) -> float:
        floor = min(self.cap, self.base * (2 ** attempt))
        if retry_after is not None:
            floor = max(floor, float(retry_after))
        return floor + random.uniform(0, floor * 0.25)   # jitter additive, cannot go under

Now the properties are assertable: delay >= retry_after, delay >= base * 2**attempt, and delay is non-decreasing in attempt until the cap. If you cannot state those three as tests, you do not have a retry policy; you have a hope.

Step 5: Inject the failures before production does

A fake endpoint with scripted behavior turns this from opinion into a check. The test plan below is the artifact I would actually merge.

1. Herd test — return 429 for 30% of calls, assert the gate never exceeds cap and no request exceeds its deadline by more than one service time.
2. Slow-drain test — multiply service time by 3 mid-run, assert the gate shrinks admissions instead of queueing.
3. Truncation test — close the stream at 60% of tokens, assert the artifact is marked unusable and never persisted.
4. Replay test — time out after the server has already executed, replay with the same idempotency key, assert exactly one side effect.
5. Floor test — property-check RetryFloor over 10,000 random (attempt, retry_after) pairs.

Acceptance rule: over a 30-minute run at the chosen cap, p99 queue wait must stay under the deadline, the 429 rate must stay under one percent, and the floor test must have zero violations. Fail any of the three and halve the cap before you touch anything else.

What I would change next

The honest weakness of this design is that mu is a single scalar standing in for a multi-dimensional system. Two lanes with different prompt sizes share one budget, so a burst of long generations starves short ones even though the request count looks fine. The fix I would build next is a two-class gate: measure mu per size bucket, reserve a floor of capacity for the short class, and let the long class consume only the remainder. That is a small change to Envelope and a large change to tail behavior.

Limitations, and who should not use this

Do not adopt this if you need a hard p99 contractual SLO — a shared free tier gives you a measurement, not a guarantee, and the envelope can only shed work, not create capacity. Do not adopt it for non-replayable side effects without idempotency keys, and do not adopt it if your evaluation depends on a stable model identity that a free tier may not promise. Teams without any retry hygiene should fix that first; admission control in front of a retry storm just makes a tidier storm.

If you want a place to run the probe without booking infrastructure, MonkeyCode's free model access and free server option are a reasonable starting substrate — bring your own harness, and replace every number in this template with your own output.

So here is the question I would put to any reviewer of this design: which event order breaks your envelope — a Retry-After shorter than your own floor, a truncation that reports success, or a capacity drop between probe runs? And when it breaks, should the gate reject, replay with a key, or compensate downstream?