Test Your Model Fallback Path Locally Before the Primary Endpoint Saturates

Every time a new open model release trends, the same operational question lands on my desk: if we route traffic to it and the endpoint saturates, what exactly happens to the requests already in flight? Most teams can answer for stateless web traffic. Far fewer can answer for LLM serving, where a "failed" request may have already burned 40 seconds of deadline slack.

This post is a local drill for that scenario: a primary model endpoint, a cheaper fallback endpoint, a queue in front of both, and a deliberate saturation event. Everything runs on a laptop or a small free server, and every number below is either an actual command output or clearly labeled expected output.

Why fallback routing is the risk nobody tests

The typical pattern after a hyped model release:

1. Team points a staging service at the new endpoint.
2. Latency looks great at 2 req/s.
3. Someone promotes it, traffic arrives, provider-side rate limiting kicks in, and requests start piling up.
4. The "fallback" config exists but has never been exercised under load — so the first real test of the retry/failover logic is the incident itself.

The failure mode I care about is not "the endpoint is slow." It is double-spend: a request that times out at the gateway, gets retried against the fallback, while the primary still completes it — now you paid for two generations and possibly returned a duplicate side effect. Fallback routing without idempotency and deadline accounting is a queue-eater.

Topology

┌────────┐   ┌─────────┐   ┌──────────────┐
│ loadgen│──▶│ gateway │──▶│ primary:8001 │  (saturating, artificial cap)
└────────┘   │  :8080  │   └──────────────┘
             │    │    │   ┌──────────────┐
             │    └──────▶│ fallback:8002│  (small, always healthy)
             └─────────┘   └──────────────┘
                  │
             queue depth, per-request deadline,
             route decision log (stdout JSON)

Two tiny mock servers stand in for real model endpoints. That is deliberate: the drill tests your routing and admission logic, not anyone's model quality. Point the same gateway at real free-tier endpoints afterwards and the mechanics are identical.

The gateway

Save as gateway.py (Python 3.11+, only stdlib + httpx):

import asyncio, json, time, uuid
import httpx

PRIMARY = "http://127.0.0.1:8001/v1/chat"
FALLBACK = "http://127.0.0.1:8002/v1/chat"
DEADLINE_MS = 8000          # end-to-end budget per request
PRIMARY_TIMEOUT_MS = 3000   # give up on primary after this

async def call(client, url, req_id, prompt, deadline):
    remaining = deadline - time.monotonic()
    if remaining <= 0:
        return {"route": "shed", "reason": "deadline_exhausted"}
    try:
        r = await client.post(
            url,
            json={"prompt": prompt, "request_id": req_id},  # idempotency key
            timeout=remaining,
        )
        return {"route": url, "status": r.status_code,
                "body": r.json()}
    except (httpx.TimeoutException, httpx.ConnectError) as e:
        return {"route": url, "error": type(e).__name__}

async def handle(prompt: str):
    req_id = str(uuid.uuid4())
    deadline = time.monotonic() + DEADLINE_MS / 1000
    async with httpx.AsyncClient() as client:
        first = await asyncio.wait_for(
            call(client, PRIMARY, req_id, prompt, deadline),
            timeout=PRIMARY_TIMEOUT_MS / 1000,
        ) if True else None
        # NOTE: asyncio.wait_for cancels the client call, but the
        # primary server may STILL be processing. The request_id
        # is what lets the primary dedupe a late retry.
        log = {"request_id": req_id, "attempts": [first]}
        if "error" in first or first.get("status", 500) >= 500:
            second = await call(client, FALLBACK, req_id, prompt, deadline)
            log["attempts"].append(second)
        log["elapsed_ms"] = None  # filled by caller
        print(json.dumps(log), flush=True)
        return log

Mock endpoints (primary.py, fallback.py) — the primary enforces a hard concurrency cap of 2 and queues everything else for 6 seconds, simulating provider-side saturation:

# primary.py
import asyncio, json
from aiohttp import web

sem = asyncio.Semaphore(2)
seen = set()  # idempotency: request_ids already completed

async def chat(req):
    body = await req.json()
    rid = body["request_id"]
    if rid in seen:
        return web.json_response({"deduped": True, "id": rid})
    async with sem:
        await asyncio.sleep(6)   # saturated: every request is slow
        seen.add(rid)
        return web.json_response({"model": "primary", "id": rid})

app = web.Application()
app.router.add_post("/v1/chat", chat)
web.run_app(app, port=8001)

The fallback is the same shape with asyncio.sleep(0.4) on port 8002.

Declared workload and expected output

Workload, declared up front so results are interpretable:

• 40 requests, arrival rate 10 req/s (4 seconds of offered load)
• Deadline budget 8000 ms end-to-end, primary attempt capped at 3000 ms
• Primary capacity: 2 concurrent × 6 s service time ≈ 0.33 req/s sustainable

Little's law says the primary will absorb ~2 requests in the window and everything else must fall back or shed. Expected output (labeled — rerun on your machine before quoting): roughly 2 requests logged with a single primary attempt, ~36–38 with two attempts ending at fallback, and 0 sheds, because the fallback's 0.4 s service time keeps total latency under the 8 s deadline.

What you should grep for in the gateway's JSON log:

python gateway_load.py | jq -r '.attempts[-1].route' | sort | uniq -c
# expected:  ~2 http://127.0.0.1:8001/v1/chat
#           ~38 http://127.0.0.1:8002/v1/chat

And the critical idempotency check — no request_id should appear with two successful generations:

python gateway_load.py | jq -r 'select(.attempts | length == 2) | .request_id' \
  | sort | uniq -d | wc -l
# must be 0 (or every duplicate must show "deduped": true server-side)

Failure injection

Run the drill three times, changing one thing:

1. Baseline (above): validates the happy failover path.
2. Fallback also degraded: set fallback sleep to 7 s. Now the deadline math matters — expected result is that late arrivals shed with deadline_exhausted instead of double-queuing. If you see requests completing in 14 s, your deadline accounting is decorative.
3. Primary hangs without erroring: make the primary accept the connection and sleep 30 s. This is the nastiest real-world case (provider overloaded but not refusing). Your 3 s wait_for must fire; if the gateway instead waits for the TCP timeout, the fallback never engages and the queue silently ages.

Telemetry fields worth keeping

Whatever your real stack is, the route-decision log needs, per request: request_id, offered_at, first_route, first_outcome, fallback_outcome, elapsed_ms, deadline_ms, and queue_wait_ms. The two fields people skip — deadline_ms and queue_wait_ms — are exactly the ones that tell you, during an incident, whether to shed load or whether you still have slack.

Thresholds: what triggers failover in production

The drill forces you to pick a number and defend it. My ordering of candidate signals:

• Deadline slack (deadline minus elapsed, minus p95 fallback latency): best signal, because it is denominated in the thing the user experiences. Fail over when slack < fallback p95.
• Queue age of oldest request: good proxy, works without per-request deadlines.
• Utilization / concurrency: weakest alone — 100% concurrency on a fast endpoint is fine; 100% on a saturated one is a meltdown.

If you can only instrument one, instrument deadline slack.

Running this on free infrastructure

The mocks and gateway fit comfortably on a small free VM. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Their free model access and free server option are a reasonable place to run the gateway-plus-real-endpoint variant of this drill — swap the mock URLs for actual model endpoints and the failover, deadline, and idempotency logic carries over unchanged. If you want to try it, the drill above is a self-contained starting point; the load script is under 60 lines.

One honest limitation of free tiers for this specific drill: provider-side rate limits may be stricter than the saturation you are simulating, which is actually fine — it exercises run 2 (both paths degraded) for free. Just do not extrapolate free-tier throughput numbers to production capacity planning.

Who should not use this approach

• Teams whose requests are not idempotent and cannot add an idempotency key server-side. Failover without dedup converts latency problems into correctness problems.
• Streaming responses with mid-stream failover expectations. This drill covers request-level failover; mid-stream resume is a different, harder problem.
• Anyone using it to benchmark a specific model. The mocks exist precisely so the routing logic is tested independently of model behavior.

Cleanup and rollback

The drill leaves no state behind if you do three things: kill the mock servers (pkill -f primary.py), truncate the seen-request-id set (it is in-memory, so restart does it), and — in a real deployment — keep the fallback route behind a config flag so rollback is FALLBACK_ENABLED=false plus a reload, not a deploy. In the incident version of this, rollback means draining: stop admitting new requests to the primary, let in-flight requests finish or shed against their deadline, then cut over. Practice the drain locally the same way — set arrival rate to 0 mid-run and watch the log for stragglers past their deadline.

The drill takes an afternoon to build. The incident it replaces takes a weekend.