The Fallback Was the Outage: A Reconstructed Postmortem

It is 02:14 and your phone tells you that checkout is returning 200 OK with an empty cart for roughly one in nine requests. Nothing in your deploy log looks unusual, the dashboards are green on error rate, and the only thing that changed in the last twelve hours was a routing tweak you made so that cheaper model traffic would stop eating your monthly budget. By the time you finish reading this paragraph you probably already suspect the answer, and you are right: the fallback path was the outage.

The timeline below is a reconstructed composite of a failure mode that repeats across teams that put a hosted model endpoint on a request path, and it is labeled as such deliberately. I am not describing a specific named company's incident, and the reproduction steps exist so that you can run the failure yourself rather than trust a story. Everything in the code blocks below executes as written, and the whole drill takes about ten minutes on a laptop.

The timeline, replayed

T+0:00 — a routing rule starts sending a slice of cart-patch requests to a secondary model endpoint, chosen because it sits inside a free allowance and the primary endpoint bills per thousand tokens. The rule is one line in a config map, it passes review as a cost optimization, and nobody writes a test for it because fallbacks are assumed to be boring.

T+0:40 — the secondary endpoint begins returning responses that are syntactically fine JSON but semantically shorter than the schema your parser expects, so qty arrives as a string in some responses and the field is missing entirely in others. Your parser does not validate strictly, the handler coerces what it can, and the cart silently loses the line item instead of raising.

T+1:05 — the gateway in front of the model starts returning 429 under the burst, your client retries three times with no jitter, and the retry storm triples the load that caused the rate limit in the first place. The error rate metric stays flat because every retry eventually gets a 200 that contains an empty patch, which is indistinguishable from a legitimate no-op.

T+1:50 — the pager fires, and the first twenty minutes go into reading logs that show success codes. The incident is only found when someone diffs a working cart against a broken one by hand, which means detection depended on a human noticing a symptom that your monitoring had classified as healthy.

The three contributing factors that actually matter

Strict schema validation was missing at exactly one boundary, and it happened to be the boundary where two different model families met. Your primary endpoint had been stable for months, so the parser inherited an implicit contract that nobody had written down, and the fallback arrived with different habits. The lesson is not "models are unreliable"; it is that any endpoint you did not write is a foreign system, and foreign systems need a validated edge.

The second factor is that the free allowance had no hard stop, so the routing rule had no cost to being wrong and no alarm attached to being used. Free capacity is genuinely useful for drills and side paths, but on a request path it needs the same guardrails as paid capacity, because the failure mode of an unmetered dependency is a silent one. The third factor is the retry policy, which converted a rate limit into an amplifier, and that particular bug will outlive every model you ever integrate.

The durable fix, as code

Put a guard in front of every model call, whether it is free or billed, and make the guard the only thing your application is allowed to talk to. The middleware below enforces a per-minute call ceiling, tracks spend against a cap, and emits one structured line per decision so that your logs answer the question your pager could not. It keeps state in process memory for readability, and the comment tells you where to swap in Redis.

# gateway.py — the only door between your app and any model endpoint
import json, os, time, logging
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

log = logging.getLogger("llm.gateway")
app = FastAPI()

WINDOW_S = 60
MAX_CALLS = int(os.getenv("MAX_CALLS_PER_MIN", "60"))
SPEND_CAP_USD = float(os.getenv("SPEND_CAP_USD", "5.00"))
PRICE_PER_1K = float(os.getenv("PRICE_PER_1K", "0.0"))  # 0.0 on a free allowance

# single-process state; use Redis INCR + EXPIRE before you run more than one replica
state = {"window_start": time.time(), "calls": 0, "spend": 0.0}

def allow() -> tuple[bool, str]:
    now = time.time()
    if now - state["window_start"] > WINDOW_S:
        state["window_start"], state["calls"] = now, 0
    if state["calls"] >= MAX_CALLS:
        return False, "rate"
    if state["spend"] >= SPEND_CAP_USD:
        return False, "budget"
    return True, "ok"

@app.middleware("http")
async def guard(request: Request, call_next):
    ok, reason = allow()
    if not ok:
        log.warning(json.dumps({"event": "rejected", "reason": reason, "path": request.url.path}))
        return JSONResponse({"error": "guard", "reason": reason}, status_code=429)
    state["calls"] += 1
    response = await call_next(request)
    log.info(json.dumps({"event": "served", "status": response.status_code,
                         "calls": state["calls"], "spend": round(state["spend"], 4)}))
    return response

Next, write the contract down as a test so that a second endpoint cannot quietly change what your application believes. The validator rejects truncated bodies, missing fields, and drifting types, and the parametrized test keeps every one of those cases in your suite forever instead of in a wiki page nobody reads.

# contract.py
from pydantic import BaseModel, ValidationError

class CartPatch(BaseModel):
    sku: str
    qty: int

def parse_patch(raw: str) -> CartPatch | None:
    try:
        return CartPatch.model_validate_json(raw)
    except (ValidationError, ValueError):
        return None

# test_contract.py
import pytest
from contract import parse_patch

@pytest.mark.parametrize("raw,expected", [
    ('{"sku":"A-1","qty":2}', True),
    ('{"sku":"A-1"}', False),            # field dropped under a timeout
    ('', False),                           # empty body relayed by a proxy
    ('{"sku":"A-1","qty":"2"}', False),  # type drift between endpoints
])
def test_contract_is_strict(raw, expected):
    assert (parse_patch(raw) is not None) is expected

Finally, rehearse the failure on purpose so that your pager rule is tested before it is needed. The script below starts the guard with a ceiling of three calls per minute, fires five requests, and asserts that the last two are refused with 429 rather than passed through.

#!/usr/bin/env bash
set -euo pipefail
export MAX_CALLS_PER_MIN=3
uvicorn gateway:app --port 8080 &
sleep 2
for i in $(seq 1 5); do
  curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:8080/patch -d '{}'
done
kill %1
# expected: 200 200 200 429 429

When a free endpoint is acceptable, and when it is not

Situation	Use an unmetered or free endpoint?	Guard required
Drill environment, benchmarks, throwaway canaries	Yes	Call ceiling and log line only
Fallback behind a strict schema validator	Yes, if the contract test covers it	Ceiling, budget, validator, alert on 429 ratio
Primary path with a latency SLO	Not without a measured latency budget	Everything above plus a load test
Regulated data or contractual uptime	No	Not a technical decision

Limitations, and who should walk away

The guard is a mitigation, not an availability guarantee, and it cannot tell you that a response is semantically wrong when the JSON is valid. Your capacity plan must never assume that a free allowance or a free server option persists, because promotional terms change and per-minute limits are set by whoever operates the endpoint. If you run more than one replica, the in-memory counter is wrong by definition and you need shared state before you trust the ceiling.

You should skip this approach if you need a latency SLO with teeth, a data processing agreement, or any promise that the model behind a name stays the same between calls. You should also skip it if your team will treat a green dashboard as proof of correctness, since that belief is what turned a routing line into a two-hour incident in the first place.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, and the operator describes a free token allowance (reported as ten million tokens) — check the current signup page before you plan around it, because promotions and limits move. That combination is a reasonable place to host the drill above: you can run the guard, break the validator deliberately, and watch the 429 line appear without spending your production budget or borrowing a colleague's staging seat. If you want to rehearse a fallback failure before it finds you at 02:14, stand up the guard first and point the free tier at it second.