Measure Your Free-Tier LLM Endpoint's Real Rate Limit Before You Build an Admission Gate

The alert that didn't mean what we thought

Timeline from a Tuesday afternoon:

• 14:02 — p99 latency on our coding-agent job queue climbs from 4s to 41s.
• 14:04 — The on-call assumes "provider is degraded" and starts draining workers.
• 14:11 — Latency recovers, but 38 jobs were needlessly rejected during the drain.
• 14:30 — Post-incident check: the provider was fine. We were being rate-limited. The queue hadn't slowed; it was retrying against a 429 wall we'd never actually measured.

The contradiction: every dashboard said "throughput dropped," and the correct operational action was the opposite of draining — it was shedding load at admission. We had a drain runbook for provider degradation and nothing for self-inflicted rate-limit saturation, because nobody had ever probed what the free tier's actual limits were.

This article is the drill I built after that incident: measure the real rate-limit boundary of a free-tier LLM endpoint first, then derive admission thresholds from observation instead of guesses.

Setup and declared conditions

Topology:

[load generator] → [admission gate (local)] → [queue: Redis list] → [worker pool: 4] → [free-tier LLM endpoint]

Declared test conditions:

• Endpoint: a free-tier LLM API. I used MonkeyCode's free model access with their free server option as the test target, since a drill that costs nothing to run is a drill that actually gets run. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
• Workload: synthetic jobs, each one chat-style request of ~200 prompt tokens, asking for ~150 output tokens.
• Machine: single 4-core VM, workers and queue co-located. No production traffic involved.
• Duration: 20 minutes per ramp stage.

I am deliberately not stating the endpoint's rate limit as a number here, for two reasons: free-tier quotas change without notice, and the entire point of the drill is that you must measure it yourself on the day you depend on it. Any number I printed would be stale by the time you read this. Treat all script output below as clearly labeled expected shapes, not my measurements of anyone's current quota.

Step 1: Probe the limit, don't read the docs

Docs lag reality; headers don't. The probe ramps request concurrency and records the first sustained 429.

#!/usr/bin/env python3
# probe_limit.py — ramp concurrency until 429s appear, record where.
import asyncio, time, statistics, json
import aiohttp

ENDPOINT = "https://YOUR-FREE-TIER-ENDPOINT/v1/chat/completions"
API_KEY  = "..."  # never commit; read from env in real use
PAYLOAD  = {"model": "...", "messages": [{"role": "user", "content": "Summarize the word 'sre'."}], "max_tokens": 40}

async def one(session, results):
    t0 = time.monotonic()
    try:
        async with session.post(ENDPOINT, json=PAYLOAD, timeout=30) as r:
            results.append({
                "status": r.status,
                "latency": time.monotonic() - t0,
                "remaining": r.headers.get("x-ratelimit-remaining-requests"),
                "reset": r.headers.get("x-ratelimit-reset-requests"),
                "retry_after": r.headers.get("retry-after"),
            })
    except Exception as e:
        results.append({"status": -1, "latency": time.monotonic() - t0, "error": str(e)})

async def stage(concurrency, seconds=120):
    results = []
    async with aiohttp.ClientSession(headers={"Authorization": f"Bearer {API_KEY}"}) as s:
        end = time.monotonic() + seconds
        while time.monotonic() < end:
            await asyncio.gather(*[one(s, results) for _ in range(concurrency)])
    ok = [r for r in results if r["status"] == 200]
    limited = [r for r in results if r["status"] == 429]
    lat = [r["latency"] for r in ok] or [0]
    print(json.dumps({
        "concurrency": concurrency,
        "total": len(results), "ok": len(ok), "http429": len(limited),
        "errors": len(results) - len(ok) - len(limited),
        "p50_s": round(statistics.median(lat), 3),
        "p99_s": round(sorted(lat)[int(len(lat)*0.99)-1], 3) if len(lat) > 100 else None,
        "sample_remaining": ok[-1].get("remaining") if ok else None,
        "sample_retry_after": limited[0].get("retry_after") if limited else None,
    }))

async def main():
    for c in [1, 2, 4, 8, 16]:
        await stage(c)

asyncio.run(main())

Expected output shape (labeled expectation, not a measured claim):

{"concurrency": 1,  "total": ~120, "ok": ~120, "http429": 0,   "p50_s": ~1.2, "sample_remaining": "58"}
{"concurrency": 2,  "total": ~240, "ok": ~238, "http429": 2,   "p50_s": ~1.3, "sample_remaining": "3"}
{"concurrency": 4,  "total": ~300, "ok": ~180, "http429": 118, "p50_s": ~1.5, "sample_retry_after": "12"}

What you're extracting:

1. The concurrency step where http429 first exceeds ~1% of total. That is your observed admission ceiling, not the documented one.
2. Whether the endpoint exposes x-ratelimit-remaining / retry-after. If it does, your gate can be reactive; if it doesn't, it must be predictive (token bucket sized from the probe).
3. Whether latency degrades before 429s start. If p99 doubles at concurrency 2 while 429s start at 4, your real ceiling is 2 — the SLO breaks before the quota does.

Step 2: Derive the admission gate from the observation

Decision table, filled in from your probe output:

Probe finding	Gate behavior	Rationale
x-ratelimit-remaining header present	Gate reads the header from every response and refuses new admissions when remaining < 20% of observed window size	Cheap, reactive, self-correcting
No rate-limit headers	Local token bucket sized at 80% of the observed 429-onset rate	Predictive; 20% headroom absorbs clock skew and burst windows
Latency degrades before 429 onset	Gate on concurrency at the degradation knee, not the quota	The SLO is the binding constraint, not the provider's counter
retry-after present on 429	Honor it exactly; park the worker, mark endpoint saturated	Prevents the retry storm that caused my Tuesday incident

Minimal gate sketch (the piece that was missing on Tuesday):

class AdmissionGate:
    def __init__(self, max_concurrent, window_remaining_floor):
        self.sem = asyncio.Semaphore(max_concurrent)
        self.remaining_floor = window_remaining_floor
        self.last_remaining = None  # updated from response headers

    def admit(self, queue_age_s: float, deadline_slack_s: float):
        if self.last_remaining is not None and self.last_remaining < self.remaining_floor:
            return False, "provider-window-exhausted"
        if self.sem.locked() and deadline_slack_s < queue_age_s:
            return False, "no-slack-left"  # job can't finish before its deadline anyway
        return True, "admit"

Note the combination: quota signal (remaining) OR deadline-slack signal. Either one alone produces Tuesday-style mistakes — draining on a healthy provider, or admitting jobs that can never make their deadline.

Step 3: Failure injection, locally, before production

You can't ask a free tier to rate-limit you on demand at 3 AM. So fake it:

• Run a local stub (e.g., a 30-line aiohttp server) that returns 429 with retry-after: 10 after N requests, and point the worker pool at it via env var.
• Verify: workers park instead of retrying, the gate flips to reject-new, queue age rises, and your alert fires on queue age vs. deadline slack, not on "throughput dropped."
• Verify the recovery: after the stub's window resets, admissions resume without manual intervention.

The metric contradiction to encode in the alert rule: throughput down + 429 rate up + provider health OK = shed at admission, do not drain.

Cleanup and rollback

• The gate is a sidecar/library, not a schema change: rollback = redeploy the worker image without the gate, queue semantics unchanged.
• Probe script leaves no state; the only residue is whatever the free tier counted against your quota — keep ramps short and infrequent (run the probe on quota-change suspicion or monthly, not per-deploy).
• If the gate over-rejects in production, lower remaining_floor first (config), not the concurrency cap (redeploy).

Limitations and who shouldn't do this

• Free tiers are not capacity plans. A measured free-tier limit tells you how to fail gracefully today, not what you can promise next quarter. If you have a real SLO with real users, this drill tells you when to reject, which may not be an acceptable answer at all.
• Point-in-time measurement. Quotas, window sizes, and header behavior change; re-run the probe when the provider announces changes and after any unexplained 429 cluster.
• Single-endpoint assumption. If you fan out across providers, the gate needs per-endpoint buckets, which this sketch doesn't have.
• Don't run the probe against a shared production quota. That's how drills become incidents.

The threshold question I'll leave you with

My gate rejects when observed remaining < 20% or when deadline slack is gone. But there's a third candidate I haven't settled: rejecting on queue age exceeding the measured retry-after window, on the theory that a job older than one penalty window is already poisoned. If you've run admission control against a rate-limited dependency, which signal actually fired first in your incidents — quota, slack, or queue age? If you want a zero-cost target to run this drill against, MonkeyCode's free server option is one place to start; the script above doesn't care which endpoint you point it at.