Same model, different relay: a 5-metric checklist (with Python) before you pick an LLM API provider

If you buy access to the "same" model through different API relays/resellers, you quickly learn they are not the same. One times out at peak hours, another is fast at the median but terrible at P95, a third silently returns errors in a 200 body.

Here is the small checklist I use before committing to one, plus a Python script you can run yourself.

1. Compare like with like

Only compare providers serving the same model name. Mixing models turns a reliability question into a quality question.

2. Measure availability, not vibes

Count a request as failed if it times out, returns non-2xx, or returns an error payload. Report success rate with a sample size: "19/20" is not the same evidence as "950/1000".

3. Look at P95, not just the average

Users feel the slow tail. The mean hides it.

4. Write down when you measured

A relay that was fine at 03:00 can be broken at 21:00. Every number needs a timestamp.

5. Missing data stays missing

If you could not test a provider, show "no data". Do not fill the gap with a guess.

A minimal probe

import time, statistics, requests, datetime as dt

def probe(base_url, key, model, n=20, timeout=60):
    lat, ok = [], 0
    for _ in range(n):
        t0 = time.perf_counter()
        try:
            r = requests.post(f"{base_url}/v1/chat/completions",
                headers={"Authorization": f"Bearer {key}"},
                json={"model": model, "max_tokens": 16,
                      "messages": [{"role": "user", "content": "Reply with OK"}]},
                timeout=timeout)
            body = r.json()
            if r.ok and "choices" in body:
                ok += 1
                lat.append(time.perf_counter() - t0)
        except Exception:
            pass
    p95 = statistics.quantiles(lat, n=20)[18] if len(lat) >= 2 else None
    return {"checked_at": dt.datetime.utcnow().isoformat() + "Z",
            "success": f"{ok}/{n}", "p95_s": p95}

Only probe endpoints you are allowed to test, and keep n small so you do not hammer anyone.

Don't want to run it yourself?

I've been using Folkbench for exactly this: it ranks relays for the same model using the same published metrics (availability, response time, price), stamps each result with its check time, keeps missing data visible, and keeps commercial placements separate from the ranking order.

How do you choose between API relays? Curious what other metrics people track.