DEV Community: Seven

"OpenAI-compatible" is a spectrum, not a boolean — here's an 11-check conformance suite

Seven — Thu, 10 Sep 2026 14:46:31 +0000

Two endpoints both say "OpenAI-compatible" on the tin. You point your app at the first one and everything works. You point it at the second one and everything works too — for about a week. Then a streamed tool call comes back with arguments split across chunks in a way your accumulator didn't expect, your JSON parser throws inside a retry loop, and the retry loop hammers the endpoint because the error body doesn't have the field your backoff code reads.

Nothing lied to you. "OpenAI-compatible" was never a boolean. It's a surface area, and every implementation covers a different subset of it.

I maintain a gateway, so I read a lot of compatibility bug reports. Disclosure: I work on daoxe, a multi-model gateway that speaks the OpenAI protocol among others. The checklist below is deliberately written so you can run it against us and against anyone else, and the script at the end doesn't know or care which endpoint you point it at. If it makes us look bad on a check, that's the correct output.

The nine surfaces

1. Streaming deltas. The reference behaviour is specific: the first chat.completion.chunk carries choices[0].delta.role = "assistant" and usually empty content, middle chunks carry delta.content fragments, the last content-bearing chunk carries finish_reason, and the stream terminates with a literal data: [DONE] line. Implementations diverge on all four. Some never send the role chunk, which breaks clients that use it to open a message. Some put finish_reason on a trailing chunk with an empty delta. Some omit [DONE] entirely and just close the connection, which is fine for a client that reads to EOF and fatal for one that blocks waiting for the sentinel.

2. Tool / function calling. Two traps here. First, function.arguments is a JSON-encoded string, not an object — endpoints that "helpfully" return a parsed object break every client that calls json.loads on it. Second, in streaming mode, tool calls arrive as fragments that you reassemble by the index field, with id and function.name typically only present on the first fragment. An endpoint that re-sends id on every fragment, or that omits index when there's only one call, will work with your naive accumulator and fail the moment a model emits two parallel calls. Also check that finish_reason is tool_calls and not stop — agent loops branch on that value.

3. response_format. Three tiers, and they're commonly conflated: no support, {"type": "json_object"} (valid JSON, any shape), and {"type": "json_schema", "json_schema": {..., "strict": true}} (constrained decoding against your schema). The dangerous middle case is an endpoint that accepts the parameter and ignores it. You get prose with a code fence around it, your parser fails one request in fifty, and it looks like a model quality problem.

4. logprobs / top_logprobs. Usually the first thing a proxy layer drops, because almost nobody notices. If you do classification by comparing token probabilities, or you use logprobs for confidence gating, this is load-bearing and you should test it explicitly.

5. temperature and sampling params. Reasoning-style models reject temperature outright on some upstreams, accept-and-ignore it on others, and honour it on a third set. All three are defensible; not knowing which one you're on is not. The same applies to top_p, presence_penalty, and max_tokens vs max_completion_tokens.

6. stop sequences. Does the endpoint honour an array of stop strings? Is the stop sequence included in or excluded from the returned content (the reference excludes it)? Does finish_reason come back as "stop"? A surprising number of shims implement stop by post-truncating the full completion, which means you pay for tokens you never see — and the usage numbers will show it.

7. Usage accounting. Non-streaming responses should carry usage.prompt_tokens, completion_tokens, total_tokens. Streaming responses only include usage if you pass stream_options: {"include_usage": true}, and then it arrives in a final chunk with an empty choices array — a shape that crashes clients which assume choices[0] always exists. If you do cost attribution per request, also check whether cached-prompt and reasoning-token breakdowns survive.

8. Error-body shape. Every retry layer you've ever written parses this, and nobody tests it. The reference is {"error": {"message": ..., "type": ..., "param": ..., "code": ...}} with an HTTP status that matches the semantics. Real-world variations: a 200 with an error object in the body (fatal for retry logic — you'll cheerfully return an error string to your user), an HTML error page from an intermediate proxy, or a 500 where a 400 belonged, which turns a permanent client error into an infinite retry storm.

9. /v1/models fidelity. Does it exist, does it return {"object": "list", "data": [{"id": ...}]}, and — the part that matters — does it list the IDs your key can actually call? A catalogue endpoint that returns everything the vendor sells, rather than everything your key is entitled to, is worse than no catalogue at all, because you'll build CI checks on top of it.

The script

Stdlib only, Python 3.8+, no install. It runs eleven checks and prints a verdict table.

#!/usr/bin/env python3
"""compat_probe.py — how OpenAI-compatible is this endpoint, really?

    export LLM_BASE_URL=https://api.example.com/v1
    export LLM_API_KEY=sk-...
    python3 compat_probe.py --model <exact-model-id>
"""
import argparse, json, os, sys, urllib.error, urllib.request

BASE = os.environ.get("LLM_BASE_URL", "").rstrip("/")
KEY = os.environ.get("LLM_API_KEY", "")
OUT = []


def _open(path, payload=None, method="GET", timeout=90):
    data = json.dumps(payload).encode() if payload is not None else None
    req = urllib.request.Request(BASE + path, data=data, method=method)
    req.add_header("Authorization", "Bearer " + KEY)
    if data is not None:
        req.add_header("Content-Type", "application/json")
    try:
        return urllib.request.urlopen(req, timeout=timeout)
    except urllib.error.HTTPError as exc:   # still a readable file object
        return exc


def call(path, payload=None, method="GET"):
    resp = _open(path, payload, method)
    raw = resp.read().decode("utf-8", "replace")
    try:
        return resp.getcode(), json.loads(raw)
    except ValueError:
        return resp.getcode(), raw


def stream(payload):
    payload = dict(payload, stream=True)
    resp = _open("/chat/completions", payload, "POST")
    chunks, saw_done = [], False
    for line in resp:
        line = line.decode("utf-8", "replace").strip()
        if not line.startswith("data:"):
            continue
        body = line[5:].strip()
        if body == "[DONE]":
            saw_done = True
            break
        try:
            chunks.append(json.loads(body))
        except ValueError:
            pass
    return chunks, saw_done


def record(name, ok, detail):
    OUT.append((name, "PASS" if ok is True else "FAIL" if ok is False else "PARTIAL", detail))


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--model", required=True)
    model = ap.parse_args().model
    ask = {"model": model, "messages": [{"role": "user", "content": "Say hi."}],
           "max_tokens": 24}

    # 1 — catalogue
    code, body = call("/models")
    ids = [m.get("id") for m in body.get("data", [])] if isinstance(body, dict) else []
    record("models_endpoint", code == 200 and bool(ids),
           f"HTTP {code}, {len(ids)} ids, target listed: {model in ids}")

    # 2 — basic completion + model echo + usage
    code, body = call("/chat/completions", ask, "POST")
    msg = (body.get("choices") or [{}])[0].get("message", {}) if isinstance(body, dict) else {}
    usage = body.get("usage") if isinstance(body, dict) else None
    record("basic_chat", bool(msg.get("content")), f"HTTP {code}")
    record("model_echo", isinstance(body, dict) and body.get("model") == model,
           f"asked {model!r}, got {body.get('model')!r}" if isinstance(body, dict) else "n/a")
    record("usage_fields", bool(usage and usage.get("total_tokens") is not None), str(usage))

    # 3 — streaming shape
    chunks, done = stream(ask)
    role = any((c.get("choices") or [{}])[0].get("delta", {}).get("role") for c in chunks)
    fin = any((c.get("choices") or [{}])[0].get("finish_reason") for c in chunks)
    record("stream_shape", bool(chunks) and role and fin and done,
           f"{len(chunks)} chunks, role_chunk={role}, finish_reason={fin}, [DONE]={done}")

    # 4 — usage on stream
    chunks, _ = stream(dict(ask, stream_options={"include_usage": True}))
    record("stream_usage", any(c.get("usage") for c in chunks),
           "final usage chunk present" if any(c.get("usage") for c in chunks) else "absent")

    # 5 — tool calling
    tool = {"type": "function", "function": {"name": "get_weather", "parameters": {
        "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}
    code, body = call("/chat/completions", dict(
        ask, messages=[{"role": "user", "content": "Weather in Osaka? Use the tool."}],
        tools=[tool], tool_choice="auto"), "POST")
    choice = (body.get("choices") or [{}])[0] if isinstance(body, dict) else {}
    calls = choice.get("message", {}).get("tool_calls") or []
    args_is_str = bool(calls) and isinstance(calls[0].get("function", {}).get("arguments"), str)
    record("tool_calls", bool(calls) and args_is_str and choice.get("finish_reason") == "tool_calls",
           f"n={len(calls)} arguments_is_string={args_is_str} finish={choice.get('finish_reason')}")

    # 6 — json mode
    code, body = call("/chat/completions", dict(
        ask, messages=[{"role": "user", "content": "Return {\"ok\": true} as JSON."}],
        response_format={"type": "json_object"}), "POST")
    text = ((body.get("choices") or [{}])[0].get("message", {}).get("content")
            if isinstance(body, dict) else "") or ""
    try:
        json.loads(text); parsed = True
    except ValueError:
        parsed = False
    record("json_object", code == 200 and parsed, f"HTTP {code}, parses={parsed}")

    # 7 — strict schema
    schema = {"type": "json_schema", "json_schema": {"name": "r", "strict": True, "schema": {
        "type": "object", "properties": {"ok": {"type": "boolean"}},
        "required": ["ok"], "additionalProperties": False}}}
    code, _ = call("/chat/completions", dict(ask, response_format=schema), "POST")
    record("json_schema_strict", code == 200, f"HTTP {code}")

    # 8 — logprobs
    code, body = call("/chat/completions", dict(ask, logprobs=True, top_logprobs=3), "POST")
    lp = ((body.get("choices") or [{}])[0].get("logprobs") if isinstance(body, dict) else None)
    record("logprobs", bool(lp and lp.get("content")), f"HTTP {code}")

    # 9 — temperature: accepted / rejected / ignored is three different worlds
    code, _ = call("/chat/completions", dict(ask, temperature=0.5), "POST")
    record("temperature", None if code == 400 else code == 200,
           "rejected with 400 (reasoning model?)" if code == 400 else f"HTTP {code}")

    # 10 — stop sequences
    code, body = call("/chat/completions", dict(
        ask, messages=[{"role": "user", "content": "Count: one two three four five"}],
        stop=["three"], max_tokens=48), "POST")
    text = ((body.get("choices") or [{}])[0].get("message", {}).get("content")
            if isinstance(body, dict) else "") or ""
    record("stop_sequences", "three" not in text, f"stop string leaked into content: {'three' in text}")

    # 11 — error shape on a model that cannot exist
    code, body = call("/chat/completions", dict(ask, model="definitely-not-a-model-xyz"), "POST")
    shaped = isinstance(body, dict) and isinstance(body.get("error"), dict)
    record("error_shape", 400 <= code < 500 and shaped, f"HTTP {code}, error object: {shaped}")

    width = max(len(n) for n, _, _ in OUT)
    for name, verdict, detail in OUT:
        print(f"{name.ljust(width)}  {verdict:<7} {detail}")
    print("\nPARTIAL is not a failure — it is a behaviour you now have to design around.")
    return 0 if all(v != "FAIL" for _, v, _ in OUT) else 1


if __name__ == "__main__":
    sys.exit(main())

Two notes on running it. It costs a handful of tiny completions, so run it against a cheap model first to confirm your base URL and key are right. And run it against the exact model ID you plan to ship, because conformance varies per model on the same endpoint — reasoning models in particular reject parameters that their non-reasoning siblings accept.

Reading the output

FAIL on basic_chat or error_shape is a real problem. FAIL on logprobs is only a problem if you use logprobs. The point of the table isn't a score; it's that you now have a written record of which surfaces you're allowed to depend on, which you can re-run after any provider change and diff.

Three habits that follow from it:

Check it into CI. One nightly run, output committed as an artifact. Compatibility regressions are silent by nature — nobody sends a changelog entry saying "we stopped forwarding logprobs".
Assert the model echo in production, not just in the probe. If you asked for one ID and the response's model field says another, you want that in your logs the day it starts happening, not the week you notice quality dropped.
Treat PARTIAL as a design input. If stream_usage is absent, your cost attribution needs a non-streaming path or a local tokenizer estimate. That's a two-hour decision now or a month-end reconciliation mystery later.

What this can't tell you

The probe tests the protocol, not the model behind it. An endpoint can pass all eleven checks and still be routing you to a smaller model, a quantized build, or a truncated context window — the wire format would be perfect either way. That's a different investigation with different tools, and the honest framing there is that behavioural probes give you signals, not proof: nobody outside the provider can see which weights served your request. I wrote up the catalogue side of that problem separately — pinning model IDs, detecting drift, and what to log when the model field stops matching what you asked for — in Model IDs are a dependency. Pin them like one..

But protocol conformance is the part you can settle in an afternoon, with a script that fits on one screen, and it's the part that will wake you up at 3am if you skip it.

This post is part of an ongoing "verify your endpoint" series. The companion piece on model-ID drift is Model IDs are a dependency. Pin them like one., and if you're evaluating a third-party endpoint before your first production call, Ten questions to answer before you route production traffic through someone else's LLM endpoint is the buyer-side checklist.

Ten questions to answer before you route production traffic through someone else's LLM endpoint

Seven — Mon, 07 Sep 2026 19:36:11 +0000

Every LLM endpoint's landing page says fast, reliable, compatible, secure. Those words are free. This is the list of questions that aren't free — the ones where the answer takes a vendor real work to be able to give, and where you can verify the answer yourself in an afternoon.

Disclosure up front: I work on daoxe, a multi-model gateway. At the end I run this checklist against it, and two rows don't pass. That's the point of publishing it — a checklist that flatters whoever wrote it isn't a checklist, it's a spec sheet.

1. Protocol coverage — native or translated?

Ask which wire protocols the endpoint implements and which ones it translates into. These are different products. A translation layer is fine for text chat and lossy for agents: tool-call block ordering, streamed tool arguments, prompt-cache markers and stop-reason fidelity all live in fields that have no equivalent on the other side.

Evidence: for the Anthropic protocol, omit the required max_tokens from a /v1/messages call. A native implementation returns 400; a translation layer often fills in a default and returns 200. For the OpenAI protocol, check whether streamed tool calls arrive as incremental fragments with an index, or all at once at the end.

Bad answer: "we're fully compatible with everything."

2. Model ID transparency

Does GET /v1/models return the exact IDs your key can call — not the vendor's full catalogue? Does the response echo a model field, and does it match what you asked for? And critically: what happens when you send an ID that doesn't exist?

Evidence: send a deliberately mangled model ID. If you get a completion instead of an error, the endpoint is picking on your behalf, and every ID you send is advisory rather than binding. That single fact changes how much of the rest of this list you can trust.

Bad answer: a model list on a marketing page, with no catalogue endpoint.

3. Provenance — where does the capacity come from?

Any aggregator is reselling access it obtained somehow. The question isn't whether there's a supply chain; it's whether they'll tell you what's in it. Some capacity is direct vendor access, some is enterprise-cloud resale, some is reverse-engineered consumer endpoints with materially different reliability. All three can be legitimate purchases — at very different risk levels, for very different workloads.

Evidence: ask for the labelling scheme, in writing, and check whether it appears in the product itself or only in a support conversation.

Bad answer: silence, or "we don't disclose our infrastructure." Undisclosed provenance means you cannot reason about your own availability.

4. Can you get your own data out?

Usage per key, per model, per day. Ideally an export rather than a dashboard screenshot. You need this for two things you'll definitely do: reconciling a bill you don't understand, and attributing spend to a team or a feature.

Evidence: try to export a month. If it's a dashboard-only number, plan on logging every request yourself from day one — you can't retrofit history.

5. Incident behaviour

How do you find out something is degraded? Three tiers, in descending order of comfort: a published availability page with history, an announcement channel, or nothing.

Here's the thing worth internalising: an availability figure on a marketing page is a claim, not a measurement. It's not necessarily false, but it isn't evidence, and you shouldn't put it in a design document. Whatever tier your vendor is in, run your own probe — a one-token completion against each model you depend on, every five minutes, appended to a JSONL file. After a month you own a success rate and a latency distribution measured from your own network, which is a strictly better artifact than any number on any vendor page.

Evidence: your own probe log. Nothing else counts.

6. Key handling

Six sub-questions: can you issue multiple keys, scope them (per model, per group, per spend cap), rotate without downtime, revoke instantly, see last-used time, and does revocation actually take effect immediately?

Evidence: issue a second key, use it once, revoke it, and immediately retry. If it still works sixty seconds later, you've learned something important about the caching layer between you and their auth.

Then audit where your keys physically live. On a normal developer laptop a gateway key ends up in an editor's settings store, a shell profile, two dotfiles, a CI secret, and at least one Docker environment. Write that list down before you distribute a key to a team, not after.

7. Data handling

Retention period for prompts and completions, whether your traffic can be used for training, whether that's per-account configurable, and what the sub-processor situation is for the upstream vendors behind the aggregator.

Get this in writing. I'm deliberately not paraphrasing anyone's policy — including my own employer's — in a blog post you might then rely on. Ask, get a written answer, keep it, and re-ask on renewal.

Bad answer: a verbal assurance in a chat window.

8. Rate limits and behaviour under load

Are limits documented per key or per model? What does the endpoint do when you exceed them — a clean 429 with Retry-After, a 500, or a hang? Do errors have a stable machine-readable shape your retry layer can classify?

Evidence: fire a small burst of concurrent requests at a cheap model and read the failures. A 429 carrying Retry-After is a mature implementation. A 500 where a 429 belonged will turn your backoff into a retry storm, because your code will classify it as transient-and-worth-hammering.

9. Billing you can reconcile

You want three things: a mechanism you can state in one sentence, a live price reference rather than a number in a PDF, and per-request usage data you can multiply out yourself.

Be actively suspicious of "cheapest". Price is the easiest thing on this list to change and the easiest to advertise; it tells you nothing about the nine criteria above it, and a number well below everyone else's floor is usually a statement about provenance (criterion 3), not about efficiency.

For the record, since I'm asking you to run this against my employer: DaoXE tops up at a flat 1:1 top-up rate across Alipay, WeChat Pay, USDT, bank card (Visa·Mastercard), Apple Pay and Google Pay — one unit of local currency buys one dollar of credit — models bill at their USD list prices, and the live numbers are at https://daoxe.com/pricing — check them there rather than trusting any figure in an article, including this one.

10. The exit plan

The last question is the one that makes the other nine survivable: how many lines of code does it take to leave?

If the answer is "change a base URL and a key in one config file", the switching cost is near zero and every other risk on this list becomes a manageable one. If the answer involves a proprietary SDK, vendor-specific request extensions, or a routing DSL you can't reproduce elsewhere, you've bought a migration.

Four habits that keep the number low: keep the base URL and model IDs in config, not in code; prefer standard protocol fields over vendor extensions; keep your own request/response logs; and maintain a golden eval set of twenty to fifty real requests so that "is the replacement good enough" is a script you run rather than an argument you have.

The scorecard, run against my own employer

#	Criterion	DaoXE	Notes
1	Protocol coverage	Pass	OpenAI (`/v1/chat/completions`, `/v1/responses`, `/v1/embeddings`, `/v1/images/generations`, `/v1/videos`), Anthropic Messages (`/v1/messages`), Gemini (`/v1beta/models/{model}:generateContent`), plus `/v1/rerank` — implemented, not translated. Verify with the `max_tokens` test above.
2	Model ID transparency	Pass	`GET /v1/models` returns the exact IDs your key can call. Catalogue size drifts constantly — hundreds of models across roughly 25 vendors — so treat any published count, including ours, as stale.
3	Provenance	Pass, with a caveat you should read	Channels are labelled in the catalogue: full-power, official relay, reverse-engineered ("quality not guaranteed"), promotional. The reverse-engineered tier genuinely exists. Labelling it is the honest part; choosing it for production is your call, and I'd say no.
4	Usage / log export	Ask	Don't take my word for the current state — ask on Telegram `@daoxe_ai` and log your own requests regardless.
5	Incident behaviour	Does not pass	There is no public status page, so run your own cron probe. The availability figure on the marketing page is a company claim, not a measurement, and I won't repeat it here as if it were one.
6	Key handling	Test it yourself	Run the issue → use → revoke → retry sequence on your own account. Note that sign-up is open, with password login, no email verification, and OAuth via GitHub, Telegram or Passkey — convenient, and worth factoring into your own account-security posture.
7	Data handling	Ask, in writing	Same rule I'd apply to any vendor. Ask `@daoxe_ai`, keep the answer.
8	Rate limits under load	Test it yourself	Burst a cheap model, read the failure bodies, check for `Retry-After`.
9	Billing	Pass on mechanism	One-sentence mechanism, live prices on the pricing page, per-request usage in the response body.
10	Exit plan	Pass	Standard protocols, model IDs are strings, base URL is one config line. Leaving is the same edit as arriving — which is the only version of this claim worth making.

One more thing that belongs on any honest vendor row: DaoXE is not available in mainland China. That's the official line and it's a hard constraint, not a caveat to work around.

Using this without turning it into a ritual

Ten criteria is too many for a side project and about right for anything with users. If you only have an hour, do these four: the mangled-model-ID test (criterion 2), the provenance question (3), start the probe (5), and count the lines it would take to leave (10). Those four catch the failures that are expensive to discover later; the rest catch the failures that are merely annoying.

And apply it symmetrically. If a vendor's answer to criterion 3 or 7 is a shrug, that's data — but so is a checklist published by a vendor that somehow scores ten out of ten on its own list.

Model IDs are a dependency. Pin them like one.

Seven — Mon, 07 Sep 2026 15:12:43 +0000

You typed claude-sonnet-4-5 from memory, it returned a completion, and you shipped. Six weeks later the outputs are subtly different, or a deploy fails with a 404 on a Tuesday morning, and the git blame points at a string literal in a file nobody has opened since.

Model IDs are a runtime dependency on someone else's release schedule, and almost nobody treats them like one. We pin requests==2.32.3 and then interpolate a model name from a config file we last checked in March.

Disclosure: I work on daoxe, a gateway that fronts hundreds of models from roughly 25 vendors — which means I get to watch upstream renames arrive from several directions at once. Everything below works against any endpoint that implements GET /v1/models.

Three ways a wrong ID fails

Mode A — a clean error. 404 or 400 with "code": "model_not_found". This is the good outcome. You find out in development, in the first second, and you fix a typo.

Mode B — alias resolution. You ask for gpt-4o, the provider resolves it to a dated build, and the response's model field says which one. Fine and normal — as long as you read the echo. If you never look, you've silently accepted whatever the alias points at this month.

Mode C — nearest-match routing. Some endpoints and proxies try to be helpful: they don't recognise your ID, so they route to the closest thing they do recognise. You asked for a flagship and got something else, with a 200 and no indication anywhere in the response that a substitution happened. This is the mode that costs real money and real trust, and it's invisible in exactly the way that matters.

The ten-second test for Mode C

Send an ID that cannot possibly exist, formed by mangling a real one:

curl -s https://api.daoxe.com/v1/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"claude-sonnet-4-5-TYPO-NOT-A-MODEL",
       "max_tokens":8,"messages":[{"role":"user","content":"hi"}]}' \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); \
print("ERROR:", d["error"]["message"][:80]) if "error" in d else print("SERVED BY:", d.get("model"))'

If you get an error, every model ID you send is authoritative. If you get a completion, every model ID you send is advisory — the endpoint reserves the right to pick for you, and no amount of pinning on your side will help until you know the rule it uses. Run this before you build anything else in this article.

Dated versus floating

Two ID styles, two different risks:

	Floating (`claude-sonnet-4-5`)	Dated (`claude-sonnet-4-5-20250929`)
Behaviour	changes under you, no code change	stable while it exists
Lifespan	long	retired on a schedule
Fails as	silent quality drift	a loud 404
Use for	exploration, prototypes	anything with a recorded eval

The policy that follows: exploration floats, production is dated, and every eval artifact records the ID that actually served it — read from the response's model field, not from what you asked for. That last part is what lets you answer "was this eval run on the same build as that one" three months later.

`GET /v1/models` is the source of truth, and it's per-key

Not the docs, not the pricing page, not a blog post. The catalogue endpoint returns the IDs your key is entitled to call, which is frequently a different set from what the vendor sells:

curl -s https://api.daoxe.com/v1/models \
  -H "Authorization: Bearer $LLM_API_KEY" \
  | python3 -c 'import json,sys; [print(m["id"]) for m in json.load(sys.stdin)["data"]]' \
  | sort

One caveat worth knowing up front: most implementations return IDs and little else. Context windows, tool support, vision support and modality are usually not in there, so you still need a small capability sidecar of your own. Don't infer capabilities from the name.

Lock the catalogue

#!/usr/bin/env python3
"""check_models.py — fail CI when a model ID this repo depends on disappears.

    python3 check_models.py --write      # re-pin (review the diff!)
    python3 check_models.py              # verify
"""
import argparse, json, os, sys, urllib.request

BASE = os.environ["LLM_BASE_URL"].rstrip("/")
KEY = os.environ["LLM_API_KEY"]


def catalogue():
    req = urllib.request.Request(BASE + "/models")
    req.add_header("Authorization", "Bearer " + KEY)
    with urllib.request.urlopen(req, timeout=30) as resp:
        return sorted({m["id"] for m in json.load(resp).get("data", []) if m.get("id")})


def required_ids(path="models.json"):
    """The IDs this repo actually uses — derived from the alias map, never hand-listed."""
    with open(path) as fh:
        return sorted(set(json.load(fh)["roles"].values()))


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--lock", default="models.lock.json")
    ap.add_argument("--write", action="store_true")
    args = ap.parse_args()

    live = catalogue()
    if args.write or not os.path.exists(args.lock):
        with open(args.lock, "w") as fh:
            json.dump({"catalogue": live}, fh, indent=2)
            fh.write("\n")
        print(f"pinned {len(live)} ids to {args.lock}")
        return 0

    with open(args.lock) as fh:
        pinned = set(json.load(fh)["catalogue"])
    live_set, needed = set(live), set(required_ids())

    for mid in sorted(live_set - pinned):
        print(f"note   new in catalogue: {mid}")
    for mid in sorted(pinned - live_set - needed):
        print(f"warn   left the catalogue (unused here): {mid}")
    gone = sorted(needed - live_set)
    for mid in gone:
        print(f"ERROR  required by models.json but not in the catalogue: {mid}")
    return 1 if gone else 0


if __name__ == "__main__":
    sys.exit(main())

Run it nightly and on every pull request:

name: model-catalogue
on:
  schedule: [{ cron: "17 6 * * *" }]
  pull_request:
  workflow_dispatch:

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: python3 check_models.py
        env:
          LLM_BASE_URL: https://api.daoxe.com/v1
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}

The two failure signals mean different things. A nightly failure means an upstream retired or renamed something you depend on, and you found out before your users did. A pull-request failure means someone added a role pointing at an ID that isn't callable — a typo caught at review time instead of at 2am.

The alias layer

Nothing in your application code should contain a model name. Code refers to roles; one file maps roles to exact IDs.

{
  "pinned_at": "2026-08-12",
  "roles": {
    "fast":   "<exact-id>",
    "deep":   "<exact-id>",
    "bulk":   "<exact-id>",
    "vision": "<exact-id>",
    "embed":  "<exact-embedding-id>"
  },
  "notes": {
    "deep": "dated build; re-pin only with an eval run linked in the PR"
  }
}

import functools, json, pathlib

@functools.lru_cache(maxsize=1)
def _roles():
    return json.loads(pathlib.Path("models.json").read_text())["roles"]

def model_for(role):
    try:
        return _roles()[role]
    except KeyError:
        raise KeyError(f"no model pinned for role {role!r} — add it to models.json") from None

Three things fall out of this for free. A rename becomes a one-line diff in a reviewable file instead of a search-and-replace across forty call sites. check_models.py can derive its required set from the same file, so the lock and the code can't disagree. And git log models.json becomes a readable history of what your product was running and when — which is the artifact you'll want the first time someone asks why output quality changed in July.

Then stop the IDs from leaking back in. One test, one line:

# tests/no_hardcoded_model_ids.sh — fails if a model-looking literal appears in src/
! rg -n --glob '!models.json' \
    -e '"(gpt|claude|gemini|deepseek|qwen|kimi|grok|llama)-[a-z0-9._-]+"' src/

When an upstream retires a model mid-project

Find out from CI, not from users. That's what the nightly is for.
Read the label before you pick a successor. Vendors publish migration notes; gateways publish catalogue metadata. Don't infer the replacement from the name — -mini, -fast, -pro and numeric bumps are marketing, not a capability contract.
Freeze a golden set. Twenty to fifty real requests from your own traffic with their recorded outputs. If you don't have one yet, this is the moment it becomes worth an afternoon; you will need it again.
Run the candidate against the golden set and diff the things that actually break in production: output format compliance, tool-call rate, refusal rate, response length, and latency. Not vibes.
One PR, one line. Change models.json, link the eval run in the description, let CI re-verify the lock.
Keep the corpse. Leave the old ID in the git history of models.lock.json and in your eval artifacts. When you're asked to reproduce a result from two quarters ago, that record is the only thing that will let you say what actually served it.

Variants and channels on a gateway

If you route through a gateway, two IDs that look like the same model may reach different upstream channels with different reliability characteristics. Some catalogues label this explicitly — the one I work on tags channels as full-power, official relay, reverse-engineered ("quality not guaranteed"), or promotional, and that labelling is the whole point: you can choose a cheaper or less stable path deliberately, or refuse to. What you should never do is pattern-match a suffix and assume you know what it means. Read the catalogue metadata, pick the exact string, pin it.

Drift under a pinned ID

Pinning protects you from renames. It does not protect you from an upstream changing what sits behind a name it didn't change. The only handle you have from the outside is behavioural: record how a pinned ID answers a fixed battery of prompts at temperature 0, re-run it on a schedule, and diff.

pipx install git+https://github.com/seven7763/llm-honesty-probe

Be precise about what that gives you: signals, not proof. No probe run from outside can prove which weights served a request, and a change in output can have several innocent explanations. What it does produce is a timestamped record that something changed on a date, which is a much stronger position to argue from than "it feels worse lately."

Pin the ID, alias the role, lock the catalogue, watch the behaviour. It's an afternoon of work, and it converts your loudest class of production surprise into a CI failure.

Model IDs are a dependency. Pin them like one.

Seven — Mon, 07 Sep 2026 11:19:26 +0000

Three ways a wrong ID fails

Mode A — a clean error. 404 or 400 with "code": "model_not_found". This is the good outcome. You find out in development, in the first second, and you fix a typo.

The ten-second test for Mode C

Send an ID that cannot possibly exist, formed by mangling a real one:

curl -s https://api.daoxe.com/v1/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"claude-sonnet-4-5-TYPO-NOT-A-MODEL",
       "max_tokens":8,"messages":[{"role":"user","content":"hi"}]}' \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); \
print("ERROR:", d["error"]["message"][:80]) if "error" in d else print("SERVED BY:", d.get("model"))'

Dated versus floating

Two ID styles, two different risks:

	Floating (`claude-sonnet-4-5`)	Dated (`claude-sonnet-4-5-20250929`)
Behaviour	changes under you, no code change	stable while it exists
Lifespan	long	retired on a schedule
Fails as	silent quality drift	a loud 404
Use for	exploration, prototypes	anything with a recorded eval

`GET /v1/models` is the source of truth, and it's per-key

Not the docs, not the pricing page, not a blog post. The catalogue endpoint returns the IDs your key is entitled to call, which is frequently a different set from what the vendor sells:

curl -s https://api.daoxe.com/v1/models \
  -H "Authorization: Bearer $LLM_API_KEY" \
  | python3 -c 'import json,sys; [print(m["id"]) for m in json.load(sys.stdin)["data"]]' \
  | sort

Lock the catalogue

#!/usr/bin/env python3
"""check_models.py — fail CI when a model ID this repo depends on disappears.

    python3 check_models.py --write      # re-pin (review the diff!)
    python3 check_models.py              # verify
"""
import argparse, json, os, sys, urllib.request

BASE = os.environ["LLM_BASE_URL"].rstrip("/")
KEY = os.environ["LLM_API_KEY"]


def catalogue():
    req = urllib.request.Request(BASE + "/models")
    req.add_header("Authorization", "Bearer " + KEY)
    with urllib.request.urlopen(req, timeout=30) as resp:
        return sorted({m["id"] for m in json.load(resp).get("data", []) if m.get("id")})


def required_ids(path="models.json"):
    """The IDs this repo actually uses — derived from the alias map, never hand-listed."""
    with open(path) as fh:
        return sorted(set(json.load(fh)["roles"].values()))


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--lock", default="models.lock.json")
    ap.add_argument("--write", action="store_true")
    args = ap.parse_args()

    live = catalogue()
    if args.write or not os.path.exists(args.lock):
        with open(args.lock, "w") as fh:
            json.dump({"catalogue": live}, fh, indent=2)
            fh.write("\n")
        print(f"pinned {len(live)} ids to {args.lock}")
        return 0

    with open(args.lock) as fh:
        pinned = set(json.load(fh)["catalogue"])
    live_set, needed = set(live), set(required_ids())

    for mid in sorted(live_set - pinned):
        print(f"note   new in catalogue: {mid}")
    for mid in sorted(pinned - live_set - needed):
        print(f"warn   left the catalogue (unused here): {mid}")
    gone = sorted(needed - live_set)
    for mid in gone:
        print(f"ERROR  required by models.json but not in the catalogue: {mid}")
    return 1 if gone else 0


if __name__ == "__main__":
    sys.exit(main())

Run it nightly and on every pull request:

name: model-catalogue
on:
  schedule: [{ cron: "17 6 * * *" }]
  pull_request:
  workflow_dispatch:

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: python3 check_models.py
        env:
          LLM_BASE_URL: https://api.daoxe.com/v1
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}

The alias layer

Nothing in your application code should contain a model name. Code refers to roles; one file maps roles to exact IDs.

{
  "pinned_at": "2026-08-12",
  "roles": {
    "fast":   "<exact-id>",
    "deep":   "<exact-id>",
    "bulk":   "<exact-id>",
    "vision": "<exact-id>",
    "embed":  "<exact-embedding-id>"
  },
  "notes": {
    "deep": "dated build; re-pin only with an eval run linked in the PR"
  }
}

import functools, json, pathlib

@functools.lru_cache(maxsize=1)
def _roles():
    return json.loads(pathlib.Path("models.json").read_text())["roles"]

def model_for(role):
    try:
        return _roles()[role]
    except KeyError:
        raise KeyError(f"no model pinned for role {role!r} — add it to models.json") from None

Then stop the IDs from leaking back in. One test, one line:

# tests/no_hardcoded_model_ids.sh — fails if a model-looking literal appears in src/
! rg -n --glob '!models.json' \
    -e '"(gpt|claude|gemini|deepseek|qwen|kimi|grok|llama)-[a-z0-9._-]+"' src/

When an upstream retires a model mid-project

Find out from CI, not from users. That's what the nightly is for.
Read the label before you pick a successor. Vendors publish migration notes; gateways publish catalogue metadata. Don't infer the replacement from the name — -mini, -fast, -pro and numeric bumps are marketing, not a capability contract.
Freeze a golden set. Twenty to fifty real requests from your own traffic with their recorded outputs. If you don't have one yet, this is the moment it becomes worth an afternoon; you will need it again.
Run the candidate against the golden set and diff the things that actually break in production: output format compliance, tool-call rate, refusal rate, response length, and latency. Not vibes.
One PR, one line. Change models.json, link the eval run in the description, let CI re-verify the lock.
Keep the corpse. Leave the old ID in the git history of models.lock.json and in your eval artifacts. When you're asked to reproduce a result from two quarters ago, that record is the only thing that will let you say what actually served it.

Variants and channels on a gateway

Drift under a pinned ID

pipx install git+https://github.com/seven7763/llm-honesty-probe

Pin the ID, alias the role, lock the catalogue, watch the behaviour. It's an afternoon of work, and it converts your loudest class of production surprise into a CI failure.

Your LLM API will fail in production. Here's the fallback system that catches it.

Seven — Thu, 03 Sep 2026 06:28:43 +0000

Every LLM API fails. Not "might fail." Fails.

OpenAI returns 429s during peak hours. Anthropic has regional outages. Google's Gemini API occasionally decides your key isn't authorized for the model you've been calling all week. A provider you depend on goes down for 45 minutes and your agent pipeline — the one you told your team was "production-ready" — sits there dead, mid-task, with no idea what happened.

You already know retry logic. This article is about what comes after retry: how to build a multi-model fallback system that keeps your application running when the primary model is unreachable, and how to avoid the mistakes that make fallback worse than just failing.

I'll show real Python patterns, the gotchas that break them in practice, and how to test your fallback without waiting for an actual outage.

Disclosure up front: I work on daoxe, a multi-model API gateway. The patterns in this article work against any provider — official APIs, third-party gateways, or self-hosted endpoints. Use them wherever your key works.

1. Why retry alone isn't enough

The simplest thing that sometimes works:

import time
import openai

def call_with_retry(model: str, messages: list, max_retries: int = 3):
    for attempt in range(max_retries):
        try:
            return openai.chat.completions.create(
                model=model, messages=messages
            )
        except openai.APIError as e:
            if attempt == max_retries - 1:
                raise
            time.sleep(2 ** attempt)

This handles transient network blips and brief rate limiting. It does not handle:

Sustained outages. If the provider is down for 30 minutes, exponential backoff just means your caller eventually raises an exception after 14 seconds instead of immediately. Your agent is still dead.
Quota exhaustion. Your key hits its monthly limit. Retrying won't fix it. It'll just burn your error budget faster.
Model deprecation. The provider sunsets the model version you pinned. Every call fails with model_not_found. Retry spins uselessly.
Regional blocks. The endpoint is up — just not from where your server lives. The client sees a timeout and retries into the same wall.

Retry is the first layer. Fallback is the second.

2. The model-level fallback pattern (simple, with sharp edges)

The basic idea: if model A fails, try model B.

import os
from openai import OpenAI

FALLBACK_CHAIN = [
    {"model": "gpt-5.5", "base_url": "https://api.openai.com/v1", "key_env": "OPENAI_API_KEY"},
    {"model": "claude-sonnet-4-6", "base_url": "https://api.anthropic.com/v1/messages",
     "key_env": "ANTHROPIC_API_KEY", "protocol": "anthropic"},
    {"model": "gemini-2.5-pro", "base_url": "https://generativelanguage.googleapis.com/v1beta",
     "key_env": "GEMINI_API_KEY", "protocol": "gemini"},
]

def call_with_fallback(messages: list, chain: list = FALLBACK_CHAIN):
    last_error = None
    for cfg in chain:
        try:
            client = OpenAI(
                base_url=cfg["base_url"],
                api_key=os.environ[cfg["key_env"]],
                timeout=30.0,
            )
            return client.chat.completions.create(
                model=cfg["model"], messages=messages
            )
        except Exception as e:
            last_error = e
            continue
    raise RuntimeError(
        f"All {len(chain)} models failed. Last error: {last_error}"
    )

This works — until it doesn't. Here's where it breaks.

Gotcha 1: Protocol incompatibility

Anthropic's API uses the Messages protocol, not Chat Completions. Calling it through an OpenAI-shaped client with base_url=https://api.anthropic.com/v1/messages produces a 404 or a garbled error. You need a protocol adapter, or you need to use a gateway that translates between protocols.

The honest fix: either maintain separate client code paths per provider, or use a gateway that presents every model behind a single protocol. Both approaches work. The single-protocol path means fewer branches to test.

Gotcha 2: The system prompt disappears

Some models handle system prompts differently. Claude native uses a top-level system parameter. OpenAI uses a system role in the messages array. Gemini puts system instructions in a systemInstruction config field. When you fall back across providers, your carefully crafted system prompt might arrive as a user message — or not at all.

Test this explicitly. Send a message that asks the model to quote the system prompt verbatim. If it can't, your fallback is silently dropping instructions.

Gotcha 3: Tool calling breaks between families

Anthropic and OpenAI have different tool calling schemas. The function definitions that work on GPT may produce tool_use blocks on Claude that your parser doesn't recognize. If your agent depends on structured tool calls, cross-provider fallback is risky without a normalization layer.

For agent pipelines that use tool calling heavily, fallback within the same model family first — GPT family before Claude family before Gemini — and only cross families when the task doesn't require structured output.

3. Routing by capability, not just availability

A smarter fallback system doesn't blindly try models in a fixed order. It selects the fallback based on what the task actually needs.

CAPABILITY_PROFILES = {
    "reasoning": ["claude-sonnet-4-6", "gpt-5.5", "gemini-2.5-pro"],
    "coding": ["claude-sonnet-4-6", "gpt-5.5", "deepseek-v4-pro"],
    "fast_chat": ["claude-haiku-4-5", "gpt-5.4", "gemini-3.5-flash"],
    "long_context": ["gemini-2.5-pro", "claude-sonnet-4-6", "gpt-5.5"],
}

MODEL_CONFIGS = {
    "claude-sonnet-4-6": {"base_url": "...", "key_env": "...", "protocol": "anthropic"},
    "gpt-5.5": {"base_url": "...", "key_env": "...", "protocol": "openai"},
    # ... etc
}

def call_by_capability(messages: list, capability: str):
    candidates = CAPABILITY_PROFILES.get(capability, [])
    if not candidates:
        raise ValueError(f"Unknown capability: {capability}")

    last_error = None
    for model_id in candidates:
        cfg = MODEL_CONFIGS[model_id]
        try:
            client = OpenAI(base_url=cfg["base_url"], api_key=os.environ[cfg["key_env"]])
            return client.chat.completions.create(model=model_id, messages=messages)
        except Exception as e:
            last_error = e
            continue
    raise RuntimeError(f"All {capability} models failed: {last_error}")

Now your code declares intent — "I need long-context reasoning" — and the dispatcher picks appropriate models. This is more resilient than a fixed order, and it makes your code self-documenting about what each call actually requires.

4. The gateway pattern: one endpoint, many models

Managing separate API keys, base URLs, and protocol adapters for a half-dozen providers is operational friction. The alternative is a single endpoint that presents many models behind one API surface:

# One base URL. One key. All models available.
client = OpenAI(
    base_url="https://api.daoxe.com/v1",
    api_key=os.environ["DAOXE_API_KEY"],
)

FALLBACK_CHAIN = [
    "claude-sonnet-4-6",   # primary
    "gpt-5.5",             # fallback 1
    "gemini-2.5-pro",      # fallback 2
]

def call_single_endpoint(messages: list, chain: list = FALLBACK_CHAIN):
    last_error = None
    for model in chain:
        try:
            return client.chat.completions.create(
                model=model, messages=messages, timeout=30.0
            )
        except Exception as e:
            last_error = e
            continue
    raise RuntimeError(f"All models in chain failed: {last_error}")

The advantage is not having to manage per-provider credentials, protocol quirks, or billing relationships. The trade-off is that the gateway itself becomes a single point of failure — if the gateway goes down, all models behind it become unreachable. (Mitigations: monitor the gateway's health separately, and keep a direct-to-provider key as an escape hatch for critical paths.)

5. Circuit breaker: when even fallback should stop

Blindly cycling through fallback models under load can make things worse. If your primary model is returning 429s because you're over your rate limit, hammering the fallback models immediately is antisocial — and it might exhaust your quota on all providers in a cascade.

A circuit breaker stops calling a model after N consecutive failures and only re-enables it after a cooldown.

import time
from collections import defaultdict

class CircuitBreaker:
    def __init__(self, failure_threshold: int = 5, cooldown_seconds: float = 60.0):
        self.failure_threshold = failure_threshold
        self.cooldown_seconds = cooldown_seconds
        self._failures: dict[str, int] = defaultdict(int)
        self._open_until: dict[str, float] = {}

    def allow(self, model: str) -> bool:
        if model in self._open_until and time.time() < self._open_until[model]:
            return False
        return True

    def record_failure(self, model: str):
        self._failures[model] += 1
        if self._failures[model] >= self.failure_threshold:
            self._open_until[model] = time.time() + self.cooldown_seconds

    def record_success(self, model: str):
        self._failures[model] = 0
        self._open_until.pop(model, None)

Wire it into the fallback dispatcher:

breaker = CircuitBreaker(failure_threshold=3, cooldown_seconds=30.0)

def call_with_breaker(messages: list, chain: list):
    for model in chain:
        if not breaker.allow(model):
            continue  # circuit open, skip
        try:
            result = client.chat.completions.create(model=model, messages=messages)
            breaker.record_success(model)
            return result
        except Exception as e:
            breaker.record_failure(model)
            continue
    raise RuntimeError("All models unavailable (circuit open or failed)")

Now when a model goes down repeatedly, the system stops trying it for 30 seconds, giving it time to recover without wasting your remaining quota on guaranteed failures.

6. Observability: you can't fix what you can't see

A fallback system without instrumentation is a debugging nightmare. When a call succeeds on the third model in the chain, you need to know that it happened and why the first two failed.

At minimum, log:

import logging
import json

logger = logging.getLogger("llm_fallback")

def call_instrumented(messages: list, chain: list):
    for i, model in enumerate(chain):
        try:
            start = time.monotonic()
            result = client.chat.completions.create(model=model, messages=messages)
            elapsed = time.monotonic() - start

            if i > 0:
                logger.warning(
                    "fallback_used",
                    extra={
                        "primary": chain[0],
                        "actual": model,
                        "fallback_depth": i,
                        "latency_s": round(elapsed, 3),
                        "prompt_tokens": result.usage.prompt_tokens,
                        "completion_tokens": result.usage.completion_tokens,
                    },
                )

            return result

        except Exception as e:
            logger.error(
                "model_call_failed",
                extra={"model": model, "attempt": i, "error": str(e)},
            )
            continue

    logger.critical("all_models_exhausted", extra={"chain": chain})
    raise RuntimeError("All models in chain failed")

Critical metric to track over time: fallback rate (what percentage of calls hit a fallback model). If this creeps above 1-2%, your primary provider has a reliability problem. If it spikes suddenly, something is broken. Either way, you want to know before your users do.

7. Testing your fallback without waiting for an outage

The worst time to discover your fallback system is broken is during an actual outage. Test it artificially.

Method 1: The unreachable endpoint

Point your primary model at a dead URL and verify the fallback fires:

# Set primary to a port where nothing listens
PRIMARY_BASE_URL="http://localhost:19999/v1" python test_fallback.py

Method 2: The error-injection proxy

Write a tiny proxy that returns errors for a percentage of requests:

# error_proxy.py — run with: python error_proxy.py
# Proxies requests to your real endpoint, but returns 503 for ~30% of them.

from http.server import HTTPServer, BaseHTTPRequestHandler
import random

class ErrorProxy(BaseHTTPRequestHandler):
    def do_POST(self):
        if random.random() < 0.3:
            self.send_response(503)
            self.end_headers()
            self.wfile.write(b'{"error": {"message": "injected failure", "type": "server_error"}}')
            return
        # ... proxy to real endpoint

Run your application against this proxy. If the fallback doesn't fire within a few requests, something is misconfigured.

Method 3: The key rotation test

Rotate your primary API key to an invalid value and verify the system fails over to the next model. This catches the case where your fallback is configured but the credentials are wrong — a failure mode that's silent until the primary actually goes down.

8. The honest limitations

No fallback system is transparent to the caller. Here's what you actually trade off:

Latency. Each fallback attempt adds connection + inference time. A fast call that normally takes 800ms can take 3+ seconds if the first two models fail. For real-time applications, set a total timeout and surface to the user that a fallback is in progress.

Cost unpredictability. Fallback models may have different pricing. Your monthly bill can spike if your primary model has a bad week and all traffic shifts to a more expensive fallback. Track spend per model and set alerts when fallback-model spend exceeds a threshold.

Behavioral drift. Different models produce different outputs. A system prompt that Claude handles elegantly might produce aggressive refusals on Gemini. A chain-of-thought instruction that works on GPT may be ignored by DeepSeek. Test your fallback models with your actual prompts before depending on them.

Determinism is gone. Even with temperature=0, different models produce different outputs. If your application depends on deterministic or near-deterministic behavior, cross-model fallback fundamentally changes the semantics of the call. Your fallback is keeping the system running, not keeping it identical.

There is no zero-dependency option. A fallback system that spans multiple providers means you depend on all of them — their uptime, their pricing changes, their model deprecations, their API migrations. Your operational surface area multiplies.

9. What to do right now

If you take nothing else from this article, do these three things today:

Identify which LLM calls in your codebase are single-model and single-provider. Anywhere you hardcode model="gpt-5.5" with no alternative path is a single point of failure. Count them.
Pick one non-critical path and add a two-model fallback chain. Not your payment processing. Not your auth flow. A summarization step, a classification call, a "suggest alternative phrasing" feature. Ship it, instrument it, watch the logs for a week. Get comfortable with the pattern before you apply it to critical paths.
Add a health-check endpoint that probes your primary model every 60 seconds and exposes the result. A simple /health/llm that calls your primary model with a trivial prompt and returns {"status": "ok", "latency_ms": 420} or {"status": "degraded", "error": "timeout"}. Wire this into whatever monitoring you already use. You want to know about provider degradation before your users do — and before your fallback system has to prove itself under fire.

The goal of a fallback system is not perfection. It's to turn a hard failure — your agent crashes, your pipeline stops, your user sees an error — into a soft degradation the caller might not even notice. That's worth the complexity.

If you liked this, you might also want llm-honesty-probe (github.com/seven7763/llm-honesty-probe), an open-source tool that checks whether your API endpoint is actually serving the model it claims. Fallback + verification = you're covered on both reliability and trust.

OpenAI cut prices 80%. Here's why I'm not celebrating.

Seven — Fri, 31 Jul 2026 16:23:40 +0000

This week, OpenAI slashed Luna (GPT-5.6) pricing by 80% — from $7 to $1.40 per million tokens. Anthropic shipped Opus 5 at half the price of its predecessor. The LLM API market is in a genuine price war, and for once, developers are winning.

I should be celebrating. I'm not. Here's why.

The hidden math of cheaper models

When official API prices drop, the economics of model-swapping change in a way nobody talks about.

A bad relay has a simple business model: bill you for Model X, serve you Model Y, pocket the difference. When Model X costs $15/M tokens and Model Y costs $0.15/M, the margin is $14.85. That's already attractive.

When Model X drops to $1.40? The absolute margin shrinks — and that's what most people see. But the relative margin — the ratio of what you're charged vs what's actually served — can actually grow if budget models get cheaper still. And they have.

The point isn't the exact math. The point is: price competition between legitimate providers doesn't eliminate the incentive to swap models. It often masks it. When everyone's prices are dropping, a relay that's 30% cheaper than official looks like aggressive discounting, not fraud.

"What model are you?" doesn't work

If you're still asking an API endpoint to identify itself, stop. That's a system prompt — trivially spoofed. Every relay operator knows to make the model say "I am Claude, created by Anthropic."

What works: behavioral fingerprinting at temp=0.

Give the model a fixed set of tasks that different models handle differently — reasoning problems, long-context recall, strict JSON formatting, refusal boundaries. Run them at temperature 0 to remove sampling noise. Diff the results against the official API.

This isn't theoretical. Here's the actual tool:

git clone https://github.com/seven7763/llm-honesty-probe
cd llm-honesty-probe
python3 -m llm_honesty_probe --self-test --card

That runs a battery of deterministic probes and produces a verdict card: PASS, SUSPICIOUS, or DEGRADED — with the specific probes that triggered each finding. No API key leaves your machine. No judgment calls required. The self-test runs without any API key and takes about 30 seconds.

To test your actual endpoint:

export LLM_ENDPOINT="https://your-endpoint/v1"
export LLM_API_KEY="sk-your-key"
python3 -m llm_honesty_probe --compare

Why this matters more now, not less

The price war is genuinely good news. Lower costs mean more experiments, more products, more people who can afford to build with LLMs. I want all of that.

But cheaper official APIs also mean more users signing up for the first time — users who don't know what "Claude" is supposed to sound like, what GPT-5.6's reasoning depth should feel like, or what latency to expect. These are exactly the users a model-swapping relay wants: no baseline for comparison.

The defense isn't trust. It's verification that costs less than being wrong.

Test your endpoint right now

If you're using any third-party API endpoint — mine included — run the probe. The self-test takes 30 seconds. The compare against your endpoint takes about 2 minutes.

If everything passes, you've got evidence — not proof, but signals — that you're getting what you pay for. If something flags, you know exactly which probe to investigate.

Either way, you know more than you did before. That's the whole point.

Disclosure: I work on daoxe, an OpenAI-compatible LLM gateway. I also maintain llm-honesty-probe, the open-source tool mentioned in this post. The probe treats every endpoint identically — including daoxe. Point it at anyone. If it flags something on my own service, I want to know before you do.

The price war is real. The trust problem is real. They're related, and fixing one doesn't automatically fix the other.

Deploy a real AI chat app in one click — key stays server-side (Vercel/Render/Railway)

Seven — Tue, 21 Jul 2026 14:40:49 +0000

Most "deploy an AI app in 5 minutes" tutorials quietly do the one thing you must never do: put an API key where a browser can see it. This post is three one-click starters that are correct by construction — the key is a deploy-time environment variable, server-side only, never in client code, never committed to git.

Each template is provider-neutral: it defaults to an OpenAI-compatible endpoint (https://daoxe.com/v1) but points at any compatible endpoint by changing one variable. Pick the one that matches what you're building.

Disclosure: I work on daoxe, the OpenAI-compatible gateway these default to. Everything below works against any compatible endpoint — the default and the "powered by" link are one-line changes.

The rule that makes these safe

An LLM API key is a billing credential. Anything shipped to a browser is public. So every template here follows the same discipline:

DAOXE_API_KEY (and any tokens) are deploy-time env vars, declared in each platform's config with no baked-in value.
.env.example files ship empty placeholders.
The key is used server-side only — an Edge function, a proxy server, or a bot process holds it; the browser never sees it.

If a tutorial ever tells you to paste your key into front-end JavaScript, close the tab.

Template 1 — Streaming chat UI on Vercel

A minimal chat UI backed by an Edge function that holds the key and streams tokens back to the browser. Good for: a demo, an internal tool, a landing-page assistant.

Repo: seven7763/daoxe-vercel-chat

Deploy: click the Vercel button in the repo README; Vercel prompts you for two env vars:

DAOXE_API_KEY — your key.
DAOXE_MODEL — an exact id from GET /v1/models.

That's it — you get a URL with a working streaming chat. The browser calls your Edge function; the Edge function calls the gateway with the key. The key never leaves the server.

Browser ──▶ /api/chat (Edge, holds key) ──▶ https://daoxe.com/v1/chat/completions
        ◀── streamed tokens ◀──────────────

Optional DAOXE_ALLOWED_MODELS lets you restrict which models the function will serve.

Template 2 — OpenAI-compatible proxy on Render

Sometimes you don't want a UI — you want to share one key with a team or a script without handing out the real credential. This template is a tiny OpenAI-compatible proxy: it exposes /v1/chat/completions, injects the real key server-side, and gates access with a token you hand out instead.

Repo: seven7763/daoxe-render-proxy

Deploy: click the Render button in the repo README; set DAOXE_API_KEY (the real key) and a share token. Now teammates point their OpenAI-compatible tools at your proxy URL with the share token — they get model access, never your billing credential. It also serves a live /healthz page.

Template 3 — Discord bot on Railway

A Discord bot that answers /ask with an LLM. Good for: team assistants, support bots, community helpers.

Repo: seven7763/daoxe-railway-discord-bot

Deploy: from the public repo, open Railway → New Project → Deploy from GitHub (or create a Template once), then set DAOXE_API_KEY, DAOXE_MODEL, and your Discord bot token. The bot process holds the key; Discord users just type /ask. (This one has a single runtime dependency, discord.js.)

Env vars cheat sheet

Variable	Where	Notes
`DAOXE_API_KEY`	all three	Billing credential. Server-side only.
`DAOXE_MODEL`	vercel-chat, railway bot	Exact id from `GET /v1/models`.
`DAOXE_ALLOWED_MODELS`	vercel-chat (optional)	Comma-separated allowlist.
Share / Discord tokens	render proxy / railway bot	Separate from the billing key.

(Repos are public. Vercel/Render one-click buttons live in each README; Railway is "Deploy from GitHub" from the public repo — no private template id required.)

What this buys you

Three one-click AI starters: Vercel (streaming chat), Render (OpenAI-compatible proxy), Railway (Discord bot).
Key never in the browser — the failure mode most tutorials ship by accident.
One key → many models behind a single OpenAI-compatible base URL, with fallback and one bill if you want it.

Don't trust the pitch — deploy one template, open the network tab, and confirm the key is not in any client request. Then point the same templates at any OpenAI-compatible endpoint and compare.

One key, two ecosystems: routing between DeepSeek/Kimi/Qwen and Claude/GPT/Gemini by task

Seven — Sun, 19 Jul 2026 22:40:57 +0000

Picking a single model family for an app is a false choice. Some Chinese-origin models are strong and cost-efficient at certain tasks; some Western models lead at others. If you're building outside mainland China, the practical move isn't to bet on one — it's to route each request to the right one, and make switching a one-line change instead of a new integration.

This guide shows how to do that through a single OpenAI-compatible endpoint, so DeepSeek, Kimi, Qwen, Claude, GPT and Gemini all sit behind one key and one bill — and how to choose between them per task without guessing.

Disclosure & scope: I work on daoxe, an OpenAI-compatible gateway. It is not available in mainland China — this article is written for developers outside it (overseas devs, overseas Chinese, and users in Russia, Brazil, Vietnam, Turkey and elsewhere) who want access to both ecosystems in one place. Everything here is standard OpenAI-compatible config; point it at any endpoint that carries the models you want.

Why route at all

Three reasons a single-model app leaves value on the table:

Task fit. Models have uneven strengths — reasoning, long-context, code, multilingual, structured output, vision. The best model for summarizing a 200-page PDF isn't necessarily the best for a tricky refactor.
Cost. For high-volume, low-stakes calls (classification, extraction, routing itself), a cheaper model is often good enough, and the savings compound. Reserve the expensive model for the calls that need it.
Resilience. If one provider is rate-limited or degraded, being able to fail over to another family keeps you shipping.

The blocker has always been integration cost: different SDKs, keys, billing, and response quirks per vendor. An OpenAI-compatible endpoint removes it — model choice becomes a string.

The mechanism: model choice is a string

Every call uses the same shape; only the model field changes:

curl https://daoxe.com/v1/chat/completions \
  -H "Authorization: Bearer $DAOXE_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"<MODEL_ID>","messages":[{"role":"user","content":"..."}]}'

List what your key can actually call — the authoritative, account-scoped source is:

curl https://daoxe.com/v1/models -H "Authorization: Bearer $DAOXE_API_KEY"

Use those exact ids. Families you'll typically see include Chinese-origin ones (DeepSeek, Kimi, Qwen, and others) and Western ones (Claude, GPT, Gemini). Don't hardcode ids from this post — they drift; read them from /v1/models.

A simple task-based router (Python)

You don't need a framework. A dict from task → model id gets you 90% of the value:

import os, httpx

BASE = "https://daoxe.com/v1"
KEY = os.environ["DAOXE_API_KEY"]

# Map task types to EXACT ids from GET /v1/models (examples — replace with yours).
ROUTES = {
    "cheap_bulk":   "deepseek-...",     # high-volume extraction/classification
    "long_context": "kimi-k2.5",        # very long inputs
    "reasoning":    "claude-sonnet-4-6",# hard multi-step problems
    "code":         "gpt-5.5",          # tricky refactors
    "default":      "claude-haiku-...", # fast, inexpensive fallback
}

def ask(task: str, prompt: str) -> str:
    model = ROUTES.get(task, ROUTES["default"])
    r = httpx.post(f"{BASE}/chat/completions",
        headers={"Authorization": f"Bearer {KEY}"},
        json={"model": model, "temperature": 0,
              "messages": [{"role": "user", "content": prompt}]}, timeout=60)
    r.raise_for_status()
    return r.json()["choices"][0]["message"]["content"]

print(ask("cheap_bulk", "Extract the invoice total as JSON."))
print(ask("reasoning",  "Prove or disprove: ..."))

Because it's one endpoint and one key, adding a route is adding a dict entry — no new SDK, no new credential. Add a try/except that falls back to another family on error and you've got cheap resilience too.

How to choose the route (without vibes)

Don't route on reputation — route on evidence for your workload:

Build a tiny eval set of real prompts per task type with known-good answers.
Run each candidate model through it, and record accuracy alongside latency (p50/p95) and token cost. (A small harness does this; I wrote up a reproducible gateway benchmark separately.)
Pick the cheapest model that clears your quality bar for each task — not the most expensive one you can afford.
Re-check periodically. New model versions ship constantly; a route that was optimal last quarter may not be now.

On price specifically: I'm not going to quote numbers here because they change. Check the provider's pricing page and put real $/1M figures next to your latency and accuracy — the right route is a three-way trade-off, not a single-axis choice.

Don't forget to verify what you're getting

Routing across families makes verification more important, not less: with several models behind one endpoint, you want to confirm each id actually serves what it claims and isn't being silently swapped or truncated. Test behavior, not the model's self-report — a tokenizer fingerprint, a capability-floor check, and a long-context needle test catch the common cheats. (There are open-source probes for this; I covered the method in a separate post.) A good endpoint is fine with you running these against it.

Honest positioning

A few things to be straight about:

This is for outside mainland China. daoxe doesn't serve mainland China; the audience here is overseas developers and users in regions where getting both ecosystems in one place — and paying for it — is otherwise fiddly.
"Chinese vs Western" is about model origin, not a quality ranking. Route on measured task fit and cost, not on where a model came from.
One key is a convenience, not magic. It removes integration and billing friction; it doesn't make a small model into a big one. Match the model to the task.

daoxe is the endpoint I use for this — OpenAI-compatible at https://daoxe.com/v1, native Anthropic Messages for Claude-native tools, one key across many models (Chinese-origin and Western), one bill, and a design meant to be verified. See the pricing page for current models and prices; use GET /v1/models for what your key can call.

TL;DR

Don't pick one family — route each task to the best/cheapest model.
One OpenAI-compatible endpoint makes model choice a string: one key, one bill, no per-vendor SDKs.
Choose routes on a small eval set (accuracy + p50/p95 + $/1M), not reputation; re-check as versions ship.
Verify each model behaviorally, and remember this is for use outside mainland China.

Add an AI chat widget to any website — without leaking your API key

Seven — Sun, 19 Jul 2026 22:31:19 +0000

Adding an AI chat box to a website is a five-minute job — and most five-minute tutorials get the one important thing wrong: they put an API key in the front-end. This walkthrough uses a tiny, dependency-free widget that refuses to let you do that, and shows the two correct ways to handle the key depending on whether the page is yours or public.

The widget is ~11 KB of vanilla JS — no build step, no framework — and it talks to any OpenAI-compatible Chat Completions endpoint. I'll use daoxe as the default (disclosure below), but it's one line to point it anywhere.

Disclosure: I work on daoxe, the widget's default endpoint. Nothing here is locked to it — change one line and it talks to any compatible endpoint.

The uncomfortable truth: anything in the browser is public

An LLM API key is a billing credential. Anything you ship to a browser — including a key in your JS bundle — is readable by every visitor. So the widget deliberately has no apiKey option in its config. Instead you pick one of two modes:

Mode	Who holds the key	Use it for	Trade-off
`byok` (default)	the visitor pastes their own key; kept only in their browser	a personal/local page only you use; dashboards; internal tools	needs the endpoint to allow browser CORS
`proxy` (recommended for public sites)	your server injects the key; the browser never sees it	any public website	you run a tiny proxy (examples included)

That single design choice is the whole point. If a widget lets you drop your own key into a public page's source, it's shipping your credential to the world.

A) BYOK mode — for a page that is yours

For a personal dashboard or an internal tool, the visitor supplies their own key, stored only in their browser's localStorage and sent straight to the endpoint:

<script src="daoxe-widget.js"></script>
<script>
  DaoxeChat.init({
    baseUrl: "https://daoxe.com/v1",
    model: "YOUR_DAOXE_MODEL_ID",   // exact id from GET /v1/models
    title: "Ask me anything"
  });
</script>

This only works if the endpoint returns permissive CORS headers (many gateways don't enable CORS on /chat/completions — that's what proxy mode is for). If you see a "network/CORS" error, switch to proxy.

B) Proxy mode — for public sites

For anything public, the browser talks to your route, and your server adds the key:

<script src="daoxe-widget.js"></script>
<script>
  DaoxeChat.init({
    mode: "proxy",
    proxyUrl: "/api/chat",          // a route you control that adds the key
    model: "YOUR_DAOXE_MODEL_ID",
    title: "Ask us anything"
  });
</script>

Ready-to-run proxies ship in examples/:

Cloudflare Worker — deploy in minutes; key stored as a Worker secret.
Node/Express — reads DAOXE_API_KEY from the environment and streams straight through.

Both keep the key server-side and can optionally allow-list which models the proxy will forward. In production, restrict Access-Control-Allow-Origin to your domain.

Visitor ──▶ /api/chat (your proxy, holds key) ──▶ https://daoxe.com/v1/chat/completions
        ◀── streamed tokens ◀───────────────────

Security details worth knowing

A chat widget renders untrusted model output into your page, so two things matter beyond the key:

The widget escapes all model output before rendering — it doesn't innerHTML raw model text. A tiny markdown-ish renderer handles code/bold/newlines on already-escaped text, and it does not execute HTML returned by the model. (If you build your own widget, do this — model output is untrusted input.)
Lock down the proxy. Restrict CORS to your domain and optionally allow-list models so a leaked proxy URL can't be used to run arbitrary expensive models on your bill.

Configuration highlights

DaoxeChat.init({ ... }) takes the usual knobs — title, subtitle, greeting, accent, position, startOpen, systemPrompt, and (BYOK only) temperature / maxTokens. Two worth calling out:

model is required — an exact id available to the key/account. Don't hardcode a list from a blog post; the authoritative list is GET {baseUrl}/models.
storageKey controls the localStorage key for BYOK; set it to null to never persist the visitor's key.

The widget is minimal by design: no file uploads, no image input, no tool calls, no cross-page persistence. Streaming uses the standard OpenAI SSE format and falls back to a single JSON read if an endpoint doesn't stream.

Why an OpenAI-compatible endpoint helps here

Because the widget speaks the standard Chat Completions shape, the same embed can serve GPT, Claude, Gemini or DeepSeek by changing a model id — one key, one bill, and you can repoint baseUrl to any provider you trust and verify. daoxe is the default (https://daoxe.com/v1, one key across many models, native Anthropic Messages too) and is not available in mainland China — but the widget privileges nothing; baseUrl and the "Powered by" link are one-line changes.

TL;DR

One <script> tag, ~11 KB, no build, any OpenAI-compatible endpoint.
Never put your key in front-end code. Use BYOK (your pages) or proxy (public sites).
Proxy examples included (Cloudflare Worker, Node/Express); lock CORS to your domain in prod.
Escape model output; require an exact model id from /v1/models.

One API key, three terminals: driving many models from the shell with llm, mods and aichat

Seven — Sun, 19 Jul 2026 22:24:18 +0000

The fastest way to use an LLM is often not a chat window — it's a pipe. git diff | ... "write a commit message", ... "explain this error" < stderr.log, cat main.py | ... "find the bug". Three of the best command-line tools for this — llm, mods, and aichat — all accept a custom OpenAI-compatible endpoint, which means one key can back all three and give you Claude, GPT, Gemini and more from the shell.

This is the terminal setup, end to end. It's genuinely tool-agnostic: every config below works against any OpenAI-compatible endpoint. I use daoxe as the example endpoint (disclosure below); swap the base URL for your own.

Disclosure: I work on daoxe, an OpenAI-compatible gateway (it also speaks Anthropic Messages natively). Everything here is standard OpenAI-compatible config — point these tools at whatever endpoint you trust.

One convention across all three: the key lives in an environment variable, DAOXE_API_KEY, never on the command line (so it doesn't land in your shell history). And model ids are account-scoped — list yours with curl https://daoxe.com/v1/models -H "Authorization: Bearer $DAOXE_API_KEY".

1. `llm` (Simon Willison) — the extensible one

llm is the most extensible of the three: a pluggy-based plugin system, a local SQLite log of everything, templates, embeddings. There are two ways to add an OpenAI-compatible gateway.

Option A — zero-YAML via a plugin

There's a small plugin, llm-daoxe, that registers the gateway's models and adds a command to pull your account's real model list:

llm install llm-daoxe           # (coming soon on PyPI; until then: clone + `pip install -e .`)
llm keys set daoxe              # store the key (or export DAOXE_API_KEY=...)
llm daoxe models --refresh     # pull your account's real models from GET /v1/models, and cache them
llm -m daoxe/claude-sonnet-4-6 "In one line, what's an OpenAI-compatible gateway?"
git diff | llm -m daoxe/gpt-5.5 -s "Write a concise commit message."

Model ids are namespaced as daoxe/<model> so they don't collide with other plugins' ids. The key is read only from llm's key store or DAOXE_API_KEY, and it's never printed.

Option B — no plugin, native config

Don't want a plugin? llm natively reads custom OpenAI-compatible models from extra-openai-models.yaml in its config dir:

- model_id: daoxe/claude-sonnet-4-6
  model_name: claude-sonnet-4-6
  api_base: https://daoxe.com/v1
  api_key_name: daoxe          # matches `llm keys set daoxe`
  can_stream: true
- model_id: daoxe/gpt-5.5
  model_name: gpt-5.5
  api_base: https://daoxe.com/v1
  api_key_name: daoxe

The plugin's advantage is just convenience: zero YAML plus llm daoxe models --refresh to auto-discover your account's models. Both paths use llm's built-in OpenAI transport, so streaming and tools behave the same as with official models.

2. `mods` (charmbracelet) — the pipe-native one

mods is built for Unix pipes and looks great doing it. Open its config with mods --settings and add a provider under apis::

default-api: daoxe            # optional: make it the default so you can skip -a
apis:
  daoxe:
    base-url: https://daoxe.com/v1
    api-key-env: DAOXE_API_KEY
    models:                  # use real ids from GET /v1/models; max-input-chars is a client-side value
      claude-sonnet-4-6:
        aliases: ["daoxe-sonnet"]
        max-input-chars: 600000
      gpt-5.5:
        aliases: ["daoxe-gpt"]
        max-input-chars: 600000

export DAOXE_API_KEY=your-daoxe-key
mods -a daoxe -m claude-sonnet-4-6 "explain this diff" < patch.diff
mods --model daoxe-sonnet "summarize" < README.md      # via alias

Gotcha: mods requires api-key-env to point at an environment variable that already exists, or it complains about a missing key. max-input-chars is a client-side truncation setting — tune it to your model, it's not a claim about the gateway.

3. `aichat` (sigoden) — the all-in-one one

aichat is a single binary that's a CLI, a REPL, and a local server. It uses an openai-compatible client type. Edit ~/.config/aichat/config.yaml:

clients:
  - type: openai-compatible
    name: daoxe
    api_base: https://daoxe.com/v1
    # omit api_key and aichat auto-reads DAOXE_API_KEY (client name → {NAME}_API_KEY)
    models:                     # real ids from GET /v1/models; tune capabilities per model
      - name: claude-sonnet-4-6
        max_input_tokens: 200000
        supports_function_calling: true
      - name: gpt-5.5
        max_input_tokens: 272000
        supports_function_calling: true

export DAOXE_API_KEY=your-daoxe-key
aichat -m daoxe:claude-sonnet-4-6 "explain this code" < main.py
aichat --list-models | grep daoxe

Nice touch: because the client is named daoxe, aichat automatically looks for DAOXE_API_KEY — you can omit the api_key field entirely. Model selection is client:model, e.g. daoxe:claude-sonnet-4-6.

Which one should you use?

Tool	Best for	Superpower
`llm`	scripting, logging, plugins	SQLite log of every call; huge plugin ecosystem
`mods`	quick pipes	gorgeous TUI output, alias/fallback config
`aichat`	interactive + server	REPL, roles, and a built-in local API server

They're not mutually exclusive — I keep all three configured against the same key. The same mental model applies to other CLIs too: anything that takes an OpenAI-compatible base_url + key + model id works the same way (point the base URL at the /v1 root, use a real id from /v1/models, keep the key in an env var). Verify each tool's current config fields before you rely on them — these tools move fast.

The reason one key matters here

CLI users tend to script things and forget them: a cron job that summarizes logs, a git hook that drafts commit messages, a shell function that explains errors. When each of those needs a different vendor key, it's a mess of secrets and bills. One OpenAI-compatible key across llm, mods, and aichat — and across every script that calls them — means one secret to rotate, one bill to read, and model choice as a string.

daoxe is the endpoint I point them at: https://daoxe.com/v1, one key across many models, plus native Anthropic Messages if a tool wants Claude's protocol. It's not available in mainland China. As always, none of the config above is daoxe-specific — and because it's a standard endpoint, you can benchmark and verify it rather than trust it.

TL;DR

llm, mods, and aichat all take a custom OpenAI-compatible endpoint — one key backs all three.
llm: a plugin (llm-daoxe, coming soon) or extra-openai-models.yaml.
mods: an apis: block; api-key-env must point at an existing env var.
aichat: an openai-compatible client; name it daoxe and it auto-reads DAOXE_API_KEY.
Keep the key in an env var, use exact ids from /v1/models, and pipe away.

Bring your own key to GitHub Copilot: Chat, CLI, App and SDK (with the honest caveat)

Seven — Sun, 19 Jul 2026 22:13:54 +0000

Most people don't know Copilot can do this yet: as of 2026, GitHub Copilot has GA'd Bring Your Own Key (BYOK). You can point Copilot Chat (in VS Code), the Copilot CLI, the Copilot app, and the Copilot SDK at any OpenAI-compatible endpoint, Anthropic, or Azure — including local runtimes and third-party gateways.

That's a big deal if you already pay for Copilot but want to run its chat and agent features on a model or a gateway of your choice. This guide covers all four surfaces, with the exact steps and one honest caveat that trips people up.

Disclosure: I work on daoxe, an OpenAI-compatible gateway that also speaks the Anthropic Messages protocol. It happens to line up neatly with Copilot's three wire protocols, so I use it as the example — but everything here works with any compatible endpoint. Substitute your own base URL.

The caveat, first (so nobody's misled)

BYOK affects Copilot Chat and agent sessions. It does not change inline completions. Those gray "Tab to accept" suggestions still run on Copilot's own infrastructure and are unaffected by your base URL. If your goal is to change what powers chat/agent, BYOK is for you. If you wanted to swap the model behind inline completions, BYOK won't do it. That's the honest scope.

Why a gateway fits Copilot's three protocols

Copilot's BYOK supports three provider "types", and a good multi-protocol gateway hits all three:

openai → the gateway's /v1/chat/completions (default).
openai with wireApi: "responses" → the gateway's /v1/responses.
anthropic → the gateway's /v1/messages (Claude's native protocol).

So one key can serve GPT-shaped, Responses-shaped, and Claude-native requests, depending on which model you pick. Now the four ways to wire it.

1. Copilot CLI (easiest — just env vars)

export COPILOT_PROVIDER_BASE_URL=https://daoxe.com/v1
export COPILOT_PROVIDER_API_KEY=$DAOXE_API_KEY     # your gateway key, from an env var
export COPILOT_MODEL=gpt-5.5                        # any exact id from GET /v1/models
# optional: COPILOT_PROVIDER_TYPE=openai (default) | anthropic | azure
copilot

For Claude's native protocol, set COPILOT_PROVIDER_TYPE=anthropic and use the gateway's Anthropic entry point. Confirm the exact path with curl first — many gateways take the host root (https://daoxe.com) and let the client append /v1/messages; some want /v1. Don't assume; check your account.

2. Copilot Chat (VS Code)

Command Palette → Chat: Manage Language Models.
Choose OpenAI Compatible / Custom Endpoint.
Base URL: https://daoxe.com/v1 (it must serve /chat/completions). VS Code probes GET /models to fill the model dropdown — if your gateway supports it, models auto-populate; otherwise type the Model ID by hand.
Paste your API Key, pick a Model ID (e.g. claude-sonnet-4-6) → Add Model.

The model now shows up in the Copilot Chat model picker alongside the built-ins.

3. Copilot App (desktop)

Settings → Model Providers → Add provider. Fill in the endpoint https://daoxe.com/v1 and your API key (or choose the anthropic type for Claude-native). Your models then appear in the model picker next to Copilot's hosted ones; the key is stored in your OS keychain.

4. Copilot SDK (for integrators)

provider: {
  type: "openai",                    // or "anthropic"
  baseUrl: "https://daoxe.com/v1",   // full path; for anthropic, use your account's path
  apiKey: process.env.DAOXE_API_KEY,
  wireApi: "completions",            // use "responses" when you need Responses-API behavior
}

Getting the model ids right

Copilot only calls what you tell it to, so use the exact ids your endpoint exposes:

curl https://daoxe.com/v1/models -H "Authorization: Bearer $DAOXE_API_KEY"

Guessed names fail silently or fall back. A gateway with an account-scoped /v1/models is ideal here, because the list you see is exactly the list you can call.

Troubleshooting

Model dropdown empty in VS Code? Your endpoint may not expose GET /models, or the key lacks access. Type the id manually and confirm the key with the curl above.
401 / auth errors? Check whether the provider expects Authorization: Bearer (OpenAI type) vs x-api-key (some Anthropic setups). Match the provider type to the protocol.
Anthropic type 404s? It's almost always the base URL shape — root vs /v1. Verify against your gateway's docs.
Chat works but inline completions "didn't change"? Expected — see the caveat. Inline stays on Copilot's models.

What to look for in a BYOK target

Since you're choosing the backend now, pick one that is:

Multi-protocol. OpenAI Chat Completions and Responses and Anthropic Messages, so all of Copilot's provider types work and Claude runs on its native protocol.
One key, many models, with an account-scoped /v1/models.
Verifiable. Fine with you benchmarking it and checking it isn't silently downgrading you (there are open-source tools for exactly this).
Reachable where official billing isn't. If your card keeps getting declined at an official checkout, a gateway that accepts other payment paths is the difference between shipping and not.

daoxe checks those boxes — OpenAI-compatible base URL (https://daoxe.com/v1), native Anthropic Messages, one key across many models — and it's not available in mainland China. But BYOK is the point: you're not locked in, so point Copilot at whatever endpoint you trust and can verify.

TL;DR

Copilot BYOK works in Chat, CLI, App, and SDK — set a base URL + key.
A multi-protocol gateway covers all three Copilot provider types (openai / responses / anthropic).
Use exact model ids from /v1/models.
Inline completions don't change — BYOK is for chat/agent only.

One key for CrewAI, AutoGen, LlamaIndex and 8 more: the base_url trick for Python agents

Seven — Sun, 19 Jul 2026 22:11:15 +0000

Here's a fact that quietly makes multi-model agent development a lot less painful: in 2026, virtually every Python agent framework natively supports pointing its underlying LLM at a custom OpenAI-compatible base_url. No new package, no fork, no framework change. You set one URL and one key, and the framework talks to whatever backend you choose.

That means a single OpenAI-compatible gateway — one key that fronts Claude, GPT, Gemini, DeepSeek, Kimi and more, with fallback and one bill — is a natural backend for all of these frameworks at once. The orchestration stays with the framework; the models come from one endpoint.

This post is the map: the exact one-liner for each framework, plus the one honest gotcha each has, because those gotchas are what actually cost you an afternoon.

Disclosure: I work on daoxe, one such gateway. But nothing below is daoxe-specific — every snippet works against any OpenAI-compatible endpoint. Swap the base URL for your own; the frameworks don't care who's behind it.

Why this works at all

Almost every framework builds on the OpenAI Chat Completions shape:

POST {base_url}/chat/completions
Authorization: Bearer <key>

The only two things that route this call to OpenAI are the base URL and the key. Change the base URL to a compatible gateway and the same orchestration code now runs against a different backend. Frameworks expose this under slightly different names — base_url, api_base, api_base_url — but it's the same idea.

Two universal rules before the snippets:

Model ids are account-scoped. Don't copy an id from a blog post. List yours with curl {base_url}/models -H "Authorization: Bearer $KEY" and use an exact id.
Keys go in env vars, never hardcoded. Every snippet below reads from the environment.

The 11 frameworks, grouped by mechanism

A) Frameworks with a direct `base_url` parameter

LangChain (+ LangGraph) — the biggest ecosystem, so start here:

from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="MODEL_ID", base_url="https://daoxe.com/v1", api_key="...")  # key via env
print(llm.invoke("ping").content)

# LangGraph: feed that llm straight into a prebuilt agent
from langgraph.prebuilt import create_react_agent
agent = create_react_agent(llm, tools=[...])

Gotcha: ChatOpenAI targets the official OpenAI spec; non-standard fields some proxies add aren't preserved. Fine for standard responses.

AutoGen (Microsoft AgentChat):

from autogen_agentchat.agents import AssistantAgent
from autogen_core.models import ModelInfo
from autogen_ext.models.openai import OpenAIChatCompletionClient

client = OpenAIChatCompletionClient(
    model="MODEL_ID", base_url="https://daoxe.com/v1", api_key="...",
    model_info=ModelInfo(vision=False, function_calling=True, json_output=True,
                         family="unknown", structured_output=True),
)
agent = AssistantAgent("assistant", model_client=client)

Gotcha: a non-OpenAI model id needs both base_url and model_info (capability flags). Omit model_info and it errors. Set the flags to match the model behind your id.

smolagents (Hugging Face):

from smolagents import CodeAgent, OpenAIModel
model = OpenAIModel(model_id="MODEL_ID", api_base="https://daoxe.com/v1", api_key="...")
print(CodeAgent(tools=[], model=model).run("ping"))

Gotcha: pre-1.9 the class was OpenAIServerModel (same params). Uses api_base, not base_url.

Haystack (deepset):

from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret

gen = OpenAIChatGenerator(model="MODEL_ID", api_base_url="https://daoxe.com/v1",
                          api_key=Secret.from_env_var("DAOXE_API_KEY"))
print(gen.run([ChatMessage.from_user("ping")])["replies"][0].text)

Gotcha: the param is api_base_url (note the extra word), and the key goes through Secret.from_env_var(...).

Agno (formerly phidata):

from agno.agent import Agent
from agno.models.openai.like import OpenAILike
Agent(model=OpenAILike(id="MODEL_ID", base_url="https://daoxe.com/v1", api_key="...")).print_response("ping")

Gotcha: use OpenAILike (from agno.models.openai.like), not the plain OpenAIChat, for a third-party endpoint.

B) Frameworks with an explicit provider object

PydanticAI — type-safe agents:

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider

model = OpenAIChatModel("MODEL_ID",
    provider=OpenAIProvider(base_url="https://daoxe.com/v1", api_key="..."))
agent = Agent(model)
print(agent.run_sync("ping").output)

Gotcha: build a generic OpenAIProvider and pass it to OpenAIChatModel — this is the same pattern the docs use for any OpenAI-compatible provider.

LlamaIndex:

from llama_index.llms.openai_like import OpenAILike  # pip install llama-index-llms-openai-like

llm = OpenAILike(model="MODEL_ID", api_base="https://daoxe.com/v1", api_key="...",
                 is_chat_model=True, is_function_calling_model=True, context_window=128000)
print(llm.complete("ping"))

Gotcha: set is_chat_model=True or it calls the /completions endpoint instead of /chat/completions. Also set a real context_window.

C) Frameworks that route through LiteLLM (need the `openai/` prefix)

CrewAI:

from crewai import LLM, Agent, Crew, Task
llm = LLM(model="openai/MODEL_ID", base_url="https://daoxe.com/v1", api_key="...")
agent = Agent(role="Greeter", goal="Greet briefly.", backstory="Concise.", llm=llm)
task = Task(description="ping", expected_output="A short greeting.", agent=agent)
print(Crew(agents=[agent], tasks=[task]).kickoff())

Gotcha: keep the openai/ prefix on the model id. Without it, CrewAI (via LiteLLM) may match a native provider client for a familiar model name and ignore base_url — you'll get a misleading "API key not valid". Use the API root (.../v1), not .../chat/completions.

DSPy:

import dspy
lm = dspy.LM("openai/MODEL_ID", api_base="https://daoxe.com/v1", api_key="...")
dspy.configure(lm=lm)
print(dspy.Predict("question -> answer")(question="ping").answer)

Gotcha: same openai/ prefix, and keep /v1 in api_base with no trailing slash — a trailing slash can double the path and 404.

D) Wrap the OpenAI client directly

Instructor — structured output validated into Pydantic models:

import instructor
from openai import OpenAI
from pydantic import BaseModel

class Out(BaseModel):
    text: str

client = instructor.from_openai(OpenAI(base_url="https://daoxe.com/v1", api_key="..."))
print(client.chat.completions.create(model="MODEL_ID", response_model=Out,
        messages=[{"role": "user", "content": "ping"}]))

Gotcha: the default (tool-calling) mode works for tool-capable models; if a model lacks tool calling, pass mode=instructor.Mode.JSON.

The gotchas, in one table

Framework	Param name	The one gotcha
LangChain	`base_url`	non-standard proxy fields not preserved
AutoGen	`base_url`	non-OpenAI id needs `model_info` too
smolagents	`api_base`	class renamed from `OpenAIServerModel`
Haystack	`api_base_url`	key via `Secret.from_env_var`
Agno	`base_url`	use `OpenAILike`, not `OpenAIChat`
PydanticAI	`OpenAIProvider(base_url=)`	generic provider, not a vendor class
LlamaIndex	`api_base`	`is_chat_model=True` or it hits `/completions`
CrewAI	`base_url`	model must be `openai/<id>`
DSPy	`api_base`	`openai/` prefix + no trailing slash
Instructor	wrap `OpenAI(base_url=)`	use `Mode.JSON` if no tool calling

Why one endpoint beats juggling keys

Multi-agent orchestration is where model sprawl hurts most: a planner on a strong model, workers on a cheap one, a critic on a third — that's three vendors, three keys, three bills, three failure modes. Pointing all of them at one OpenAI-compatible endpoint means:

One key, one bill across every framework and every agent role.
Model choice is a string, not a new integration. Swap MODEL_ID and the same agent runs on Claude, GPT, or Gemini.
You're not locked to a framework. The same key runs the same task across all 11.

And because it's a standard endpoint, you can verify it rather than trust it — run a probe against it and against the official API and compare (I wrote a separate piece on catching silent model swaps).

daoxe is the endpoint I use for this: OpenAI-compatible at https://daoxe.com/v1, plus native Anthropic Messages for Claude-native tools, one key across many models. It's not available in mainland China. But the whole point of this article is that none of the code above is daoxe-specific — point it at any compatible endpoint you trust.

TL;DR

Every framework here takes a custom OpenAI-compatible endpoint natively — it's a one-liner.
Watch the param name (base_url vs api_base vs api_base_url) and the per-framework gotcha.
CrewAI/DSPy need the openai/ prefix; AutoGen needs model_info; LlamaIndex needs is_chat_model=True.
One key, one bill, model choice as a string — and you can verify the backend instead of trusting it.

DEV Community: Seven

"OpenAI-compatible" is a spectrum, not a boolean — here's an 11-check conformance suite

The nine surfaces

The script

Reading the output

What this can't tell you

Ten questions to answer before you route production traffic through someone else's LLM endpoint

1. Protocol coverage — native or translated?

2. Model ID transparency

3. Provenance — where does the capacity come from?

4. Can you get your own data out?

5. Incident behaviour

6. Key handling

7. Data handling

8. Rate limits and behaviour under load

9. Billing you can reconcile

10. The exit plan

The scorecard, run against my own employer

Using this without turning it into a ritual

Model IDs are a dependency. Pin them like one.

Three ways a wrong ID fails

The ten-second test for Mode C

Dated versus floating

GET /v1/models is the source of truth, and it's per-key

Lock the catalogue

The alias layer

When an upstream retires a model mid-project

Variants and channels on a gateway

Drift under a pinned ID

Model IDs are a dependency. Pin them like one.

Three ways a wrong ID fails

The ten-second test for Mode C

Dated versus floating

GET /v1/models is the source of truth, and it's per-key

Lock the catalogue

The alias layer

When an upstream retires a model mid-project

Variants and channels on a gateway

Drift under a pinned ID

Your LLM API will fail in production. Here's the fallback system that catches it.

1. Why retry alone isn't enough

2. The model-level fallback pattern (simple, with sharp edges)

Gotcha 1: Protocol incompatibility

Gotcha 2: The system prompt disappears

Gotcha 3: Tool calling breaks between families

3. Routing by capability, not just availability

4. The gateway pattern: one endpoint, many models

5. Circuit breaker: when even fallback should stop

6. Observability: you can't fix what you can't see

7. Testing your fallback without waiting for an outage

Method 1: The unreachable endpoint

Method 2: The error-injection proxy

Method 3: The key rotation test

8. The honest limitations

9. What to do right now

OpenAI cut prices 80%. Here's why I'm not celebrating.

The hidden math of cheaper models

"What model are you?" doesn't work

Why this matters more now, not less

Test your endpoint right now

Deploy a real AI chat app in one click — key stays server-side (Vercel/Render/Railway)

The rule that makes these safe

Template 1 — Streaming chat UI on Vercel

Template 2 — OpenAI-compatible proxy on Render

Template 3 — Discord bot on Railway

Env vars cheat sheet

What this buys you

One key, two ecosystems: routing between DeepSeek/Kimi/Qwen and Claude/GPT/Gemini by task

Why route at all

The mechanism: model choice is a string

A simple task-based router (Python)

How to choose the route (without vibes)

Don't forget to verify what you're getting

Honest positioning

TL;DR

Add an AI chat widget to any website — without leaking your API key

The uncomfortable truth: anything in the browser is public

A) BYOK mode — for a page that is yours

B) Proxy mode — for public sites

Security details worth knowing

`GET /v1/models` is the source of truth, and it's per-key

`GET /v1/models` is the source of truth, and it's per-key

1. `llm` (Simon Willison) — the extensible one

2. `mods` (charmbracelet) — the pipe-native one

3. `aichat` (sigoden) — the all-in-one one

A) Frameworks with a direct `base_url` parameter

C) Frameworks that route through LiteLLM (need the `openai/` prefix)