DEV Community: Seven

Four ways reasoning models hide their thinking (and what that does to your bill)

Seven — Wed, 16 Sep 2026 08:40:23 +0000

Same question, four endpoints: "show your reasoning." Four different answers. Not four opinions — four wire formats, four billing behaviors, and four ways your audit trail goes blind. This piece is the map I wish I had before I assumed any of them. Disclosure: I work on daoxe, an OpenAI-compatible gateway — one key, many models, all four shapes behind one endpoint. Bias declared up front; the captures below include results that make my own gateway look wrong, and every probe here runs the same way against anyone's endpoint.

Shape 1 — shared budget (GLM)

GLM's reasoning-capable IDs run the chain of thought inside the completion. max_tokens is the budget for thinking and the answer together. Undersize it and you don't get an error — you get empty content with finish_reason: "length". The model "said" nothing because it spent everything thinking.

The trap is this: it looks exactly like a dead endpoint. The tell is in the finish reason — length means budget, not outage. Size the cap for the thinking first; the answer gets what's left.

Behind the gateway, same shape, same warning label — and the accounting gets weirder. I sent max_tokens: 8 at glm-4.7-flash through my own endpoint (free-group capture, 2026-09-16; signals not proof):

"message": { "content": "", "reasoning_content": "1.  **Analyze the User" },
"finish_reason": "length"

The budget-trap textbook sample, alive at my own party: eight tokens of pure thinking, the answer starved to an empty string. One detail to note for Shape 4, though: that same response reported completion_tokens_details.reasoning_tokens: 0 — even with the whole budget visibly spent on thinking. A counter that is present is not a counter that works; verify before you build a dashboard on it.

Shape 2 — separate field (DeepSeek)

DeepSeek's R-series puts reasoning in a dedicated reasoning_content field on the message, separate from content. Clean — until the second turn of a tool loop. DeepSeek's API documents that the previous assistant message must be echoed back including its reasoning_content, or the request fails with a 400.

Build your history from the raw message objects, never from a trimmed summary of the visible text. The moment a "helpful" history compactor strips the reasoning field, every subsequent call fails in a way that looks like a gateway problem.

Except the split itself is a property of the endpoint you're actually talking to, not of the model's name. I called deepseek-r1-32b through my gateway twice (free-group captures, 2026-09-16; signals not proof). Both times reasoning_content was absent entirely — the thinking came back fused into content, inside think-style delimiters:

content: "<think>\nFirst, the farmer initially has 17 sheep.\n…\n</think>\n\n**Solution:** …"

The 400-echo rule is DeepSeek's documented API contract — against the official endpoint. On a translated channel that never produced the field in the first place, there is nothing to echo and nothing to enforce, and my captures confirmed it: no error, just a fused blob. The lesson generalizes past my gateway: before your multi-turn logic depends on reasoning_content existing, check that your endpoint actually returns it. The model ID is not the contract; the wire is.

Shape 3 — explicit flag (MiniMax)

MiniMax reasoning IDs keep thinking out of the answer text only when asked: pass extra_body={"reasoning_split": True} and the chain of thought arrives in reasoning_details. Without the flag, the split is not guaranteed — reasoning may stay fused into content or be dropped.

If your pipeline needs machine-separable reasoning, set the flag on every call. Hope is not a configuration — and on some endpoints, neither is the flag.

No MiniMax model exists in my free-group catalog, so I probed the flag mechanism on two reasoning IDs that do, mimo-v2.5 and qwen3-30b (free-group captures, 2026-09-16; signals not proof). On my gateway, mimo-v2.5 with reasoning_split: true and without it produced the same response shape — thinking fused into content, no separate field, no error, no acknowledgement. The flag was silently swallowed. Then qwen3-30b with chat_template_kwargs: {"enable_thinking": false}: thinking came back anyway in a reasoning_content field, while content was empty and finish_reason was "stop". A wrong answer that looks like a finished one.

Two explicit instructions in the request, both silently overridden. That's the honest qualifier for Shape 3 as a family: the flag says what the vendor's docs promise; it does not guarantee what your endpoint honors. Verify the response shape every time — a configuration you can't observe being applied is a wish, not a setting.

Shape 4 — usage accounting only (Grok)

xAI reports thinking as an accounting detail, not a message field: usage.completion_tokens_details.reasoning_tokens is meant to count the completion tokens consumed by reasoning. The chain of thought itself is never returned.

That should make Grok the cheapest of the four to log — the numbers arrive in your usage stream already — and the least inspectable, since there is nothing to read, only to count. If your billing dashboards read only top-level usage numbers, thinking spend is invisible to them.

"Should." Here is one of three grok-4.3 calls through my gateway (free-group capture, 2026-09-16; signals not proof). The message: visible step-by-step reasoning inlined in content, no reasoning field anywhere. The usage:

"completion_tokens_details": { "reasoning_tokens": 0 }

The field is present and the count is zero on all three captures. So the Shape 4 caveat, earned the hard way: field present ≠ counting happening. An audit trail built on "the field exists, therefore the tokens are attributed" inherits whatever the implementation quietly doesn't do. Check that the counter actually moves before you trust it to allocate cost.

What each shape costs you

	Visible reasoning	Separable for parsing	Correct billing surface
GLM	sometimes (`reasoning_content`)	per-request	inside `max_tokens`
DeepSeek	per docs: always (field)	yes, where honored	separate field
MiniMax	only with the flag	yes, flag honored	standard
Grok	never	only via counter	`reasoning_tokens`, if it counts

Cost attribution, audit trails, client parsing and history management all break differently against each row. The GLM budget trap and the DeepSeek echo requirement are the two that turn into 2am pages; the Grok accounting gap is the one that turns into a quietly wrong bill. And the qualifier column is the point: "per docs", "where honored", "if it counts" — every row assumes a contract, and the captures above show a contract on each row moving under translation.

A four-line probe

You do not have to memorize any of this. Send one fixed prompt at temperature 0 and dump the three surfaces that differ:

import os, openai
c = openai.OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["KEY"])
r = c.chat.completions.create(model=os.environ["MODEL"], temperature=0,
    max_tokens=2048,  # room for GLM-style shared budgets
    messages=[{"role": "user", "content": "Reason step by step: what is 17*23?"}])
m = r.choices[0].message
print("finish_reason:", r.choices[0].finish_reason)
print("message keys:  ", sorted(m.model_dump().keys()))       # reasoning_content? reasoning_details?
print("usage details: ", r.usage.completion_tokens_details)   # reasoning_tokens?

Reading the output: empty content with finish_reason: "length" is Shape 1 (raise the cap). A reasoning_content key on the message is Shape 2 — and from then on, echo it in multi-turn. A reasoning_details field means Shape 3 — set the flag and verify it stuck. reasoning_tokens in the usage details with nothing on the message is Shape 4 — and a zero there does not mean nothing was thought; I watched Shape 4 channels report 0 while reasoning in plain sight. Run this when you pin the model, and again whenever the provider ships a new version; the shape is part of the contract, and contracts move.

The general rule

Never assume reasoning is visible, separable, or billable the same way twice. The four shapes above are current as of this writing, and providers move — which is exactly why the probe comes before the assumption, every time.

Anthropic Messages is a different protocol, not a flag — and shims leak

Seven — Wed, 16 Sep 2026 01:44:50 +0000

The failure has a shape. Your Claude-based agent connects to the new endpoint, answers the first question well, executes a tool, comes back, executes another — and then somewhere around step seven it stops calling tools and starts narrating that it would call a tool. Or it loses the thread of a long file edit. Or your prompt-cache hit rate, which you were counting on, is quietly zero and nothing in the response says so.

What usually happened is that the endpoint doesn't speak Anthropic's Messages protocol. It speaks OpenAI chat completions and translates. Disclosure: I work on daoxe, a gateway that implements /v1/messages alongside the OpenAI protocol — which is exactly why I've spent an unreasonable amount of time in the gap between the two. The tests at the end are written so you can point them at any endpoint, including mine.

To be fair to shims: if all you do is single-turn chat with text in and text out, a translation layer is genuinely fine, and it's how a lot of tooling gets broad model coverage cheaply. The argument below is about agents.

The two protocols disagree about what a message is

In OpenAI's chat completions, an assistant turn is content: string | null plus a separate tool_calls array. In Anthropic Messages, an assistant turn is an ordered array of typed content blocks: text, tool_use, thinking, image, and others, interleaved in the order the model produced them.

That ordering is real information. "Let me check the config file" → tool_use → "and also the lockfile" → tool_use is one assistant turn with four blocks. Flatten it into a string plus a call array and you've thrown away which text belonged before which call. On a single turn nobody notices. Fifteen turns into an agent loop, the transcript the model sees is subtly not the transcript it wrote, and behaviour drifts.

tool_use and tool_result are shaped differently on purpose

Three concrete mismatches:

input is a JSON object. Anthropic's tool_use block carries input: {...}. OpenAI carries function.arguments as a JSON-encoded string. A shim has to serialise and deserialise on every hop. Usually that round-trips fine; when it doesn't — key ordering, float formatting, non-ASCII escaping — the bytes change, and byte-identical prefixes are exactly what prompt caching depends on.
tool_result belongs to a user turn. In Anthropic's model, the tool output comes back as a tool_result content block inside a user message, referencing tool_use_id. OpenAI uses a dedicated role: "tool" message. Mapping between them is mechanical but lossy in one direction: Anthropic's tool_result has an is_error: true flag, and OpenAI's has nowhere to put it. Through a shim, a failed tool call arrives as an ordinary string that happens to start with "Error:", and the model has to infer from prose what the protocol was willing to tell it directly.
Multiple parallel tool_use blocks in one assistant turn, each needing its own tool_result block in the following user turn, in an order the API validates. Shims that queue calls one at a time serialise what should have been parallel.

Streaming is a typed event protocol

Anthropic's SSE stream isn't a sequence of interchangeable chunks. It's message_start, then per block content_block_start / content_block_delta / content_block_stop carrying an index, then message_delta (which is where stop_reason and the final output token count live), then message_stop, with ping and error events possible anywhere.

The delta types matter too: text_delta for prose, input_json_delta for tool arguments streaming in as partial JSON, thinking_delta and signature_delta for extended thinking. The Anthropic SDK's streaming helpers accumulate these into typed objects for you. A shim re-wrapping OpenAI chunks can usually produce plausible text_delta events; what it typically cannot produce faithfully is incremental input_json_delta, so tool arguments show up all at once at the end. If you render tool calls live in a UI, that's a visible regression. If you gate on partial arguments, it's a broken feature.

`system` is a parameter, and that's where caching lives

Anthropic takes the system prompt as a top-level system field, either a string or an array of content blocks. The array form is the important one, because that's where you attach cache_control: {"type": "ephemeral"} to mark a cacheable prefix.

There is no place in OpenAI's message list to express that. So a shim converts your system blocks into a {"role": "system"} message and the cache markers evaporate. Nothing errors. Your requests just cost what they cost and take as long as they take, and usage.cache_read_input_tokens — the field that would have told you — isn't in an OpenAI-shaped response at all. For a coding agent that resends a large system prompt and file context on every turn, this is the single most expensive silent difference in this article.

`stop_reason` has six values

Anthropic returns end_turn, max_tokens, stop_sequence, tool_use, pause_turn, or refusal. OpenAI returns stop, length, tool_calls, or content_filter.

Agent loops branch on stop_reason == "tool_use". Through a shim that maps everything into OpenAI's smaller vocabulary and back, you can end up with end_turn on a turn that actually contained a tool call, at which point the loop exits and the agent "decides" to stop working. pause_turn and refusal have no OpenAI equivalent whatsoever and get mapped to something approximate.

Thinking blocks are signed

With extended thinking enabled, the response contains thinking blocks with a signature. On subsequent turns of a tool-use loop you must pass those blocks back verbatim, signature included. A shim that flattens thinking into text, or drops it as "internal", breaks that contract — you'll get an error on the next turn, or thinking will be silently disabled, and the model you're paying reasoning tokens for will stop reasoning.

One thing my captures taught me: the signature field is itself part of what an implementation can quietly not produce. On my gateway's /v1/messages, the thinking block came back with its text but no signature key at all (free-group capture, 2026-09-16; signals not proof). The shape looks right; the contract that depends on the shape is not fully honoured. If you build anything that replays thinking blocks, check for the field's presence before you trust the round trip.

What it costs you

Protocol feature	Through a shim	How you notice
Ordered content blocks	Flattened	Drift in long agent transcripts
`is_error` on tool_result	Lost	Model retries a tool that can't succeed
`input_json_delta`	Batched to the end	Tool args appear all at once in the UI
`cache_control`	Dropped	Nothing — no error, no cache field, higher cost
`stop_reason: tool_use`	Approximated	Agent loop exits early
`thinking` + signature	Stripped	Next-turn error, or thinking silently off
`max_tokens` enforced	Varies	A gateway can pass shape tests and still not enforce this one

Five tests

Test 1 is the one-liner, with a caveat I earned by running it. Anthropic's API documents max_tokens as required, and a strict native endpoint returns a 400 when you omit it. But a 200 is not a shim confession: enforcement varies across implementations — my own gateway's /v1/messages path answered the request happily without max_tokens (free-tier capture from 2026-09-16, signals not proof). So read this one one-sidedly: a 400 with an Anthropic-shaped error body is a strong native signal; a 200 sends you to tests 2–5.

BASE=https://api.daoxe.com          # note: no /v1 suffix here
curl -s -o /dev/null -w '%{http_code}\n' "$BASE/v1/messages" \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"model":"<exact-id>","messages":[{"role":"user","content":"hi"}]}'
# 400 + Anthropic error shape: strong native signal
# 200: inconclusive — enforcement varies; run the rest

Test 2 — the error envelope. Anthropic's shape is {"type":"error","error":{"type":"invalid_request_error","message":"..."}}. An OpenAI-shaped {"error":{"code":...}} is a tell.

curl -s "$BASE/v1/messages" -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" -H "content-type: application/json" \
  -d '{"model":"<exact-id>","max_tokens":16,"messages":[],"nonsense_field":1}' | jq .

The envelope is the better tell than any single field inside it. On my gateway the Messages path returned the right outer envelope but leaked its internals ("type":"new_api_error", not the Messages error vocabulary), while the chat-completions path returned the OpenAI-shaped code field — one request each, and you can see which door you came in.

Test 3 — tool_use fidelity. Check that stop_reason is tool_use and that the block's input is an object, not a string.

curl -s "$BASE/v1/messages" -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" -H "content-type: application/json" \
  -d '{
    "model":"<exact-id>","max_tokens":256,
    "tools":[{"name":"get_weather","description":"Current weather",
      "input_schema":{"type":"object","properties":{"city":{"type":"string"}},
      "required":["city"]}}],
    "tool_choice":{"type":"tool","name":"get_weather"},
    "messages":[{"role":"user","content":"Weather in Osaka?"}]
  }' | jq '{stop_reason, input_type: (.content[] | select(.type=="tool_use") | .input | type)}'
# want: {"stop_reason":"tool_use","input_type":"object"}

The input_type half is the one that usually separates the two worlds — a string where an object belongs is a translation scar. The stop_reason half is the stricter criterion, and don't assume any particular endpoint passes it: on my gateway's /v1/messages, forced tool_choice produced a proper tool_use block with an object input, but stop_reason: "end_turn" (free-group capture, 2026-09-16; signals, not proof). That is exactly the failure described above, arriving through the right-shaped door.

Test 4 — streaming event names. Grep the raw stream for the typed events.

curl -sN "$BASE/v1/messages" -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" -H "content-type: application/json" \
  -d '{"model":"<exact-id>","max_tokens":64,"stream":true,
       "messages":[{"role":"user","content":"Count to five."}]}' \
  | grep -o '^event: .*' | sort -u
# want at least: message_start, content_block_start, content_block_delta,
#                content_block_stop, message_delta, message_stop

Of the five, this one drew the cleanest line when I ran both paths of my gateway on the same day: the /v1/messages side emitted all six event names; the /v1/chat/completions side emitted zero event: lines — plain data: chunks, because that's what the OpenAI wire format is. (Free-group captures, 2026-09-16; signals not proof.) A shim can fake most of the field names one response at a time; re-typing a whole event protocol under every chunk is work almost nobody does.

Test 5 — prompt caching. Send a long system prompt as a block array with cache_control, twice, and read the usage fields. (Providers enforce a minimum cacheable prefix length, so make the system text genuinely long — a few thousand tokens.)

REQ='{"model":"<exact-id>","max_tokens":16,
 "system":[{"type":"text","text":"<several-thousand-tokens-of-stable-context>",
            "cache_control":{"type":"ephemeral"}}],
 "messages":[{"role":"user","content":"ok"}]}'
for i in 1 2; do
  curl -s "$BASE/v1/messages" -H "x-api-key: $ANTHROPIC_API_KEY" \
    -H "anthropic-version: 2023-06-01" -H "content-type: application/json" \
    -d "$REQ" | jq '.usage | {cache_creation_input_tokens, cache_read_input_tokens}'
done
# an OpenAI-shaped response has no cache fields at all — that absence is the tell
# if the fields are present, what you have learned is that the *schema* is Anthropic's.
# You have not learned that caching works. Keep reading.

That last line is a correction I have to make to my own draft. The expected pattern is creation > 0 on the first call and read > 0 on the second. I ran this twice against my gateway's /v1/messages path (free-tier capture, 2026-09-16) and got creation=0, read=5472 on pass one, and creation=0, read=3355 on pass two. A non-zero read count before any creation count is not prompt caching — it looks like an upstream implicit cache being echoed into Anthropic-shaped field names. Which is a more interesting failure than the easy one: an endpoint can present the correct field names, and even plausible-looking numbers, and still not honour cache_control at all. The presence of a field tells you about the schema. It tells you nothing about the semantics, and it is exactly the kind of signal an agent has no way to sanity-check at runtime. So the honest version of this test is: absence of the fields proves a shim; presence proves nothing — verify actual caching against your bill or the provider's own docs.

Read the battery as signal, not as a quiz score. Positive tells are stronger than negative ones: a tool_use block whose input is an object, all six stream event names, an Anthropic error envelope — each one is hard to fake and easy to check. Absences and 200s are weaker than they look: an endpoint can emit the right field names, as mine's caching counters and missing thinking signature show. The tests that catch a shim are the ones that look for what's not there — no event: lines, no cache fields, a string where an object belongs — because those are the parts a translation layer has to actively rebuild to hide.

Claude Code against a custom endpoint

Claude Code reads the base URL and credential from the environment:

export ANTHROPIC_BASE_URL=https://api.daoxe.com
export ANTHROPIC_AUTH_TOKEN=sk-...     # sent as Authorization: Bearer
# or: export ANTHROPIC_API_KEY=sk-...  # sent as x-api-key
export ANTHROPIC_MODEL=<exact-id-from-the-catalogue>
claude

Two notes. The base URL has no /v1 suffix — Claude Code appends /v1/messages itself, and a doubled /v1/v1/messages is the most common first-run mistake. And Claude Code uses a second small model for background work; the environment variable that names it has been renamed across releases, so check claude config in your version rather than copying a variable name from a blog post.

The honest version

A translation shim is a reasonable engineering decision for text chat, and if that's your workload you will never see any of this. The moment you're running a tool loop, streaming tool arguments to a UI, relying on cached prefixes, or using extended thinking, the protocol stops being an implementation detail — every one of those features lives in a field that has no home on the other side of the translation.

Run the five tests against whatever you're about to build on. The whole battery is a few minutes of curl, and every one of the degradations above is a runtime bug you'd otherwise discover in production, three months later, with no error message pointing at the protocol.

We pointed our model-honesty probe at our own gateway. It said SUSPICIOUS.

Seven — Tue, 15 Sep 2026 13:16:29 +0000

We built a CLI tool that gives you signals about whether an LLM API endpoint is serving the model it claims — or quietly handing you a smaller, quantized, or context-truncated substitute. In early September we pointed it at our own gateway.

It flagged us. SUSPICIOUS — Hard to explain if the claim were true.

This post is the full record: what the tool said, why it was wrong, how we knew the tool was the wrong one, and what changed. Every detail below is quoted from the raw JSON of those runs, and the fix itself is an open commit in the tool's public repo — so you can check my quoting instead of trusting it.

Disclosure: I work on daoxe, the gateway that got flagged. The tool is provider-neutral and open source — github.com/seven7763/llm-honesty-probe — and this is written as an engineer's post-mortem, not a victory lap. There is no version of this story where we come out looking flawless; the honest one starts with our own tool failing on us.

What the probe actually does

One paragraph of background, then we get to the incident. The probe runs a fixed battery of behavioral checks against any OpenAI- or Anthropic-compatible endpoint: a capability floor (easy tasks with answers the tool already knows), long-context recall (it plants the needle, so it knows the right answer), output determinism and backend-identity stability at temperature 0, a tokenizer fingerprint from usage deltas, and a self-report it deliberately weights near zero because "what model are you?" is trivially spoofed. Your key is read only from an environment variable, the endpoint host is masked on shareable verdict cards, and every signal is explicitly labeled what it is: heuristics, not proof.

The run

2026-09-04, against one of our own groups — glm-5.3-flash, the group this working session was itself running on at the time:

python3 -m llm_honesty_probe \
  --base-url https://jp.daoxe.com/v1 --claimed-model glm-5.3-flash --card

The card came back:

===  SUSPICIOUS  ===
Hard to explain if the claim were true.

[-] Tokenizer fingerprint
      No verdict — add --compare for a definitive check.
[!] Capability floor
      Failed 2/4 easy reasoning tasks a full-tier model rarely misses. (medium)
[!] Long context
      Lost a needle in long context (possible truncation). (medium)
[+] Consistency
      Stable model field / fingerprint across calls.
[-] Self-report (spoofable, low weight)
      Claimed model family unknown; can't compare self-report.

Two failed signals, both "possible downgrade", neither deniable-looking. The arithmetic task answered 396. Multistep answered 58. The string-reversal task answered… nothing. The character-counting task: nothing. An empty content with finish_reason: "length". The strict-JSON test — the prompt demands exactly {"a":42,"b":"ok"} — returned {"a":.

And then we ran it a second time, same version, same endpoint, and the card shifted under us: one floor task tripped instead of two; the strict-JSON test passed this time; the needle at 8,000 characters still failed; and the report's own headline softened to "Only low-confidence flags; likely noise, but worth a second run." The three temperature-0 calls came back identical this time — all three of them the empty string. The same traffic gave the same tool two different stories. This is the part where a marketing team would quietly not publish the card.

Three tells

What turned the investigation from "is our gateway dishonest?" into "is our probe dishonest?" was three details in the raw JSON, none of which the card shows:

1. The server quoted our request back at us. The tokenizer probe reported itself inconclusive with this detail string:

Can't fingerprint tokenizer: HTTP 400: {"error":{"message":"max_tokens
must be greater than 2 …","type":"invalid_request"}}

We had our own code send max_tokens=1 for the fingerprint calls — an old OpenAI-compat habit. Our gateway requires max_tokens > 2. No server lies like that; that 400 is the sound of our tool misusing our API contract.

2. The truncated JSON ended exactly at the budget. {"a": is three tokens of an answer that needs about eight. Three tokens in, because the probe — at that time — capped those completions at 64 tokens (32 on the long-context calls). An answer that stops precisely at the budget with finish_reason: "length" is the signature of running out of allowed tokens, not of a weak model. The empty replies in the reasoning tasks had the same shape: content: "", finish_reason: "length".

3. The asymmetry. If the backend had really been swapped for a weaker model, why did arithmetic (396) and multistep (58) pass while reversal and counting came back empty? A weaker model gets them wrong — it doesn't get them blank. "Wrong" is a capability signal. "Empty at exactly the token limit" is a plumbing signal.

The root cause was in the routing, and it was innocent: this group can serve reasoning-style backends, which spend completion tokens on hidden thinking before any visible answer. The budget went to thinking. The probe read the empty visible reply as "failed a task a full-tier model rarely misses." That is a false positive — a bug in our caller-side assumptions, filed against our server. One check in the raw archive settles who was wrong: the second v0.2.0 run — where the budget-starved replies mostly stopped happening — scored the capability floor merely inconclusive, "could be formatting rather than capability," while the needle call at 8k still failed at the then-32-token needle cap. Same bug, different face.

The fix, and the re-run

Same day, v0.2.1 (commit cb15e61):

A shared guard — budget_starved() — that recognizes "OK, but empty content with finish_reason: length" and forces those probes to inconclusive. Budget starvation is a property of the caller's max_tokens, not of the server's honesty, and it must never be able to produce SUSPICIOUS.
Default per-call budgets raised so short factual answers actually fit (our re-run passed --max-tokens 512).
The tokenizer probe now requests 4 tokens instead of 1, clearing the gateway-validation 400.
The built-in mock server grew a reasoning=True mode that reproduces the starvation shape, plus regression tests asserting it can never flag suspicion. 20/20 pass.

Re-run with the fixed tool: needle recall at 2,000 and 8,000 characters — both retrieved. Capability floor: 4/4, exact JSON. Consistency: identical output on all 3 calls at temperature 0. Verdict:

===  PASS  ===
No downgrade signals in this run.

Note what is not on that card. The tokenizer fingerprint still says "No verdict — add --compare," because in single-endpoint mode there's no reference to diff against; it now measures the vector cleanly, finds its closest public match, and still refuses to call it — by design, the claimed family has no public reference table entry. Self-report remains inconclusive and low-weight forever, because it's the one signal a dishonest endpoint can pre-write. We did not upgrade the accusation into a certificate. We turned a wrong red into an honest gray.

Why the wrong flag is the better story

Here's the uncomfortable logic. Any tool can print PASS; it's one string. A tool that has been publicly wrong about a concrete endpoint, caught, root-caused, fixed in an open commit, and re-run has shown you something no clean card can: its red flags are fallible, so they get investigated instead of believed, and the investigation itself is checkable. If the only version of this story you'd ever seen were "we ran the probe on ourselves and it passed," you should trust it less — that's what a rigged scoreboard looks like.

The false positive also taught us something about who this tool hurts. We built it to catch relays that quietly downgrade you. A miss leaves a bad provider unpunished today and time-series diffs catch it tomorrow. A false accusation frames an honest provider with a screenshot someone posts in a forum — and once a community learns that these cards convict the innocent, every flag starts getting ignored, including the true ones. Over-flagging doesn't just hurt the flagged; it degrades the signal for everyone. "Treat a flag as a reason to look closer, not a conviction" is easy to print and much harder to engineer for, because the defaults are asymmetric: the probe weights failure more than success, so the failure modes you feed it decide who gets accused. budget_starved() is the fix for one such mode. It is not the fix for all of them.

What we are not claiming

The discipline of "signals, not proof" points at us twice:

The SUSPICIOUS was wrong. That does not mean the current PASS is right in any certificate sense — it means this battery, run once, on one group, showed no downgrade signals at that moment. A single run is a snapshot; silent degradation is a time-series problem, which is what the --json diff mode in cron is for.
One group is not the gateway. Our other groups went untested — the test account simply didn't have the balance for more.
Differential mode is the stronger tool, and we didn't use it: point --compare at the official endpoint and the tokenizer question can become a real verdict instead of an inconclusive. We'd rather say "not settled" than let our own marketing settle it.
A provider that mirrored the official tokenizer, matched the capability floor, and held long context would pass this battery. We can't prove intent from the outside — ours or anyone's.

So: clone it, run it against us, run it against the endpoint you actually pay for, and compare both cards.

git clone https://github.com/seven7763/llm-honesty-probe
cd llm-honesty-probe
python3 -m llm_honesty_probe --self-test --card   # no key needed — see both verdict shapes first

Questions, corrections, or "your probe wrongly flagged my honest relay": the public support channel is Telegram @daoxe_ai, and the issue tracker is on the repo. The bug reports we want most are the ones about false positives — the next wrong SUSPICIOUS is exactly as damaging as the one we just ate.

Five coding agents, one endpoint: where the base URL field actually lives

Seven — Mon, 14 Sep 2026 14:53:30 +0000

Every one of these tools supports a custom OpenAI-compatible endpoint. That's not the interesting part. The interesting part is that the field is in five different places, three of them undocumented in the UI, and each tool fails in a way that doesn't look like a configuration error.

I run a gateway, so I see the support tickets from all five. Disclosure: I work on daoxe. Nothing below is specific to it — swap in any endpoint that speaks /v1/chat/completions. Menu paths in desktop tools drift every few releases, so the paths below were re-verified against each vendor's current documentation on 2026-09-09; if a path has moved since, the gotcha underneath it almost certainly hasn't.

Deliberately not covered: GitHub Copilot. Its BYOK flow is a different mechanism with different limits and it deserves its own write-up. Everything here is the "custom base URL" family.

Ground rule: never type a model ID from memory

Before touching any tool, get the exact strings your key can call:

curl -s https://api.daoxe.com/v1/models \
  -H "Authorization: Bearer $LLM_API_KEY" | python3 -m json.tool | grep '"id"'

Most of the failures below trace back to an ID that is almost right. claude-sonnet-4-5 and claude-sonnet-4-5-20250929 are not interchangeable, and a tool that gets a 404 from a model list it can't see will usually surface it as "connection failed".

And those IDs are a moving target in their own right — vendors rename, deprecate, and quietly re-point them. I wrote up that side of the problem — pinning IDs, detecting drift, what to log when the model field stops matching what you asked for — in Model IDs are a dependency. Pin them like one..

Cursor

Where: Settings → Models. Enter your key in the OpenAI API Key box, expand it, enable Override OpenAI Base URL, and set it to your endpoint's /v1 root. Then add your model with + Add model, typing the exact ID.

Three gotchas, in the order people hit them:

Verification calls a model, not the base URL. The "Verify" button issues a real completion. If the default OpenAI model names are still enabled in the list and your endpoint doesn't serve those exact IDs, verification fails even though your custom model works perfectly. Add your model first, toggle the stock ones off, then verify.
A custom key does not make every feature custom. Chat and inline edit go to your endpoint. Tab completion and the editor's own indexing/retrieval features are Cursor-side services and keep working the way they always did. This surprises people who expected the custom key to cover their entire bill.
Traffic shape isn't obvious. Depending on your privacy settings, requests may still transit Cursor's infrastructure rather than going browser-direct to your endpoint. If that matters to you — corporate egress rules, IP allowlisting on the endpoint side — test it with a throwaway key before you assume.

Symptom of a wrong ID: a generic red toast with no upstream error text. Cursor swallows the body.

Cline

Where: the Cline sidebar → settings gear → API Provider → OpenAI Compatible. Fill Base URL (.../v1), API Key, Model. Then open the model configuration section underneath, which is where the real work is.

Fill in the context window yourself. For a model Cline doesn't recognise, it can't infer the window, and its context manager will either truncate too early (you lose file content mid-task) or too late (the upstream returns a context-length error halfway through a multi-file edit). This field is the single highest-value thing on the page.
Set the image-support flag honestly. Turning it on for a text-only model produces a 400 the first time you paste a screenshot, several steps into a task.
Malformed diffs are a model problem wearing a config costume. "The SEARCH block doesn't match" repeated three times in a row usually means the model behind your ID isn't strong enough at exact-match diff editing, not that the endpoint is broken. Try the same ID via curl with a diff-editing prompt before you blame the wiring.
Cline shows a per-request cost estimate derived from the price fields you type in. Those are local display values; they don't come from the endpoint and they don't affect billing.

Nice property: Cline surfaces the raw request in an expandable block in the chat — when something's wrong, look there first; it's the best request log of the five. If your build doesn't show it, the proxy trick at the end of this article covers you.

Continue

Where: ~/.continue/config.yaml (older installs use config.json).

name: gateway
version: 0.0.1
schema: v1
models:
  - name: Deep model
    provider: openai
    model: <exact-id-from-/v1/models>
    apiBase: https://api.daoxe.com/v1
    apiKey: ${{ secrets.LLM_API_KEY }}
    roles: [chat, edit, apply]
  - name: Embeddings
    provider: openai
    model: <exact-embedding-id>
    apiBase: https://api.daoxe.com/v1
    apiKey: ${{ secrets.LLM_API_KEY }}
    roles: [embed]

provider: openai plus apiBase is the generic path. You are not claiming the model is from OpenAI; you're telling Continue which wire protocol to speak. Include /v1 in apiBase — Continue appends /chat/completions to whatever you give it, and a missing /v1 produces a 404 that looks like an auth failure.
Do not give a chat model the autocomplete role. This is the most expensive mistake in this article. Autocomplete fires continuously as you type and expects a fill-in-the-middle model; a chat model will return prose, Continue will show nonsense ghost text, and you'll generate a request every few keystrokes. If you have no FIM-capable model, leave autocomplete unconfigured.
embed is separate. If @codebase returns irrelevant chunks, check that an embeddings model is actually configured rather than silently falling back.

Where to look when it breaks: enable the Continue: Enable Console setting and run Focus on Continue Console View from the command palette for prompt logs; the core log lives at ~/.continue/logs/core.log.

Zed

Where: zed: open settings (the JSON file) — or, in recent builds, agent: open settings → Add Provider under LLM Providers, which writes the same block for you. Zed does not discover models — you declare them, either way.

{
  "language_models": {
    "openai_compatible": {
      "gateway": {
        "api_url": "https://api.daoxe.com/v1",
        "available_models": [
          {
            "name": "<exact-id-from-/v1/models>",
            "display_name": "Deep model",
            "max_tokens": 200000,
            "max_output_tokens": 8192,
            "capabilities": { "tools": true, "images": false }
          }
        ]
      }
    }
  }
}

The API key is not in this file. Zed resolves the key from an environment variable named after the provider (DAOXE_API_KEY for a provider id daoxe) — set it in your shell or launch environment. A perfectly correct settings.json with no key set produces "no models available" and no error.
max_tokens is the context window and it is mandatory. Zed has no catalogue to look it up in. Guess high and you get upstream 400s on long sessions; guess low and Zed refuses to attach files it could have.
Declaring "tools": true for a model that doesn't do tool calling gives you an agent that opens a thread, thinks, and does nothing. Check tool support against the endpoint first.
The provider key under language_models has been renamed at least once across Zed versions. If the block above is rejected, run zed: open default settings and copy the current key name from there rather than trusting any blog post, including this one.

Aider

Where: environment variables, CLI flags, or .aider.conf.yml. All three work; pick one and stick to it.

export OPENAI_API_BASE=https://api.daoxe.com/v1
export OPENAI_API_KEY=sk-...
aider --model openai/<exact-id> --weak-model openai/<cheap-exact-id>

Or, checked into the repo root as .aider.conf.yml:

model: openai/<exact-id>
weak-model: openai/<cheap-exact-id>
openai-api-base: https://api.daoxe.com/v1
edit-format: diff

The openai/ prefix is not decoration. Aider routes through LiteLLM, and the prefix is what says "treat this as a generic OpenAI-protocol endpoint". Drop it and LiteLLM tries to pattern-match your ID against its own registry, which can send the request somewhere else entirely or attach parameters your upstream rejects.
--weak-model is the classic delayed failure. Aider uses a second, cheaper model for commit messages and chat summarisation. If you don't set it, the default may not exist on your endpoint — and because it's only used after your first successful edit, everything looks fine right up until the commit step. Set it explicitly.
Unknown models trigger metadata warnings. Silence them properly with .aider.model.metadata.json in the repo root:

{
  "openai/<exact-id>": {
    "max_input_tokens": 200000,
    "max_output_tokens": 8192,
    "input_cost_per_token": 0,
    "output_cost_per_token": 0,
    "litellm_provider": "openai",
    "mode": "chat"
  }
}

Leaving the cost fields at 0 disables aider's local spend estimate, which is better than a wrong estimate. Check your real usage on the provider's side.

If streaming misbehaves through a corporate proxy, --no-stream isolates it in one flag.

Symptom → cause

What you see	Where it usually comes from
"Connection failed" with no detail	Wrong model ID, or `apiBase` missing `/v1`
Works, then dies at commit / summary time	Aider `--weak-model` not set
Ghost text is prose, token use spikes	Chat model given Continue's `autocomplete` role
Agent opens a thread and stalls	`"tools": true` declared for a non-tool model (Zed)
Context errors mid-task, only on big files	Context window not set (Cline) or `max_tokens` wrong (Zed)
"SEARCH block doesn't match", repeatedly	Model capability, not configuration
Verify button fails, manual calls work	Cursor validating a stock model ID your endpoint doesn't serve
No models listed, config looks perfect	Zed key not entered in the panel UI

When the tool won't tell you what it sent

Three of these five hide the request body. The fastest way out is to put something in the middle: run a local HTTP proxy, point the tool's base URL at http://127.0.0.1:8080/v1, and forward to the real endpoint. Ten minutes of setup answers questions that hours of toggling settings won't — particularly "is it even sending the model ID I typed" and "which extra parameters is this tool adding".

The part that isn't configuration

All five of these will connect in under five minutes once you know where the field is. What actually determines whether the setup is good is whether the model behind that ID does diff editing well, holds a long tool-use loop without drifting, and returns the context window it claims. That's an evaluation problem, and it's the one worth your afternoon — the config is a ten-line diff.

One practical consequence of doing it this way: the model becomes a config string in five files instead of five separate integrations, so trying a different one next month is an edit, not a migration. That's the whole reason to route through a single endpoint, and it's a much more honest reason than any benchmark table.

This post is part of an ongoing "verify your endpoint" series. Once you've pointed a tool at an endpoint, the next question is whether it actually honours the OpenAI wire format — streaming shape, tool calls, error bodies — which is what "OpenAI-compatible" is a spectrum, not a boolean tests with an eleven-check script. And because every setup above hinges on an exact model ID, Model IDs are a dependency. Pin them like one. covers the catalogue side: pinning, drift detection, and what to log when the model field stops matching.

"OpenAI-compatible" is a spectrum, not a boolean — here's an 11-check conformance suite

Seven — Thu, 10 Sep 2026 14:46:31 +0000

Two endpoints both say "OpenAI-compatible" on the tin. You point your app at the first one and everything works. You point it at the second one and everything works too — for about a week. Then a streamed tool call comes back with arguments split across chunks in a way your accumulator didn't expect, your JSON parser throws inside a retry loop, and the retry loop hammers the endpoint because the error body doesn't have the field your backoff code reads.

Nothing lied to you. "OpenAI-compatible" was never a boolean. It's a surface area, and every implementation covers a different subset of it.

I maintain a gateway, so I read a lot of compatibility bug reports. Disclosure: I work on daoxe, a multi-model gateway that speaks the OpenAI protocol among others. The checklist below is deliberately written so you can run it against us and against anyone else, and the script at the end doesn't know or care which endpoint you point it at. If it makes us look bad on a check, that's the correct output.

The nine surfaces

1. Streaming deltas. The reference behaviour is specific: the first chat.completion.chunk carries choices[0].delta.role = "assistant" and usually empty content, middle chunks carry delta.content fragments, the last content-bearing chunk carries finish_reason, and the stream terminates with a literal data: [DONE] line. Implementations diverge on all four. Some never send the role chunk, which breaks clients that use it to open a message. Some put finish_reason on a trailing chunk with an empty delta. Some omit [DONE] entirely and just close the connection, which is fine for a client that reads to EOF and fatal for one that blocks waiting for the sentinel.

2. Tool / function calling. Two traps here. First, function.arguments is a JSON-encoded string, not an object — endpoints that "helpfully" return a parsed object break every client that calls json.loads on it. Second, in streaming mode, tool calls arrive as fragments that you reassemble by the index field, with id and function.name typically only present on the first fragment. An endpoint that re-sends id on every fragment, or that omits index when there's only one call, will work with your naive accumulator and fail the moment a model emits two parallel calls. Also check that finish_reason is tool_calls and not stop — agent loops branch on that value.

3. response_format. Three tiers, and they're commonly conflated: no support, {"type": "json_object"} (valid JSON, any shape), and {"type": "json_schema", "json_schema": {..., "strict": true}} (constrained decoding against your schema). The dangerous middle case is an endpoint that accepts the parameter and ignores it. You get prose with a code fence around it, your parser fails one request in fifty, and it looks like a model quality problem.

4. logprobs / top_logprobs. Usually the first thing a proxy layer drops, because almost nobody notices. If you do classification by comparing token probabilities, or you use logprobs for confidence gating, this is load-bearing and you should test it explicitly.

5. temperature and sampling params. Reasoning-style models reject temperature outright on some upstreams, accept-and-ignore it on others, and honour it on a third set. All three are defensible; not knowing which one you're on is not. The same applies to top_p, presence_penalty, and max_tokens vs max_completion_tokens.

6. stop sequences. Does the endpoint honour an array of stop strings? Is the stop sequence included in or excluded from the returned content (the reference excludes it)? Does finish_reason come back as "stop"? A surprising number of shims implement stop by post-truncating the full completion, which means you pay for tokens you never see — and the usage numbers will show it.

7. Usage accounting. Non-streaming responses should carry usage.prompt_tokens, completion_tokens, total_tokens. Streaming responses only include usage if you pass stream_options: {"include_usage": true}, and then it arrives in a final chunk with an empty choices array — a shape that crashes clients which assume choices[0] always exists. If you do cost attribution per request, also check whether cached-prompt and reasoning-token breakdowns survive.

8. Error-body shape. Every retry layer you've ever written parses this, and nobody tests it. The reference is {"error": {"message": ..., "type": ..., "param": ..., "code": ...}} with an HTTP status that matches the semantics. Real-world variations: a 200 with an error object in the body (fatal for retry logic — you'll cheerfully return an error string to your user), an HTML error page from an intermediate proxy, or a 500 where a 400 belonged, which turns a permanent client error into an infinite retry storm.

9. /v1/models fidelity. Does it exist, does it return {"object": "list", "data": [{"id": ...}]}, and — the part that matters — does it list the IDs your key can actually call? A catalogue endpoint that returns everything the vendor sells, rather than everything your key is entitled to, is worse than no catalogue at all, because you'll build CI checks on top of it.

The script

Stdlib only, Python 3.8+, no install. It runs eleven checks and prints a verdict table.

#!/usr/bin/env python3
"""compat_probe.py — how OpenAI-compatible is this endpoint, really?

    export LLM_BASE_URL=https://api.example.com/v1
    export LLM_API_KEY=sk-...
    python3 compat_probe.py --model <exact-model-id>
"""
import argparse, json, os, sys, urllib.error, urllib.request

BASE = os.environ.get("LLM_BASE_URL", "").rstrip("/")
KEY = os.environ.get("LLM_API_KEY", "")
OUT = []


def _open(path, payload=None, method="GET", timeout=90):
    data = json.dumps(payload).encode() if payload is not None else None
    req = urllib.request.Request(BASE + path, data=data, method=method)
    req.add_header("Authorization", "Bearer " + KEY)
    if data is not None:
        req.add_header("Content-Type", "application/json")
    try:
        return urllib.request.urlopen(req, timeout=timeout)
    except urllib.error.HTTPError as exc:   # still a readable file object
        return exc


def call(path, payload=None, method="GET"):
    resp = _open(path, payload, method)
    raw = resp.read().decode("utf-8", "replace")
    try:
        return resp.getcode(), json.loads(raw)
    except ValueError:
        return resp.getcode(), raw


def stream(payload):
    payload = dict(payload, stream=True)
    resp = _open("/chat/completions", payload, "POST")
    chunks, saw_done = [], False
    for line in resp:
        line = line.decode("utf-8", "replace").strip()
        if not line.startswith("data:"):
            continue
        body = line[5:].strip()
        if body == "[DONE]":
            saw_done = True
            break
        try:
            chunks.append(json.loads(body))
        except ValueError:
            pass
    return chunks, saw_done


def record(name, ok, detail):
    OUT.append((name, "PASS" if ok is True else "FAIL" if ok is False else "PARTIAL", detail))


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--model", required=True)
    model = ap.parse_args().model
    ask = {"model": model, "messages": [{"role": "user", "content": "Say hi."}],
           "max_tokens": 24}

    # 1 — catalogue
    code, body = call("/models")
    ids = [m.get("id") for m in body.get("data", [])] if isinstance(body, dict) else []
    record("models_endpoint", code == 200 and bool(ids),
           f"HTTP {code}, {len(ids)} ids, target listed: {model in ids}")

    # 2 — basic completion + model echo + usage
    code, body = call("/chat/completions", ask, "POST")
    msg = (body.get("choices") or [{}])[0].get("message", {}) if isinstance(body, dict) else {}
    usage = body.get("usage") if isinstance(body, dict) else None
    record("basic_chat", bool(msg.get("content")), f"HTTP {code}")
    record("model_echo", isinstance(body, dict) and body.get("model") == model,
           f"asked {model!r}, got {body.get('model')!r}" if isinstance(body, dict) else "n/a")
    record("usage_fields", bool(usage and usage.get("total_tokens") is not None), str(usage))

    # 3 — streaming shape
    chunks, done = stream(ask)
    role = any((c.get("choices") or [{}])[0].get("delta", {}).get("role") for c in chunks)
    fin = any((c.get("choices") or [{}])[0].get("finish_reason") for c in chunks)
    record("stream_shape", bool(chunks) and role and fin and done,
           f"{len(chunks)} chunks, role_chunk={role}, finish_reason={fin}, [DONE]={done}")

    # 4 — usage on stream
    chunks, _ = stream(dict(ask, stream_options={"include_usage": True}))
    record("stream_usage", any(c.get("usage") for c in chunks),
           "final usage chunk present" if any(c.get("usage") for c in chunks) else "absent")

    # 5 — tool calling
    tool = {"type": "function", "function": {"name": "get_weather", "parameters": {
        "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}
    code, body = call("/chat/completions", dict(
        ask, messages=[{"role": "user", "content": "Weather in Osaka? Use the tool."}],
        tools=[tool], tool_choice="auto"), "POST")
    choice = (body.get("choices") or [{}])[0] if isinstance(body, dict) else {}
    calls = choice.get("message", {}).get("tool_calls") or []
    args_is_str = bool(calls) and isinstance(calls[0].get("function", {}).get("arguments"), str)
    record("tool_calls", bool(calls) and args_is_str and choice.get("finish_reason") == "tool_calls",
           f"n={len(calls)} arguments_is_string={args_is_str} finish={choice.get('finish_reason')}")

    # 6 — json mode
    code, body = call("/chat/completions", dict(
        ask, messages=[{"role": "user", "content": "Return {\"ok\": true} as JSON."}],
        response_format={"type": "json_object"}), "POST")
    text = ((body.get("choices") or [{}])[0].get("message", {}).get("content")
            if isinstance(body, dict) else "") or ""
    try:
        json.loads(text); parsed = True
    except ValueError:
        parsed = False
    record("json_object", code == 200 and parsed, f"HTTP {code}, parses={parsed}")

    # 7 — strict schema
    schema = {"type": "json_schema", "json_schema": {"name": "r", "strict": True, "schema": {
        "type": "object", "properties": {"ok": {"type": "boolean"}},
        "required": ["ok"], "additionalProperties": False}}}
    code, _ = call("/chat/completions", dict(ask, response_format=schema), "POST")
    record("json_schema_strict", code == 200, f"HTTP {code}")

    # 8 — logprobs
    code, body = call("/chat/completions", dict(ask, logprobs=True, top_logprobs=3), "POST")
    lp = ((body.get("choices") or [{}])[0].get("logprobs") if isinstance(body, dict) else None)
    record("logprobs", bool(lp and lp.get("content")), f"HTTP {code}")

    # 9 — temperature: accepted / rejected / ignored is three different worlds
    code, _ = call("/chat/completions", dict(ask, temperature=0.5), "POST")
    record("temperature", None if code == 400 else code == 200,
           "rejected with 400 (reasoning model?)" if code == 400 else f"HTTP {code}")

    # 10 — stop sequences
    code, body = call("/chat/completions", dict(
        ask, messages=[{"role": "user", "content": "Count: one two three four five"}],
        stop=["three"], max_tokens=48), "POST")
    text = ((body.get("choices") or [{}])[0].get("message", {}).get("content")
            if isinstance(body, dict) else "") or ""
    record("stop_sequences", "three" not in text, f"stop string leaked into content: {'three' in text}")

    # 11 — error shape on a model that cannot exist
    code, body = call("/chat/completions", dict(ask, model="definitely-not-a-model-xyz"), "POST")
    shaped = isinstance(body, dict) and isinstance(body.get("error"), dict)
    record("error_shape", 400 <= code < 500 and shaped, f"HTTP {code}, error object: {shaped}")

    width = max(len(n) for n, _, _ in OUT)
    for name, verdict, detail in OUT:
        print(f"{name.ljust(width)}  {verdict:<7} {detail}")
    print("\nPARTIAL is not a failure — it is a behaviour you now have to design around.")
    return 0 if all(v != "FAIL" for _, v, _ in OUT) else 1


if __name__ == "__main__":
    sys.exit(main())

Two notes on running it. It costs a handful of tiny completions, so run it against a cheap model first to confirm your base URL and key are right. And run it against the exact model ID you plan to ship, because conformance varies per model on the same endpoint — reasoning models in particular reject parameters that their non-reasoning siblings accept.

Reading the output

FAIL on basic_chat or error_shape is a real problem. FAIL on logprobs is only a problem if you use logprobs. The point of the table isn't a score; it's that you now have a written record of which surfaces you're allowed to depend on, which you can re-run after any provider change and diff.

Three habits that follow from it:

Check it into CI. One nightly run, output committed as an artifact. Compatibility regressions are silent by nature — nobody sends a changelog entry saying "we stopped forwarding logprobs".
Assert the model echo in production, not just in the probe. If you asked for one ID and the response's model field says another, you want that in your logs the day it starts happening, not the week you notice quality dropped.
Treat PARTIAL as a design input. If stream_usage is absent, your cost attribution needs a non-streaming path or a local tokenizer estimate. That's a two-hour decision now or a month-end reconciliation mystery later.

What this can't tell you

The probe tests the protocol, not the model behind it. An endpoint can pass all eleven checks and still be routing you to a smaller model, a quantized build, or a truncated context window — the wire format would be perfect either way. That's a different investigation with different tools, and the honest framing there is that behavioural probes give you signals, not proof: nobody outside the provider can see which weights served your request. I wrote up the catalogue side of that problem separately — pinning model IDs, detecting drift, and what to log when the model field stops matching what you asked for — in Model IDs are a dependency. Pin them like one..

But protocol conformance is the part you can settle in an afternoon, with a script that fits on one screen, and it's the part that will wake you up at 3am if you skip it.

This post is part of an ongoing "verify your endpoint" series. The companion piece on model-ID drift is Model IDs are a dependency. Pin them like one., and if you're evaluating a third-party endpoint before your first production call, Ten questions to answer before you route production traffic through someone else's LLM endpoint is the buyer-side checklist.

Ten questions to answer before you route production traffic through someone else's LLM endpoint

Seven — Mon, 07 Sep 2026 19:36:11 +0000

Every LLM endpoint's landing page says fast, reliable, compatible, secure. Those words are free. This is the list of questions that aren't free — the ones where the answer takes a vendor real work to be able to give, and where you can verify the answer yourself in an afternoon.

Disclosure up front: I work on daoxe, a multi-model gateway. At the end I run this checklist against it, and two rows don't pass. That's the point of publishing it — a checklist that flatters whoever wrote it isn't a checklist, it's a spec sheet.

1. Protocol coverage — native or translated?

Ask which wire protocols the endpoint implements and which ones it translates into. These are different products. A translation layer is fine for text chat and lossy for agents: tool-call block ordering, streamed tool arguments, prompt-cache markers and stop-reason fidelity all live in fields that have no equivalent on the other side.

Evidence: for the Anthropic protocol, omit the required max_tokens from a /v1/messages call. A native implementation returns 400; a translation layer often fills in a default and returns 200. For the OpenAI protocol, check whether streamed tool calls arrive as incremental fragments with an index, or all at once at the end.

Bad answer: "we're fully compatible with everything."

2. Model ID transparency

Does GET /v1/models return the exact IDs your key can call — not the vendor's full catalogue? Does the response echo a model field, and does it match what you asked for? And critically: what happens when you send an ID that doesn't exist?

Evidence: send a deliberately mangled model ID. If you get a completion instead of an error, the endpoint is picking on your behalf, and every ID you send is advisory rather than binding. That single fact changes how much of the rest of this list you can trust.

Bad answer: a model list on a marketing page, with no catalogue endpoint.

3. Provenance — where does the capacity come from?

Any aggregator is reselling access it obtained somehow. The question isn't whether there's a supply chain; it's whether they'll tell you what's in it. Some capacity is direct vendor access, some is enterprise-cloud resale, some is reverse-engineered consumer endpoints with materially different reliability. All three can be legitimate purchases — at very different risk levels, for very different workloads.

Evidence: ask for the labelling scheme, in writing, and check whether it appears in the product itself or only in a support conversation.

Bad answer: silence, or "we don't disclose our infrastructure." Undisclosed provenance means you cannot reason about your own availability.

4. Can you get your own data out?

Usage per key, per model, per day. Ideally an export rather than a dashboard screenshot. You need this for two things you'll definitely do: reconciling a bill you don't understand, and attributing spend to a team or a feature.

Evidence: try to export a month. If it's a dashboard-only number, plan on logging every request yourself from day one — you can't retrofit history.

5. Incident behaviour

How do you find out something is degraded? Three tiers, in descending order of comfort: a published availability page with history, an announcement channel, or nothing.

Here's the thing worth internalising: an availability figure on a marketing page is a claim, not a measurement. It's not necessarily false, but it isn't evidence, and you shouldn't put it in a design document. Whatever tier your vendor is in, run your own probe — a one-token completion against each model you depend on, every five minutes, appended to a JSONL file. After a month you own a success rate and a latency distribution measured from your own network, which is a strictly better artifact than any number on any vendor page.

Evidence: your own probe log. Nothing else counts.

6. Key handling

Six sub-questions: can you issue multiple keys, scope them (per model, per group, per spend cap), rotate without downtime, revoke instantly, see last-used time, and does revocation actually take effect immediately?

Evidence: issue a second key, use it once, revoke it, and immediately retry. If it still works sixty seconds later, you've learned something important about the caching layer between you and their auth.

Then audit where your keys physically live. On a normal developer laptop a gateway key ends up in an editor's settings store, a shell profile, two dotfiles, a CI secret, and at least one Docker environment. Write that list down before you distribute a key to a team, not after.

7. Data handling

Retention period for prompts and completions, whether your traffic can be used for training, whether that's per-account configurable, and what the sub-processor situation is for the upstream vendors behind the aggregator.

Get this in writing. I'm deliberately not paraphrasing anyone's policy — including my own employer's — in a blog post you might then rely on. Ask, get a written answer, keep it, and re-ask on renewal.

Bad answer: a verbal assurance in a chat window.

8. Rate limits and behaviour under load

Are limits documented per key or per model? What does the endpoint do when you exceed them — a clean 429 with Retry-After, a 500, or a hang? Do errors have a stable machine-readable shape your retry layer can classify?

Evidence: fire a small burst of concurrent requests at a cheap model and read the failures. A 429 carrying Retry-After is a mature implementation. A 500 where a 429 belonged will turn your backoff into a retry storm, because your code will classify it as transient-and-worth-hammering.

9. Billing you can reconcile

You want three things: a mechanism you can state in one sentence, a live price reference rather than a number in a PDF, and per-request usage data you can multiply out yourself.

Be actively suspicious of "cheapest". Price is the easiest thing on this list to change and the easiest to advertise; it tells you nothing about the nine criteria above it, and a number well below everyone else's floor is usually a statement about provenance (criterion 3), not about efficiency.

For the record, since I'm asking you to run this against my employer: DaoXE tops up at a flat 1:1 top-up rate across Alipay, WeChat Pay, USDT, bank card (Visa·Mastercard), Apple Pay and Google Pay — one unit of local currency buys one dollar of credit — models bill at their USD list prices, and the live numbers are at https://daoxe.com/pricing — check them there rather than trusting any figure in an article, including this one.

10. The exit plan

The last question is the one that makes the other nine survivable: how many lines of code does it take to leave?

If the answer is "change a base URL and a key in one config file", the switching cost is near zero and every other risk on this list becomes a manageable one. If the answer involves a proprietary SDK, vendor-specific request extensions, or a routing DSL you can't reproduce elsewhere, you've bought a migration.

Four habits that keep the number low: keep the base URL and model IDs in config, not in code; prefer standard protocol fields over vendor extensions; keep your own request/response logs; and maintain a golden eval set of twenty to fifty real requests so that "is the replacement good enough" is a script you run rather than an argument you have.

The scorecard, run against my own employer

#	Criterion	DaoXE	Notes
1	Protocol coverage	Pass	OpenAI (`/v1/chat/completions`, `/v1/responses`, `/v1/embeddings`, `/v1/images/generations`, `/v1/videos`), Anthropic Messages (`/v1/messages`), Gemini (`/v1beta/models/{model}:generateContent`), plus `/v1/rerank` — implemented, not translated. Verify with the `max_tokens` test above.
2	Model ID transparency	Pass	`GET /v1/models` returns the exact IDs your key can call. Catalogue size drifts constantly — hundreds of models across roughly 25 vendors — so treat any published count, including ours, as stale.
3	Provenance	Pass, with a caveat you should read	Channels are labelled in the catalogue: full-power, official relay, reverse-engineered ("quality not guaranteed"), promotional. The reverse-engineered tier genuinely exists. Labelling it is the honest part; choosing it for production is your call, and I'd say no.
4	Usage / log export	Ask	Don't take my word for the current state — ask on Telegram `@daoxe_ai` and log your own requests regardless.
5	Incident behaviour	Does not pass	There is no public status page, so run your own cron probe. The availability figure on the marketing page is a company claim, not a measurement, and I won't repeat it here as if it were one.
6	Key handling	Test it yourself	Run the issue → use → revoke → retry sequence on your own account. Note that sign-up is open, with password login, no email verification, and OAuth via GitHub, Telegram or Passkey — convenient, and worth factoring into your own account-security posture.
7	Data handling	Ask, in writing	Same rule I'd apply to any vendor. Ask `@daoxe_ai`, keep the answer.
8	Rate limits under load	Test it yourself	Burst a cheap model, read the failure bodies, check for `Retry-After`.
9	Billing	Pass on mechanism	One-sentence mechanism, live prices on the pricing page, per-request usage in the response body.
10	Exit plan	Pass	Standard protocols, model IDs are strings, base URL is one config line. Leaving is the same edit as arriving — which is the only version of this claim worth making.

One more thing that belongs on any honest vendor row: DaoXE is not available in mainland China. That's the official line and it's a hard constraint, not a caveat to work around.

Using this without turning it into a ritual

Ten criteria is too many for a side project and about right for anything with users. If you only have an hour, do these four: the mangled-model-ID test (criterion 2), the provenance question (3), start the probe (5), and count the lines it would take to leave (10). Those four catch the failures that are expensive to discover later; the rest catch the failures that are merely annoying.

And apply it symmetrically. If a vendor's answer to criterion 3 or 7 is a shrug, that's data — but so is a checklist published by a vendor that somehow scores ten out of ten on its own list.

Model IDs are a dependency. Pin them like one.

Seven — Mon, 07 Sep 2026 15:12:43 +0000

You typed claude-sonnet-4-5 from memory, it returned a completion, and you shipped. Six weeks later the outputs are subtly different, or a deploy fails with a 404 on a Tuesday morning, and the git blame points at a string literal in a file nobody has opened since.

Model IDs are a runtime dependency on someone else's release schedule, and almost nobody treats them like one. We pin requests==2.32.3 and then interpolate a model name from a config file we last checked in March.

Disclosure: I work on daoxe, a gateway that fronts hundreds of models from roughly 25 vendors — which means I get to watch upstream renames arrive from several directions at once. Everything below works against any endpoint that implements GET /v1/models.

Three ways a wrong ID fails

Mode A — a clean error. 404 or 400 with "code": "model_not_found". This is the good outcome. You find out in development, in the first second, and you fix a typo.

Mode B — alias resolution. You ask for gpt-4o, the provider resolves it to a dated build, and the response's model field says which one. Fine and normal — as long as you read the echo. If you never look, you've silently accepted whatever the alias points at this month.

Mode C — nearest-match routing. Some endpoints and proxies try to be helpful: they don't recognise your ID, so they route to the closest thing they do recognise. You asked for a flagship and got something else, with a 200 and no indication anywhere in the response that a substitution happened. This is the mode that costs real money and real trust, and it's invisible in exactly the way that matters.

The ten-second test for Mode C

Send an ID that cannot possibly exist, formed by mangling a real one:

curl -s https://api.daoxe.com/v1/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"claude-sonnet-4-5-TYPO-NOT-A-MODEL",
       "max_tokens":8,"messages":[{"role":"user","content":"hi"}]}' \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); \
print("ERROR:", d["error"]["message"][:80]) if "error" in d else print("SERVED BY:", d.get("model"))'

If you get an error, every model ID you send is authoritative. If you get a completion, every model ID you send is advisory — the endpoint reserves the right to pick for you, and no amount of pinning on your side will help until you know the rule it uses. Run this before you build anything else in this article.

Dated versus floating

Two ID styles, two different risks:

	Floating (`claude-sonnet-4-5`)	Dated (`claude-sonnet-4-5-20250929`)
Behaviour	changes under you, no code change	stable while it exists
Lifespan	long	retired on a schedule
Fails as	silent quality drift	a loud 404
Use for	exploration, prototypes	anything with a recorded eval

The policy that follows: exploration floats, production is dated, and every eval artifact records the ID that actually served it — read from the response's model field, not from what you asked for. That last part is what lets you answer "was this eval run on the same build as that one" three months later.

`GET /v1/models` is the source of truth, and it's per-key

Not the docs, not the pricing page, not a blog post. The catalogue endpoint returns the IDs your key is entitled to call, which is frequently a different set from what the vendor sells:

curl -s https://api.daoxe.com/v1/models \
  -H "Authorization: Bearer $LLM_API_KEY" \
  | python3 -c 'import json,sys; [print(m["id"]) for m in json.load(sys.stdin)["data"]]' \
  | sort

One caveat worth knowing up front: most implementations return IDs and little else. Context windows, tool support, vision support and modality are usually not in there, so you still need a small capability sidecar of your own. Don't infer capabilities from the name.

Lock the catalogue

#!/usr/bin/env python3
"""check_models.py — fail CI when a model ID this repo depends on disappears.

    python3 check_models.py --write      # re-pin (review the diff!)
    python3 check_models.py              # verify
"""
import argparse, json, os, sys, urllib.request

BASE = os.environ["LLM_BASE_URL"].rstrip("/")
KEY = os.environ["LLM_API_KEY"]


def catalogue():
    req = urllib.request.Request(BASE + "/models")
    req.add_header("Authorization", "Bearer " + KEY)
    with urllib.request.urlopen(req, timeout=30) as resp:
        return sorted({m["id"] for m in json.load(resp).get("data", []) if m.get("id")})


def required_ids(path="models.json"):
    """The IDs this repo actually uses — derived from the alias map, never hand-listed."""
    with open(path) as fh:
        return sorted(set(json.load(fh)["roles"].values()))


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--lock", default="models.lock.json")
    ap.add_argument("--write", action="store_true")
    args = ap.parse_args()

    live = catalogue()
    if args.write or not os.path.exists(args.lock):
        with open(args.lock, "w") as fh:
            json.dump({"catalogue": live}, fh, indent=2)
            fh.write("\n")
        print(f"pinned {len(live)} ids to {args.lock}")
        return 0

    with open(args.lock) as fh:
        pinned = set(json.load(fh)["catalogue"])
    live_set, needed = set(live), set(required_ids())

    for mid in sorted(live_set - pinned):
        print(f"note   new in catalogue: {mid}")
    for mid in sorted(pinned - live_set - needed):
        print(f"warn   left the catalogue (unused here): {mid}")
    gone = sorted(needed - live_set)
    for mid in gone:
        print(f"ERROR  required by models.json but not in the catalogue: {mid}")
    return 1 if gone else 0


if __name__ == "__main__":
    sys.exit(main())

Run it nightly and on every pull request:

name: model-catalogue
on:
  schedule: [{ cron: "17 6 * * *" }]
  pull_request:
  workflow_dispatch:

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: python3 check_models.py
        env:
          LLM_BASE_URL: https://api.daoxe.com/v1
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}

The two failure signals mean different things. A nightly failure means an upstream retired or renamed something you depend on, and you found out before your users did. A pull-request failure means someone added a role pointing at an ID that isn't callable — a typo caught at review time instead of at 2am.

The alias layer

Nothing in your application code should contain a model name. Code refers to roles; one file maps roles to exact IDs.

{
  "pinned_at": "2026-08-12",
  "roles": {
    "fast":   "<exact-id>",
    "deep":   "<exact-id>",
    "bulk":   "<exact-id>",
    "vision": "<exact-id>",
    "embed":  "<exact-embedding-id>"
  },
  "notes": {
    "deep": "dated build; re-pin only with an eval run linked in the PR"
  }
}

import functools, json, pathlib

@functools.lru_cache(maxsize=1)
def _roles():
    return json.loads(pathlib.Path("models.json").read_text())["roles"]

def model_for(role):
    try:
        return _roles()[role]
    except KeyError:
        raise KeyError(f"no model pinned for role {role!r} — add it to models.json") from None

Three things fall out of this for free. A rename becomes a one-line diff in a reviewable file instead of a search-and-replace across forty call sites. check_models.py can derive its required set from the same file, so the lock and the code can't disagree. And git log models.json becomes a readable history of what your product was running and when — which is the artifact you'll want the first time someone asks why output quality changed in July.

Then stop the IDs from leaking back in. One test, one line:

# tests/no_hardcoded_model_ids.sh — fails if a model-looking literal appears in src/
! rg -n --glob '!models.json' \
    -e '"(gpt|claude|gemini|deepseek|qwen|kimi|grok|llama)-[a-z0-9._-]+"' src/

When an upstream retires a model mid-project

Find out from CI, not from users. That's what the nightly is for.
Read the label before you pick a successor. Vendors publish migration notes; gateways publish catalogue metadata. Don't infer the replacement from the name — -mini, -fast, -pro and numeric bumps are marketing, not a capability contract.
Freeze a golden set. Twenty to fifty real requests from your own traffic with their recorded outputs. If you don't have one yet, this is the moment it becomes worth an afternoon; you will need it again.
Run the candidate against the golden set and diff the things that actually break in production: output format compliance, tool-call rate, refusal rate, response length, and latency. Not vibes.
One PR, one line. Change models.json, link the eval run in the description, let CI re-verify the lock.
Keep the corpse. Leave the old ID in the git history of models.lock.json and in your eval artifacts. When you're asked to reproduce a result from two quarters ago, that record is the only thing that will let you say what actually served it.

Variants and channels on a gateway

If you route through a gateway, two IDs that look like the same model may reach different upstream channels with different reliability characteristics. Some catalogues label this explicitly — the one I work on tags channels as full-power, official relay, reverse-engineered ("quality not guaranteed"), or promotional, and that labelling is the whole point: you can choose a cheaper or less stable path deliberately, or refuse to. What you should never do is pattern-match a suffix and assume you know what it means. Read the catalogue metadata, pick the exact string, pin it.

Drift under a pinned ID

Pinning protects you from renames. It does not protect you from an upstream changing what sits behind a name it didn't change. The only handle you have from the outside is behavioural: record how a pinned ID answers a fixed battery of prompts at temperature 0, re-run it on a schedule, and diff.

pipx install git+https://github.com/seven7763/llm-honesty-probe

Be precise about what that gives you: signals, not proof. No probe run from outside can prove which weights served a request, and a change in output can have several innocent explanations. What it does produce is a timestamped record that something changed on a date, which is a much stronger position to argue from than "it feels worse lately."

Pin the ID, alias the role, lock the catalogue, watch the behaviour. It's an afternoon of work, and it converts your loudest class of production surprise into a CI failure.

Model IDs are a dependency. Pin them like one.

Seven — Mon, 07 Sep 2026 11:19:26 +0000

Three ways a wrong ID fails

Mode A — a clean error. 404 or 400 with "code": "model_not_found". This is the good outcome. You find out in development, in the first second, and you fix a typo.

The ten-second test for Mode C

Send an ID that cannot possibly exist, formed by mangling a real one:

curl -s https://api.daoxe.com/v1/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"claude-sonnet-4-5-TYPO-NOT-A-MODEL",
       "max_tokens":8,"messages":[{"role":"user","content":"hi"}]}' \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); \
print("ERROR:", d["error"]["message"][:80]) if "error" in d else print("SERVED BY:", d.get("model"))'

Dated versus floating

Two ID styles, two different risks:

	Floating (`claude-sonnet-4-5`)	Dated (`claude-sonnet-4-5-20250929`)
Behaviour	changes under you, no code change	stable while it exists
Lifespan	long	retired on a schedule
Fails as	silent quality drift	a loud 404
Use for	exploration, prototypes	anything with a recorded eval

`GET /v1/models` is the source of truth, and it's per-key

Not the docs, not the pricing page, not a blog post. The catalogue endpoint returns the IDs your key is entitled to call, which is frequently a different set from what the vendor sells:

curl -s https://api.daoxe.com/v1/models \
  -H "Authorization: Bearer $LLM_API_KEY" \
  | python3 -c 'import json,sys; [print(m["id"]) for m in json.load(sys.stdin)["data"]]' \
  | sort

Lock the catalogue

#!/usr/bin/env python3
"""check_models.py — fail CI when a model ID this repo depends on disappears.

    python3 check_models.py --write      # re-pin (review the diff!)
    python3 check_models.py              # verify
"""
import argparse, json, os, sys, urllib.request

BASE = os.environ["LLM_BASE_URL"].rstrip("/")
KEY = os.environ["LLM_API_KEY"]


def catalogue():
    req = urllib.request.Request(BASE + "/models")
    req.add_header("Authorization", "Bearer " + KEY)
    with urllib.request.urlopen(req, timeout=30) as resp:
        return sorted({m["id"] for m in json.load(resp).get("data", []) if m.get("id")})


def required_ids(path="models.json"):
    """The IDs this repo actually uses — derived from the alias map, never hand-listed."""
    with open(path) as fh:
        return sorted(set(json.load(fh)["roles"].values()))


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--lock", default="models.lock.json")
    ap.add_argument("--write", action="store_true")
    args = ap.parse_args()

    live = catalogue()
    if args.write or not os.path.exists(args.lock):
        with open(args.lock, "w") as fh:
            json.dump({"catalogue": live}, fh, indent=2)
            fh.write("\n")
        print(f"pinned {len(live)} ids to {args.lock}")
        return 0

    with open(args.lock) as fh:
        pinned = set(json.load(fh)["catalogue"])
    live_set, needed = set(live), set(required_ids())

    for mid in sorted(live_set - pinned):
        print(f"note   new in catalogue: {mid}")
    for mid in sorted(pinned - live_set - needed):
        print(f"warn   left the catalogue (unused here): {mid}")
    gone = sorted(needed - live_set)
    for mid in gone:
        print(f"ERROR  required by models.json but not in the catalogue: {mid}")
    return 1 if gone else 0


if __name__ == "__main__":
    sys.exit(main())

Run it nightly and on every pull request:

name: model-catalogue
on:
  schedule: [{ cron: "17 6 * * *" }]
  pull_request:
  workflow_dispatch:

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: python3 check_models.py
        env:
          LLM_BASE_URL: https://api.daoxe.com/v1
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}

The alias layer

Nothing in your application code should contain a model name. Code refers to roles; one file maps roles to exact IDs.

{
  "pinned_at": "2026-08-12",
  "roles": {
    "fast":   "<exact-id>",
    "deep":   "<exact-id>",
    "bulk":   "<exact-id>",
    "vision": "<exact-id>",
    "embed":  "<exact-embedding-id>"
  },
  "notes": {
    "deep": "dated build; re-pin only with an eval run linked in the PR"
  }
}

import functools, json, pathlib

@functools.lru_cache(maxsize=1)
def _roles():
    return json.loads(pathlib.Path("models.json").read_text())["roles"]

def model_for(role):
    try:
        return _roles()[role]
    except KeyError:
        raise KeyError(f"no model pinned for role {role!r} — add it to models.json") from None

Then stop the IDs from leaking back in. One test, one line:

# tests/no_hardcoded_model_ids.sh — fails if a model-looking literal appears in src/
! rg -n --glob '!models.json' \
    -e '"(gpt|claude|gemini|deepseek|qwen|kimi|grok|llama)-[a-z0-9._-]+"' src/

When an upstream retires a model mid-project

Find out from CI, not from users. That's what the nightly is for.
Read the label before you pick a successor. Vendors publish migration notes; gateways publish catalogue metadata. Don't infer the replacement from the name — -mini, -fast, -pro and numeric bumps are marketing, not a capability contract.
Freeze a golden set. Twenty to fifty real requests from your own traffic with their recorded outputs. If you don't have one yet, this is the moment it becomes worth an afternoon; you will need it again.
Run the candidate against the golden set and diff the things that actually break in production: output format compliance, tool-call rate, refusal rate, response length, and latency. Not vibes.
One PR, one line. Change models.json, link the eval run in the description, let CI re-verify the lock.
Keep the corpse. Leave the old ID in the git history of models.lock.json and in your eval artifacts. When you're asked to reproduce a result from two quarters ago, that record is the only thing that will let you say what actually served it.

Variants and channels on a gateway

Drift under a pinned ID

pipx install git+https://github.com/seven7763/llm-honesty-probe

Pin the ID, alias the role, lock the catalogue, watch the behaviour. It's an afternoon of work, and it converts your loudest class of production surprise into a CI failure.

Your LLM API will fail in production. Here's the fallback system that catches it.

Seven — Thu, 03 Sep 2026 06:28:43 +0000

Every LLM API fails. Not "might fail." Fails.

OpenAI returns 429s during peak hours. Anthropic has regional outages. Google's Gemini API occasionally decides your key isn't authorized for the model you've been calling all week. A provider you depend on goes down for 45 minutes and your agent pipeline — the one you told your team was "production-ready" — sits there dead, mid-task, with no idea what happened.

You already know retry logic. This article is about what comes after retry: how to build a multi-model fallback system that keeps your application running when the primary model is unreachable, and how to avoid the mistakes that make fallback worse than just failing.

I'll show real Python patterns, the gotchas that break them in practice, and how to test your fallback without waiting for an actual outage.

Disclosure up front: I work on daoxe, a multi-model API gateway. The patterns in this article work against any provider — official APIs, third-party gateways, or self-hosted endpoints. Use them wherever your key works.

1. Why retry alone isn't enough

The simplest thing that sometimes works:

import time
import openai

def call_with_retry(model: str, messages: list, max_retries: int = 3):
    for attempt in range(max_retries):
        try:
            return openai.chat.completions.create(
                model=model, messages=messages
            )
        except openai.APIError as e:
            if attempt == max_retries - 1:
                raise
            time.sleep(2 ** attempt)

This handles transient network blips and brief rate limiting. It does not handle:

Sustained outages. If the provider is down for 30 minutes, exponential backoff just means your caller eventually raises an exception after 14 seconds instead of immediately. Your agent is still dead.
Quota exhaustion. Your key hits its monthly limit. Retrying won't fix it. It'll just burn your error budget faster.
Model deprecation. The provider sunsets the model version you pinned. Every call fails with model_not_found. Retry spins uselessly.
Regional blocks. The endpoint is up — just not from where your server lives. The client sees a timeout and retries into the same wall.

Retry is the first layer. Fallback is the second.

2. The model-level fallback pattern (simple, with sharp edges)

The basic idea: if model A fails, try model B.

import os
from openai import OpenAI

FALLBACK_CHAIN = [
    {"model": "gpt-5.5", "base_url": "https://api.openai.com/v1", "key_env": "OPENAI_API_KEY"},
    {"model": "claude-sonnet-4-6", "base_url": "https://api.anthropic.com/v1/messages",
     "key_env": "ANTHROPIC_API_KEY", "protocol": "anthropic"},
    {"model": "gemini-2.5-pro", "base_url": "https://generativelanguage.googleapis.com/v1beta",
     "key_env": "GEMINI_API_KEY", "protocol": "gemini"},
]

def call_with_fallback(messages: list, chain: list = FALLBACK_CHAIN):
    last_error = None
    for cfg in chain:
        try:
            client = OpenAI(
                base_url=cfg["base_url"],
                api_key=os.environ[cfg["key_env"]],
                timeout=30.0,
            )
            return client.chat.completions.create(
                model=cfg["model"], messages=messages
            )
        except Exception as e:
            last_error = e
            continue
    raise RuntimeError(
        f"All {len(chain)} models failed. Last error: {last_error}"
    )

This works — until it doesn't. Here's where it breaks.

Gotcha 1: Protocol incompatibility

Anthropic's API uses the Messages protocol, not Chat Completions. Calling it through an OpenAI-shaped client with base_url=https://api.anthropic.com/v1/messages produces a 404 or a garbled error. You need a protocol adapter, or you need to use a gateway that translates between protocols.

The honest fix: either maintain separate client code paths per provider, or use a gateway that presents every model behind a single protocol. Both approaches work. The single-protocol path means fewer branches to test.

Gotcha 2: The system prompt disappears

Some models handle system prompts differently. Claude native uses a top-level system parameter. OpenAI uses a system role in the messages array. Gemini puts system instructions in a systemInstruction config field. When you fall back across providers, your carefully crafted system prompt might arrive as a user message — or not at all.

Test this explicitly. Send a message that asks the model to quote the system prompt verbatim. If it can't, your fallback is silently dropping instructions.

Gotcha 3: Tool calling breaks between families

Anthropic and OpenAI have different tool calling schemas. The function definitions that work on GPT may produce tool_use blocks on Claude that your parser doesn't recognize. If your agent depends on structured tool calls, cross-provider fallback is risky without a normalization layer.

For agent pipelines that use tool calling heavily, fallback within the same model family first — GPT family before Claude family before Gemini — and only cross families when the task doesn't require structured output.

3. Routing by capability, not just availability

A smarter fallback system doesn't blindly try models in a fixed order. It selects the fallback based on what the task actually needs.

CAPABILITY_PROFILES = {
    "reasoning": ["claude-sonnet-4-6", "gpt-5.5", "gemini-2.5-pro"],
    "coding": ["claude-sonnet-4-6", "gpt-5.5", "deepseek-v4-pro"],
    "fast_chat": ["claude-haiku-4-5", "gpt-5.4", "gemini-3.5-flash"],
    "long_context": ["gemini-2.5-pro", "claude-sonnet-4-6", "gpt-5.5"],
}

MODEL_CONFIGS = {
    "claude-sonnet-4-6": {"base_url": "...", "key_env": "...", "protocol": "anthropic"},
    "gpt-5.5": {"base_url": "...", "key_env": "...", "protocol": "openai"},
    # ... etc
}

def call_by_capability(messages: list, capability: str):
    candidates = CAPABILITY_PROFILES.get(capability, [])
    if not candidates:
        raise ValueError(f"Unknown capability: {capability}")

    last_error = None
    for model_id in candidates:
        cfg = MODEL_CONFIGS[model_id]
        try:
            client = OpenAI(base_url=cfg["base_url"], api_key=os.environ[cfg["key_env"]])
            return client.chat.completions.create(model=model_id, messages=messages)
        except Exception as e:
            last_error = e
            continue
    raise RuntimeError(f"All {capability} models failed: {last_error}")

Now your code declares intent — "I need long-context reasoning" — and the dispatcher picks appropriate models. This is more resilient than a fixed order, and it makes your code self-documenting about what each call actually requires.

4. The gateway pattern: one endpoint, many models

Managing separate API keys, base URLs, and protocol adapters for a half-dozen providers is operational friction. The alternative is a single endpoint that presents many models behind one API surface:

# One base URL. One key. All models available.
client = OpenAI(
    base_url="https://api.daoxe.com/v1",
    api_key=os.environ["DAOXE_API_KEY"],
)

FALLBACK_CHAIN = [
    "claude-sonnet-4-6",   # primary
    "gpt-5.5",             # fallback 1
    "gemini-2.5-pro",      # fallback 2
]

def call_single_endpoint(messages: list, chain: list = FALLBACK_CHAIN):
    last_error = None
    for model in chain:
        try:
            return client.chat.completions.create(
                model=model, messages=messages, timeout=30.0
            )
        except Exception as e:
            last_error = e
            continue
    raise RuntimeError(f"All models in chain failed: {last_error}")

The advantage is not having to manage per-provider credentials, protocol quirks, or billing relationships. The trade-off is that the gateway itself becomes a single point of failure — if the gateway goes down, all models behind it become unreachable. (Mitigations: monitor the gateway's health separately, and keep a direct-to-provider key as an escape hatch for critical paths.)

5. Circuit breaker: when even fallback should stop

Blindly cycling through fallback models under load can make things worse. If your primary model is returning 429s because you're over your rate limit, hammering the fallback models immediately is antisocial — and it might exhaust your quota on all providers in a cascade.

A circuit breaker stops calling a model after N consecutive failures and only re-enables it after a cooldown.

import time
from collections import defaultdict

class CircuitBreaker:
    def __init__(self, failure_threshold: int = 5, cooldown_seconds: float = 60.0):
        self.failure_threshold = failure_threshold
        self.cooldown_seconds = cooldown_seconds
        self._failures: dict[str, int] = defaultdict(int)
        self._open_until: dict[str, float] = {}

    def allow(self, model: str) -> bool:
        if model in self._open_until and time.time() < self._open_until[model]:
            return False
        return True

    def record_failure(self, model: str):
        self._failures[model] += 1
        if self._failures[model] >= self.failure_threshold:
            self._open_until[model] = time.time() + self.cooldown_seconds

    def record_success(self, model: str):
        self._failures[model] = 0
        self._open_until.pop(model, None)

Wire it into the fallback dispatcher:

breaker = CircuitBreaker(failure_threshold=3, cooldown_seconds=30.0)

def call_with_breaker(messages: list, chain: list):
    for model in chain:
        if not breaker.allow(model):
            continue  # circuit open, skip
        try:
            result = client.chat.completions.create(model=model, messages=messages)
            breaker.record_success(model)
            return result
        except Exception as e:
            breaker.record_failure(model)
            continue
    raise RuntimeError("All models unavailable (circuit open or failed)")

Now when a model goes down repeatedly, the system stops trying it for 30 seconds, giving it time to recover without wasting your remaining quota on guaranteed failures.

6. Observability: you can't fix what you can't see

A fallback system without instrumentation is a debugging nightmare. When a call succeeds on the third model in the chain, you need to know that it happened and why the first two failed.

At minimum, log:

import logging
import json

logger = logging.getLogger("llm_fallback")

def call_instrumented(messages: list, chain: list):
    for i, model in enumerate(chain):
        try:
            start = time.monotonic()
            result = client.chat.completions.create(model=model, messages=messages)
            elapsed = time.monotonic() - start

            if i > 0:
                logger.warning(
                    "fallback_used",
                    extra={
                        "primary": chain[0],
                        "actual": model,
                        "fallback_depth": i,
                        "latency_s": round(elapsed, 3),
                        "prompt_tokens": result.usage.prompt_tokens,
                        "completion_tokens": result.usage.completion_tokens,
                    },
                )

            return result

        except Exception as e:
            logger.error(
                "model_call_failed",
                extra={"model": model, "attempt": i, "error": str(e)},
            )
            continue

    logger.critical("all_models_exhausted", extra={"chain": chain})
    raise RuntimeError("All models in chain failed")

Critical metric to track over time: fallback rate (what percentage of calls hit a fallback model). If this creeps above 1-2%, your primary provider has a reliability problem. If it spikes suddenly, something is broken. Either way, you want to know before your users do.

7. Testing your fallback without waiting for an outage

The worst time to discover your fallback system is broken is during an actual outage. Test it artificially.

Method 1: The unreachable endpoint

Point your primary model at a dead URL and verify the fallback fires:

# Set primary to a port where nothing listens
PRIMARY_BASE_URL="http://localhost:19999/v1" python test_fallback.py

Method 2: The error-injection proxy

Write a tiny proxy that returns errors for a percentage of requests:

# error_proxy.py — run with: python error_proxy.py
# Proxies requests to your real endpoint, but returns 503 for ~30% of them.

from http.server import HTTPServer, BaseHTTPRequestHandler
import random

class ErrorProxy(BaseHTTPRequestHandler):
    def do_POST(self):
        if random.random() < 0.3:
            self.send_response(503)
            self.end_headers()
            self.wfile.write(b'{"error": {"message": "injected failure", "type": "server_error"}}')
            return
        # ... proxy to real endpoint

Run your application against this proxy. If the fallback doesn't fire within a few requests, something is misconfigured.

Method 3: The key rotation test

Rotate your primary API key to an invalid value and verify the system fails over to the next model. This catches the case where your fallback is configured but the credentials are wrong — a failure mode that's silent until the primary actually goes down.

8. The honest limitations

No fallback system is transparent to the caller. Here's what you actually trade off:

Latency. Each fallback attempt adds connection + inference time. A fast call that normally takes 800ms can take 3+ seconds if the first two models fail. For real-time applications, set a total timeout and surface to the user that a fallback is in progress.

Cost unpredictability. Fallback models may have different pricing. Your monthly bill can spike if your primary model has a bad week and all traffic shifts to a more expensive fallback. Track spend per model and set alerts when fallback-model spend exceeds a threshold.

Behavioral drift. Different models produce different outputs. A system prompt that Claude handles elegantly might produce aggressive refusals on Gemini. A chain-of-thought instruction that works on GPT may be ignored by DeepSeek. Test your fallback models with your actual prompts before depending on them.

Determinism is gone. Even with temperature=0, different models produce different outputs. If your application depends on deterministic or near-deterministic behavior, cross-model fallback fundamentally changes the semantics of the call. Your fallback is keeping the system running, not keeping it identical.

There is no zero-dependency option. A fallback system that spans multiple providers means you depend on all of them — their uptime, their pricing changes, their model deprecations, their API migrations. Your operational surface area multiplies.

9. What to do right now

If you take nothing else from this article, do these three things today:

Identify which LLM calls in your codebase are single-model and single-provider. Anywhere you hardcode model="gpt-5.5" with no alternative path is a single point of failure. Count them.
Pick one non-critical path and add a two-model fallback chain. Not your payment processing. Not your auth flow. A summarization step, a classification call, a "suggest alternative phrasing" feature. Ship it, instrument it, watch the logs for a week. Get comfortable with the pattern before you apply it to critical paths.
Add a health-check endpoint that probes your primary model every 60 seconds and exposes the result. A simple /health/llm that calls your primary model with a trivial prompt and returns {"status": "ok", "latency_ms": 420} or {"status": "degraded", "error": "timeout"}. Wire this into whatever monitoring you already use. You want to know about provider degradation before your users do — and before your fallback system has to prove itself under fire.

The goal of a fallback system is not perfection. It's to turn a hard failure — your agent crashes, your pipeline stops, your user sees an error — into a soft degradation the caller might not even notice. That's worth the complexity.

If you liked this, you might also want llm-honesty-probe (github.com/seven7763/llm-honesty-probe), an open-source tool that checks whether your API endpoint is actually serving the model it claims. Fallback + verification = you're covered on both reliability and trust.

OpenAI cut prices 80%. Here's why I'm not celebrating.

Seven — Fri, 31 Jul 2026 16:23:40 +0000

This week, OpenAI slashed Luna (GPT-5.6) pricing by 80% — from $7 to $1.40 per million tokens. Anthropic shipped Opus 5 at half the price of its predecessor. The LLM API market is in a genuine price war, and for once, developers are winning.

I should be celebrating. I'm not. Here's why.

The hidden math of cheaper models

When official API prices drop, the economics of model-swapping change in a way nobody talks about.

A bad relay has a simple business model: bill you for Model X, serve you Model Y, pocket the difference. When Model X costs $15/M tokens and Model Y costs $0.15/M, the margin is $14.85. That's already attractive.

When Model X drops to $1.40? The absolute margin shrinks — and that's what most people see. But the relative margin — the ratio of what you're charged vs what's actually served — can actually grow if budget models get cheaper still. And they have.

The point isn't the exact math. The point is: price competition between legitimate providers doesn't eliminate the incentive to swap models. It often masks it. When everyone's prices are dropping, a relay that's 30% cheaper than official looks like aggressive discounting, not fraud.

"What model are you?" doesn't work

If you're still asking an API endpoint to identify itself, stop. That's a system prompt — trivially spoofed. Every relay operator knows to make the model say "I am Claude, created by Anthropic."

What works: behavioral fingerprinting at temp=0.

Give the model a fixed set of tasks that different models handle differently — reasoning problems, long-context recall, strict JSON formatting, refusal boundaries. Run them at temperature 0 to remove sampling noise. Diff the results against the official API.

This isn't theoretical. Here's the actual tool:

git clone https://github.com/seven7763/llm-honesty-probe
cd llm-honesty-probe
python3 -m llm_honesty_probe --self-test --card

That runs a battery of deterministic probes and produces a verdict card: PASS, SUSPICIOUS, or DEGRADED — with the specific probes that triggered each finding. No API key leaves your machine. No judgment calls required. The self-test runs without any API key and takes about 30 seconds.

To test your actual endpoint:

export LLM_ENDPOINT="https://your-endpoint/v1"
export LLM_API_KEY="sk-your-key"
python3 -m llm_honesty_probe --compare

Why this matters more now, not less

The price war is genuinely good news. Lower costs mean more experiments, more products, more people who can afford to build with LLMs. I want all of that.

But cheaper official APIs also mean more users signing up for the first time — users who don't know what "Claude" is supposed to sound like, what GPT-5.6's reasoning depth should feel like, or what latency to expect. These are exactly the users a model-swapping relay wants: no baseline for comparison.

The defense isn't trust. It's verification that costs less than being wrong.

Test your endpoint right now

If you're using any third-party API endpoint — mine included — run the probe. The self-test takes 30 seconds. The compare against your endpoint takes about 2 minutes.

If everything passes, you've got evidence — not proof, but signals — that you're getting what you pay for. If something flags, you know exactly which probe to investigate.

Either way, you know more than you did before. That's the whole point.

Disclosure: I work on daoxe, an OpenAI-compatible LLM gateway. I also maintain llm-honesty-probe, the open-source tool mentioned in this post. The probe treats every endpoint identically — including daoxe. Point it at anyone. If it flags something on my own service, I want to know before you do.

The price war is real. The trust problem is real. They're related, and fixing one doesn't automatically fix the other.

Deploy a real AI chat app in one click — key stays server-side (Vercel/Render/Railway)

Seven — Tue, 21 Jul 2026 14:40:49 +0000

Most "deploy an AI app in 5 minutes" tutorials quietly do the one thing you must never do: put an API key where a browser can see it. This post is three one-click starters that are correct by construction — the key is a deploy-time environment variable, server-side only, never in client code, never committed to git.

Each template is provider-neutral: it defaults to an OpenAI-compatible endpoint (https://daoxe.com/v1) but points at any compatible endpoint by changing one variable. Pick the one that matches what you're building.

Disclosure: I work on daoxe, the OpenAI-compatible gateway these default to. Everything below works against any compatible endpoint — the default and the "powered by" link are one-line changes.

The rule that makes these safe

An LLM API key is a billing credential. Anything shipped to a browser is public. So every template here follows the same discipline:

DAOXE_API_KEY (and any tokens) are deploy-time env vars, declared in each platform's config with no baked-in value.
.env.example files ship empty placeholders.
The key is used server-side only — an Edge function, a proxy server, or a bot process holds it; the browser never sees it.

If a tutorial ever tells you to paste your key into front-end JavaScript, close the tab.

Template 1 — Streaming chat UI on Vercel

A minimal chat UI backed by an Edge function that holds the key and streams tokens back to the browser. Good for: a demo, an internal tool, a landing-page assistant.

Repo: seven7763/daoxe-vercel-chat

Deploy: click the Vercel button in the repo README; Vercel prompts you for two env vars:

DAOXE_API_KEY — your key.
DAOXE_MODEL — an exact id from GET /v1/models.

That's it — you get a URL with a working streaming chat. The browser calls your Edge function; the Edge function calls the gateway with the key. The key never leaves the server.

Browser ──▶ /api/chat (Edge, holds key) ──▶ https://daoxe.com/v1/chat/completions
        ◀── streamed tokens ◀──────────────

Optional DAOXE_ALLOWED_MODELS lets you restrict which models the function will serve.

Template 2 — OpenAI-compatible proxy on Render

Sometimes you don't want a UI — you want to share one key with a team or a script without handing out the real credential. This template is a tiny OpenAI-compatible proxy: it exposes /v1/chat/completions, injects the real key server-side, and gates access with a token you hand out instead.

Repo: seven7763/daoxe-render-proxy

Deploy: click the Render button in the repo README; set DAOXE_API_KEY (the real key) and a share token. Now teammates point their OpenAI-compatible tools at your proxy URL with the share token — they get model access, never your billing credential. It also serves a live /healthz page.

Template 3 — Discord bot on Railway

A Discord bot that answers /ask with an LLM. Good for: team assistants, support bots, community helpers.

Repo: seven7763/daoxe-railway-discord-bot

Deploy: from the public repo, open Railway → New Project → Deploy from GitHub (or create a Template once), then set DAOXE_API_KEY, DAOXE_MODEL, and your Discord bot token. The bot process holds the key; Discord users just type /ask. (This one has a single runtime dependency, discord.js.)

Env vars cheat sheet

Variable	Where	Notes
`DAOXE_API_KEY`	all three	Billing credential. Server-side only.
`DAOXE_MODEL`	vercel-chat, railway bot	Exact id from `GET /v1/models`.
`DAOXE_ALLOWED_MODELS`	vercel-chat (optional)	Comma-separated allowlist.
Share / Discord tokens	render proxy / railway bot	Separate from the billing key.

(Repos are public. Vercel/Render one-click buttons live in each README; Railway is "Deploy from GitHub" from the public repo — no private template id required.)

What this buys you

Three one-click AI starters: Vercel (streaming chat), Render (OpenAI-compatible proxy), Railway (Discord bot).
Key never in the browser — the failure mode most tutorials ship by accident.
One key → many models behind a single OpenAI-compatible base URL, with fallback and one bill if you want it.

Don't trust the pitch — deploy one template, open the network tab, and confirm the key is not in any client request. Then point the same templates at any OpenAI-compatible endpoint and compare.

One key, two ecosystems: routing between DeepSeek/Kimi/Qwen and Claude/GPT/Gemini by task

Seven — Sun, 19 Jul 2026 22:40:57 +0000

Picking a single model family for an app is a false choice. Some Chinese-origin models are strong and cost-efficient at certain tasks; some Western models lead at others. If you're building outside mainland China, the practical move isn't to bet on one — it's to route each request to the right one, and make switching a one-line change instead of a new integration.

This guide shows how to do that through a single OpenAI-compatible endpoint, so DeepSeek, Kimi, Qwen, Claude, GPT and Gemini all sit behind one key and one bill — and how to choose between them per task without guessing.

Disclosure & scope: I work on daoxe, an OpenAI-compatible gateway. It is not available in mainland China — this article is written for developers outside it (overseas devs, overseas Chinese, and users in Russia, Brazil, Vietnam, Turkey and elsewhere) who want access to both ecosystems in one place. Everything here is standard OpenAI-compatible config; point it at any endpoint that carries the models you want.

Why route at all

Three reasons a single-model app leaves value on the table:

Task fit. Models have uneven strengths — reasoning, long-context, code, multilingual, structured output, vision. The best model for summarizing a 200-page PDF isn't necessarily the best for a tricky refactor.
Cost. For high-volume, low-stakes calls (classification, extraction, routing itself), a cheaper model is often good enough, and the savings compound. Reserve the expensive model for the calls that need it.
Resilience. If one provider is rate-limited or degraded, being able to fail over to another family keeps you shipping.

The blocker has always been integration cost: different SDKs, keys, billing, and response quirks per vendor. An OpenAI-compatible endpoint removes it — model choice becomes a string.

The mechanism: model choice is a string

Every call uses the same shape; only the model field changes:

curl https://daoxe.com/v1/chat/completions \
  -H "Authorization: Bearer $DAOXE_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"<MODEL_ID>","messages":[{"role":"user","content":"..."}]}'

List what your key can actually call — the authoritative, account-scoped source is:

curl https://daoxe.com/v1/models -H "Authorization: Bearer $DAOXE_API_KEY"

Use those exact ids. Families you'll typically see include Chinese-origin ones (DeepSeek, Kimi, Qwen, and others) and Western ones (Claude, GPT, Gemini). Don't hardcode ids from this post — they drift; read them from /v1/models.

A simple task-based router (Python)

You don't need a framework. A dict from task → model id gets you 90% of the value:

import os, httpx

BASE = "https://daoxe.com/v1"
KEY = os.environ["DAOXE_API_KEY"]

# Map task types to EXACT ids from GET /v1/models (examples — replace with yours).
ROUTES = {
    "cheap_bulk":   "deepseek-...",     # high-volume extraction/classification
    "long_context": "kimi-k2.5",        # very long inputs
    "reasoning":    "claude-sonnet-4-6",# hard multi-step problems
    "code":         "gpt-5.5",          # tricky refactors
    "default":      "claude-haiku-...", # fast, inexpensive fallback
}

def ask(task: str, prompt: str) -> str:
    model = ROUTES.get(task, ROUTES["default"])
    r = httpx.post(f"{BASE}/chat/completions",
        headers={"Authorization": f"Bearer {KEY}"},
        json={"model": model, "temperature": 0,
              "messages": [{"role": "user", "content": prompt}]}, timeout=60)
    r.raise_for_status()
    return r.json()["choices"][0]["message"]["content"]

print(ask("cheap_bulk", "Extract the invoice total as JSON."))
print(ask("reasoning",  "Prove or disprove: ..."))

Because it's one endpoint and one key, adding a route is adding a dict entry — no new SDK, no new credential. Add a try/except that falls back to another family on error and you've got cheap resilience too.

How to choose the route (without vibes)

Don't route on reputation — route on evidence for your workload:

Build a tiny eval set of real prompts per task type with known-good answers.
Run each candidate model through it, and record accuracy alongside latency (p50/p95) and token cost. (A small harness does this; I wrote up a reproducible gateway benchmark separately.)
Pick the cheapest model that clears your quality bar for each task — not the most expensive one you can afford.
Re-check periodically. New model versions ship constantly; a route that was optimal last quarter may not be now.

On price specifically: I'm not going to quote numbers here because they change. Check the provider's pricing page and put real $/1M figures next to your latency and accuracy — the right route is a three-way trade-off, not a single-axis choice.

Don't forget to verify what you're getting

Routing across families makes verification more important, not less: with several models behind one endpoint, you want to confirm each id actually serves what it claims and isn't being silently swapped or truncated. Test behavior, not the model's self-report — a tokenizer fingerprint, a capability-floor check, and a long-context needle test catch the common cheats. (There are open-source probes for this; I covered the method in a separate post.) A good endpoint is fine with you running these against it.

Honest positioning

A few things to be straight about:

This is for outside mainland China. daoxe doesn't serve mainland China; the audience here is overseas developers and users in regions where getting both ecosystems in one place — and paying for it — is otherwise fiddly.
"Chinese vs Western" is about model origin, not a quality ranking. Route on measured task fit and cost, not on where a model came from.
One key is a convenience, not magic. It removes integration and billing friction; it doesn't make a small model into a big one. Match the model to the task.

daoxe is the endpoint I use for this — OpenAI-compatible at https://daoxe.com/v1, native Anthropic Messages for Claude-native tools, one key across many models (Chinese-origin and Western), one bill, and a design meant to be verified. See the pricing page for current models and prices; use GET /v1/models for what your key can call.

TL;DR

Don't pick one family — route each task to the best/cheapest model.
One OpenAI-compatible endpoint makes model choice a string: one key, one bill, no per-vendor SDKs.
Choose routes on a small eval set (accuracy + p50/p95 + $/1M), not reputation; re-check as versions ship.
Verify each model behaviorally, and remember this is for use outside mainland China.

DEV Community: Seven

Four ways reasoning models hide their thinking (and what that does to your bill)

Shape 1 — shared budget (GLM)

Shape 2 — separate field (DeepSeek)

Shape 3 — explicit flag (MiniMax)

Shape 4 — usage accounting only (Grok)

What each shape costs you

A four-line probe

The general rule

Anthropic Messages is a different protocol, not a flag — and shims leak

The two protocols disagree about what a message is

tool_use and tool_result are shaped differently on purpose

Streaming is a typed event protocol

system is a parameter, and that's where caching lives

stop_reason has six values

Thinking blocks are signed

What it costs you

Five tests

Claude Code against a custom endpoint

The honest version

We pointed our model-honesty probe at our own gateway. It said SUSPICIOUS.

What the probe actually does

The run

Three tells

The fix, and the re-run

Why the wrong flag is the better story

What we are not claiming

Five coding agents, one endpoint: where the base URL field actually lives

Ground rule: never type a model ID from memory

Cursor

Cline

Continue

Zed

Aider

Symptom → cause

When the tool won't tell you what it sent

The part that isn't configuration

"OpenAI-compatible" is a spectrum, not a boolean — here's an 11-check conformance suite

The nine surfaces

The script

Reading the output

What this can't tell you

Ten questions to answer before you route production traffic through someone else's LLM endpoint

1. Protocol coverage — native or translated?

2. Model ID transparency

3. Provenance — where does the capacity come from?

4. Can you get your own data out?

5. Incident behaviour

6. Key handling

7. Data handling

8. Rate limits and behaviour under load

9. Billing you can reconcile

10. The exit plan

The scorecard, run against my own employer

Using this without turning it into a ritual

Model IDs are a dependency. Pin them like one.

Three ways a wrong ID fails

The ten-second test for Mode C

Dated versus floating

GET /v1/models is the source of truth, and it's per-key

Lock the catalogue

The alias layer

When an upstream retires a model mid-project

Variants and channels on a gateway

Drift under a pinned ID

Model IDs are a dependency. Pin them like one.

Three ways a wrong ID fails

The ten-second test for Mode C

Dated versus floating

GET /v1/models is the source of truth, and it's per-key

Lock the catalogue

The alias layer

When an upstream retires a model mid-project

Variants and channels on a gateway

Drift under a pinned ID

Your LLM API will fail in production. Here's the fallback system that catches it.

1. Why retry alone isn't enough

2. The model-level fallback pattern (simple, with sharp edges)

Gotcha 1: Protocol incompatibility

Gotcha 2: The system prompt disappears

`system` is a parameter, and that's where caching lives

`stop_reason` has six values

`GET /v1/models` is the source of truth, and it's per-key

`GET /v1/models` is the source of truth, and it's per-key