Every LLM endpoint's landing page says fast, reliable, compatible, secure. Those words are free. This is the list of questions that aren't free — the ones where the answer takes a vendor real work to be able to give, and where you can verify the answer yourself in an afternoon.
Disclosure up front: I work on daoxe, a multi-model gateway. At the end I run this checklist against it, and two rows don't pass. That's the point of publishing it — a checklist that flatters whoever wrote it isn't a checklist, it's a spec sheet.
1. Protocol coverage — native or translated?
Ask which wire protocols the endpoint implements and which ones it translates into. These are different products. A translation layer is fine for text chat and lossy for agents: tool-call block ordering, streamed tool arguments, prompt-cache markers and stop-reason fidelity all live in fields that have no equivalent on the other side.
Evidence: for the Anthropic protocol, omit the required max_tokens from a /v1/messages call. A native implementation returns 400; a translation layer often fills in a default and returns 200. For the OpenAI protocol, check whether streamed tool calls arrive as incremental fragments with an index, or all at once at the end.
Bad answer: "we're fully compatible with everything."
2. Model ID transparency
Does GET /v1/models return the exact IDs your key can call — not the vendor's full catalogue? Does the response echo a model field, and does it match what you asked for? And critically: what happens when you send an ID that doesn't exist?
Evidence: send a deliberately mangled model ID. If you get a completion instead of an error, the endpoint is picking on your behalf, and every ID you send is advisory rather than binding. That single fact changes how much of the rest of this list you can trust.
Bad answer: a model list on a marketing page, with no catalogue endpoint.
3. Provenance — where does the capacity come from?
Any aggregator is reselling access it obtained somehow. The question isn't whether there's a supply chain; it's whether they'll tell you what's in it. Some capacity is direct vendor access, some is enterprise-cloud resale, some is reverse-engineered consumer endpoints with materially different reliability. All three can be legitimate purchases — at very different risk levels, for very different workloads.
Evidence: ask for the labelling scheme, in writing, and check whether it appears in the product itself or only in a support conversation.
Bad answer: silence, or "we don't disclose our infrastructure." Undisclosed provenance means you cannot reason about your own availability.
4. Can you get your own data out?
Usage per key, per model, per day. Ideally an export rather than a dashboard screenshot. You need this for two things you'll definitely do: reconciling a bill you don't understand, and attributing spend to a team or a feature.
Evidence: try to export a month. If it's a dashboard-only number, plan on logging every request yourself from day one — you can't retrofit history.
5. Incident behaviour
How do you find out something is degraded? Three tiers, in descending order of comfort: a published availability page with history, an announcement channel, or nothing.
Here's the thing worth internalising: an availability figure on a marketing page is a claim, not a measurement. It's not necessarily false, but it isn't evidence, and you shouldn't put it in a design document. Whatever tier your vendor is in, run your own probe — a one-token completion against each model you depend on, every five minutes, appended to a JSONL file. After a month you own a success rate and a latency distribution measured from your own network, which is a strictly better artifact than any number on any vendor page.
Evidence: your own probe log. Nothing else counts.
6. Key handling
Six sub-questions: can you issue multiple keys, scope them (per model, per group, per spend cap), rotate without downtime, revoke instantly, see last-used time, and does revocation actually take effect immediately?
Evidence: issue a second key, use it once, revoke it, and immediately retry. If it still works sixty seconds later, you've learned something important about the caching layer between you and their auth.
Then audit where your keys physically live. On a normal developer laptop a gateway key ends up in an editor's settings store, a shell profile, two dotfiles, a CI secret, and at least one Docker environment. Write that list down before you distribute a key to a team, not after.
7. Data handling
Retention period for prompts and completions, whether your traffic can be used for training, whether that's per-account configurable, and what the sub-processor situation is for the upstream vendors behind the aggregator.
Get this in writing. I'm deliberately not paraphrasing anyone's policy — including my own employer's — in a blog post you might then rely on. Ask, get a written answer, keep it, and re-ask on renewal.
Bad answer: a verbal assurance in a chat window.
8. Rate limits and behaviour under load
Are limits documented per key or per model? What does the endpoint do when you exceed them — a clean 429 with Retry-After, a 500, or a hang? Do errors have a stable machine-readable shape your retry layer can classify?
Evidence: fire a small burst of concurrent requests at a cheap model and read the failures. A 429 carrying Retry-After is a mature implementation. A 500 where a 429 belonged will turn your backoff into a retry storm, because your code will classify it as transient-and-worth-hammering.
9. Billing you can reconcile
You want three things: a mechanism you can state in one sentence, a live price reference rather than a number in a PDF, and per-request usage data you can multiply out yourself.
Be actively suspicious of "cheapest". Price is the easiest thing on this list to change and the easiest to advertise; it tells you nothing about the nine criteria above it, and a number well below everyone else's floor is usually a statement about provenance (criterion 3), not about efficiency.
For the record, since I'm asking you to run this against my employer: DaoXE tops up at a flat 1:1 top-up rate across Alipay, WeChat Pay, USDT, bank card (Visa·Mastercard), Apple Pay and Google Pay — one unit of local currency buys one dollar of credit — models bill at their USD list prices, and the live numbers are at https://daoxe.com/pricing — check them there rather than trusting any figure in an article, including this one.
10. The exit plan
The last question is the one that makes the other nine survivable: how many lines of code does it take to leave?
If the answer is "change a base URL and a key in one config file", the switching cost is near zero and every other risk on this list becomes a manageable one. If the answer involves a proprietary SDK, vendor-specific request extensions, or a routing DSL you can't reproduce elsewhere, you've bought a migration.
Four habits that keep the number low: keep the base URL and model IDs in config, not in code; prefer standard protocol fields over vendor extensions; keep your own request/response logs; and maintain a golden eval set of twenty to fifty real requests so that "is the replacement good enough" is a script you run rather than an argument you have.
The scorecard, run against my own employer
| # | Criterion | DaoXE | Notes |
|---|---|---|---|
| 1 | Protocol coverage | Pass | OpenAI (/v1/chat/completions, /v1/responses, /v1/embeddings, /v1/images/generations, /v1/videos), Anthropic Messages (/v1/messages), Gemini (/v1beta/models/{model}:generateContent), plus /v1/rerank — implemented, not translated. Verify with the max_tokens test above. |
| 2 | Model ID transparency | Pass |
GET /v1/models returns the exact IDs your key can call. Catalogue size drifts constantly — hundreds of models across roughly 25 vendors — so treat any published count, including ours, as stale. |
| 3 | Provenance | Pass, with a caveat you should read | Channels are labelled in the catalogue: full-power, official relay, reverse-engineered ("quality not guaranteed"), promotional. The reverse-engineered tier genuinely exists. Labelling it is the honest part; choosing it for production is your call, and I'd say no. |
| 4 | Usage / log export | Ask | Don't take my word for the current state — ask on Telegram @daoxe_ai and log your own requests regardless. |
| 5 | Incident behaviour | Does not pass | There is no public status page, so run your own cron probe. The availability figure on the marketing page is a company claim, not a measurement, and I won't repeat it here as if it were one. |
| 6 | Key handling | Test it yourself | Run the issue → use → revoke → retry sequence on your own account. Note that sign-up is open, with password login, no email verification, and OAuth via GitHub, Telegram or Passkey — convenient, and worth factoring into your own account-security posture. |
| 7 | Data handling | Ask, in writing | Same rule I'd apply to any vendor. Ask @daoxe_ai, keep the answer. |
| 8 | Rate limits under load | Test it yourself | Burst a cheap model, read the failure bodies, check for Retry-After. |
| 9 | Billing | Pass on mechanism | One-sentence mechanism, live prices on the pricing page, per-request usage in the response body. |
| 10 | Exit plan | Pass | Standard protocols, model IDs are strings, base URL is one config line. Leaving is the same edit as arriving — which is the only version of this claim worth making. |
One more thing that belongs on any honest vendor row: DaoXE is not available in mainland China. That's the official line and it's a hard constraint, not a caveat to work around.
Using this without turning it into a ritual
Ten criteria is too many for a side project and about right for anything with users. If you only have an hour, do these four: the mangled-model-ID test (criterion 2), the provenance question (3), start the probe (5), and count the lines it would take to leave (10). Those four catch the failures that are expensive to discover later; the rest catch the failures that are merely annoying.
And apply it symmetrically. If a vendor's answer to criterion 3 or 7 is a shrug, that's data — but so is a checklist published by a vendor that somehow scores ten out of ten on its own list.
Top comments (0)