We pointed our model-honesty probe at our own gateway. It said SUSPICIOUS.

We built a CLI tool that gives you signals about whether an LLM API endpoint is serving the model it claims — or quietly handing you a smaller, quantized, or context-truncated substitute. In early September we pointed it at our own gateway.

It flagged us. SUSPICIOUS — Hard to explain if the claim were true.

This post is the full record: what the tool said, why it was wrong, how we knew the tool was the wrong one, and what changed. Every detail below is quoted from the raw JSON of those runs, and the fix itself is an open commit in the tool's public repo — so you can check my quoting instead of trusting it.

Disclosure: I work on daoxe, the gateway that got flagged. The tool is provider-neutral and open source — github.com/seven7763/llm-honesty-probe — and this is written as an engineer's post-mortem, not a victory lap. There is no version of this story where we come out looking flawless; the honest one starts with our own tool failing on us.

What the probe actually does

One paragraph of background, then we get to the incident. The probe runs a fixed battery of behavioral checks against any OpenAI- or Anthropic-compatible endpoint: a capability floor (easy tasks with answers the tool already knows), long-context recall (it plants the needle, so it knows the right answer), output determinism and backend-identity stability at temperature 0, a tokenizer fingerprint from usage deltas, and a self-report it deliberately weights near zero because "what model are you?" is trivially spoofed. Your key is read only from an environment variable, the endpoint host is masked on shareable verdict cards, and every signal is explicitly labeled what it is: heuristics, not proof.

The run

2026-09-04, against one of our own groups — glm-5.3-flash, the group this working session was itself running on at the time:

python3 -m llm_honesty_probe \
  --base-url https://jp.daoxe.com/v1 --claimed-model glm-5.3-flash --card

The card came back:

===  SUSPICIOUS  ===
Hard to explain if the claim were true.

[-] Tokenizer fingerprint
      No verdict — add --compare for a definitive check.
[!] Capability floor
      Failed 2/4 easy reasoning tasks a full-tier model rarely misses. (medium)
[!] Long context
      Lost a needle in long context (possible truncation). (medium)
[+] Consistency
      Stable model field / fingerprint across calls.
[-] Self-report (spoofable, low weight)
      Claimed model family unknown; can't compare self-report.

Two failed signals, both "possible downgrade", neither deniable-looking. The arithmetic task answered 396. Multistep answered 58. The string-reversal task answered… nothing. The character-counting task: nothing. An empty content with finish_reason: "length". The strict-JSON test — the prompt demands exactly {"a":42,"b":"ok"} — returned {"a":.

And then we ran it a second time, same version, same endpoint, and the card shifted under us: one floor task tripped instead of two; the strict-JSON test passed this time; the needle at 8,000 characters still failed; and the report's own headline softened to "Only low-confidence flags; likely noise, but worth a second run." The three temperature-0 calls came back identical this time — all three of them the empty string. The same traffic gave the same tool two different stories. This is the part where a marketing team would quietly not publish the card.

Three tells

What turned the investigation from "is our gateway dishonest?" into "is our probe dishonest?" was three details in the raw JSON, none of which the card shows:

1. The server quoted our request back at us. The tokenizer probe reported itself inconclusive with this detail string:

Can't fingerprint tokenizer: HTTP 400: {"error":{"message":"max_tokens
must be greater than 2 …","type":"invalid_request"}}

We had our own code send max_tokens=1 for the fingerprint calls — an old OpenAI-compat habit. Our gateway requires max_tokens > 2. No server lies like that; that 400 is the sound of our tool misusing our API contract.

2. The truncated JSON ended exactly at the budget. {"a": is three tokens of an answer that needs about eight. Three tokens in, because the probe — at that time — capped those completions at 64 tokens (32 on the long-context calls). An answer that stops precisely at the budget with finish_reason: "length" is the signature of running out of allowed tokens, not of a weak model. The empty replies in the reasoning tasks had the same shape: content: "", finish_reason: "length".

3. The asymmetry. If the backend had really been swapped for a weaker model, why did arithmetic (396) and multistep (58) pass while reversal and counting came back empty? A weaker model gets them wrong — it doesn't get them blank. "Wrong" is a capability signal. "Empty at exactly the token limit" is a plumbing signal.

The root cause was in the routing, and it was innocent: this group can serve reasoning-style backends, which spend completion tokens on hidden thinking before any visible answer. The budget went to thinking. The probe read the empty visible reply as "failed a task a full-tier model rarely misses." That is a false positive — a bug in our caller-side assumptions, filed against our server. One check in the raw archive settles who was wrong: the second v0.2.0 run — where the budget-starved replies mostly stopped happening — scored the capability floor merely inconclusive, "could be formatting rather than capability," while the needle call at 8k still failed at the then-32-token needle cap. Same bug, different face.

The fix, and the re-run

Same day, v0.2.1 (commit cb15e61):

• A shared guard — budget_starved() — that recognizes "OK, but empty content with finish_reason: length" and forces those probes to inconclusive. Budget starvation is a property of the caller's max_tokens, not of the server's honesty, and it must never be able to produce SUSPICIOUS.
• Default per-call budgets raised so short factual answers actually fit (our re-run passed --max-tokens 512).
• The tokenizer probe now requests 4 tokens instead of 1, clearing the gateway-validation 400.
• The built-in mock server grew a reasoning=True mode that reproduces the starvation shape, plus regression tests asserting it can never flag suspicion. 20/20 pass.

Re-run with the fixed tool: needle recall at 2,000 and 8,000 characters — both retrieved. Capability floor: 4/4, exact JSON. Consistency: identical output on all 3 calls at temperature 0. Verdict:

===  PASS  ===
No downgrade signals in this run.

Note what is not on that card. The tokenizer fingerprint still says "No verdict — add --compare," because in single-endpoint mode there's no reference to diff against; it now measures the vector cleanly, finds its closest public match, and still refuses to call it — by design, the claimed family has no public reference table entry. Self-report remains inconclusive and low-weight forever, because it's the one signal a dishonest endpoint can pre-write. We did not upgrade the accusation into a certificate. We turned a wrong red into an honest gray.

Why the wrong flag is the better story

Here's the uncomfortable logic. Any tool can print PASS; it's one string. A tool that has been publicly wrong about a concrete endpoint, caught, root-caused, fixed in an open commit, and re-run has shown you something no clean card can: its red flags are fallible, so they get investigated instead of believed, and the investigation itself is checkable. If the only version of this story you'd ever seen were "we ran the probe on ourselves and it passed," you should trust it less — that's what a rigged scoreboard looks like.

The false positive also taught us something about who this tool hurts. We built it to catch relays that quietly downgrade you. A miss leaves a bad provider unpunished today and time-series diffs catch it tomorrow. A false accusation frames an honest provider with a screenshot someone posts in a forum — and once a community learns that these cards convict the innocent, every flag starts getting ignored, including the true ones. Over-flagging doesn't just hurt the flagged; it degrades the signal for everyone. "Treat a flag as a reason to look closer, not a conviction" is easy to print and much harder to engineer for, because the defaults are asymmetric: the probe weights failure more than success, so the failure modes you feed it decide who gets accused. budget_starved() is the fix for one such mode. It is not the fix for all of them.

What we are not claiming

The discipline of "signals, not proof" points at us twice:

• The SUSPICIOUS was wrong. That does not mean the current PASS is right in any certificate sense — it means this battery, run once, on one group, showed no downgrade signals at that moment. A single run is a snapshot; silent degradation is a time-series problem, which is what the --json diff mode in cron is for.
• One group is not the gateway. Our other groups went untested — the test account simply didn't have the balance for more.
• Differential mode is the stronger tool, and we didn't use it: point --compare at the official endpoint and the tokenizer question can become a real verdict instead of an inconclusive. We'd rather say "not settled" than let our own marketing settle it.
• A provider that mirrored the official tokenizer, matched the capability floor, and held long context would pass this battery. We can't prove intent from the outside — ours or anyone's.

So: clone it, run it against us, run it against the endpoint you actually pay for, and compare both cards.

git clone https://github.com/seven7763/llm-honesty-probe
cd llm-honesty-probe
python3 -m llm_honesty_probe --self-test --card   # no key needed — see both verdict shapes first

Questions, corrections, or "your probe wrongly flagged my honest relay": the public support channel is Telegram @daoxe_ai, and the issue tracker is on the repo. The bug reports we want most are the ones about false positives — the next wrong SUSPICIOUS is exactly as damaging as the one we just ate.