An API can return a valid response, use the model name you requested, and still leave an important question unanswered:
Is the endpoint behaving like the model it claims to serve?
A single answer cannot settle this. Style guessing is unreliable, benchmark prompts are easy to overfit, and providers can update models without notice. A useful test needs to be cheap, repeatable, and honest about uncertainty.
This is the workflow we are using for the AllRouter Public Model Fingerprint Lab. AllRouter is not treated as the judge. It is one endpoint under the same public test.
The idea: compare distributions, not prose
The paper One Token Is Enough shows that models have surprisingly stable preferences when they answer simple one-token questions repeatedly.
Ask for a random number, color, letter, city, or coin flip 25 times. One answer means almost nothing. The distribution across many answers is a behavioral fingerprint.
A practical audit looks like this:
- Select 8 or more task-language cells.
- Sample each cell repeatedly at a fixed temperature.
- Normalize the one-word answers.
- Compare the observed distributions with a trusted reference using Jensen-Shannon divergence.
- Repeat the run on another day before drawing a strong conclusion.
The paper reports an equal error rate of about 10.6% with 8 cells and about 7.3% with 40 cells. That is useful evidence, not perfect identification.
Four verdicts, not a binary accusation
The open-source implementation we are using reports four states:
-
match: behavior is statistically consistent with the reference. -
uncertain: the result is in the gray zone; collect more samples or repeat later. -
mismatch: behavior differs substantially; investigate before concluding why. -
insufficient: there are not enough valid comparable samples.
A mismatch does not prove fraud. Model updates, quantization, hidden system prompts, reasoning fallbacks, and load-balancing across several backends can all move the fingerprint.
That limitation is part of the result, not fine print.
Run the test locally
The MIT-licensed llm-fingerprint-detector provides a TypeScript CLI and library for OpenAI-compatible endpoints.
The fixed experiment definitions, public report schema, and blank result card are in the public protocol directory.
Keep the API key in a local environment variable. Do not paste it into a form, chat, issue, or public log.
git clone https://github.com/ToseaAI/llm-fingerprint-detector.git
cd llm-fingerprint-detector
npm install
npm run build
export LLM_FINGERPRINT_API_KEY='your-local-key'
node dist/cli.js fingerprint \
--base-url https://your-endpoint.example.com/v1 \
--model your-model-id \
--out observed.json
To verify against a trusted reference:
node dist/cli.js verify \
--base-url https://your-endpoint.example.com/v1 \
--model your-model-id \
--reference trusted-reference.json
Record the model ID, test-set version, timestamp, temperature, sample count, comparable cells, mean JSD, split-half consistency, warnings, and a hash of the response summary. Never publish the key or private prompts.
The test I want providers to publish
A provider should not publish only a green badge. A useful public report should include:
| Field | Why it matters |
|---|---|
| Test-set version | Prevents silent changes to the experiment |
| Collection time | Makes model updates and drift visible |
| Declared model ID | Defines the claim being tested |
| Mean JSD and comparable cells | Shows the basis for the verdict |
| Split-half JSD | Reveals whether the endpoint disagrees with itself |
| Error and invalid-sample rates | Exposes reliability problems |
| Reproduction command | Lets a third party repeat the run |
| Limitations | Prevents overclaiming |
We will run the same test on AllRouter at a fixed time for seven days and publish match, uncertain, mismatch, or insufficient without forcing a positive conclusion.
Why this can be more useful than a generic benchmark
Benchmarks ask whether a model is capable. A routing audit asks whether the endpoint stays consistent with the model and configuration you chose.
That matters for coding agents. A silent route change can alter formatting discipline, tool-use behavior, latency, refusal boundaries, and the amount of review a patch needs. Even when a response is syntactically valid, the workflow cost can change.
The identity test should therefore sit next to a small task suite:
- preserve a public API during a refactor;
- handle missing configuration explicitly;
- follow a strict JSON schema;
- separate risk analysis from the edit;
- keep the diff inside the requested boundary.
The fingerprint tells you whether behavior changed. The task suite tells you whether that change matters to your workflow.
Re-run the same experiment on AllRouter
If you want to compare the same OpenAI-compatible workflow on AllRouter, use the campaign-specific entry below:
Run the same test on AllRouter
The goal is not “trust this router.” The goal is the opposite: make the result reproducible enough that trust is not the only evidence available.
In the next report, we will publish the seven-day drift format, including latency, error rate, split-half consistency, and the exact decision thresholds.
Top comments (0)