Your LLM Detector Is 99% Confident. It May Still Be Wrong

A black-box model detector returns the following result:
Model A: 99.1%
Model B: 0.6%
Model C: 0.3%
The obvious interpretation is that the endpoint has a 99.1% probability of running Model A.

In many systems, that interpretation is wrong.

The number may only mean that Model A received 99.1% of the normalized probability mass among the candidates included in the detector’s reference set. If the actual model is absent from that set, the detector can remain highly confident while assigning the endpoint to the wrong model.

This is not a minor wording issue. It is the difference between:
“Model A is the closest candidate we know.”

and:
“The endpoint has been verified as Model A.”

The first is a ranking result. The second is an identity claim.

Any serious attempt to identify models behind black-box APIs must keep those claims separate.
The practical problem
When developers call an LLM through an API provider, the model name is usually supplied as metadata:
{
"model": "model-a"
}
That field is a provider assertion. It is not independently verifiable evidence of the model that processed the request.

Several operational situations can create a difference between the advertised model and the observed endpoint:

• traffic may be routed across multiple model versions;
• a provider may silently update the underlying deployment;
• a smaller model may handle part of the traffic;
• a proxy may rewrite model identifiers;
• inference parameters may differ from the official endpoint;
• a model family may contain several behaviorally similar checkpoints. Black-box fingerprinting attempts to investigate this problem by measuring output behavior rather than trusting the model label.

A detector may send controlled prompts, extract statistical features from the responses, and compare them with samples collected from known reference endpoints.

That can produce useful evidence. But the meaning of the resulting probability depends on the statistical problem the detector was designed to solve.
Ranking is not authentication
Assume a detector has a reference set containing (K) known model candidates:
[
\mathcal{R} = {R_1, R_2, \ldots, R_K}
]
For an observed response (x), the detector extracts a feature vector:
[
z = f(x)
]
It then assigns each candidate a similarity or compatibility score:
[
s_i = S(z, R_i)
]
A common final step is to convert those scores into a normalized distribution using softmax:
[
p_i =
\frac{\exp(\tau s_i)}
{\sum_{j=1}^{K}\exp(\tau s_j)}
]
where (\tau) controls how sharply the probability mass concentrates around the highest-scoring candidates.

The output satisfies:
[
\sum_{i=1}^{K} p_i = 1
]
This mathematical property is convenient for ranking candidates. It also creates a dangerous interpretation problem.

The softmax must distribute all probability mass among the available candidates. It has no inherent mechanism for saying:
None of the above.

Even if every reference model is a poor match, one candidate can still receive most of the normalized probability.
A simple example
Imagine a detector whose reference set contains only three model families:
Claude
GPT
Gemini
Now suppose the tested endpoint is powered by an unrelated model that has never appeared in the detector’s reference data.

The detector may still return:
Claude: 97%
GPT: 2%
Gemini: 1%
This result does not imply a 97% probability that the endpoint is Claude.

It means that, after comparing the sample only with Claude, GPT, and Gemini references, Claude received the strongest relative score.

The more precise interpretation is:
Conditional on selecting one of the candidates represented in this reference set, the observed behavior is most compatible with the Claude reference distribution.

The missing condition matters:
[
P(R_i \mid x,\; y \in \mathcal{R})
\neq
P(y = R_i \mid x)
]
The expression on the left is a closed-set classification result. The expression on the right is a claim about the model’s real identity.

They are not equivalent.
Why softmax can be confidently wrong
Softmax does not measure whether the best candidate is a good match in absolute terms. It measures how the candidate scores compare with one another.

Consider two score vectors:
Case 1: [10.0, 2.0, 1.0]
Case 2: [-90.0, -98.0, -99.0]
In both cases, the first candidate leads the second by eight score units. Depending on the temperature, both can produce nearly identical normalized probabilities.

But the cases may have very different meanings.

In the first case, the best candidate may genuinely match the observed sample. In the second, every candidate may be far outside its expected distribution. Softmax alone does not preserve that distinction.

This creates several failure modes.
The true model is absent
A model outside the reference set is assigned to the least incompatible known candidate.
The reference set changes
Adding or removing candidates changes the denominator of the softmax. The reported probability can therefore change even when the observed response and the best candidate’s raw score remain unchanged.
The temperature changes
A calibration temperature can make an output appear more or less decisive without changing the candidate ranking.

For scores (s_i), increasing (\tau) generally produces a sharper distribution. A visually impressive 99% may partly reflect calibration choices rather than stronger evidence.
All candidates are poor matches
Relative normalization can conceal poor absolute compatibility. The winner can dominate simply because every alternative is worse.
Closely related models share features
Models from the same family may be statistically difficult to separate. A high aggregate family match does not automatically support exact checkpoint attribution.
A concrete implementation
I encountered this interpretation problem while working on the behavioral model-fingerprinting system used in Oken’s evaluation lab.

The implementation uses constrained, semantically minimal generation tasks to collect behavioral samples from a black-box endpoint. It then compares properties such as:

• marginal output preferences;
• position-dependent behavior;
• transition magnitudes;
• repetition patterns;
• local sequence structure. These observations are compared with a frozen reference bank to produce a candidate ranking.

This is useful for asking:
Which reference model behaves most similarly to this endpoint under the current evaluation protocol?

It does not, by itself, answer:
Which model was definitively executed on the provider’s server?

The distinction became especially important when presenting the final normalized probability. A concentrated closed-set result may be meaningful as evidence of relative similarity, but it should not be labeled as an independently verified probability of model identity.

That limitation does not make behavioral fingerprinting useless. It defines the type of evidence the method can provide.
What should a detector report?
A technically responsible interface should not reduce the result to a model name and a large percentage.

At minimum, it should expose several different quantities.

1. Candidate ranking The interface should show the leading candidates, not only the winner.
2. Model A
3. Model B

4. Model C
   The ranking answers:
   Which represented candidates are most compatible with the observed behavior?

5. Relative probability
   If a softmax probability is shown, it should be explicitly labeled:
   Relative probability within the current reference set

It should not be labeled:
Probability that this is Model A

unless that interpretation has been independently validated.

1. Score margin The difference between the first and second candidates provides information about local ambiguity: [ \Delta = s_{\text{top-1}} - s_{\text{top-2}} ] A small margin suggests that the detector cannot reliably distinguish the leading candidates.

A large margin shows that the top candidate dominates nearby alternatives, but it still does not solve the unknown-model problem.

1. Absolute compatibility A detector also needs a measure of how well the observation fits the expected distribution of the winning candidate.

Possible quantities include:

• distance to the candidate distribution;
• local reference density;
• likelihood under a fitted class-conditional model;
• conformity score;
• distance from the nearest reference samples;
• distance from the candidate centroid after nuisance correction. Absolute compatibility asks a different question: Is this sample plausible for the predicted candidate at all?

1. Data-quality diagnostics A result may be unreliable because the response was incomplete, malformed, refused, rewritten, or inconsistent with the sampling protocol.

The interface should report:

• number of valid observations;
• parsing failures;
• out-of-range outputs;
• missing samples;
• protocol deviations;
• inference settings, where known. A precise classifier cannot recover information that was never collected correctly.
• Reference provenance Every result should be tied to:
• detector version;
• reference-bank version;
• reference collection date;
• prompt or protocol version;
• scoring configuration;
• relevant inference parameters. Without this information, the result cannot be reproduced or meaningfully compared over time. Closed-set and open-set recognition A closed-set classifier assumes that every evaluated sample belongs to one of the known classes.

For model fingerprinting, that means assuming:
[
y \in \mathcal{R}
]
This assumption is often unrealistic.

The LLM ecosystem changes quickly. New checkpoints appear, providers apply private modifications, and routing layers may combine several deployments. A detector should expect to encounter endpoints that do not correspond cleanly to any reference candidate.

An open-set system must therefore solve two problems:

1. Does the observation belong to the distribution covered by the reference bank?
2. If it does, which candidate provides the best match? This suggests a decision rule of the form: [ \hat{y} = \begin{cases} \arg\max_i s_i, & \text{if } A(x) \geq \lambda \ \text{UNKNOWN}, & \text{otherwise} \end{cases} ] where (A(x)) is an absolute acceptance statistic and (\lambda) is a threshold calibrated on independent data.

The difficult part is not adding an UNKNOWN label to the interface. The difficult part is validating when that label should be used.
Why a probability threshold is not enough
A tempting rule is:
If the top probability is below 60%, return UNKNOWN.
This is generally insufficient.

An unknown model can receive a top probability of 99% if it happens to be much closer to one reference candidate than to the others.

Conversely, a known model may receive only 45% when:

• two adjacent versions are difficult to distinguish;
• sampling parameters have changed;
• the system prompt modifies its behavior;
• the endpoint has drifted since reference collection;
• the response contains limited usable evidence. A threshold on normalized probability mostly measures competition among candidates. It does not necessarily measure whether the sample belongs to any known class.

Open-set rejection should be based on statistics whose behavior has been evaluated on both known and unknown models.
How to validate unknown-model rejection
A credible evaluation must freeze the detector before testing it.

The model identities, reference samples, feature transformations, scoring rules, calibration method, and rejection thresholds should be fixed before the test labels are revealed.

The evaluation set should include at least three groups.
Known models
These models are represented in the reference bank, but their evaluation responses must come from independent collection runs.

This measures ordinary closed-set identification.
Nearby unknown models
These may include adjacent checkpoints, fine-tuned variants, quantized deployments, or related models deliberately excluded from the reference bank.

They test the hardest boundary: whether the detector can avoid confidently assigning a near neighbor to a known identity.
Distant unknown models
These models come from families not represented in the bank.

They help measure whether the rejection mechanism recognizes clearly out-of-distribution behavior.

The evaluation should report more than Top-1 accuracy.

Useful open-set metrics include:

• true-positive rate for known models;
• false-accept rate for unknown models;
• false-reject rate for known models;
• AUROC;
• AUPRC;
• FPR at a specified true-positive rate;
• open-set classification rate;
• per-family rejection performance. The operating threshold should be chosen according to the application.

A low-stakes comparison tool may tolerate some false acceptance. A system used for provider auditing should prefer inconclusive results over unsupported identity claims.
Calibration requires independent evidence
A classifier is calibrated when predictions assigned probability (q) are correct approximately (q) of the time.

For example, among predictions reported at approximately 80%, around 80% should be correct under the defined evaluation distribution.

This property cannot be inferred from the softmax formula. It must be measured.

Common calibration metrics include:

• Expected Calibration Error;
• Brier score;
• negative log-likelihood;
• reliability diagrams;
• class-conditional calibration error. Calibration must also be evaluated under distribution shift.

A probability calibrated on one set of models, prompts, temperatures, and serving providers may not remain calibrated after those conditions change.

For black-box model detection, calibration should ideally be tested across:

• collection dates;
• API providers;
• sampling parameters;
• prompt variants;
• model families;
• deployment configurations. Otherwise, a percentage may be numerically precise but operationally misleading. Model identity may not be the right target There is a deeper question: should a black-box detector try to infer exact model identity at all?

In many real applications, the user’s concern is not the provider’s internal checkpoint hash. The concern is whether the endpoint still behaves consistently with the service that was purchased.

That changes the evaluation target.

Instead of asking:
Is this endpoint definitely Model A?

we can ask:

• Is its behavior still consistent with the Model A reference distribution?
• Has the endpoint changed significantly since the previous evaluation?
• Is it closer to a lower-cost candidate than to the advertised reference?
• Does its behavior vary across repeated requests?
• Is there evidence of mixed routing?
• Has the endpoint drifted after a provider update? These are monitoring questions rather than identity-authentication questions.

Behavioral fingerprinting is often better suited to answering them.
Detecting drift rather than declaring identity
Suppose an endpoint is evaluated every week under a frozen protocol.

Let (z_t) denote its behavioral representation at time (t). Instead of assigning a definitive identity, the system can monitor:
[
D(z_t, z_{\text{reference}})
]
and:
[
D(z_t, z_{t-1})
]
A significant increase may indicate:

• a model update;
• a routing change;
• a modified system prompt;
• different inference parameters;
• provider-side post-processing;
• degraded protocol compliance. This does not reveal the cause automatically. It does provide an observable signal that can trigger additional testing.

For API buyers, that may be more actionable than an overconfident model label.
What a professional conclusion looks like
A responsible report should use language such as:
Under the frozen prompt protocol, quality rules, detector version, and reference bank, the observed responses were most similar to Model A. Model A also showed a substantial score margin over the next represented candidate. This result supports behavioral consistency with the Model A reference distribution, but does not independently verify the endpoint’s internal deployment identity.

If the absolute fit is poor, it should say:
Model A was the highest-ranked represented candidate, but the observation showed weak compatibility with the known reference distributions. The result should be treated as inconclusive.

If related versions cannot be separated, it should say:
The responses are consistent with the Model A family, but the current evidence does not reliably distinguish between the included versions.

Scientific credibility often depends less on producing stronger conclusions and more on refusing to claim more than the evidence supports.
Conclusion
A 99% result can be mathematically correct and semantically misleading at the same time.

Closed-set probability answers a conditional ranking question:
Assuming the relevant model is represented in this reference set, which candidate is preferred?

It does not automatically answer:
What is the probability that this API truly runs the predicted model?

To support the second claim, a detector needs more than softmax:

• independent blind evaluation;
• absolute compatibility measurements;
• unknown-model testing;
• validated rejection thresholds;
• probability calibration;
• versioned reference data;
• explicit uncertainty reporting. Black-box behavioral fingerprinting can still be useful. It can reveal similarity, inconsistency, drift, and possible routing anomalies when direct deployment evidence is unavailable.

Its value depends on stating the result correctly.

The goal should not be to make the detector sound certain. The goal should be to make its uncertainty measurable.

Comments

[Launch Gate]

The softmax sum-to-1 is the structural tell: it deletes the most important answer the detector could give. "None of the above" is not a candidate you add to the reference set, it's a different output class - a rejection - and a system that can't emit it will manufacture confidence out of whatever candidates it has. Ranking tool wearing an identity tool's clothes, as you put it.

The metadata field point deserves the same treatment in pipeline code generally: the model name in the response is a provider assertion, and any system that logs it as ground truth is recording a claim as if it were a measurement. The healthy pattern is two columns - asserted and observed - with the divergence itself being the alertable signal.

The real damage happens downstream: 99.1% gets copied into an incident ticket as identity, and from there the wrong model is a fact everyone builds on.

[okenai]

Exactly. "None of the above" needs to be a separate rejection decision, not
just another candidate in the same softmax. The system should first test
whether the observation is compatible with the reference bank and only then
rank candidates within it.

I also like the asserted-versus-observed distinction. The provider model name
is a claim; the fingerprint result is a versioned observation. The useful
signal is the divergence between them.

And your downstream point is critical: once "99.1% relative weight within a
closed reference set" becomes "99.1% Model A" in an incident ticket, the
qualification disappears and the inference gets treated as ground truth.

That suggests the output schema should keep the asserted model, reference
match, closed-set probability, absolute compatibility, rejection status, and
detector/reference versions as separate fields. Thanks for making that
operational consequence explicit.