<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: okenai</title>
    <description>The latest articles on DEV Community by okenai (@okenai_1ec49d66dbe8d17916).</description>
    <link>https://dev.to/okenai_1ec49d66dbe8d17916</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4210087%2Fa6c2ae13-aa7d-43f8-a140-2a580d51f8bf.png</url>
      <title>DEV Community: okenai</title>
      <link>https://dev.to/okenai_1ec49d66dbe8d17916</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/okenai_1ec49d66dbe8d17916"/>
    <language>en</language>
    <item>
      <title>Beyond Fixed Probes: Making LLM API Evaluation Harder to Game</title>
      <dc:creator>okenai</dc:creator>
      <pubDate>Mon, 16 Nov 2026 07:42:50 +0000</pubDate>
      <link>https://dev.to/okenai_1ec49d66dbe8d17916/beyond-fixed-probes-making-llm-api-evaluation-harder-to-game-381d</link>
      <guid>https://dev.to/okenai_1ec49d66dbe8d17916/beyond-fixed-probes-making-llm-api-evaluation-harder-to-game-381d</guid>
      <description>&lt;p&gt;A model fingerprinting test can return a convincing result while missing an important failure mode:&lt;br&gt;
What if the provider knows when it is being tested?&lt;/p&gt;

&lt;p&gt;A provider could recognize a fixed fingerprinting prompt, route that request to the genuine model, and send ordinary production traffic to a cheaper or older model.&lt;/p&gt;

&lt;p&gt;The test would pass. The endpoint would still behave differently for real users.&lt;/p&gt;

&lt;p&gt;This possibility does not invalidate black-box evaluation. It defines a stronger requirement for the evaluation protocol:&lt;br&gt;
The test should be difficult to recognize, reproducible across controls, and informative when request routing changes.&lt;/p&gt;

&lt;p&gt;This question came up while working on the model evaluation workflow at Oken. Oken compares AI model and API behavior through controlled black-box tests. The next challenge is making those tests more robust against provider-side routing decisions and ordinary model drift.&lt;br&gt;
A fixed probe is easy to recognize&lt;br&gt;
A fixed fingerprinting task has several properties that make it vulnerable to pattern matching:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the prompt text stays the same;&lt;/li&gt;
&lt;li&gt;the output length is predictable;&lt;/li&gt;
&lt;li&gt;the value range is fixed;&lt;/li&gt;
&lt;li&gt;the response format is unusual;&lt;/li&gt;
&lt;li&gt;the request structure looks different from ordinary application traffic;&lt;/li&gt;
&lt;li&gt;the provider can collect and identify the test over time.
For example, a provider could detect a known integer-generation prompt and apply a special route:
Recognized fingerprint probe
    ↓
Route to the advertised model
    ↓
Return behavior consistent with the reference
Normal application requests could follow a different route:
Ordinary production request
    ↓
Route to a cheaper or older deployment
In that situation, the detector measures the provider’s response to the probe rather than the behavior most users receive in production.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A detector should therefore avoid treating one successful probe as conclusive evidence.&lt;br&gt;
Fresh content preserves the measurement target&lt;br&gt;
The first improvement is to generate fresh probe content for every run.&lt;/p&gt;

&lt;p&gt;The surface form can change while the statistical measurement target remains stable. For an integer-sequence task, this could include varying:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the exact prompt wording;&lt;/li&gt;
&lt;li&gt;sequence length;&lt;/li&gt;
&lt;li&gt;numeric range;&lt;/li&gt;
&lt;li&gt;formatting instructions;&lt;/li&gt;
&lt;li&gt;equivalent task descriptions;&lt;/li&gt;
&lt;li&gt;session order;&lt;/li&gt;
&lt;li&gt;prompt position.
The purpose is not to add randomness for its own sake. The purpose is to prevent the evaluator from relying on one recognizable string.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A useful protocol should preserve the behavioral dimensions being measured:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;marginal value preferences;&lt;/li&gt;
&lt;li&gt;positional effects;&lt;/li&gt;
&lt;li&gt;transition magnitude;&lt;/li&gt;
&lt;li&gt;repetition behavior;&lt;/li&gt;
&lt;li&gt;local sequence structure.
At the same time, it should change the details that a routing layer could memorize.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The design principle is:&lt;br&gt;
Keep the measurement objective stable while changing the observable form of the probe.&lt;/p&gt;

&lt;p&gt;Fresh content does not make a test impossible to target. It raises the cost of recognizing the test and reduces dependence on a single known prompt.&lt;br&gt;
Request shape is another routing signal&lt;br&gt;
Changing the prompt content may not be enough.&lt;/p&gt;

&lt;p&gt;Providers can route requests based on metadata and request structure, including:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;system instructions;&lt;/li&gt;
&lt;li&gt;tool definitions;&lt;/li&gt;
&lt;li&gt;streaming mode;&lt;/li&gt;
&lt;li&gt;response format;&lt;/li&gt;
&lt;li&gt;context length;&lt;/li&gt;
&lt;li&gt;requested capabilities;&lt;/li&gt;
&lt;li&gt;endpoint parameters;&lt;/li&gt;
&lt;li&gt;application-specific headers.
This suggests a paired test.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The same semantic probe can be sent using two request envelopes:&lt;br&gt;
Request A&lt;br&gt;
Minimal evaluation envelope&lt;/p&gt;

&lt;p&gt;Request B&lt;br&gt;
Production-shaped envelope with:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a normal system prompt;&lt;/li&gt;
&lt;li&gt;tool schemas;&lt;/li&gt;
&lt;li&gt;application context;&lt;/li&gt;
&lt;li&gt;streaming enabled;&lt;/li&gt;
&lt;li&gt;ordinary request metadata.
The probe content remains equivalent, but its external shape resembles a real production request.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the endpoint behaves differently across the two envelopes, that difference becomes a useful signal for investigation.&lt;/p&gt;

&lt;p&gt;A simplified result could look like this:&lt;br&gt;
Same probe content&lt;br&gt;
Minimal envelope:      matches reference model A&lt;br&gt;
Production envelope:   matches reference model B&lt;/p&gt;

&lt;p&gt;Observed divergence:   substantial&lt;br&gt;
This does not prove selective routing by itself. The request envelope can change model behavior naturally. A system prompt may alter instruction following, tool definitions may change the model’s context, and streaming may affect generation or post-processing.&lt;/p&gt;

&lt;p&gt;The correct interpretation is:&lt;br&gt;
The endpoint shows request-shape-dependent behavioral differences that are consistent with a routing or configuration effect and require further testing.&lt;/p&gt;

&lt;p&gt;That wording preserves the signal without turning it into an unsupported accusation.&lt;br&gt;
Same content, two shapes, repeated many times&lt;br&gt;
A single paired comparison is still vulnerable to noise.&lt;/p&gt;

&lt;p&gt;A stronger design repeats the same experiment across multiple fresh probes:&lt;br&gt;
Fresh probe 1:&lt;br&gt;
  minimal envelope&lt;br&gt;
  production-shaped envelope&lt;/p&gt;

&lt;p&gt;Fresh probe 2:&lt;br&gt;
  minimal envelope&lt;br&gt;
  production-shaped envelope&lt;/p&gt;

&lt;p&gt;Fresh probe 3:&lt;br&gt;
  minimal envelope&lt;br&gt;
  production-shaped envelope&lt;br&gt;
The evaluator can then measure:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;within-envelope variation;&lt;/li&gt;
&lt;li&gt;between-envelope variation;&lt;/li&gt;
&lt;li&gt;consistency across repeated pairs;&lt;/li&gt;
&lt;li&gt;distance from the trusted reference;&lt;/li&gt;
&lt;li&gt;whether the effect appears only for one prompt variant;&lt;/li&gt;
&lt;li&gt;whether the effect persists across time.
The important quantity is not simply whether two outputs differ. All independent generations differ.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The useful question is whether the difference between request shapes is larger and more stable than the normal variation expected under a trusted control.&lt;br&gt;
A same-day trusted control&lt;br&gt;
Reference-bank distance creates another interpretation problem.&lt;/p&gt;

&lt;p&gt;Suppose a tested endpoint becomes more distant from its stored reference. Several explanations are possible:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the provider substituted the model;&lt;/li&gt;
&lt;li&gt;the provider changed the inference configuration;&lt;/li&gt;
&lt;li&gt;the model itself received an update;&lt;/li&gt;
&lt;li&gt;the reference bank became stale;&lt;/li&gt;
&lt;li&gt;the test protocol changed;&lt;/li&gt;
&lt;li&gt;the endpoint is naturally variable.
A distance from an old reference bank cannot separate these explanations on its own.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One practical control is to call a trusted endpoint serving the claimed model on the same day and under the same protocol.&lt;/p&gt;

&lt;p&gt;The evaluator can then compare:&lt;br&gt;
Suspect endpoint → current observation&lt;br&gt;
Trusted endpoint  → current control&lt;br&gt;
Stored reference → historical baseline&lt;br&gt;
The useful signal becomes differential behavior:&lt;br&gt;
Difference(suspect, trusted control)&lt;br&gt;
rather than only:&lt;br&gt;
Difference(suspect, historical reference)&lt;br&gt;
The trusted endpoint does not need to be perfectly current. It needs to provide a contemporary baseline for the model under the same testing conditions.&lt;/p&gt;

&lt;p&gt;This helps distinguish provider-specific divergence from general model drift.&lt;/p&gt;

&lt;p&gt;The control itself requires provenance:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;provider identity;&lt;/li&gt;
&lt;li&gt;model assertion;&lt;/li&gt;
&lt;li&gt;test timestamp;&lt;/li&gt;
&lt;li&gt;request parameters;&lt;/li&gt;
&lt;li&gt;detector version;&lt;/li&gt;
&lt;li&gt;reference-bank version;&lt;/li&gt;
&lt;li&gt;response hashes.
Without that information, the control becomes another unverified assumption.
Test-retest spread is evidence
A detector should report how stable the same endpoint is across repeated runs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example, the same endpoint could be evaluated twice within a short interval:&lt;br&gt;
Run 1: closest to Model A&lt;br&gt;
Run 2: closest to Model A&lt;br&gt;
Run 3: closest to Model B&lt;br&gt;
This may indicate:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;unstable routing;&lt;/li&gt;
&lt;li&gt;high sampling variance;&lt;/li&gt;
&lt;li&gt;insufficient probe length;&lt;/li&gt;
&lt;li&gt;inconsistent provider configuration;&lt;/li&gt;
&lt;li&gt;an incomplete reference bank;&lt;/li&gt;
&lt;li&gt;a mixed deployment.
The spread itself is informative.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A result that changes dramatically within minutes should be treated differently from a result that remains stable across independent runs.&lt;/p&gt;

&lt;p&gt;A useful report should therefore include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;repeated candidate rankings;&lt;/li&gt;
&lt;li&gt;score-margin variation;&lt;/li&gt;
&lt;li&gt;distance variation;&lt;/li&gt;
&lt;li&gt;compatibility variation;&lt;/li&gt;
&lt;li&gt;response-quality variation;&lt;/li&gt;
&lt;li&gt;time between runs.
This is more informative than showing only the strongest result from a single run.
The evaluator should distinguish three sources of change
A practical monitoring system needs to separate at least three kinds of change.
Provider-specific divergence
The tested endpoint changes while the trusted same-day control remains stable.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is more consistent with a provider-side routing or configuration difference.&lt;br&gt;
Shared model drift&lt;br&gt;
Both the tested endpoint and the trusted control move in a similar direction relative to the historical reference.&lt;/p&gt;

&lt;p&gt;This is more consistent with a model update or a change shared by both deployments.&lt;br&gt;
Measurement instability&lt;br&gt;
The tested endpoint and the control both show large run-to-run variation under the same protocol.&lt;/p&gt;

&lt;p&gt;This suggests that the probe, sampling process, or test configuration may not be stable enough for a strong conclusion.&lt;/p&gt;

&lt;p&gt;These categories are not automatically identifiable from one run. They require repeated measurements and controls.&lt;br&gt;
What Oken is designed to help with&lt;br&gt;
Oken’s model evaluation workflow provides a practical way to compare black-box endpoints with reference model behavior.&lt;/p&gt;

&lt;p&gt;The workflow can be used to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;compare API providers;&lt;/li&gt;
&lt;li&gt;inspect whether two endpoints with the same model name behave similarly;&lt;/li&gt;
&lt;li&gt;establish a behavioral baseline;&lt;/li&gt;
&lt;li&gt;investigate possible model-version drift;&lt;/li&gt;
&lt;li&gt;repeat tests over time;&lt;/li&gt;
&lt;li&gt;examine candidate rankings and quality diagnostics.
The model fingerprint result should be treated as a behavioral measurement tied to a specific protocol and reference-bank version.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It can help identify an endpoint that deserves further investigation. It does not independently reveal the provider’s internal routing configuration.&lt;/p&gt;

&lt;p&gt;You can explore the evaluation workflow at &lt;a href="https://www.oken.ai/zh" rel="noopener noreferrer"&gt;https://www.oken.ai/zh&lt;/a&gt;.&lt;br&gt;
A more complete result record&lt;br&gt;
A production-oriented evaluator should preserve enough information to reproduce and audit the result.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
{&lt;br&gt;
  "provider_asserted_model": "model-a",&lt;br&gt;
  "top_reference_match": "model-a",&lt;br&gt;
  "closed_set_probability": 0.94,&lt;br&gt;
  "absolute_compatibility": 0.81,&lt;br&gt;
  "request_shape": "production_like",&lt;br&gt;
  "trusted_control_status": "passed",&lt;br&gt;
  "test_retest_spread": 0.07,&lt;br&gt;
  "detector_version": "detector-1.2.0",&lt;br&gt;
  "reference_bank_version": "bank-2026-11-01",&lt;br&gt;
  "calibration_version": "calibration-2026-11-02",&lt;br&gt;
  "decision": "behaviorally_consistent"&lt;br&gt;
}&lt;br&gt;
For a suspicious result:&lt;br&gt;
{&lt;br&gt;
  "provider_asserted_model": "model-a",&lt;br&gt;
  "top_reference_match": "model-b",&lt;br&gt;
  "closed_set_probability": 0.88,&lt;br&gt;
  "absolute_compatibility": 0.42,&lt;br&gt;
  "request_shape": "production_like",&lt;br&gt;
  "trusted_control_status": "passed",&lt;br&gt;
  "test_retest_spread": 0.31,&lt;br&gt;
  "decision": "requires_investigation"&lt;br&gt;
}&lt;br&gt;
The output should preserve the difference between:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the model the provider claims to serve;&lt;/li&gt;
&lt;li&gt;the candidate that ranked first;&lt;/li&gt;
&lt;li&gt;the absolute compatibility of the sample;&lt;/li&gt;
&lt;li&gt;the stability of the result;&lt;/li&gt;
&lt;li&gt;the behavior under different request shapes;&lt;/li&gt;
&lt;li&gt;the version of the evidence used to reach the decision.
What would count as stronger evidence?
A responsible evaluation should avoid treating one divergence as proof of selective routing.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Stronger evidence would require a repeated pattern:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;fresh probes produce the same result;&lt;/li&gt;
&lt;li&gt;the effect appears across multiple request variants;&lt;/li&gt;
&lt;li&gt;production-shaped requests behave differently from minimal requests;&lt;/li&gt;
&lt;li&gt;the trusted control remains stable;&lt;/li&gt;
&lt;li&gt;the difference exceeds normal test-retest variation;&lt;/li&gt;
&lt;li&gt;the pattern persists across time;&lt;/li&gt;
&lt;li&gt;the result is consistent across multiple API keys or sessions;&lt;/li&gt;
&lt;li&gt;ordinary production traffic shows related behavior.
Even then, the conclusion should describe the observation rather than claim access to the provider’s internal routing logic.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A suitable report might say:&lt;br&gt;
Across repeated fresh probes, the endpoint showed a stable request-shape-dependent behavioral difference relative to a same-day trusted control. The pattern is consistent with routing or configuration differences and warrants further investigation.&lt;/p&gt;

&lt;p&gt;This is a stronger and more defensible statement than:&lt;br&gt;
The provider is definitely routing test requests to the real model.&lt;/p&gt;

&lt;p&gt;Conclusion&lt;br&gt;
A fixed probe can be measured, recognized, and targeted.&lt;/p&gt;

&lt;p&gt;A stronger black-box evaluation protocol should use fresh probe content, multiple request shapes, same-day controls, and repeated runs. It should measure test-retest spread and preserve the versions and provenance required to interpret the result.&lt;/p&gt;

&lt;p&gt;The objective is not to make the test impossible to game. No black-box test can guarantee that against a provider with complete control of the endpoint.&lt;/p&gt;

&lt;p&gt;The objective is to make selective behavior more difficult to hide and easier to detect.&lt;/p&gt;

&lt;p&gt;For developers comparing AI API providers, behavioral fingerprinting can provide a useful external signal when deployment logs and routing records are unavailable. Oken provides a practical workflow for running these comparisons and tracking provider-level differences over time.&lt;/p&gt;

&lt;p&gt;The result should be read as evidence about observed behavior under a defined protocol. Its strength comes from fresh probes, controls, repeated measurements, and transparent uncertainty.&lt;/p&gt;

&lt;p&gt;Disclosure: Oken is the platform discussed in this article. Its evaluation workflow reports behavioral similarity against reference samples and should not be interpreted as cryptographic proof of the server-side model identity.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>api</category>
      <category>llm</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>How to Verify an AI API Beyond Its Model Name: Behavioral Fingerprinting with Oken</title>
      <dc:creator>okenai</dc:creator>
      <pubDate>Fri, 13 Nov 2026 07:26:47 +0000</pubDate>
      <link>https://dev.to/okenai_1ec49d66dbe8d17916/how-to-verify-an-ai-api-beyond-its-model-name-behavioral-fingerprinting-with-oken-4ci</link>
      <guid>https://dev.to/okenai_1ec49d66dbe8d17916/how-to-verify-an-ai-api-beyond-its-model-name-behavioral-fingerprinting-with-oken-4ci</guid>
      <description>&lt;p&gt;When you call an AI model through an API provider, the model name in the request is usually just a claim.&lt;/p&gt;

&lt;p&gt;The request may specify:&lt;/p&gt;

&lt;p&gt;But that field does not independently prove which deployment actually processed the request.&lt;br&gt;
{&lt;br&gt;
  "model": "model-a"&lt;br&gt;
}&lt;br&gt;
The endpoint could be using:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a different model version;&lt;/li&gt;
&lt;li&gt;an older checkpoint;&lt;/li&gt;
&lt;li&gt;a lower-cost model;&lt;/li&gt;
&lt;li&gt;a mixed routing strategy;&lt;/li&gt;
&lt;li&gt;different inference settings;&lt;/li&gt;
&lt;li&gt;provider-side post-processing.
This matters because model names often determine both pricing and expected capability. If an API is advertised as a flagship model but behaves more like a smaller or older model, the difference is important for both developers and organizations.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Oken is built to make this kind of provider comparison easier. In addition to comparing model and API pricing, Oken includes a black-box behavioral fingerprinting workflow that compares an endpoint’s output behavior with a versioned reference bank.&lt;/p&gt;

&lt;p&gt;The goal is not to provide cryptographic proof of model identity. The goal is to give API users a reproducible signal when an endpoint appears inconsistent with the model it claims to serve.&lt;br&gt;
Why model names are not enough&lt;br&gt;
Most API users evaluate a provider using a combination of:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;input token price;&lt;/li&gt;
&lt;li&gt;output token price;&lt;/li&gt;
&lt;li&gt;cached input pricing;&lt;/li&gt;
&lt;li&gt;context limits;&lt;/li&gt;
&lt;li&gt;rate limits;&lt;/li&gt;
&lt;li&gt;latency;&lt;/li&gt;
&lt;li&gt;benchmark performance;&lt;/li&gt;
&lt;li&gt;model availability.
These are all useful, but they do not fully answer a basic question:
Is the endpoint actually behaving like the model being advertised?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Traditional capability tests are not always sufficient.&lt;/p&gt;

&lt;p&gt;A model may perform well on one benchmark and poorly on another. Results may also depend on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;system prompts;&lt;/li&gt;
&lt;li&gt;reasoning settings;&lt;/li&gt;
&lt;li&gt;temperature;&lt;/li&gt;
&lt;li&gt;retrieval;&lt;/li&gt;
&lt;li&gt;tool use;&lt;/li&gt;
&lt;li&gt;prompt formatting;&lt;/li&gt;
&lt;li&gt;benchmark contamination;&lt;/li&gt;
&lt;li&gt;output post-processing.
A few correct answers cannot establish model identity. Different models can solve the same task correctly, while the same model can produce different results under different configurations.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is why behavioral comparison can be useful as a complementary evaluation method.&lt;br&gt;
What behavioral fingerprinting measures&lt;br&gt;
Oken’s evaluation workflow uses controlled, semantically minimal generation tasks.&lt;/p&gt;

&lt;p&gt;One example is an integer-generation task in which the endpoint is asked to produce a long sequence of bounded integers. The task has no meaningful factual answer. The model is instructed to choose each value independently and avoid obvious arithmetic or repeating patterns.&lt;/p&gt;

&lt;p&gt;The purpose is not to test mathematical ability. It is to reduce the influence of:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;domain knowledge;&lt;/li&gt;
&lt;li&gt;factual recall;&lt;/li&gt;
&lt;li&gt;writing style;&lt;/li&gt;
&lt;li&gt;reasoning quality;&lt;/li&gt;
&lt;li&gt;retrieval;&lt;/li&gt;
&lt;li&gt;subjective grading.
Even when a model is asked to produce a random-looking sequence, it does not become a perfect random-number generator.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Its output can still reflect:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;training data;&lt;/li&gt;
&lt;li&gt;tokenizer behavior;&lt;/li&gt;
&lt;li&gt;alignment;&lt;/li&gt;
&lt;li&gt;decoding strategy;&lt;/li&gt;
&lt;li&gt;inference configuration;&lt;/li&gt;
&lt;li&gt;hidden prompt instructions;&lt;/li&gt;
&lt;li&gt;model-specific generation habits.
A single number is not useful for identification. A long sequence provides many observations that can be analyzed together.
The features behind the comparison
The detector does not simply count which integer appears most often.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It compares several types of behavioral features.&lt;br&gt;
Number preferences&lt;br&gt;
Does the endpoint favor certain values, ranges, or final digits?&lt;/p&gt;

&lt;p&gt;The system can measure whether the distribution is relatively uniform, skewed, clustered, or unusually sparse.&lt;br&gt;
Positional behavior&lt;br&gt;
Does the distribution change between the beginning, middle, and end of the response?&lt;/p&gt;

&lt;p&gt;Some endpoints may show position-dependent effects, such as early stabilization, late drift, or changing final-digit preferences.&lt;br&gt;
Transition magnitude&lt;br&gt;
How large are the changes between adjacent or lagged values?&lt;/p&gt;

&lt;p&gt;The detector can measure:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;small and large jumps;&lt;/li&gt;
&lt;li&gt;upward and downward movement;&lt;/li&gt;
&lt;li&gt;repeated values;&lt;/li&gt;
&lt;li&gt;short-term correlation;&lt;/li&gt;
&lt;li&gt;center-seeking behavior;&lt;/li&gt;
&lt;li&gt;unusually smooth or unusually volatile transitions.
Repetition patterns
Repetition does not only mean two identical values appearing next to each other.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The system can also examine:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;nearby-value repetition;&lt;/li&gt;
&lt;li&gt;final-digit repetition;&lt;/li&gt;
&lt;li&gt;repeated movement direction;&lt;/li&gt;
&lt;li&gt;repeated step sizes;&lt;/li&gt;
&lt;li&gt;local diversity within a window.
A model may try to avoid repetition because repetition does not “look random.” That avoidance itself can become a measurable statistical pattern.
Local sequence structure
Two endpoints can have similar overall frequency distributions but different ordering behavior.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example, one model may generate values that are approximately independent, while another may repeatedly move between high and low ranges.&lt;/p&gt;

&lt;p&gt;Transition statistics can reveal patterns that are invisible in a simple histogram.&lt;br&gt;
Why the workflow uses multiple samples&lt;br&gt;
A single response can be affected by sampling noise.&lt;/p&gt;

&lt;p&gt;For that reason, the evaluation workflow uses multiple independent sessions. Each session receives its own task, and the first response is retained without manually repairing or extending it.&lt;/p&gt;

&lt;p&gt;The system also performs quality checks, including:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;valid output count;&lt;/li&gt;
&lt;li&gt;value range;&lt;/li&gt;
&lt;li&gt;parsing errors;&lt;/li&gt;
&lt;li&gt;missing responses;&lt;/li&gt;
&lt;li&gt;malformed content;&lt;/li&gt;
&lt;li&gt;protocol violations.
These checks matter because a detector cannot recover information that was never collected correctly.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Passing the quality check does not mean that the model followed every instruction perfectly. It means that enough usable data exists for comparison.&lt;br&gt;
How Oken reports the result&lt;br&gt;
The collected responses are compared with reference samples from known model endpoints.&lt;/p&gt;

&lt;p&gt;The result may include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Top-1 candidate;&lt;/li&gt;
&lt;li&gt;Top-2 and additional candidates;&lt;/li&gt;
&lt;li&gt;internal score;&lt;/li&gt;
&lt;li&gt;relative probability within the reference bank;&lt;/li&gt;
&lt;li&gt;score margin;&lt;/li&gt;
&lt;li&gt;data-quality diagnostics;&lt;/li&gt;
&lt;li&gt;detector and reference-bank versions.
A result might look like:
Top-1: Model A
Top-2: Model B
Relative probability within reference bank: 94%
This should be interpreted carefully.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The result means:&lt;br&gt;
Under the current protocol and reference bank, the observed behavior is most similar to Model A.&lt;/p&gt;

&lt;p&gt;It does not mean:&lt;br&gt;
The endpoint has been cryptographically verified as Model A.&lt;/p&gt;

&lt;p&gt;The probability is a closed-set relative probability. It is normalized over the candidates represented in the reference bank.&lt;/p&gt;

&lt;p&gt;If the actual model is not included in that bank, the system may still assign a high relative probability to the closest known candidate.&lt;/p&gt;

&lt;p&gt;That is a limitation of closed-set classification, not a reason to hide the limitation.&lt;br&gt;
How to run a check&lt;br&gt;
You can try the workflow through the &lt;a href="https://www.oken.ai/zh/evaluation-lab?from=header" rel="noopener noreferrer"&gt;https://www.oken.ai/zh/evaluation-lab?from=header&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The basic process is:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Open the model evaluation workflow.&lt;/li&gt;
&lt;li&gt;Enter the compatible API endpoint.&lt;/li&gt;
&lt;li&gt;Provide the required API key.&lt;/li&gt;
&lt;li&gt;Select the model the provider claims to serve.&lt;/li&gt;
&lt;li&gt;Run the independent sampling process.&lt;/li&gt;
&lt;li&gt;Review the candidate ranking and quality diagnostics.&lt;/li&gt;
&lt;li&gt;Compare the observed behavior with the provider’s model assertion.
For repeated monitoring, save the following with each result:&lt;/li&gt;
&lt;li&gt;test date;&lt;/li&gt;
&lt;li&gt;detector version;&lt;/li&gt;
&lt;li&gt;reference-bank version;&lt;/li&gt;
&lt;li&gt;prompt protocol;&lt;/li&gt;
&lt;li&gt;inference parameters;&lt;/li&gt;
&lt;li&gt;raw response hashes, where available;&lt;/li&gt;
&lt;li&gt;complete candidate ranking.
This allows later results to be compared against a known evaluation state.
When this is useful
Behavioral fingerprinting can be useful when you are:&lt;/li&gt;
&lt;li&gt;comparing AI API aggregators;&lt;/li&gt;
&lt;li&gt;evaluating a relay or proxy endpoint;&lt;/li&gt;
&lt;li&gt;investigating a possible model downgrade;&lt;/li&gt;
&lt;li&gt;monitoring a provider after a silent update;&lt;/li&gt;
&lt;li&gt;comparing the same model name across providers;&lt;/li&gt;
&lt;li&gt;checking whether an endpoint has drifted over time;&lt;/li&gt;
&lt;li&gt;establishing a behavioral baseline before production use.
For example, suppose a provider advertises a flagship model, but repeated tests consistently place the endpoint closer to a smaller or older reference model.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;That does not automatically prove why the difference exists. Possible explanations include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;model substitution;&lt;/li&gt;
&lt;li&gt;routing changes;&lt;/li&gt;
&lt;li&gt;different inference parameters;&lt;/li&gt;
&lt;li&gt;system prompt differences;&lt;/li&gt;
&lt;li&gt;model-version drift;&lt;/li&gt;
&lt;li&gt;output transformation;&lt;/li&gt;
&lt;li&gt;an incomplete reference library.
But it does provide a reproducible signal that justifies further investigation.
What the test cannot prove
The method has clear boundaries.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It cannot independently prove:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the exact server-side checkpoint;&lt;/li&gt;
&lt;li&gt;the provider’s internal routing configuration;&lt;/li&gt;
&lt;li&gt;that no middleware rewrites the response;&lt;/li&gt;
&lt;li&gt;that the model version will remain unchanged;&lt;/li&gt;
&lt;li&gt;that an unknown model will not resemble a known candidate;&lt;/li&gt;
&lt;li&gt;that a high relative probability equals real-world identity accuracy.
It also cannot replace:&lt;/li&gt;
&lt;li&gt;provider deployment logs;&lt;/li&gt;
&lt;li&gt;signed model metadata;&lt;/li&gt;
&lt;li&gt;trusted execution evidence;&lt;/li&gt;
&lt;li&gt;contractual audits;&lt;/li&gt;
&lt;li&gt;cryptographic attestation.
The appropriate role is external behavioral evaluation.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It provides evidence about consistency, similarity, and drift when direct server-side evidence is unavailable.&lt;br&gt;
Why the limitations are part of the product&lt;br&gt;
A useful evaluation tool should not only produce a model name. It should also show when the evidence is weak.&lt;/p&gt;

&lt;p&gt;In a future version, a more complete decision record might look like:&lt;br&gt;
{&lt;br&gt;
  "provider_asserted_model": "model-a",&lt;br&gt;
  "top_reference_match": "model-a",&lt;br&gt;
  "closed_set_probability": 0.94,&lt;br&gt;
  "absolute_compatibility": 0.31,&lt;br&gt;
  "decision": "UNKNOWN",&lt;br&gt;
  "detector_version": "detector-1.2.0",&lt;br&gt;
  "reference_bank_version": "bank-2026-11-01"&lt;br&gt;
}&lt;br&gt;
This separates four different concepts:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;what the provider claims;&lt;/li&gt;
&lt;li&gt;which known candidate ranked first;&lt;/li&gt;
&lt;li&gt;how strong the relative ranking was;&lt;/li&gt;
&lt;li&gt;whether the sample was compatible enough to accept.
That distinction is important because a model detector should be allowed to return UNKNOWN.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A high-ranking candidate is not always a valid attribution.&lt;br&gt;
Conclusion&lt;br&gt;
The model name in an API request is useful metadata, but it should not be treated as independently verified identity evidence.&lt;/p&gt;

&lt;p&gt;Behavioral fingerprinting offers another way to evaluate an endpoint. By collecting controlled outputs and comparing their statistical behavior with a versioned reference bank, users can investigate possible model substitutions, routing changes, and endpoint drift.&lt;/p&gt;

&lt;p&gt;The result should be interpreted as behavioral consistency, not cryptographic authentication.&lt;/p&gt;

&lt;p&gt;If you are comparing AI API providers or want to check whether an endpoint behaves consistently with the model it advertises, you can explore the evaluation workflow at:&lt;/p&gt;

&lt;p&gt;Oken AI&lt;/p&gt;

&lt;p&gt;Disclosure: Oken is the platform discussed in this article. The fingerprinting workflow reports behavioral similarity against a reference bank and should not be interpreted as definitive proof of the server-side model identity.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>api</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>Your LLM Detector Is 99% Confident. It May Still Be Wrong</title>
      <dc:creator>okenai</dc:creator>
      <pubDate>Wed, 11 Nov 2026 08:27:52 +0000</pubDate>
      <link>https://dev.to/okenai_1ec49d66dbe8d17916/your-llm-detector-is-99-confident-it-may-still-be-wrong-49co</link>
      <guid>https://dev.to/okenai_1ec49d66dbe8d17916/your-llm-detector-is-99-confident-it-may-still-be-wrong-49co</guid>
      <description>&lt;p&gt;A black-box model detector returns the following result:&lt;br&gt;
Model A: 99.1%&lt;br&gt;
Model B: 0.6%&lt;br&gt;
Model C: 0.3%&lt;br&gt;
The obvious interpretation is that the endpoint has a 99.1% probability of running Model A.&lt;/p&gt;

&lt;p&gt;In many systems, that interpretation is wrong.&lt;/p&gt;

&lt;p&gt;The number may only mean that Model A received 99.1% of the normalized probability mass among the candidates included in the detector’s reference set. If the actual model is absent from that set, the detector can remain highly confident while assigning the endpoint to the wrong model.&lt;/p&gt;

&lt;p&gt;This is not a minor wording issue. It is the difference between:&lt;br&gt;
“Model A is the closest candidate we know.”&lt;/p&gt;

&lt;p&gt;and:&lt;br&gt;
“The endpoint has been verified as Model A.”&lt;/p&gt;

&lt;p&gt;The first is a ranking result. The second is an identity claim.&lt;/p&gt;

&lt;p&gt;Any serious attempt to identify models behind black-box APIs must keep those claims separate.&lt;br&gt;
The practical problem&lt;br&gt;
When developers call an LLM through an API provider, the model name is usually supplied as metadata:&lt;br&gt;
{&lt;br&gt;
  "model": "model-a"&lt;br&gt;
}&lt;br&gt;
That field is a provider assertion. It is not independently verifiable evidence of the model that processed the request.&lt;/p&gt;

&lt;p&gt;Several operational situations can create a difference between the advertised model and the observed endpoint:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;traffic may be routed across multiple model versions;&lt;/li&gt;
&lt;li&gt;a provider may silently update the underlying deployment;&lt;/li&gt;
&lt;li&gt;a smaller model may handle part of the traffic;&lt;/li&gt;
&lt;li&gt;a proxy may rewrite model identifiers;&lt;/li&gt;
&lt;li&gt;inference parameters may differ from the official endpoint;&lt;/li&gt;
&lt;li&gt;a model family may contain several behaviorally similar checkpoints.
Black-box fingerprinting attempts to investigate this problem by measuring output behavior rather than trusting the model label.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A detector may send controlled prompts, extract statistical features from the responses, and compare them with samples collected from known reference endpoints.&lt;/p&gt;

&lt;p&gt;That can produce useful evidence. But the meaning of the resulting probability depends on the statistical problem the detector was designed to solve.&lt;br&gt;
Ranking is not authentication&lt;br&gt;
Assume a detector has a reference set containing (K) known model candidates:&lt;br&gt;
[&lt;br&gt;
\mathcal{R} = {R_1, R_2, \ldots, R_K}&lt;br&gt;
]&lt;br&gt;
For an observed response (x), the detector extracts a feature vector:&lt;br&gt;
[&lt;br&gt;
z = f(x)&lt;br&gt;
]&lt;br&gt;
It then assigns each candidate a similarity or compatibility score:&lt;br&gt;
[&lt;br&gt;
s_i = S(z, R_i)&lt;br&gt;
]&lt;br&gt;
A common final step is to convert those scores into a normalized distribution using softmax:&lt;br&gt;
[&lt;br&gt;
p_i =&lt;br&gt;
\frac{\exp(\tau s_i)}&lt;br&gt;
{\sum_{j=1}^{K}\exp(\tau s_j)}&lt;br&gt;
]&lt;br&gt;
where (\tau) controls how sharply the probability mass concentrates around the highest-scoring candidates.&lt;/p&gt;

&lt;p&gt;The output satisfies:&lt;br&gt;
[&lt;br&gt;
\sum_{i=1}^{K} p_i = 1&lt;br&gt;
]&lt;br&gt;
This mathematical property is convenient for ranking candidates. It also creates a dangerous interpretation problem.&lt;/p&gt;

&lt;p&gt;The softmax must distribute all probability mass among the available candidates. It has no inherent mechanism for saying:&lt;br&gt;
None of the above.&lt;/p&gt;

&lt;p&gt;Even if every reference model is a poor match, one candidate can still receive most of the normalized probability.&lt;br&gt;
A simple example&lt;br&gt;
Imagine a detector whose reference set contains only three model families:&lt;br&gt;
Claude&lt;br&gt;
GPT&lt;br&gt;
Gemini&lt;br&gt;
Now suppose the tested endpoint is powered by an unrelated model that has never appeared in the detector’s reference data.&lt;/p&gt;

&lt;p&gt;The detector may still return:&lt;br&gt;
Claude: 97%&lt;br&gt;
GPT: 2%&lt;br&gt;
Gemini: 1%&lt;br&gt;
This result does not imply a 97% probability that the endpoint is Claude.&lt;/p&gt;

&lt;p&gt;It means that, after comparing the sample only with Claude, GPT, and Gemini references, Claude received the strongest relative score.&lt;/p&gt;

&lt;p&gt;The more precise interpretation is:&lt;br&gt;
Conditional on selecting one of the candidates represented in this reference set, the observed behavior is most compatible with the Claude reference distribution.&lt;/p&gt;

&lt;p&gt;The missing condition matters:&lt;br&gt;
[&lt;br&gt;
P(R_i \mid x,\; y \in \mathcal{R})&lt;br&gt;
\neq&lt;br&gt;
P(y = R_i \mid x)&lt;br&gt;
]&lt;br&gt;
The expression on the left is a closed-set classification result. The expression on the right is a claim about the model’s real identity.&lt;/p&gt;

&lt;p&gt;They are not equivalent.&lt;br&gt;
Why softmax can be confidently wrong&lt;br&gt;
Softmax does not measure whether the best candidate is a good match in absolute terms. It measures how the candidate scores compare with one another.&lt;/p&gt;

&lt;p&gt;Consider two score vectors:&lt;br&gt;
Case 1: [10.0, 2.0, 1.0]&lt;br&gt;
Case 2: [-90.0, -98.0, -99.0]&lt;br&gt;
In both cases, the first candidate leads the second by eight score units. Depending on the temperature, both can produce nearly identical normalized probabilities.&lt;/p&gt;

&lt;p&gt;But the cases may have very different meanings.&lt;/p&gt;

&lt;p&gt;In the first case, the best candidate may genuinely match the observed sample. In the second, every candidate may be far outside its expected distribution. Softmax alone does not preserve that distinction.&lt;/p&gt;

&lt;p&gt;This creates several failure modes.&lt;br&gt;
The true model is absent&lt;br&gt;
A model outside the reference set is assigned to the least incompatible known candidate.&lt;br&gt;
The reference set changes&lt;br&gt;
Adding or removing candidates changes the denominator of the softmax. The reported probability can therefore change even when the observed response and the best candidate’s raw score remain unchanged.&lt;br&gt;
The temperature changes&lt;br&gt;
A calibration temperature can make an output appear more or less decisive without changing the candidate ranking.&lt;/p&gt;

&lt;p&gt;For scores (s_i), increasing (\tau) generally produces a sharper distribution. A visually impressive 99% may partly reflect calibration choices rather than stronger evidence.&lt;br&gt;
All candidates are poor matches&lt;br&gt;
Relative normalization can conceal poor absolute compatibility. The winner can dominate simply because every alternative is worse.&lt;br&gt;
Closely related models share features&lt;br&gt;
Models from the same family may be statistically difficult to separate. A high aggregate family match does not automatically support exact checkpoint attribution.&lt;br&gt;
A concrete implementation&lt;br&gt;
I encountered this interpretation problem while working on the behavioral model-fingerprinting system used in Oken’s evaluation lab.&lt;/p&gt;

&lt;p&gt;The implementation uses constrained, semantically minimal generation tasks to collect behavioral samples from a black-box endpoint. It then compares properties such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;marginal output preferences;&lt;/li&gt;
&lt;li&gt;position-dependent behavior;&lt;/li&gt;
&lt;li&gt;transition magnitudes;&lt;/li&gt;
&lt;li&gt;repetition patterns;&lt;/li&gt;
&lt;li&gt;local sequence structure.
These observations are compared with a frozen reference bank to produce a candidate ranking.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is useful for asking:&lt;br&gt;
Which reference model behaves most similarly to this endpoint under the current evaluation protocol?&lt;/p&gt;

&lt;p&gt;It does not, by itself, answer:&lt;br&gt;
Which model was definitively executed on the provider’s server?&lt;/p&gt;

&lt;p&gt;The distinction became especially important when presenting the final normalized probability. A concentrated closed-set result may be meaningful as evidence of relative similarity, but it should not be labeled as an independently verified probability of model identity.&lt;/p&gt;

&lt;p&gt;That limitation does not make behavioral fingerprinting useless. It defines the type of evidence the method can provide.&lt;br&gt;
What should a detector report?&lt;br&gt;
A technically responsible interface should not reduce the result to a model name and a large percentage.&lt;/p&gt;

&lt;p&gt;At minimum, it should expose several different quantities.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Candidate ranking
The interface should show the leading candidates, not only the winner.&lt;/li&gt;
&lt;li&gt;Model A&lt;/li&gt;
&lt;li&gt;Model B&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Model C&lt;br&gt;
The ranking answers:&lt;br&gt;
Which represented candidates are most compatible with the observed behavior?&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Relative probability&lt;br&gt;
If a softmax probability is shown, it should be explicitly labeled:&lt;br&gt;
Relative probability within the current reference set&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;It should not be labeled:&lt;br&gt;
Probability that this is Model A&lt;/p&gt;

&lt;p&gt;unless that interpretation has been independently validated.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Score margin
The difference between the first and second candidates provides information about local ambiguity:
[
\Delta = s_{\text{top-1}} - s_{\text{top-2}}
]
A small margin suggests that the detector cannot reliably distinguish the leading candidates.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A large margin shows that the top candidate dominates nearby alternatives, but it still does not solve the unknown-model problem.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Absolute compatibility
A detector also needs a measure of how well the observation fits the expected distribution of the winning candidate.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Possible quantities include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;distance to the candidate distribution;&lt;/li&gt;
&lt;li&gt;local reference density;&lt;/li&gt;
&lt;li&gt;likelihood under a fitted class-conditional model;&lt;/li&gt;
&lt;li&gt;conformity score;&lt;/li&gt;
&lt;li&gt;distance from the nearest reference samples;&lt;/li&gt;
&lt;li&gt;distance from the candidate centroid after nuisance correction.
Absolute compatibility asks a different question:
Is this sample plausible for the predicted candidate at all?&lt;/li&gt;
&lt;/ul&gt;

&lt;ol&gt;
&lt;li&gt;Data-quality diagnostics
A result may be unreliable because the response was incomplete, malformed, refused, rewritten, or inconsistent with the sampling protocol.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The interface should report:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;number of valid observations;&lt;/li&gt;
&lt;li&gt;parsing failures;&lt;/li&gt;
&lt;li&gt;out-of-range outputs;&lt;/li&gt;
&lt;li&gt;missing samples;&lt;/li&gt;
&lt;li&gt;protocol deviations;&lt;/li&gt;
&lt;li&gt;inference settings, where known.
A precise classifier cannot recover information that was never collected correctly.&lt;/li&gt;
&lt;li&gt;Reference provenance
Every result should be tied to:&lt;/li&gt;
&lt;li&gt;detector version;&lt;/li&gt;
&lt;li&gt;reference-bank version;&lt;/li&gt;
&lt;li&gt;reference collection date;&lt;/li&gt;
&lt;li&gt;prompt or protocol version;&lt;/li&gt;
&lt;li&gt;scoring configuration;&lt;/li&gt;
&lt;li&gt;relevant inference parameters.
Without this information, the result cannot be reproduced or meaningfully compared over time.
Closed-set and open-set recognition
A closed-set classifier assumes that every evaluated sample belongs to one of the known classes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For model fingerprinting, that means assuming:&lt;br&gt;
[&lt;br&gt;
y \in \mathcal{R}&lt;br&gt;
]&lt;br&gt;
This assumption is often unrealistic.&lt;/p&gt;

&lt;p&gt;The LLM ecosystem changes quickly. New checkpoints appear, providers apply private modifications, and routing layers may combine several deployments. A detector should expect to encounter endpoints that do not correspond cleanly to any reference candidate.&lt;/p&gt;

&lt;p&gt;An open-set system must therefore solve two problems:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Does the observation belong to the distribution covered by the reference bank?&lt;/li&gt;
&lt;li&gt;If it does, which candidate provides the best match?
This suggests a decision rule of the form:
[
\hat{y} =
\begin{cases}
\arg\max_i s_i, &amp;amp; \text{if } A(x) \geq \lambda \
\text{UNKNOWN}, &amp;amp; \text{otherwise}
\end{cases}
]
where (A(x)) is an absolute acceptance statistic and (\lambda) is a threshold calibrated on independent data.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The difficult part is not adding an UNKNOWN label to the interface. The difficult part is validating when that label should be used.&lt;br&gt;
Why a probability threshold is not enough&lt;br&gt;
A tempting rule is:&lt;br&gt;
If the top probability is below 60%, return UNKNOWN.&lt;br&gt;
This is generally insufficient.&lt;/p&gt;

&lt;p&gt;An unknown model can receive a top probability of 99% if it happens to be much closer to one reference candidate than to the others.&lt;/p&gt;

&lt;p&gt;Conversely, a known model may receive only 45% when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;two adjacent versions are difficult to distinguish;&lt;/li&gt;
&lt;li&gt;sampling parameters have changed;&lt;/li&gt;
&lt;li&gt;the system prompt modifies its behavior;&lt;/li&gt;
&lt;li&gt;the endpoint has drifted since reference collection;&lt;/li&gt;
&lt;li&gt;the response contains limited usable evidence.
A threshold on normalized probability mostly measures competition among candidates. It does not necessarily measure whether the sample belongs to any known class.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Open-set rejection should be based on statistics whose behavior has been evaluated on both known and unknown models.&lt;br&gt;
How to validate unknown-model rejection&lt;br&gt;
A credible evaluation must freeze the detector before testing it.&lt;/p&gt;

&lt;p&gt;The model identities, reference samples, feature transformations, scoring rules, calibration method, and rejection thresholds should be fixed before the test labels are revealed.&lt;/p&gt;

&lt;p&gt;The evaluation set should include at least three groups.&lt;br&gt;
Known models&lt;br&gt;
These models are represented in the reference bank, but their evaluation responses must come from independent collection runs.&lt;/p&gt;

&lt;p&gt;This measures ordinary closed-set identification.&lt;br&gt;
Nearby unknown models&lt;br&gt;
These may include adjacent checkpoints, fine-tuned variants, quantized deployments, or related models deliberately excluded from the reference bank.&lt;/p&gt;

&lt;p&gt;They test the hardest boundary: whether the detector can avoid confidently assigning a near neighbor to a known identity.&lt;br&gt;
Distant unknown models&lt;br&gt;
These models come from families not represented in the bank.&lt;/p&gt;

&lt;p&gt;They help measure whether the rejection mechanism recognizes clearly out-of-distribution behavior.&lt;/p&gt;

&lt;p&gt;The evaluation should report more than Top-1 accuracy.&lt;/p&gt;

&lt;p&gt;Useful open-set metrics include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;true-positive rate for known models;&lt;/li&gt;
&lt;li&gt;false-accept rate for unknown models;&lt;/li&gt;
&lt;li&gt;false-reject rate for known models;&lt;/li&gt;
&lt;li&gt;AUROC;&lt;/li&gt;
&lt;li&gt;AUPRC;&lt;/li&gt;
&lt;li&gt;FPR at a specified true-positive rate;&lt;/li&gt;
&lt;li&gt;open-set classification rate;&lt;/li&gt;
&lt;li&gt;per-family rejection performance.
The operating threshold should be chosen according to the application.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A low-stakes comparison tool may tolerate some false acceptance. A system used for provider auditing should prefer inconclusive results over unsupported identity claims.&lt;br&gt;
Calibration requires independent evidence&lt;br&gt;
A classifier is calibrated when predictions assigned probability (q) are correct approximately (q) of the time.&lt;/p&gt;

&lt;p&gt;For example, among predictions reported at approximately 80%, around 80% should be correct under the defined evaluation distribution.&lt;/p&gt;

&lt;p&gt;This property cannot be inferred from the softmax formula. It must be measured.&lt;/p&gt;

&lt;p&gt;Common calibration metrics include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Expected Calibration Error;&lt;/li&gt;
&lt;li&gt;Brier score;&lt;/li&gt;
&lt;li&gt;negative log-likelihood;&lt;/li&gt;
&lt;li&gt;reliability diagrams;&lt;/li&gt;
&lt;li&gt;class-conditional calibration error.
Calibration must also be evaluated under distribution shift.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A probability calibrated on one set of models, prompts, temperatures, and serving providers may not remain calibrated after those conditions change.&lt;/p&gt;

&lt;p&gt;For black-box model detection, calibration should ideally be tested across:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;collection dates;&lt;/li&gt;
&lt;li&gt;API providers;&lt;/li&gt;
&lt;li&gt;sampling parameters;&lt;/li&gt;
&lt;li&gt;prompt variants;&lt;/li&gt;
&lt;li&gt;model families;&lt;/li&gt;
&lt;li&gt;deployment configurations.
Otherwise, a percentage may be numerically precise but operationally misleading.
Model identity may not be the right target
There is a deeper question: should a black-box detector try to infer exact model identity at all?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In many real applications, the user’s concern is not the provider’s internal checkpoint hash. The concern is whether the endpoint still behaves consistently with the service that was purchased.&lt;/p&gt;

&lt;p&gt;That changes the evaluation target.&lt;/p&gt;

&lt;p&gt;Instead of asking:&lt;br&gt;
Is this endpoint definitely Model A?&lt;/p&gt;

&lt;p&gt;we can ask:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Is its behavior still consistent with the Model A reference distribution?&lt;/li&gt;
&lt;li&gt;Has the endpoint changed significantly since the previous evaluation?&lt;/li&gt;
&lt;li&gt;Is it closer to a lower-cost candidate than to the advertised reference?&lt;/li&gt;
&lt;li&gt;Does its behavior vary across repeated requests?&lt;/li&gt;
&lt;li&gt;Is there evidence of mixed routing?&lt;/li&gt;
&lt;li&gt;Has the endpoint drifted after a provider update?
These are monitoring questions rather than identity-authentication questions.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Behavioral fingerprinting is often better suited to answering them.&lt;br&gt;
Detecting drift rather than declaring identity&lt;br&gt;
Suppose an endpoint is evaluated every week under a frozen protocol.&lt;/p&gt;

&lt;p&gt;Let (z_t) denote its behavioral representation at time (t). Instead of assigning a definitive identity, the system can monitor:&lt;br&gt;
[&lt;br&gt;
D(z_t, z_{\text{reference}})&lt;br&gt;
]&lt;br&gt;
and:&lt;br&gt;
[&lt;br&gt;
D(z_t, z_{t-1})&lt;br&gt;
]&lt;br&gt;
A significant increase may indicate:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a model update;&lt;/li&gt;
&lt;li&gt;a routing change;&lt;/li&gt;
&lt;li&gt;a modified system prompt;&lt;/li&gt;
&lt;li&gt;different inference parameters;&lt;/li&gt;
&lt;li&gt;provider-side post-processing;&lt;/li&gt;
&lt;li&gt;degraded protocol compliance.
This does not reveal the cause automatically. It does provide an observable signal that can trigger additional testing.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For API buyers, that may be more actionable than an overconfident model label.&lt;br&gt;
What a professional conclusion looks like&lt;br&gt;
A responsible report should use language such as:&lt;br&gt;
Under the frozen prompt protocol, quality rules, detector version, and reference bank, the observed responses were most similar to Model A. Model A also showed a substantial score margin over the next represented candidate. This result supports behavioral consistency with the Model A reference distribution, but does not independently verify the endpoint’s internal deployment identity.&lt;/p&gt;

&lt;p&gt;If the absolute fit is poor, it should say:&lt;br&gt;
Model A was the highest-ranked represented candidate, but the observation showed weak compatibility with the known reference distributions. The result should be treated as inconclusive.&lt;/p&gt;

&lt;p&gt;If related versions cannot be separated, it should say:&lt;br&gt;
The responses are consistent with the Model A family, but the current evidence does not reliably distinguish between the included versions.&lt;/p&gt;

&lt;p&gt;Scientific credibility often depends less on producing stronger conclusions and more on refusing to claim more than the evidence supports.&lt;br&gt;
Conclusion&lt;br&gt;
A 99% result can be mathematically correct and semantically misleading at the same time.&lt;/p&gt;

&lt;p&gt;Closed-set probability answers a conditional ranking question:&lt;br&gt;
Assuming the relevant model is represented in this reference set, which candidate is preferred?&lt;/p&gt;

&lt;p&gt;It does not automatically answer:&lt;br&gt;
What is the probability that this API truly runs the predicted model?&lt;/p&gt;

&lt;p&gt;To support the second claim, a detector needs more than softmax:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;independent blind evaluation;&lt;/li&gt;
&lt;li&gt;absolute compatibility measurements;&lt;/li&gt;
&lt;li&gt;unknown-model testing;&lt;/li&gt;
&lt;li&gt;validated rejection thresholds;&lt;/li&gt;
&lt;li&gt;probability calibration;&lt;/li&gt;
&lt;li&gt;versioned reference data;&lt;/li&gt;
&lt;li&gt;explicit uncertainty reporting.
Black-box behavioral fingerprinting can still be useful. It can reveal similarity, inconsistency, drift, and possible routing anomalies when direct deployment evidence is unavailable.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Its value depends on stating the result correctly.&lt;/p&gt;

&lt;p&gt;The goal should not be to make the detector sound certain. The goal should be to make its uncertainty measurable.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>datascience</category>
    </item>
    <item>
      <title>Oken_Behavioral_Similarity_Methodology</title>
      <dc:creator>okenai</dc:creator>
      <pubDate>Mon, 09 Nov 2026 07:21:05 +0000</pubDate>
      <link>https://dev.to/okenai_1ec49d66dbe8d17916/okenbehavioralsimilaritymethodology-1llj</link>
      <guid>https://dev.to/okenai_1ec49d66dbe8d17916/okenbehavioralsimilaritymethodology-1llj</guid>
      <description>&lt;p&gt;Behavioral Similarity Profiling for Black-Box LLM APIs&lt;br&gt;
A constrained numerical-generation approach to comparing API behavior with reference models&lt;br&gt;
Technical note  |  Oken Evaluation Lab&lt;br&gt;
Abstract&lt;br&gt;
Commercial LLM APIs expose model names, but API clients generally cannot independently verify which underlying model, checkpoint, or serving configuration processed a request. This creates a practical transparency problem for developers comparing providers, monitoring API behavior, or investigating possible routing changes.&lt;br&gt;
This note describes an experimental method for comparing the observable behavior of a black-box API endpoint with a bank of reference endpoints. The method uses constrained numerical-generation tasks: the target is asked to produce bounded integer sequences, and the resulting sequences are summarized with statistical features such as value frequencies, position-dependent distributions, transitions, and repetition patterns. A ranking pipeline then compares the observed feature representation with reference samples.&lt;br&gt;
The method is intended to measure behavioral similarity under a specified protocol. It does not cryptographically attest to model identity, reveal provider-side routing, or prove that an endpoint is running a particular model. The current implementation uses a closed reference set, so its normalized scores must be interpreted as relative comparisons among available candidates. This note describes the methodology and its limitations; it does not report validated identification accuracy.&lt;br&gt;
Disclosure&lt;br&gt;
The authors are affiliated with Oken. The implementation described here is part of the Oken Evaluation Lab. This is a methodology description from the platform team, not an independent third-party audit or validation of any provider.&lt;br&gt;
Evaluation interface: &lt;a href="https://www.oken.ai/zh/evaluation-lab/full?from=provider_drawer" rel="noopener noreferrer"&gt;https://www.oken.ai/zh/evaluation-lab/full?from=provider_drawer&lt;/a&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Problem statement and scope
A model name in an API response or product listing is a provider-supplied label. In many commercial settings, the client cannot inspect the provider's internal routing logs, deployment metadata, model weights, or serving infrastructure. As a result, a client may have limited ability to verify whether the observed endpoint behavior remains consistent with a previously measured reference.
Potential scenarios include routing some requests to a smaller model, serving an older checkpoint, changing the underlying model without notice, mixing multiple models behind one endpoint, or changing inference settings in ways that materially affect output behavior.
The question addressed here is deliberately narrower than model verification:
Can a black-box evaluator detect whether an endpoint's observed output behavior is statistically more similar to one candidate in a known reference set than to the other candidates, under a fixed generation protocol?
The method does not measure general intelligence, answer quality, factuality, instruction-following quality, or internal model identity. It produces a behavioral comparison against a specified reference bank.&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Terminology and interpretation&lt;br&gt;
We use the term behavioral similarity profiling to describe the method. The phrase behavioral fingerprinting can be used informally, but it should not be read as a claim that every model has a unique, permanent, or cryptographically verifiable fingerprint.&lt;br&gt;
Term&lt;br&gt;
Meaning in this note&lt;br&gt;
What it does not establish&lt;br&gt;
Black-box endpoint&lt;br&gt;
An API endpoint for which the evaluator observes request inputs and returned outputs, but not internal serving state.&lt;br&gt;
It does not mean that no provider-side information exists.&lt;br&gt;
Behavioral similarity&lt;br&gt;
Statistical resemblance between observed output features and reference samples under a specified protocol.&lt;br&gt;
It is not proof of model identity.&lt;br&gt;
Model attribution&lt;br&gt;
Ranking candidate reference models by similarity to the observed behavior.&lt;br&gt;
It is not equivalent to identity verification.&lt;br&gt;
Reference bank&lt;br&gt;
A versioned collection of samples gathered from endpoints with assigned labels.&lt;br&gt;
It does not guarantee that upstream labels are correct.&lt;br&gt;
Relative similarity weight&lt;br&gt;
A normalized score among the candidates currently in the reference set.&lt;br&gt;
It is not the empirical probability that a candidate is the true model.&lt;br&gt;
Open-set detection&lt;br&gt;
A mechanism for recognizing when an observation may not belong to any known candidate class.&lt;br&gt;
It is not provided merely by applying Softmax to a closed candidate set.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Method overview&lt;br&gt;
Let M denote the target endpoint and let R1, ..., Rk denote reference endpoints with assigned labels. For a fixed protocol P, the evaluator collects responses from M, extracts a feature representation f(P, M), and compares that representation with the corresponding reference representations.&lt;br&gt;
The current task is a closed-set ranking problem:&lt;br&gt;
ŷ = argmaxᵢ S(f(P, M), Rᵢ)&lt;br&gt;
Here, S is a similarity score defined by the current scoring implementation. The output identifies the most similar candidate within the available reference set. It does not recover the provider's internal routing state.&lt;br&gt;
The appropriate interpretation is:&lt;br&gt;
Under protocol P, detector version D, and reference-bank version R, the endpoint's observed behavior was most similar to candidate Rᵢ.&lt;br&gt;
The result should not be described as:&lt;br&gt;
The endpoint has been proven to run model Rᵢ.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Threat model and assumptions&lt;br&gt;
The primary use case is to identify behavioral changes that may be consistent with model substitution or unexpected routing. Examples include:&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;A less expensive model being served behind a more expensive model label.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Some traffic being routed to an older checkpoint.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;An endpoint changing its underlying model over time.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Multiple models being mixed behind one API endpoint.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;An undocumented inference configuration changing the endpoint's output distribution.&lt;br&gt;
The method assumes that the endpoint returns model-generated output and that no intermediary is deliberately transforming the output to imitate a reference distribution. A provider or middleware layer that knows the protocol could potentially detect the task and generate outputs designed to match a target distribution. The method is therefore a behavioral consistency test, not an adversarially secure attestation mechanism.&lt;br&gt;
The observed behavior may reflect several components together: model weights, tokenizer, decoding parameters, prompt wrappers, safety layers, serving stack, and provider-side routing. Unless these factors are controlled or separately measured, a difference in output behavior cannot automatically be attributed to a change in model identity.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Why use constrained numerical-generation tasks?&lt;br&gt;
The protocol asks the endpoint to generate a sequence of bounded integers while avoiding obvious deterministic patterns. A simplified task asks for approximately 300 integers between 1 and 355, with instructions not to count upward or downward, use arithmetic progressions, cycles, repeated blocks, code, tools, or an external random-number generator.&lt;br&gt;
These are low-semantic-content generation tasks, not tasks with literally no semantics: the model still has to interpret the instruction, respect the numerical range, and produce the requested format. The aim is to reduce dependence on domain knowledge, factual recall, writing style, retrieval, and subjective answer grading.&lt;br&gt;
The working hypothesis is that an LLM asked to generate a random-looking sequence does not behave as an ideal uniform random-number generator. Its output can remain conditioned by learned token distributions, tokenization, alignment, decoding policy, context, and deployment configuration. These influences may produce measurable differences in value frequencies, position-dependent distributions, repetition patterns, and local transitions.&lt;br&gt;
A single integer has little discriminatory value. The method therefore analyzes the joint statistical structure of many positions and compares repeated responses. This is a hypothesis to be tested empirically; the existence of measurable differences does not by itself establish reliable model identification.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Sampling protocol&lt;br&gt;
The current evaluation protocol uses three separately initiated generation runs. Each run is started in a separate conversation with an independently varied but semantically equivalent prompt. Each prompt requests approximately 300 bounded integers and prohibits tools, code, and external randomness. Only the first response is retained, and the parser does not repair or extend the output.&lt;br&gt;
Separate conversations isolate the conversational context across runs. This should not be interpreted as proof that the resulting sequences are statistically independent: all runs may still share model weights, serving infrastructure, decoding settings, system-level instructions, and other common factors.&lt;br&gt;
The three responses are analyzed separately rather than concatenated. Concatenation would introduce artificial transitions at sequence boundaries and could distort position-dependent or transition features.&lt;br&gt;
Before scoring, each response passes a quality-control stage that records:&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Expected and observed sequence length.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Number of valid integers.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Out-of-range values.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Unparseable material.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Whether the minimum sample threshold was met.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Whether each of the three responses is independently usable.&lt;br&gt;
Passing quality control means only that a response contains enough parseable data for scoring. It does not mean that every prompt constraint was followed perfectly. The rationale for using three runs, and the sensitivity of results to larger sample counts, should be evaluated through repeatability experiments. The current description does not establish that three runs are optimal.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Feature representation&lt;br&gt;
The current detector organizes the observed behavior into five feature families. These features describe output statistics; they should not be interpreted as direct measurements of the model's internal mechanisms.&lt;br&gt;
7.1 Marginal distribution characteristics&lt;br&gt;
This family includes frequencies across the integer range, coarse numerical regions, and final digits. It measures whether an endpoint's generated sequences show recurring over- or under-representation of particular values, ranges, or final digits.&lt;br&gt;
7.2 Position-dependent distribution characteristics&lt;br&gt;
The sequence is analyzed by stage and by normalized position. This can capture differences between early and late positions, distributional shifts across the sequence, and position-dependent final-digit frequencies. Terms such as stabilization or compensation should be used only when a specific mechanism has been demonstrated; otherwise, the analysis should describe the observed distributional change.&lt;br&gt;
7.3 Transition features&lt;br&gt;
The detector measures changes at multiple lags, not only between adjacent values. Current feature descriptions include normalized lag correlations, mean absolute displacement, positive- and zero-step proportions, small-step frequencies, and large standardized jumps. For reproducibility, the implementation should specify the lag values, normalization method, and exact definition of each transition statistic.&lt;br&gt;
7.4 Repetition features&lt;br&gt;
Exact value repetition is only one component. The representation also measures repetition or clustering of nearby values, final digits, movement directions, step sizes, and local numerical buckets. These features are intended to capture local patterns that may not be visible in a simple frequency histogram.&lt;br&gt;
7.5 Sequence-structure features&lt;br&gt;
Coarse numerical ranges and final digits are represented using transition matrices. The detector compares observed transitions with a reference distribution or an independence-based expectation. The exact binning scheme, matrix order, smoothing procedure, and deviation metric should be documented to make this component reproducible.&lt;br&gt;
In the current implementation, the positional branch is described as containing 751 features per response:&lt;br&gt;
Feature block&lt;br&gt;
Dimensions&lt;br&gt;
Smoothed frequencies over integers 1–355&lt;br&gt;
355&lt;br&gt;
Segmented position and final-digit distributions&lt;br&gt;
74&lt;br&gt;
Continuous position-dependent kernels&lt;br&gt;
102&lt;br&gt;
Lag, repetition, jump, and transition features&lt;br&gt;
220&lt;br&gt;
Total&lt;br&gt;
751&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The 751 dimensions are not 751 independent pieces of identity evidence. Some features may be correlated, and the feature count alone says nothing about generalization performance. Because the feature dimension is larger than the number of positions in a single approximately 300-integer response, evaluation should explicitly examine feature correlation, regularization, feature selection, and the risk of overfitting. The current description does not provide those results.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;Candidate ranking and score interpretation&lt;br&gt;
The current scoring pipeline combines several statistical views. The frequency branch is described as including a discriminative projection based on the sequence prefix, distances to empirical reference samples, and a nuisance-adjusted centroid comparison. The positional branch uses the position and sequence features described above.&lt;br&gt;
The high-level fusion is currently described as:&lt;br&gt;
Sᵢ = 0.5 × Sᵢ(frequency) + 0.5 × Sᵢ(positional)&lt;br&gt;
The frequency component is described as:&lt;br&gt;
Sᵢ(frequency) = 0.5 × Sᵢ(LDA) + 0.25 × Sᵢ(KNN) + 0.25 × Sᵢ(robust)&lt;br&gt;
These expressions summarize the current high-level weighting. A reproducible technical specification should additionally define each component's raw output, any distance-to-similarity conversion, normalization range, treatment of missing features, and the empirical basis for the weights. This is particularly important because some methods naturally produce distances, for which smaller values indicate greater similarity, while the fused score must have a consistent direction.&lt;br&gt;
The current implementation converts candidate scores to a Softmax-normalized distribution:&lt;br&gt;
pᵢ = exp(τSᵢ) / Σⱼ exp(τSⱼ)&lt;br&gt;
where τ is a temperature parameter calibrated using reference data. Unless calibration has been validated on held-out data with appropriate metrics, pᵢ should be described as a relative similarity weight within the current candidate set, not as the empirical probability that the endpoint is truly running candidate i.&lt;br&gt;
Output&lt;br&gt;
Interpretation&lt;br&gt;
Score&lt;br&gt;
An internal similarity statistic produced by the scoring implementation.&lt;br&gt;
Score margin&lt;br&gt;
The separation between the leading candidate and one or more alternatives.&lt;br&gt;
Softmax-normalized relative weight&lt;br&gt;
A normalized comparison among candidates currently in the reference set.&lt;br&gt;
Empirical identification accuracy&lt;br&gt;
A separately measured evaluation metric; it cannot be inferred from the normalized weight.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Reference bank and reproducibility&lt;br&gt;
The current frozen reference bank is described as containing:&lt;br&gt;
Reference-bank item&lt;br&gt;
Current count&lt;br&gt;
Candidate model identities&lt;br&gt;
59&lt;br&gt;
Collection batches&lt;br&gt;
143&lt;br&gt;
Reference responses&lt;br&gt;
2,198&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A result should be associated with at least the following metadata:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Detector version and reference-bank version.&lt;/li&gt;
&lt;li&gt;Reference collection time.&lt;/li&gt;
&lt;li&gt;Prompt hashes and raw-response hashes.&lt;/li&gt;
&lt;li&gt;Inference parameters, where available.&lt;/li&gt;
&lt;li&gt;Candidate ranking and score components.&lt;/li&gt;
&lt;li&gt;Quality-control diagnostics.
The detector, reference samples, ranking components, and evaluation scripts are intended to be bound through version manifests and file hashes. This helps prevent a result from being silently reinterpreted after the reference set or scoring implementation changes. It does not independently verify that the upstream provider's reference-model label is correct.&lt;/li&gt;
&lt;li&gt;Closed-set limitation and unknown models
The current implementation ranks candidates from a predefined reference bank. If the true model is absent from that bank, the system may still assign a high relative weight to one of the available candidates because the Softmax operation distributes weight across the candidates it receives.
The quantity is therefore closer to:
P(Rᵢ | x, y ∈ R)
than to:
P(true model = Rᵢ | x)
The condition that the true model belongs to the reference set is essential. A production open-set detector would require a separately validated rejection mechanism. Possible research directions include absolute-distance thresholds, density-based methods, conformal prediction, held-out unknown models, or an explicitly trained out-of-distribution component. These approaches have different assumptions and guarantees; they are not interchangeable. No universally valid UNKNOWN threshold is claimed here.&lt;/li&gt;
&lt;li&gt;Sources of uncertainty and confounding factors
11.1 Model-family overlap
Adjacent versions and models sharing a base model may have similar output distributions. Family-level attribution may be more stable than exact checkpoint attribution, but this must be measured rather than assumed.
11.2 Inference parameters
Temperature, top-p, reasoning effort, penalties, and provider-specific sampling logic can change output distributions. A difference in the observed fingerprint may therefore reflect a configuration change rather than a model change.
11.3 Prompt wrappers and system instructions
System and developer messages, formatting wrappers, and safety layers can affect instruction following and local sequence structure. Where possible, these factors should be held constant or recorded.
11.4 Temporal drift
Providers may update checkpoints, routing policies, safety layers, or serving infrastructure over time. A stale reference bank may lose discriminatory value. Reference samples should therefore be time-stamped and refreshed under a documented process.
11.5 Quantization and serving-stack differences
Different inference stacks may produce measurable behavior differences even when deployments use nominally related weights. The method may detect serving differences without being able to attribute them uniquely to a model identity change.
11.6 Protocol awareness and adversarial adaptation
A model or middleware layer that recognizes the task may generate algorithmic randomness or imitate a reference distribution. This limits the method's resistance to deliberate adaptation.
11.7 Reference contamination and data leakage
Repeated tuning against the same reference samples can cause the detector to learn collection-specific artifacts rather than generalizable model behavior. Training, calibration, and evaluation data should be separated by collection batch, and ideally by time period and serving configuration where feasible.
11.8 Confounding between model identity and serving configuration
The observed output is produced by a system that includes the model and its serving configuration. To support model-level claims, experiments should test the same model under multiple parameter settings and different models under matched settings. Without such controls, the safest conclusion concerns endpoint behavior under the tested conditions, not model identity in isolation.&lt;/li&gt;
&lt;li&gt;What evidence is needed to validate the method?
A plausible mechanism is not evidence that a particular detector achieves useful identification performance. A credible evaluation should freeze the protocol, feature extraction, scoring pipeline, and reference bank before running a blinded test. The evaluation set should include known endpoints, repeated runs, adjacent model versions, and models deliberately excluded from the reference bank.
At minimum, a report should include:&lt;/li&gt;
&lt;li&gt;Top-1 and Top-k accuracy.&lt;/li&gt;
&lt;li&gt;Family-level and version-level accuracy.&lt;/li&gt;
&lt;li&gt;Macro- and micro-averaged precision, recall, and F1 where applicable.&lt;/li&gt;
&lt;li&gt;Per-model confidence intervals and a complete confusion matrix.&lt;/li&gt;
&lt;li&gt;Calibration error, with the calibration procedure and held-out evaluation clearly described.&lt;/li&gt;
&lt;li&gt;Score-margin distributions and repeatability across independent runs.&lt;/li&gt;
&lt;li&gt;Sensitivity to temperature, top-p, reasoning effort, and other available inference parameters.&lt;/li&gt;
&lt;li&gt;Performance under temporal drift and across serving providers.&lt;/li&gt;
&lt;li&gt;False-accept and false-reject rates for out-of-reference models.&lt;/li&gt;
&lt;li&gt;Ablation results for each feature family and scoring component.
The split should be made by collection batch rather than randomly splitting individual responses from the same batch. Otherwise, training and test sets may share temporal, provider, or configuration artifacts. Where practical, evaluation should also include held-out time periods and held-out serving configurations.
High-value stress tests include:&lt;/li&gt;
&lt;li&gt;Adjacent versions from the same model family.&lt;/li&gt;
&lt;li&gt;The same model served with different inference configurations.&lt;/li&gt;
&lt;li&gt;Different models evaluated under matched inference configurations.&lt;/li&gt;
&lt;li&gt;Unknown models excluded from the reference bank.&lt;/li&gt;
&lt;li&gt;Mixed routing between two known models.&lt;/li&gt;
&lt;li&gt;Repeated evaluation after a provider silently changes an endpoint.&lt;/li&gt;
&lt;li&gt;The same nominal model accessed through different providers or serving stacks.
Until such results are reported, the system should be described as an experimental behavioral similarity detector with a defined methodology, not as a generally validated model-identity verifier.&lt;/li&gt;
&lt;li&gt;Appropriate use and non-use
Potentially useful applications include:&lt;/li&gt;
&lt;li&gt;Monitoring whether an endpoint's observable behavior changes materially over time.&lt;/li&gt;
&lt;li&gt;Comparing an API endpoint with a known native or provider reference under a controlled protocol.&lt;/li&gt;
&lt;li&gt;Flagging possible routing or downgrade anomalies for further investigation.&lt;/li&gt;
&lt;li&gt;Prioritizing providers or endpoints for additional technical review.&lt;/li&gt;
&lt;li&gt;Tracking behavioral drift across repeated evaluations.
The method should not replace:&lt;/li&gt;
&lt;li&gt;Provider-side routing and request logs.&lt;/li&gt;
&lt;li&gt;Signed deployment metadata or trusted execution evidence.&lt;/li&gt;
&lt;li&gt;Contractual audits and provider disclosures.&lt;/li&gt;
&lt;li&gt;Cryptographic model attestation.&lt;/li&gt;
&lt;li&gt;Independent evaluation of task quality, latency, reliability, and factuality.
A behavioral comparison is one piece of evidence. Strong claims about model substitution should be supported by repeated tests and independent operational evidence. A similarity result alone should not be used to accuse a provider of misrepresentation.&lt;/li&gt;
&lt;li&gt;Practical interpretation of an individual result
For an individual endpoint evaluation, the recommended wording is:
Under the specified prompt protocol, quality-control rules, detector version, and frozen reference bank, the endpoint's observed responses were most similar to candidate X. This result indicates behavioral similarity to X under the tested conditions; it does not prove the endpoint's internal model identity or establish that the provider has substituted a model.
If the endpoint's behavior changes materially over time, the result should be treated as a signal for follow-up investigation. The next steps may include repeating the test, checking inference parameters, comparing independent reference endpoints, reviewing provider documentation, and requesting operational evidence from the provider.
Conclusion
Constrained numerical-generation tasks provide one possible way to observe statistical differences between black-box LLM API endpoints. By analyzing value frequencies, position-dependent distributions, transitions, repetition patterns, and other sequence features, an evaluator can compare an endpoint's observable behavior with a bank of reference samples.
The central claim is deliberately limited: under a fixed protocol, numerical-generation behavior may provide useful signals for comparing endpoints and monitoring behavioral changes. Whether those signals are sufficiently stable and discriminative for reliable model-level attribution is an empirical question. It must be evaluated separately for each detector version, reference bank, inference configuration, provider environment, and evaluation period.
The current method is best understood as an experimental behavioral similarity tool. It is not a cryptographic identity check, and its closed-set ranking should not be interpreted as proof that an endpoint runs a particular model. Publishing blinded evaluation results, repeatability analysis, open-set rejection performance, and sensitivity tests will be essential to establishing its practical value.
We welcome technical feedback, particularly on feature validation, batch-independent evaluation, confounding between model identity and serving configuration, open-set rejection, adversarial robustness, and the distinction between model-family and exact-version attribution.
Appendix A. Suggested minimum reporting template
The following table is a reporting template. Values should be populated only after the corresponding experiments have been run; no performance values are inferred or fabricated in this document.
Evaluation item
What should be reported
Dataset composition
Models, providers, versions, collection dates, number of batches and responses.
Protocol
Prompt variants, requested sequence length, parsing rules, sampling parameters, number of runs.
Data split
Batch-level train/calibration/test split and any held-out time periods or providers.
Classification
Top-1, Top-k, family-level and version-level results.
Error analysis
Confusion matrix, per-class errors, false accepts and false rejects.
Calibration
Calibration method, calibration error, reliability analysis, and held-out results.
Repeatability
Rank consistency and score variation across repeated runs.
Robustness
Sensitivity to inference parameters, prompt variants, provider and serving changes.
Ablation
Performance with each feature family and scoring component removed.
Open-set behavior
Unknown-model rejection method, threshold selection, false-accept rate.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Appendix B. Implementation details to document for reproducibility&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Exact integer parsing and invalid-output handling rules.&lt;/li&gt;
&lt;li&gt;Numerical binning scheme and smoothing method.&lt;/li&gt;
&lt;li&gt;Position segmentation and kernel definitions.&lt;/li&gt;
&lt;li&gt;Lag values and transition-statistic formulas.&lt;/li&gt;
&lt;li&gt;Repetition-window definitions and bucket construction.&lt;/li&gt;
&lt;li&gt;Feature scaling, selection, regularization, and missing-value handling.&lt;/li&gt;
&lt;li&gt;LDA configuration and training procedure.&lt;/li&gt;
&lt;li&gt;KNN distance metric, neighbor count, and aggregation method.&lt;/li&gt;
&lt;li&gt;Definition of the robust comparison branch and nuisance factors.&lt;/li&gt;
&lt;li&gt;Score normalization and distance-to-similarity transformations.&lt;/li&gt;
&lt;li&gt;Weight-selection procedure and Softmax temperature calibration.&lt;/li&gt;
&lt;li&gt;Reference-bank labeling, refresh policy, and versioning procedure.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>python</category>
      <category>agents</category>
      <category>tools</category>
    </item>
    <item>
      <title>I Built a Tool to Compare AI Model API Prices and Provider Availability</title>
      <dc:creator>okenai</dc:creator>
      <pubDate>Tue, 27 Oct 2026 08:25:26 +0000</pubDate>
      <link>https://dev.to/okenai_1ec49d66dbe8d17916/i-built-a-tool-to-compare-ai-model-api-prices-and-provider-availability-2mio</link>
      <guid>https://dev.to/okenai_1ec49d66dbe8d17916/i-built-a-tool-to-compare-ai-model-api-prices-and-provider-availability-2mio</guid>
      <description>&lt;p&gt;If you use multiple AI model APIs, you have probably run into the same problem:&lt;/p&gt;

&lt;p&gt;The same model can be offered by many providers, but their pricing, availability, and token usage may vary significantly. Comparing them manually is time-consuming, and the cheapest option is not always the most reliable one.&lt;/p&gt;

&lt;p&gt;That is why I built &lt;a href="https://www.oken.ai/zh" rel="noopener noreferrer"&gt;Oken.ai&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Oken.ai is a platform for comparing AI model API providers across the market. It currently helps developers:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Compare input, output, and cached-token prices&lt;/li&gt;
&lt;li&gt;Find providers offering the same AI model&lt;/li&gt;
&lt;li&gt;Monitor API route availability&lt;/li&gt;
&lt;li&gt;Evaluate model consistency&lt;/li&gt;
&lt;li&gt;Measure actual token consumption&lt;/li&gt;
&lt;li&gt;Filter models by company, capability, and billing type&lt;/li&gt;
&lt;li&gt;Track newly released models from OpenAI, Anthropic, Google, DeepSeek, and others&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example, if you want to use GPT, Claude, Gemini, DeepSeek, or another model, you can quickly see which providers offer it, compare their prices, and check whether their API routes are currently available.&lt;/p&gt;

&lt;p&gt;The goal is simple: help developers choose AI model services based on transparent pricing and measurable availability, rather than marketing claims alone.&lt;/p&gt;

&lt;p&gt;You can try it here:&lt;/p&gt;

&lt;p&gt;👉 &lt;a href="https://www.oken.ai/zh" rel="noopener noreferrer"&gt;https://www.oken.ai/zh&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I would appreciate feedback from the developer community:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;What information do you need when choosing an AI API provider?&lt;/li&gt;
&lt;li&gt;Which models or providers should be added?&lt;/li&gt;
&lt;li&gt;What would make the comparison more useful for your workflow?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Disclosure: I am involved with Oken.ai and am sharing it to collect feedback and improve the platform.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>api</category>
      <category>llm</category>
    </item>
    <item>
      <title>中转站</title>
      <dc:creator>okenai</dc:creator>
      <pubDate>Tue, 27 Oct 2026 06:47:51 +0000</pubDate>
      <link>https://dev.to/okenai_1ec49d66dbe8d17916/zhong-zhuan-zhan-24bf</link>
      <guid>https://dev.to/okenai_1ec49d66dbe8d17916/zhong-zhuan-zhan-24bf</guid>
      <description></description>
    </item>
  </channel>
</rss>
