DEV Community: palapalapala

Can a Single Token Reveal Which Model Is Behind an API?

palapalapala — Mon, 27 Jul 2026 15:53:17 +0000

Can a Single Token Reveal Which Model Is Behind an API?

A Small Experiment in Model Identity

If you frequently use large language model APIs from different cloud providers, you have probably encountered something like this: the provider’s dashboard lists a long series of model names—“Standard,” “Advanced,” “Pro,” “Turbo,” and “Mini”—yet there is almost no practical way to verify whether those names actually correspond to different models.

They might be independently trained or fine-tuned models. But they could also be the same backend model packaged under different names, pricing tiers, and rate limits.

From an API user’s perspective, the system is largely a black box. You send a request and receive a response, but you have little visibility into where that request was actually routed.

This is not a conspiracy theory. It is a practical engineering problem.

A model name may be an alias. Requests may be routed through staged rollouts or different quantized variants. A provider may also silently update a model and continue offering it under a new name.

As users, we lack a low-cost, reproducible way to cross-check a simple question:

Do these two model names actually point to the same thing?

That is the question explored by One Token Is Enough, an open-source project recently developed by a friend of mine.

Project repository:

https://github.com/vivgrid/llm-fingerprinting

Before getting into the details, I want to make the tone of this article clear.

This is not a story about “exposing fraud” or “catching dishonest providers.” It is better understood as a methodological exploration: an attempt to use statistics to produce a measurable signal about model identity.

A signal is not proof, and it certainly is not a verdict. I will return to that distinction later in the article.

The Starting Point: A Clever Idea From a Research Paper

The project is based on a paper titled:

“One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions”
(arXiv:2607.10252)

The paper’s central idea is surprisingly simple:

If you repeatedly ask a model the same simple question, its answering habits form a measurable statistical pattern.

Imagine asking a group of people to perform several tasks:

Pick a random number between 1 and 100
Choose a random letter
Name a random color
Name a random animal
Flip a coin and say heads or tails

If you ask one person only once, you learn almost nothing. They might say “37,” “red,” or “cat,” but an isolated answer carries very little information.

If you ask the same question thousands of times, however, unconscious preferences may begin to emerge.

One person may choose “7” unusually often. Another may almost never select the letter “Q.” Someone else may consistently think of “dog” before any other animal.

Large language models behave in a similar way.

Their outputs are not uniformly random. Instead, they are sampled from probability distributions shaped by training data, model parameters, and decoding strategies.

By repeatedly sampling a model with the same prompt, we can construct a distribution of its answers to that prompt.

That distribution is what the paper describes as a behavioral fingerprint.

Why “One Token” Is the Key Design Choice

There is an important point that can easily be misunderstood:

“One token” means that each request asks the model to generate only one output token. It does not mean that an entire fingerprint can be created from a single request.

A complete fingerprint is constructed roughly as follows:

Prepare a set of simple probing prompts, such as: “Choose a random number between 1 and 100. Answer with the number only.”
Send each prompt to the target model many times as independent requests, requesting exactly one output token each time.
Count how frequently each token appears and use those frequencies to construct an empirical probability distribution (P).
Repeat the process across multiple prompts and combine the resulting distributions into a complete fingerprint.

Why collect only one token instead of asking the model to generate a full sentence?

There are several practical advantages.

Lower Cost

Generating one token is far cheaper than generating a paragraph, making large-scale repeated sampling possible within a limited budget.

Cleaner Signals

Full sentences introduce additional variables that may be unrelated to model identity, including writing style, sentence length, formatting, and phrasing.

A single token is closer to a direct sample from the model’s underlying output distribution.

Faster Sampling

Hundreds or thousands of probes can be completed more quickly, making it easier to produce a statistically stable distribution within a reasonable amount of time.

In other words, using one token reduces the cost of each individual sample, making large-scale repeated sampling practical.

The fingerprint itself is still based on a distribution, not on any single isolated answer.

A Concrete Example: How Models “Randomly” Choose Numbers

To make the method more intuitive, consider a phenomenon that can be observed in practice.

Suppose you ask a model:

Choose a random number between 1 and 100.

If the model were performing truly uniform random sampling, every number would have a probability close to (1/100).

In practice, however, language models are not random-number generators. Their choices often exhibit statistical preferences.

Some numbers may appear more frequently because they are commonly used as arbitrary examples in training data. Numbers such as “42,” “7,” and “100,” for example, carry strong cultural or linguistic associations and may be selected more often than other numbers.

There is nothing mysterious about this bias.

It comes from the statistical patterns the model has learned from language, not from a dedicated random-number algorithm.

The important observation is this:

If two models with different names produce almost identical number-selection distributions across hundreds or thousands of samples, that similarity is statistically noteworthy.

It becomes even more noteworthy when the same consistency appears across several independent probing dimensions—numbers, letters, colors, animals, and coin flips.

At that point, explaining the result as a coincidence becomes increasingly difficult.

This is the signal the method attempts to capture: not a single matching answer, but a behavioral pattern that remains consistent across multiple dimensions and large numbers of repeated samples.

Measuring Similarity: Jensen–Shannon Divergence

Once we have answer distributions from two models, the next question is:

How can we objectively express their similarity as a single number?

The project uses Jensen–Shannon divergence, or JS divergence, a symmetric and bounded measure of distance between probability distributions.

For two probability distributions (P) and (Q), JS divergence is defined as:

[
JSD(P \parallel Q) = \frac{1}{2} D_{KL}(P \parallel M) + \frac{1}{2} D_{KL}(Q \parallel M)
]

where:

[
M = \frac{1}{2}(P + Q)
]

is the average of the two distributions, and (D_{KL}) is the Kullback–Leibler divergence:

[
D_{KL}(P \parallel M) = \sum_{i} P(i) \log \frac{P(i)}{M(i)}
]

There are several practical reasons for using JS divergence instead of KL divergence directly.

Symmetry

[
JSD(P \parallel Q) = JSD(Q \parallel P)
]

This matches the symmetric nature of asking how similar two models are.

Boundedness

When logarithms use base 2, JS divergence falls within the interval ([0, 1]).

This makes it easier to define thresholds and compare results across different probing tasks.

Numerical Stability

Unlike KL divergence, JS divergence does not become infinite when one distribution assigns zero probability to a category.

For every pair of models, the tool calculates JS divergence for each probing dimension and then aggregates the results into an overall similarity report.

The closer the divergence is to 0, the more similar the distributions are. The closer it is to the upper bound, the more clearly they differ.

Split-Half Testing: Is the Fingerprint Stable?

The project also includes an important internal validation mechanism: a split-half consistency test.

The samples collected from a single model are randomly divided into two halves. A separate distribution is calculated from each half, and the divergence between them is measured.

If the two halves of the same model already produce a relatively large divergence, the current sample size is probably insufficient to create a stable fingerprint.

In that situation, any comparison between different models should be treated with particular caution.

This test helps distinguish meaningful behavioral differences from statistical noise caused by insufficient sampling.

What the Project Actually Implements

With the methodology covered, let us return to the engineering implementation.

The repository currently includes the following components.

A Probe Library

The project contains a 40-cell probing matrix:

10 task categories, including random numbers, letters, colors, animals, and coin flips
4 languages: English, Russian, Chinese, and Arabic

The multilingual design helps examine whether a model’s behavioral characteristics change when the language changes. It also creates room for adding more probing dimensions in the future.

Two Zero-Dependency Implementations

The project provides two implementations:

A Python 3.8+ version that uses only the standard library and requires no third-party packages
A Bun/TypeScript version that works without running npm install

The two implementations provide matching functionality while supporting different development environments and workflows.

An Offline Demo Mode

The repository includes a deterministic mock backend that can run the complete workflow without consuming API credits or requiring an API key.

Every stage—from generating fingerprints to performing comparative analysis—can be tested locally.

This is useful for understanding how the tool behaves and for validating the workflow before connecting it to a real API.

A Command-Line Interface

The CLI provides five subcommands:

audit
fingerprint
compare
verify
selftest

These commands can be used to generate audit reports, create individual model fingerprints, compare multiple fingerprints, verify a fingerprint against a reference, and run internal consistency checks.

Output Formats

The tool supports both:

Structured JSON fingerprint files for programmatic processing and long-term storage
Human-readable audit reports for quickly reviewing the results

API Compatibility

The tool is designed for any endpoint compatible with the OpenAI API format.

The repository uses Vivgrid as its default example endpoint, but this is only a demonstration configuration.

By changing the base URL, the tool can target any OpenAI-compatible service, including locally deployed inference servers.

An Honest Discussion of the Limitations

I believe this section is more important than the feature list itself.

This Is a Statistical Test, Not Cryptographic Proof

JS divergence provides a quantitative description of similarity. It does not authenticate a model’s identity.

If two distributions are highly similar, the appropriate conclusion is:

This result may be worth investigating further.

It is not:

These endpoints have been proven to use the same backend model.

Models From the Same Family May Naturally Behave Similarly

If two models were fine-tuned from closely related base models, it is entirely reasonable for them to produce similar behavioral distributions on simple probing tasks.

That similarity is not evidence of fraud.

Many Factors Can Shift a Fingerprint Without Changing the Underlying Model

These factors include:

A system prompt injected by the server
Sampling parameters such as temperature and top_p
Request-routing strategies, including load balancing across different inference instances
Differences between quantized model variants
Silent updates made by the provider

Any of these factors can alter the observed output distribution.

In practice, a behavioral fingerprint may reflect the behavior of the entire API endpoint under a specific configuration, rather than the model weights alone.

Reference Fingerprints Must Be Regenerated Regularly

Models change, services are reconfigured, and providers update their infrastructure.

A fingerprint generated today may no longer be representative several weeks later. Treating a single comparison as a permanent conclusion would be methodologically unsound.

Thresholds Must Be Calibrated for Each Endpoint and Sample Size

There is no universal similarity threshold that works for every model, endpoint, and probing task.

Larger sample sizes reduce statistical noise, but they also increase API costs. Smaller sample sizes are cheaper, but the resulting conclusions are less reliable.

This is a trade-off that must be evaluated for each use case. The tool cannot make that decision on the user’s behalf.

For all these reasons, the most honest conclusion the tool can provide is limited to something like this:

Under the current sampling configuration, these two endpoints exhibit a certain level of similarity in their behavioral output distributions, and the result may—or may not—justify further investigation.

Nothing more.

Final Thoughts: More of an Invitation Than a Launch

The purpose of sharing this project is not so much to announce a finished product as it is to introduce a methodology that is still being refined.

It may be useful to researchers, API users, inference-service providers, and developers interested in a broader question:

How can we verify behavioral consistency in black-box systems?

The project’s author would be especially interested in feedback on the following topics:

Methodology

Is JS divergence the best measure for this type of comparison?

Are there better metrics for comparing discrete probability distributions?

Thresholds and Sample Sizes

How many samples are needed to reach statistically credible conclusions at a reasonable cost?

Should different probing tasks use different thresholds?

Normalization

Providers may use different default sampling parameters, including temperature and top_p.

How should these settings be normalized before comparison to make the results fairer?

Probe Design

Do the current 10 task categories have obvious blind spots?

What types of prompts might provide stronger discriminative power?

Real-World Results

If anyone has tested the tool against production APIs, what did you observe?

Both supporting results and critical findings are welcome.

For anyone who wants to try it directly, the repository’s offline mock mode allows you to run the complete workflow without spending money or providing an API key.

You can see how fingerprints are generated and what the comparison reports look like:

https://github.com/vivgrid/llm-fingerprinting

The codebase, issues, and pull requests are all open.

Whether you identify a methodological flaw, propose a better statistical approach, or contribute new probing dimensions, feedback can be shared directly with the project’s author through GitHub issues and pull requests.

If you have thoughts about the paper itself—arXiv:2607.10252—or about potential applications of black-box behavioral fingerprinting in areas such as model provenance, model ownership, and model-distillation detection, feel free to share them in the comments or open an issue in the repository.

Making AI Agent Configurations Stable with an LLM Gateway

palapalapala — Mon, 26 Jan 2026 16:59:57 +0000

When building AI agents, one recurring problem is configuration churn.

LLMs evolve quickly: models are upgraded, providers change APIs, and endpoints get deprecated.
But agent configurations usually move much slower. Over time, this mismatch leads to fragile setups and frequent config changes.

In this article, I’m sharing an approach I came across while exploring ways to keep agent configurations stable: using an LLM gateway as an abstraction layer, with Clawdbot as the concrete example and Vivgrid as the gateway.

The Core Idea

Instead of binding an AI agent directly to a specific LLM provider or model, you route all model requests through an LLM gateway.

This way:

Agent logic stays stable
Model upgrades happen behind the scenes
Configuration files change far less often

Vivgrid acts as that gateway, while Clawdbot remains focused on agent behavior rather than model infrastructure.

What This Setup Achieves

After this configuration, Clawdbot will:

Use a single, stable endpoint for model access
Avoid vendor lock-in
Stay operational as underlying models evolve
Reduce the need for frequent config edits and restarts

Prerequisites

Before starting, make sure you have:

Clawdbot installed
A Vivgrid API key
Completed Clawdbot onboarding

Run:

clawdbot onboar

Step 1: Configure Vivgrid as a Model Provider

Clawdbot defines model providers in its configuration file.

Config file location:

~/.clawdbot/clawdbot.json

Add Vivgrid under models.providers:

"providers": {
  "vivgrid": {
    "baseUrl": "https://api.vivgrid.com/v1",
    "apiKey": "viv-clawdbot-xxxxxxxxxxxxxxxxxxxx",
    "api": "openai-completions",
    "models": [
      {
        "id": "managed",
        "name": "managed",
        "contextWindow": 128000
      }
    ]
  }
}

This tells Clawdbot to route all model requests through Vivgrid, instead of calling a specific LLM provider directly.

Step 2: Set Vivgrid as the Primary Model

Next, update the default model used by the agent:

"model": {
  "primary": "vivgrid/managed"
}

This is the key change.

Clawdbot is no longer tied to a specific model like GPT-4 or Claude.
Instead, it relies on Vivgrid’s managed routing layer, which can be updated independently.

Step 3: Restart and Verify

Restart the Clawdbot daemon:

clawdbot daemon restart

Then follow the logs to verify requests are flowing correctly:

clawdbot logs --follow

If everything is set up correctly, you should see model requests being routed through Vivgrid.

Why This Works Well in Production

Using an LLM gateway creates a clean separation of responsibilities:

Clawdbot handles agent behavior and logic
Vivgrid handles model selection, routing, and upgrades

This approach makes it easier to:

Upgrade models without touching agent config
Reduce downtime caused by model migrations
Keep agent systems stable as LLMs evolve rapidly

Handling Long Configuration Files

If your full clawdbot.json is lengthy, it’s usually better not to paste the entire file into the article.

A common approach is:

Keep only the essential snippets inline
Link to the complete configuration via GitHub or Gist for reference

This keeps the article readable while still being reproducible.

Final Thoughts

LLMs will continue to change faster than most agent systems.

Introducing an LLM gateway is a practical way to decouple agents from model infrastructure, reduce configuration churn, and build more stable AI systems.

Curious how others handle model upgrades and vendor lock-in when building AI agents in production.
Would love to hear different approaches in the comments.