Every model release ships with a benchmark table. MMLU up 2.3 points. GPQA up 4.1. The implication is always that the new model is meaningfully better, and that you, a user of these things, will notice.
I wanted to test the part the benchmarks don't cover: can a person actually tell?
So I built a game. Five prompts. Two models. Ten answers, shuffled, shown one at a time on a floating 3D card. You guess which model wrote each one. At the end it scores your intuition and tells you how often you were just flipping a coin.
The first time I ran it, the answer was: always. All five prompts came back byte-identical from both models.
This is a write-up of how it works, what it found, and the design decisions that make the finding trustworthy rather than a cute demo.
The finding
Default configuration: deepseek-v4-flash-0731 vs deepseek-v4.1-flash, both on the same endpoint, temperature 0.
All five prompts returned byte-identical text from both models. Not similar. Identical — same sha256, all five pairs.
Two things are simultaneously true, and this is the interesting part:
-
The two model ids are genuinely distinct. The provider accepts both, and rejects invented names —
totally-bogus-model-xyzreturnsmodel_not_found. So they are not the same string, and the endpoint is not silently ignoring themodelfield. -
On this endpoint they behaved as one model. Every response came back self-reporting as
deepseek-v4.1-flash, including the responses to requests fordeepseek-v4-flash-0731.
So: two distinct, valid, separately-billed model identifiers, served by one endpoint, produced identical bytes on every prompt I tried.
Now — I want to be careful here, because there are several explanations and I can only distinguish some of them from outside:
- The two ids may route to the same weights.
- They may route to different weights that happen to be near-identical at temperature 0.
- The endpoint may map one id onto the other.
What I can say with confidence is what I measured: the ids are distinct, the endpoint echoes one name for both, and the outputs are byte-identical at temperature 0 on these five prompts. The app reports exactly that and no more.
And the honest caveat: at temperature 1.0 the two models diverge on 3 of the 5 prompts. Sampling noise is doing work that the models aren't. A single sample at temperature 0 is one data point, and the app should not pretend otherwise.
Cross-provider is what makes the game playable
Once I added multi-provider support, I could put a local model against a hosted one:
| Configuration | Identical pairs (of 5) |
|---|---|
deepseek-v4-flash-0731 vs deepseek-v4.1-flash, both Particle.ai, temp 0 |
5 / 5 |
deepseek-v4-flash-0731 (Particle.ai) vs google/gemma-4-e4b (local LM Studio), temp 0 |
1 / 5 |
Two sibling models from the same provider: indistinguishable. A hosted model against a small
local one: 1 pair out of 5, and now the guessing is real. That contrast is the whole product.
Architecture
┌──────────────────────────────────────────────────────────────────────────────┐
│ Browser — localhost:5173 │
│ │
│ React 18 + TypeScript │
│ ┌────────────────┐ ┌──────────────────┐ ┌────────────────────────────┐ │
│ │ App.tsx │ │ SettingsPanel │ │ ScoreScreen │ │
│ │ phase machine │ │ provider CRUD │ │ score · breakdown · truth │ │
│ │ │ │ presets · probe │ │ receipt · PNG share card │ │
│ └───────┬────────┘ └──────────────────┘ └────────────────────────────┘ │
│ │ │
│ │ QuizScene.ts — three.js │
│ │ RoundedBoxGeometry card · real 3D flip · parallax tilt │
│ │ glowing guess panels · pulsing ring · IDENTICAL stamp │
│ │ │
│ │ localStorage: wmwt.config.v2 │
└───────────┼──────────────────────────────────────────────────────────────────┘
│ fetch /api/* (Vite dev proxy → 127.0.0.1:3001)
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ Quiz server — 127.0.0.1:3001 │
│ Node + TypeScript + Hono │
│ │
│ index.ts routes · CORS · the public-payload allowlist │
│ quiz.ts config resolution · generation · ground-truth binding │
│ modelClient.ts callModel · sha256 · retries · model discovery │
│ store.ts in-memory store · shuffle · score · verdict · JSONL log │
│ prompts.ts 5 prompts · system prompt · provider presets │
│ │
│ ★ The only thing that knows which model wrote which text ★ │
└──────┬───────────────────────────────────┬───────────────────────────────────┘
│ POST {baseUrl}/chat/completions │ GET {baseUrl}/models
│ plain fetch, no SDK │ (discovery, proxied for CORS)
▼ ▼
┌────────────────────────┐ ┌──────────────────────────────────────────────────┐
│ Provider A │ │ Provider B │
│ Particle.ai (hosted) │ │ LM Studio / Ollama / llama.cpp / OpenAI / ... │
│ needs an API key │ │ local: no key needed, no CORS headers │
└────────────────────────┘ └──────────────────────────────────────────────────┘
│
▼
data/quiz-log.jsonl · data/quiz-<id>.json · data/verification-report.json
The shape of that diagram is the security model. The server is the only thing that knows which model wrote which text. The browser is told the model names — they have to label the buttons — but never the per-item attribution until a guess is recorded.
Stack choices
| Layer | Choice | Why |
|---|---|---|
| Frontend | Vite 6, React 18, TS 5.7 strict | Fast HMR; strict TS catches the wire-shape mistakes that would leak truth |
| 3D | three.js 0.171 | Real 3D flip, not a CSS rotateY
|
| Backend | Hono 4 on @hono/node-server
|
Tiny, typed, no framework ceremony |
| Model calls | plain fetch
|
No SDK means any OpenAI-compatible endpoint works, including local ones |
| Storage | in-memory Map + JSONL |
No database for a toy; the JSONL is the durable artifact |
The "no SDK" decision turned out to matter more than I expected. Because callModel speaks raw HTTP, adding LM Studio and Ollama support was a config change, not an integration project.
The decisions that make the finding trustworthy
A game that says "you can't tell them apart" is only interesting if the game isn't cheating. These are the parts I'd defend in review.
1. Ground truth is bound at call time
The single most important invariant. The slot that produced a piece of text is the slot recorded on that item. Nothing downstream re-derives attribution. Nothing infers it from the text. There is no classifier, no heuristic, no "this one feels more like the newer model."
// server/src/quiz.ts
const results = await Promise.all(
(['A', 'B'] as const).map(async (slot) => {
const model = slot === 'A' ? config.modelA : config.modelB;
const provider = slot === 'A' ? config.providerA : config.providerB;
const itemId = newId(9);
const { text, record } = await callModel({
config, provider, model, slot, prompt, promptIndex, itemId,
});
return { slot, model, itemId, text, record };
}),
);
const [first, second] = results;
// Identical detection is a sha256 comparison of the exact output bytes.
const identical = first.record.sha256 === second.record.sha256;
results.forEach((r, index) => {
items.push({
itemId: r.itemId,
prompt,
text: r.text,
trueModel: r.slot, // ← bound here, at call time
trueModelName: modelNames[r.slot],
sha256: r.record.sha256,
identicalPair: identical,
pairId,
promptIndex,
});
});
The slot is a closure variable. It cannot drift from the request that produced the text, because it is the request.
2. Public payloads are built by allowlist, not by deletion
The obvious way to hide the answer is delete item.trueModel. That leaks by default: the day someone adds a field to QuizItem, it ships to the browser unless they remember to delete it too.
Instead the public shape is constructed field by field:
// server/src/index.ts
function toPublicItem(item: QuizItem): { itemId: string; prompt: string; text: string } {
return { itemId: item.itemId, prompt: item.prompt, text: item.text };
}
Three fields, named explicitly. A new field on QuizItem is private until someone deliberately adds it here. This is the same reasoning as allowlisting in an ORM serializer or a GraphQL resolver — it just matters more when the hidden field is the answer to the game.
Verified: both the quiz-creation response and the pre-guess GET expose exactly itemId, prompt, text.
3. Identical pairs are detected by hash and kept
export function sha256(text: string): string {
return createHash('sha256').update(text, 'utf8').digest('hex');
}
Equal hashes mean byte-identical output, not "similar." And crucially, identical pairs are not filtered out — they're the most revealing rounds in the quiz. The reveal stamps them IDENTICAL in amber, and the verdict counts them:
"You could not tell them apart — 80% of your guesses were coin flips. 1 of the 10 items were byte-identical from both models — those rounds were coin flips by construction."
That second sentence is the one that lands. It's the difference between "you did badly" and "this round was unwinnable."
4. Reasoning is counted, never captured
Reasoning models return a reasoning_content field containing their scratchpad. It's fascinating and it is also a massive attribution leak — the reasoning trace often names the model or reveals a distinctive style. So: read it only to confirm it exists, then drop it on the floor. Keep the count.
interface ChatChoiceMessage {
content?: unknown;
/** Present on some providers. Read ONLY to prove we never surface it. */
reasoning_content?: unknown;
}
const reasoningTokens = numberOrZero(
usage.completion_tokens_details?.reasoning_tokens ?? usage.reasoning_tokens,
);
The app can prove this is a real distinction, not a claim: with reasoning enabled the same endpoint returns 35 reasoning tokens and 104 characters of reasoning_content; with it disabled, zero and empty. A full quiz with reasoning on logged 2,146 reasoning tokens across 10 calls (123–525 each) — and not one character of the text. The verifier asserts reasoning_content appears nowhere in any response body.
5. The score is computed server-side
From stored guess records, not from anything the client reports back:
export function computeScore(quiz: StoredQuiz) {
let score = 0;
let identicalItemsGuessedCorrectly = 0;
for (const item of quiz.items) {
const guess = quiz.guesses.get(item.itemId);
if (!guess) continue;
if (guess.correct) {
score += 1;
if (item.identicalPair) identicalItemsGuessedCorrectly += 1;
}
}
return { score, total: quiz.items.length, misses: quiz.guesses.size - score, identicalItemsGuessedCorrectly };
}
Guessing is idempotent — re-guessing returns the stored verdict rather than double-counting. A client that lies about its own score changes nothing.
6. The Receipt
The claim "these two models are the same" is extraordinary, so the app prints its evidence on the score screen:
Receipt
10 live calls to the provider — every item above came from one of them.
1,676 completion tokens, 1,111 of them reasoning tokens (counted from
usage.completion_tokens_details.reasoning_tokens; the reasoning text itself is never stored or shown).
Model A on Particle.ai (https://api.particle.ai/v1): requested
deepseek-v4-flash-0731, provider reported deepseek-v4.1-flash — the provider did not echo the model you asked for Model B on LM Studio (local) (http://localhost:1234/v1, local): requested google/gemma-4-e4b, provider reported google/gemma-4-e4b
That's the whole argument in five lines: real calls, counted tokens, requested vs. reported model, and the endpoint each one came from. It's auditable rather than asserted. And it's generated from the same CallRecord objects the verifier checks, so it can't drift from reality without the tests failing.
Multi-provider: the feature that made the game good
Originally the app took a single base URL and two model names. That was a mistake — it assumed the interesting comparison is two siblings on one endpoint.
A provider is now a first-class object, and each model slot binds to one:
export interface ProviderConfig {
id: string;
label: string;
baseUrl: string;
apiKey: string;
headers?: Record<string, string>;
disableReasoning: boolean;
}
export interface QuizConfig {
providers: ProviderConfig[];
modelA: string;
providerAId: string;
modelB: string;
providerBId: string;
temperature: number;
maxTokens: number;
}
The server receives each slot already bound to a concrete endpoint, so it stays stateless with respect to the provider list:
export function toServerConfig(config: QuizConfig): ServerQuizConfig {
return {
providerA: providerFor(config, 'A'),
modelA: config.modelA,
providerB: providerFor(config, 'B'),
modelB: config.modelB,
temperature: config.temperature,
maxTokens: config.maxTokens,
};
}
Four problems that only show up with local models
1. Local endpoints need no API key — and some reject a bad one. Sending
Authorization: Bearer with an empty token makes some runtimes unhappy. So the header is
omitted entirely when the key is blank:
const headers: Record<string, string> = {
'Content-Type': 'application/json',
...(provider.headers ?? {}),
};
if (provider.apiKey.trim().length > 0) {
headers.Authorization = `Bearer ${provider.apiKey.trim()}`;
}
2. Local runtimes send no CORS headers, so the browser can't call GET /models directly. Model discovery is therefore proxied through the server:
POST /api/providers/models { provider } → GET {baseUrl}/models
It handles the OpenAI shape ({data:[{id}]}), the Ollama-native shape ({models:[{name}]}),
and bare string arrays, because all three turn up in practice.
3. "Disable reasoning" had to move from global to per-provider. One model I tested rejects the toggle outright, in its own words:
"This model always reasons and does not support disabling reasoning; do not send a reasoning-off selector."
A global toggle can't express that when slot A and slot B are on different endpoints. Provider errors are surfaced verbatim, so you see that sentence rather than a generic 400.
4. Reasoning models can eat the entire budget and return nothing. Measured on LM Studio at
max_tokens: 1600:
| Model | Result |
|---|---|
google/gemma-4-e4b |
works — 76 chars, 190 reasoning tokens |
qwen/qwen3.5-9b |
empty content — spent all 1599 tokens reasoning |
zai-org/glm-4.7-flash |
empty content — spent all 1599 tokens reasoning |
LM Studio ignored chat_template_kwargs: {enable_thinking: false}. The fix is a retry with a
doubled budget, and an error message that names the actual remedy:
if (emptyRetries < 1 && attempts < MAX_HTTP_ATTEMPTS) {
emptyRetries += 1;
retried = true;
budget = config.maxTokens * 2;
log?.(`empty content from ${model} on ${provider.label} — retrying with a doubled budget of ${budget} tokens`);
continue;
}
// …
const reasoningHint = lastEmptyReason.includes('reasoning_tokens=0') ? ''
: ' The model spent its entire budget on reasoning and never emitted an answer — ' +
'raise Max Tokens in Settings, or enable "Disable reasoning" for this provider.';
One more thing I didn't expect to need: retries
A live 429 from the hosted provider killed an entire quiz mid-verification. Ten calls is
enough requests that rate limits are a real hazard. So transient statuses now back off:
const TRANSIENT_STATUS = new Set([408, 409, 425, 429, 500, 502, 503, 504, 529]);
const MAX_HTTP_ATTEMPTS = 3;
if (TRANSIENT_STATUS.has(res.status) && attempts < MAX_HTTP_ATTEMPTS) {
const waitMs = retryAfterMs(res) ?? 500 * 2 ** (attempts - 1);
log?.(`transient ${res.status} from ${provider.label} — retrying in ${waitMs}ms …`);
await sleep(waitMs);
continue;
}
Retry-After is honoured when present, in both the seconds and HTTP-date forms, capped at 30s.
The 3D card
The card is a real 3D object. A RoundedBoxGeometry box carries six materials — four sides, a front face, and an invisible rear face — so the flip reveals a separate back plate rotated π behind it:
const boxGeo = new RoundedBoxGeometry(CARD_W, CARD_H, CARD_D, 6, 0.16);
// BoxGeometry material order: +x, -x, +y, -y, +z (front), -z (back)
const boxMaterials = [side, side, side, side, frontMaterial, invisibleMaterial];
const backGeo = new THREE.PlaneGeometry(CARD_W - 0.08, CARD_H - 0.08);
const backPlate = new THREE.Mesh(backGeo, this.backMaterial);
backPlate.rotation.y = Math.PI;
backPlate.position.z = -CARD_D / 2 - 0.004;
The flip eases toward π, and pointer parallax fades out as it progresses — otherwise the tilt fights the rotation and the card looks like it's wobbling rather than turning:
this.flip += (this.targetFlip - this.flip) * Math.min(1, dt * 4.2);
const flipProgress = Math.min(1, this.flip / Math.PI);
const tiltFade = 1 - flipProgress;
this.cardGroup.rotation.y = this.flip + this.tilt.y * tiltFade;
// IDENTICAL stamp pops in once the flip is mostly done.
const stampGate = flipProgress > 0.55 ? this.stampTarget : 0;
OrbitControls are disabled during play and enabled on the score screen, so the audience can't orbit around and read the back of the card before guessing.
I verified the flip by pixel analysis rather than by trusting my eyes on a screenshot: the card face reads rgb(185,185,201) before the flip, the back face averages luminance 37.6 after it (mean absolute delta 138/255), and the amber IDENTICAL stamp covers 5,344 px — 0.72% of the card region — appearing only after reveal.
Verification: 49 assertions, run against a live quiz
Claims in a README are cheap. server/src/verify.ts runs a real quiz end to end over HTTP and
asserts every one of them, in 8 sections:
1. Generate a quiz from real model calls
PASS 10 items returned
PASS every item has non-empty real text — min chars=66
PASS slot A served by the configured provider — Particle.ai @ https://api.particle.ai/v1
PASS slot B served by the configured provider — LM Studio (local) @ http://localhost:1234/v1
2. Pre-guess payload does not leak the answer
PASS GET body contains no "trueModel" key
PASS GET body contains no "sha256"
PASS GET items expose only itemId/prompt/text — keys=itemId,prompt,text
5. Attribution matches the actual call that produced each item
PASS item 2TT1PfIAcE9n ← B (google/gemma-4-e4b) — provider=LM Studio (local)
requested=google/gemma-4-e4b reported=google/gemma-4-e4b sha256=3d95979d3c1d…
PASS item _4na2pidfFkU ← A (deepseek-v4-flash-0731) — provider=Particle.ai
requested=deepseek-v4-flash-0731 reported=deepseek-v4.1-flash sha256=5eb74cd2d127…
6. Identical pairs detected by sha256 and flagged
PASS flagged item z6AbtBxiNQ1b really is byte-identical to its counterpart
PASS no identical pair was left unflagged — 0 unflagged identical item(s)
7. Reasoning is counted, never captured
PASS reasoning token counts come from usage.completion_tokens_details — total=1113
PASS no reasoning_content anywhere in the payload
PASS the two slots were answered by two different endpoints
Note the shape of section 5: it re-computes sha256(row.text) from the returned text and compares it to the hash recorded at call time. That's an independent check that the text shown to the player is the text the model produced — not a re-generation, not a cache, not a substitution.
And section 6's last check is the one that keeps the flag honest: it verifies that no identical pair was left unflagged. Flagging is not allowed to be selective.
Run it yourself:
cd server
PARTICLE_AI_API_KEY=... \
VERIFY_BASE_URL_B='http://localhost:1234/v1' \
VERIFY_LABEL_B='LM Studio (local)' \
VERIFY_MODEL_B='google/gemma-4-e4b' \
pnpm verify
It writes data/verification-report.json and exits non-zero on any failure. Current status: 49/49 passed on a cross-provider quiz.
What I'd tell someone building this
Hash, don't eyeball. The moment I compared sha256 values instead of reading outputs, the finding went from "these look similar" to "these are the same bytes." It also made the identical-pair feature trivial: one comparison, no threshold, no fuzz.
Design so the leak is impossible, not merely absent. The allowlist projection and the call-time slot binding aren't defensive coding, they're the product. A guessing game whose answer can leak isn't a measurement.
Bind truth at the moment of the call. Every alternative — classifying outputs, inferring from style, matching against a reference — introduces exactly the bias the experiment is trying to measure.
Build the receipt. If your app makes a surprising claim, print the evidence next to it. The Receipt turned "trust me" into "check me," and it caught the same-model-name behaviour I would otherwise have missed entirely.
Report your own error bars. 10 items is a small sample. At n=10, 50% and 60% are not reliably distinguishable. The app should say so — and if I extend this, a confidence interval on the score is the first thing I'd add.
Test against a real local runtime. Every genuinely interesting bug in this project — empty content from reasoning models, ignored reasoning toggles, missing CORS headers, enable_thinking rejections — only appeared when I pointed it at LM Studio. A hosted API would have hidden all four.
Try it
git clone https://github.com/harishkotra/which-model-wrote-this
cd which-model-wrote-this
pnpm install
pnpm dev
Open the web URL, click Settings, add a provider, paste your key if it's a hosted one. If you have LM Studio running, add it as a second provider and put the two slots on different endpoints — that's the configuration where the game actually works.
Then play it and find out whether you can tell. Most people can't.
Code & more: https://www.dailybuild.xyz/project/260-which-model-wrote-this
Top comments (0)