How a frozen particle cloud became the most convincing argument I could make about model sameness and the three bugs that nearly made it lie.
The question nobody answers with a picture
Every few weeks a new model ships with a new name, a new version number, and a benchmark table showing it is 3% better at something. The table is usually true and almost never useful. What I actually want to know is simpler and harder:
How much of this model is new?
Not "is it better" — benchmarks cover that. I want to know whether the thing I'm paying more for is a genuinely different model or the same weights wearing a new hat. That is a question about sameness, and the standard tools for answering it are all textual: benchmark deltas, eval scores, vibes.
So I built something that answers it visually. The idea is almost embarrassingly simple:
Send the same prompt to two models. Turn each answer into a cloud of particles — one particle per sampled point. Then try to morph cloud A into cloud B.
If the two answers are byte-identical, the particles do not move at all.
That frozen frame is the whole product. A cloud that locks in place, desaturates, and gets a big IDENTICAL stamp with a counter reading 0 particles moved.
It's called The Identical Twins Test, and this post is about how it works, why the visual framing matters more than I expected, and the three bugs that nearly made it lie.
Why particles, and why "nothing moving" is the point
Most data visualisations are built to show activity. A dashboard that animates is a dashboard that looks like it's working. That instinct is exactly backwards for this problem.
The finding here is an absence. Two models producing the same bytes is a non-event — it's the thing you'd skip past in a diff. If I rendered it as, say, a green checkmark, nobody would feel anything. A checkmark is a claim; a frozen cloud is evidence.
So the entire design goal was to make the absence of motion the loudest thing on screen:
- The cloud locks — not slows down, locks.
- It desaturates, going grey, like something powered off.
- The camera stops orbiting (more on this below — it was a bug at first).
- A large
IDENTICALbadge stamps over the frame. - A counter reads
0 particles moved, and that zero is measured, not hardcoded.
And the signature UI element is a per-character sha256 diff strip: one tick per character position, lit where the two answers differ. On an identical run, the strip is completely dark. A second, independent confirmation that doesn't require reading a number.
The contrast case matters too. When the answers genuinely differ, the cloud explodes outward and reforms into the new shape. Because you've just watched a cloud refuse to move, watching one tear itself apart reads as violent.
Architecture
Two processes, and the split is deliberate.
┌──────────────────────────────────────────────────────────────────────────┐
│ Browser — localhost:5173 │
│ │
│ ┌────────────────────────┐ ┌───────────────────────────────────────┐ │
│ │ React shell │ │ WebGL stage │ │
│ │ App.tsx │ │ particles.ts │ │
│ │ • phase state machine│ │ • 220,000-particle pool │ │
│ │ • run history │ │ • ShaderMaterial + morph │ │
│ │ • JSON export │ │ • lock() / morphTo() │ │
│ └───────────┬────────────┘ └───────────────▲───────────────────────┘ │
│ │ │ SampleResult │
│ │ ┌───────────┴───────────┐ │
│ │ │ sampler.ts │ │
│ │ │ text → canvas → pts │ │
│ │ └───────────────────────┘ │
│ │ fetch('/api/compare') ← Vite dev proxy │
└──────────────┼───────────────────────────────────────────────────────────┘
│
┌──────────────▼───────────────────────────────────────────────────────────┐
│ Node server — localhost:3001 │
│ │
│ index.ts POST /api/compare POST /api/test-connection │
│ model-client.ts callModel(), testModel() │
│ hash.ts sha256(), newRunId(), newNonce() │
└──────────────┬───────────────────────────────────────────────────────────┘
│ fetch POST {baseUrl}/chat/completions
▼
┌──────────────────────────────────────────────┐
│ Any OpenAI-compatible provider │
│ default: https://api.particle.ai/v1 │
└──────────────────────────────────────────────┘
Why is there a server at all? Three reasons, and I want to be explicit because "why isn't this a static site" is the first question anyone asks:
-
The verdict is computed server-side.
hashEqualis decided from the response bytes on the server, so the client cannot accidentally — or conveniently — misreport it. -
The API key is never bundled. It lives in
localStorage, goes tolocalhost:3001, and gets forwarded from there. - CORS. Most providers don't send permissive CORS headers, so a browser-only version dies on the first request.
Stack: Vite 8 + React 19 + TypeScript 5.9 on the front, Express 5 on Node on the back, three.js for the particles. No component library, no state manager, no CSS framework, no OpenAI SDK.
That last omission is deliberate. The provider call is a plain fetch to /chat/completions, which is what makes the app work against any OpenAI-compatible endpoint — particle.ai, OpenAI, Groq, Together, a local llama.cpp server — by changing one text field. An SDK would have bought me convenience and cost me the entire "works with anything" property.
The verdict is a byte comparison
Here's the core of it:
// server/index.ts
const [a, b] = await Promise.all([
callModel('A', { ...shared, model: modelA }),
callModel('B', { ...shared, model: modelB }),
]);
// The verdict is a byte comparison, not a similarity score.
const hashEqual = a.sha256 === b.sha256;
Both models get the same system prompt, the same user message, and temperature 0. Temperature 0 matters: the app is asking a question about determinism, so a divergence should be a real difference between the models rather than sampling noise.
Note what this is not: there's no embedding distance, no fuzzy matching, no similarity threshold. Identical bytes or nothing. That's a much stronger claim, and it's the one worth making visually.
Bug #1: the nonce, or how a cache could fake the entire result
This is the bug that would have destroyed the project's credibility, and it's the one I'm most glad I caught before shipping.
If the provider caches responses, sending the same prompt twice can return the same bytes twice — even when the two models would have answered differently. That manufactures an IDENTICAL verdict out of nothing.
Sit with how bad that is. The app's entire purpose is to detect sameness. A cache makes it detect sameness falsely. Every "finding" it produced would be an artifact of the infrastructure, and the more convincing the visual, the more effectively it would be lying.
The fix is a fresh random nonce on every call:
// server/hash.ts
export function newNonce(): string {
return randomBytes(4).toString('hex');
}
// server/index.ts — the nonce is part of the user message
const userPrompt = `${prompt}\n\n[nonce ${nonce}]`;
Two calls can never share a cache key. The nonce is shown in the UI and included in the JSON export, so any run can be audited. I verified this by running the same prompt five times: five distinct nonces, five identical verdicts, one distinct digest. The stability is real, and it isn't a cache.
Bug #2: the camera was moving
The first version froze the particles. It looked right. I was happy with it.
Then I measured it. The locked cloud drifted 0.77 pixels over 2.5 seconds.
The particles weren't moving — but the camera was still orbiting. So the frame was changing, which means a viewer watching the "frozen" cloud sees motion, and a viewer who sees motion concludes something happened. Which is precisely the opposite of the claim.
"Nothing moves" has to be literally true, so lock() now stops the camera too:
lock(): void {
this.morphing = false;
this.phase = 'identical';
this.material.uniforms.uFreeze.value = 1;
this.material.uniforms.uSaturation.value = 0.55;
this.material.uniforms.uBurst.value = 0;
this.material.uniforms.uProgress.value = 0;
this.targetPositions.set(this.basePositions);
// Stop the camera too. "The particles do not move" has to be literally true,
// so nothing in the frame is allowed to drift — including the viewpoint.
this.orbitSpeed = 0;
this.movedCount = 0;
this.peakDisplacement = 0;
}
Drift after the fix: 0.064 px over 2.5 seconds. Under a tenth of a pixel. No measurable motion.
There's a subtlety in the shader worth calling out. I wanted a gentle breathing motion so a frozen cloud doesn't look like a dead render — but the breathing must not change any particle's position:
// A little breathing motion so a frozen cloud still reads as alive, but the
// particles' *positions* never change when frozen.
float breathe = sin(uTime * 0.9 + aSeed * 6.2831) * 0.06 * (1.0 - uFreeze);
displaced += dir * breathe;
The (1.0 - uFreeze) term zeroes the breathing the instant the cloud locks. The cloud still shimmers very slightly via point size, but no particle moves.
Bug #3: measuring the wrong thing
Here's a trap worth knowing if you ever test WebGL.
The obvious way to verify the cloud rendered is to call gl.readPixels on the canvas and count lit pixels. It returns all zeros. The drawing buffer isn't preserved after compositing, so by the time your test code looks, there's nothing there.
My first test suite confidently reported "0 lit sample points" on a canvas that was visibly full of particles. I spent a while debugging a renderer that was working perfectly.
The fix: measure from actual screenshots. That's how I got the drift numbers above — capture frames, diff them, compute the cloud's centroid movement. It's slower and uglier than reading a buffer, and it's the only thing that measures what the user actually sees.
The particle system
A fixed pool of 220,000 particles in a single THREE.Points with a custom ShaderMaterial.
Why a fixed pool
Reallocating geometry per run means a GPU buffer upload on every comparison and GC hitches during the morph — exactly when the animation has to be smooth. A fixed pool means the only per-run work is writing into two Float32Arrays.
Unused particles aren't hidden with a draw-range trick. They're pushed 90 world units out of frame, which keeps the shader branch-free:
// Particles that do not exist in the incoming layout are pushed out of frame
// rather than left sitting on top of the new text.
float inactive = 1.0 - aActive;
base += normalize(base + vec3(0.001)) * inactive * 90.0;
The morph
Every particle interpolates from its A position to its B position, with a radial burst peaking at the midpoint:
vec3 base = mix(position, aTarget, uProgress);
// Burst impulse: radial kick, peaked at the midpoint of the morph.
vec3 dir = normalize(base + vec3(0.001));
float wobble = hash(vec3(aSeed * 91.7, aSeed * 13.3, uTime * 0.0));
float impulse = uBurst * uBurstRadius * (0.45 + 0.55 * wobble);
vec3 displaced = base + dir * impulse;
The CPU drives two envelopes — easeInOutCubic for position, sin² for the burst:
const t = Math.min((now - this.morphStart) / this.morphDuration, 1);
// easeInOutCubic: slow out of A, fast through the middle, settle into B.
const eased = t < 0.5 ? 4 * t * t * t : 1 - Math.pow(-2 * t + 2, 3) / 2;
this.material.uniforms.uProgress.value = eased;
// Burst envelope peaks at the midpoint of the morph.
const burst = Math.sin(Math.PI * t) ** 2;
this.material.uniforms.uBurst.value = burst;
The wobble term is what stops the burst looking like a perfect sphere inflating. Each particle gets a deterministic per-seed multiplier between 0.45 and 1.0, so the cloud frays instead of expanding uniformly. That one line is most of the difference between "impressive" and "cheap".
The counter is a measurement
0 particles moved is the headline number, so it can't be a hardcoded zero. It's computed by comparing every particle's destination against its origin:
for (let i = 0; i < POOL_SIZE; i += 1) {
const dx = /* … */;
if (dx * dx + dy * dy + dz * dz > 0.0004) moved += 1;
}
this.movedCount = moved;
The epsilon absorbs float noise from normalisation. If the layouts are genuinely identical, every particle lands exactly where it started and the count is genuinely zero. If I ever break the morph, the counter will tell on me.
Turning text into particles
sampler.ts does the boring-but-critical work:
- Draw the text into an offscreen 1400×760 canvas, wrapped and centred.
- Read pixels back, sample on a 4px grid; every point where red > 110 becomes one particle target.
- Normalise into centred world space of height 24 units, with ±0.8 of z-jitter for depth.
The bug that measurement caught and review never would
The first version used a fixed font size. It looked fine in code. Then I measured particle counts per preset:
| Answer | Particles (before) | Particles (after) |
|---|---|---|
"Paris" |
48 — an empty frame | 4,044 |
| A tall column of 20 digits | 145 — a faint streak | 2,206 |
A five-character answer produced 48 particles. The cloud was effectively invisible, and the app looked broken for exactly the short, punchy prompts a user is most likely to try first.
Fixed by fitting the text to the frame:
// Largest font that fits, tried from biggest to smallest.
const FONT_CANDIDATES = [150, 120, 96, 76, 60, 48, 38, 30, 24, 19, 15, 12];
// Then: if the first pass came back thin, resample on a finer grid.
const MIN_PARTICLES = 2_500;
const finerStep = Math.max(1, Math.floor(step * Math.sqrt(count / MIN_PARTICLES)));
Every preset now lands between roughly 2,200 and 7,400 particles.
One subtlety I nearly got wrong: the text block is centred as a whole, not line by line. Per-line centring is the obvious implementation and it destroys the internal alignment ASCII art depends on — a cat drawn in ASCII comes out with its ears in the wrong place. Since ASCII art is one of the most visually satisfying prompt types here, that mattered.
Not leaking chain-of-thought
Reasoning models can return a reasoning_content field alongside content. That's chain-of-thought, and it has no business in a log, an HTTP response, or the DOM.
The defence is structural, not a filter. The field doesn't exist in any type in the codebase, and the one function that reads a completion never looks at it:
interface ChatCompletionResponse {
model?: string;
choices?: Array<{
message?: {
content?: unknown;
// reasoning_content is intentionally NOT read. See extractContent().
};
finish_reason?: string | null;
}>;
usage?: {
completion_tokens_details?: { reasoning_tokens?: number };
};
}
Only the token count crosses the wire, from usage.completion_tokens_details.reasoning_tokens. There's nothing to leak because nothing ever reads it. I verified this with a canary string in a test provider: it never appeared in any response, log, or DOM node.
A filter would have been the obvious approach and the wrong one — filters fail open when someone adds a new code path. A field that doesn't exist can't be accidentally serialised.
"Test connection" — the button that saves the first five minutes
The app requires your own API key. That means the very first thing a new user does is type a key and hit Run, and if anything is wrong — typo'd key, missing /v1, wrong model name — they get a failure and no idea which of three things caused it.
So there's a Test connection button that sends one tiny request (16 max tokens, costs almost nothing) to each model and classifies the failure:
if (status === 401 || status === 403) {
return 'The API key was rejected. Check that the key is complete and belongs to this base URL.';
}
if (status === 404) {
if (lower.includes('model')) {
return `The provider does not recognise the model "${model}". Check the model name.`;
}
return 'The endpoint was not found. Check the base URL — it should end in /v1 for most providers.';
}
A bad key and a bad model name are different problems with different fixes. Conflating them into "request failed" wastes the user's time at the exact moment they're deciding whether your app works.
Two details I'm happy with:
- The result clears the moment you edit any connection setting. A stale "OK" can never describe a config you've since changed — a small thing that prevents a genuinely confusing failure mode.
- Probes run concurrently, so testing takes as long as the slower model, not both.
What I'd tell someone building something similar
Make the absence loud. The instinct to visualise activity is strong and often wrong. If your finding is a non-event, the design problem is making that non-event feel like a verdict, and that's a harder and more interesting problem than adding more motion.
Measure the thing you're claiming. "The particles don't move" is a claim, and 0.77px of camera drift made it false. Every headline number in this app — the moved count, the drift, the particle counts — is measured rather than asserted. It caught three bugs that looked fine in review.
Structural safety beats filters. Not having reasoning_content in a type is stronger than scrubbing it before logging. Filters fail open; absent fields can't be serialised by accident.
Beware the bug that fakes your own finding. The nonce exists because a cache could have manufactured IDENTICAL verdicts. When your app's purpose is detecting a specific condition, ask what could produce that condition falsely — that's where the credibility risk lives.
Try it
The app is open source, runs locally in two commands, and works against any OpenAI-compatible provider.
git clone https://github.com/harishkotra/token-morph
cd token-morph
npm install
npm run dev
Open localhost:5173, add your key in Settings, hit Test connection, then pick the "Recursion, one sentence" preset and watch a cloud refuse to move.
The most dramatic preset is "Count to 20, one per line" — the two models lay the numbers out so differently that the cloud's silhouette barely survives the morph. The most subtle is "Ocean in 200 words", where the shape holds and only the density changes.
If you find a model pair that's byte-identical on something it really shouldn't be, I'd love to see it.
Code & more: https://www.dailybuild.xyz/project/254-token-morph
Top comments (0)