A 3D heatmap of 320 model calls, and the four bugs that almost made it lie.
Temperature 0 is supposed to mean deterministic. Same prompt in, same bytes out, every time.
It's the setting people reach for when they need reproducible output, and it's the setting most of us assume makes an LLM behave like a function.
I wanted to know: is that actually true, and if not, which prompts break it?
So I built a small experiment. Eight prompts, two models, ten repetitions each — 160 calls per run, 320 total. Then I rendered the whole thing as a 3D heatmap where colour is determinism and height is how many reasoning tokens the model burned getting there.
Here's what came out, and what it cost me to trust it.
The design
Each bar is one prompt × one model. Colour is the determinism score:
determinismScore = count(most common sha256) / reps
That's a deliberately brutal metric. I hash the raw response text and count how many of the ten repetitions produced byte-identical output. Two responses differing by a single trailing space are different responses. If a model produces the same answer with different formatting, it scores low — and that's the point, because "the same answer" and "the same bytes" are very different guarantees for anything downstream that caches, diffs, or replays output.
Height is the mean reasoning tokens. That pairing is the whole visual argument: a bar that is tall and red is a prompt that is expensive and unstable — the worst quadrant, and the one you want to find before it finds you in production.
The prompt spectrum
Eight prompts chosen to span the range from "there is exactly one right answer" to "there is no right answer":
| # | Prompt | Intent |
|---|---|---|
| 1 | List the first 12 primes, comma-separated, nothing else | Exact recall |
| 2 | What is 847 × 293? Show steps, final number on its own line | Maths |
| 3 | Output a 5×5 multiplication table as a Markdown table | Spatial / formatting |
| 4 | Explain what a hash function is in one sentence | Constrained prose |
| 5 | Draw an ASCII cat, 5 lines | Structured art |
| 6 | Write a haiku about the ocean | Creative, fixed form |
| 7 | Tell me a joke about programmers | Creative, conventional form |
| 8 | Describe a city that exists between sleeping and waking | Open creative |
My hypothesis, stated before running anything: prompts 1–4 would be green (deterministic), 6 and 8 would be red, and the newer model's column would be greener than the older one's.
Two of those three predictions were wrong. More on that below.
The thing that makes or breaks the experiment: nonces
Before any of the visuals, there's a subtlety that invalidates the entire experiment if you get it wrong.
If you send the identical request ten times, a provider-side response cache will happily return the identical bytes ten times. Your heatmap then measures the cache, not the model. You'd get a beautiful, entirely green grid and conclude that LLMs are perfectly deterministic - a completely false result produced by a completely reasonable methodology.
The fix is to make every request genuinely unique while keeping the semantic content identical:
// server/src/sweep.ts
export function withNonce(base: string, nonce: string): string {
return `${base}\n\n<!--nonce:${nonce}-->`;
}
Each call appends an HTML comment containing 8 bytes of fresh randomness. The model sees a different string every time; the task is unchanged. If a cache is in play, it can't match.
And because "unique nonce" is the load-bearing assumption, it's asserted, not assumed:
// server/src/nonce.ts
issue(): string {
for (let i = 0; i < 1000; i++) {
const nonce = randomHex(8);
if (!this.seen.has(nonce)) {
this.seen.add(nonce);
this.issuedCount++;
return nonce;
}
this.duplicates.add(nonce);
}
throw new Error('nonce generation failed: 1000 consecutive collisions');
}
The registry is seeded from the persisted log on startup, so uniqueness holds across restarts - not just within one process. And the audit re-checks it from the raw file afterwards. If any nonce is reused, the numbers are discarded, not caveated. I verified the check actually
fires by injecting a duplicate into a copy of the log: the audit prints FAIL — determinism numbers are invalid and must be discarded and exits 1. A check that has never failed is not a check.
What the data actually showed
Two independent runs, 160 calls each. Model A = deepseek-v4-flash-0731, Model B =
deepseek-v4.1-flash, temperature 0.
| Prompt | Predicted | A det | B det | A reasoning | B reasoning |
|---|---|---|---|---|---|
| Exact recall | green | 0.60 | 0.60 | 53 | 53 |
| Maths | green | 0.60 | 0.70 | 168 | 195 |
| Spatial / formatting | green | 0.70 | 0.90 | 343 | 328 |
| One-sentence explanation | green | 0.30 | 0.40 | 79 | 80 |
| ASCII drawing | amber | 0.40 | 0.40 | 216 | 164 |
| Haiku | red | 0.10 | 0.10 | 373 | 321 |
| Joke | red | 1.00 | 0.90 | 59 | 51 |
| Open creative | red | 0.10 | 0.10 | 1390 | 1598 |
Prediction 1: the creative prompts are red and tall — TRUE
The haiku scored 0.10 on both models. The open-creative prompt scored 0.10 on both. And the open-creative cell is the tallest bar in the grid: ~1,400–1,600 mean reasoning tokens, against ~50 for the joke.
That's the expensive-and-unstable quadrant, confirmed. The model thinks hardest exactly where it has the least reason to converge.
Prediction 2: exact recall and maths are green — FALSE
This is the interesting one.
"List the first 12 prime numbers" scored 0.60. The model never got the primes wrong. Not once. What varied was the comma spacing:
2,3,5,7,11,13,17,19,23,29,31,37
2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37
A coin flip between two typographic conventions, on a question with exactly one correct answer. The content was perfectly deterministic. The bytes were not.
Maths scored 0.60–0.70 for the same reason. The answer was always correct — 248,171, every single time. But the working was formatted three different ways: (847 × 200), 847 × 200, and 847×200.
So my mental model was wrong. The green cells aren't "closed-form questions". They're questions with exactly one natural rendering. The Markdown table (0.70–0.90) and the joke (0.90–1.00) both have a single strongly conventional form, and the model reproduces it. Primes and arithmetic have a correct answer but no canonical format, and the model free-varies across the space of acceptable renderings.
That's a more useful finding than the one I set out to confirm, and it has a direct practical consequence: if you're caching or diffing LLM output, correctness is not the property you should be relying on. A model that is right every time can still produce different bytes every time.
Prediction 3: the newer model is more deterministic — FALSE
Across both runs: Model A mean 0.494, Model B mean 0.476. A 0.018 gap. Model B was ahead in only 5 of 16 cells, with 7 ties.
There is no clear older-to-newer determinism progression in this data. I'd have liked a clean
"newer model = more reliable" story, but the honest read is that the interesting axis here is
the prompt, not the model. The spread between the best and worst prompt (1.00 vs 0.10) is
fifty times larger than the spread between the two models.
Reproducibility
I ran the whole thing twice. Pearson r = 0.93 on per-prompt determinism. Haiku and
open-creative pinned at 0.10 both times. Max per-cell drift was 0.30, concentrated in the
mid-range prompts. The shape of the landscape is stable even where individual cells move —
which is what you want from a measurement you're going to make claims from.
The architecture
┌──────────────────────────────────────────────────────────────┐
│ BROWSER Vite + React + TypeScript :5173 │
│ │
│ App.tsx ──► Settings / Prompts / Overlay / Tooltip / Table │
│ ├──► heatmap.ts InstancedMesh + OrbitControls + PNG │
│ └──► color.ts Oklab perceptual ramp │
└───────────────────────┬──────────────────────────────────────┘
│ fetch /api/* SSE /stream, /replay
▼
┌──────────────────────────────────────────────────────────────┐
│ SERVER Hono + TypeScript :3001 │
│ │
│ index.ts routes, SSE, key resolution │
│ manager.ts sequential engine, events, cancel, replay │
│ provider.ts fetch → /chat/completions, retry, accounting │
│ nonce.ts nonce registry (asserted unique) │
│ sweep.ts aggregation + nonce audit │
│ store.ts append-only JSONL, crash-tolerant reader │
│ audit.ts independent re-derivation from raw JSONL │
└───────────────────────┬──────────────────────────────────────┘
│ one call at a time, never parallel
▼
POST {baseUrl}/chat/completions
user: <prompt>
<!--nonce:9f3ac41b7e0d2c85-->
│
▼
server/data/sweeps.jsonl ──► exports/ (PNG + JSONL + manifest)
Calls are strictly sequential
Every call runs one at a time. Parallelising would be faster and would ruin the data: latency
and token accounting get distorted by contention, and the whole point is that the numbers are
trustworthy. A 160-call sweep takes 6–10 minutes, and that's the correct trade.
Storage is append-only JSONL
Every record is appended the moment it completes. Nothing is ever rewritten. A crash loses at
most the in-flight call:
// server/src/store.ts
export async function appendLines(lines: JsonlLine[]): Promise<void> {
if (lines.length === 0) return;
await ensureDataDir();
const payload = lines.map((l) => JSON.stringify(l)).join('\n') + '\n';
await appendFile(JSONL_PATH, payload, 'utf8');
}
The reader skips a torn final line rather than corrupting the file, so a hard kill mid-write is
survivable. The JSONL is the source of truth — the UI is a view over it, and a page reload
re-attaches to a sweep in progress.
Progress streams over SSE
A long sweep must never look frozen. The server emits snapshot on connect (so a late
subscriber is never behind), then progress and cell events as each call lands, plus a 5s
heartbeat.
Reasoning text is never stored
The provider returns reasoning_content alongside the answer. I record only its token
count:
function extractReasoningTokens(json: ChatResponse): number {
const n = json.usage?.completion_tokens_details?.reasoning_tokens;
return typeof n === 'number' && Number.isFinite(n) ? n : 0;
}
The CallRecord type has no field capable of holding the reasoning text. Not "we don't log it" —
there is nowhere to put it. That's a structural guarantee rather than a discipline, and the
audit scans the raw file to confirm it holds.
Four bugs that would have made this lie
The interesting part of this project wasn't writing it. It was the four defects that only
surfaced by actually running the thing and checking the output, rather than assuming it worked.
1. The route that silently swallowed page reloads
/api/sweep/latest was registered after /api/sweep/:id. Hono matches in registration order,
so latest was captured as an id. Every page reload silently failed to re-attach to the running
sweep. No error, no warning — it just quietly did nothing. Fix: register the literal route first.
2. Tone mapping made the legend a liar
I had ACES filmic tone mapping enabled, which is a sensible default for pretty 3D. But this is a
data visualisation: a cell's rendered colour has to equal the colour shown in the legend. Tone
mapping was desaturating everything, so the green cells weren't the green in the ramp.
I only caught it because I wrote a pixel-level assertion — decode the exported PNG, count green
pixels, fail if there are none. After switching to NoToneMapping, green pixels went from
237 to 6,551. The bug was invisible to the eye and obvious to the assertion.
3. The CLI reported a number that was always 1.00
The headless runner printed each cell's determinism as soon as the first repetition landed. With
one sample, the most common hash is trivially the only hash — every cell reads 1.00. It looked
like perfect determinism everywhere. Fix: wait for all reps before reporting a cell.
4. A dropped event caused an infinite loop
The terminal done event was written like this:
void stream.writeSSE({ event, data: JSON.stringify(payload) });
if (event === 'done') { unsubscribe(); resolve(); }
The write isn't awaited. The stream closes the instant the handler returns, so done never
flushed. The browser's EventSource saw the connection drop and — correctly, per spec —
auto-reconnected, which restarted the replay from the beginning. Forever.
It presented as a hung UI. The fix is to await the write before resolving:
void (async () => {
try { await stream.writeSSE({ event, data: JSON.stringify(payload) }); }
catch { /* client already gone */ }
if (event === 'done') { clearInterval(heartbeat); unsubscribe(); resolve(); }
})();
Four bugs, none of which threw an error. Every one of them produced plausible-looking output
that was wrong.
Verification, so you don't have to trust me
The repo ships the raw evidence and an independent audit:
pnpm audit # re-derives every claim from the raw JSONL
pnpm verify:ui # drives the real UI in a browser, hovers a cell, exports the PNG
pnpm verify:png # decodes the exported bitmap and asserts it's readable
pnpm audit re-reads the JSONL and checks four things independently:
| Claim | Result |
|---|---|
| Every call carries a unique nonce | PASS — 320 calls, 320 distinct nonces, 0 reused |
| Determinism derives from real sha256 of responses | PASS — 320/320 hashes match recomputed |
Reasoning tokens from usage.completion_tokens_details
|
PASS — 320/320 well-formed |
reasoning_content never stored |
PASS — absent from every record and the raw file |
The exported PNG and the JSONL live in exports/ alongside a manifest, so the claim is
auditable rather than asserted.
What I'd tell you to take away
Determinism is a property of the prompt, not the model. The spread across prompts (1.00 to
0.10) dwarfs the spread across models (0.494 vs 0.476). If you care about reproducibility,
choose your prompts accordingly — and measure, because intuition is a poor guide here. I
predicted two of my eight prompts wrong.
Correctness and reproducibility are different guarantees. A model can be right every single
time and still emit different bytes every single time. If you're caching, diffing, or replaying
output, you need the second property, and you don't get it for free at temperature 0.
Creative prompts are where the compute goes. The open-creative prompt burned ~30× the
reasoning tokens of the joke, at the same determinism score. Both are red; only one is
expensive. The heatmap makes that visible in a way a table doesn't.
And the meta-lesson: every one of the four bugs above produced output that looked fine. The
nonce assertion, the pixel assertion, and the independent audit weren't ceremony — they were the
only reason I caught them.
Run it yourself
git clone https://github.com/harishkotra/determinism-heatmap.git
cd determinism-heatmap
pnpm install
pnpm start # → http://localhost:5173
Point it at any OpenAI-compatible endpoint via Settings — base URL, key, and both model names
are editable. The heatmap on load is replayed from disk at zero token cost; press ▶ Run sweep to take fresh measurements, or ↻ Replay to watch a finished run fill in cell by
cell without spending anything.
Change the prompts. That's where the interesting variance is.
Code & more: https://www.dailybuild.xyz/project/255-determinism-heatmap
Top comments (0)