A snapshot test that fails on rewording is not detecting a regression. It is detecting five different sources of variation, only one of which you control, and knowing which is which is what stops a team spending a sprint on it.
The sampler is outside the model
A language model returns a probability distribution over the next token. Something else picks one, and that picker is the sampler, governed by temperature, top_p and top_k. At any temperature above zero the pick is a draw from a distribution, so two identical requests diverge as soon as one token differs — and because generation is autoregressive, one different token changes the context for every token after it. A single early divergence produces an entirely different second half. That is why the diff looks catastrophic for a change that is semantically nothing.
This is also why the size of a snapshot diff carries almost no information about the size of the underlying change. A one-word substitution in sentence two and a completely wrong answer produce diffs of similar shape.
Why temperature zero is not determinism
Setting temperature to 0 requests greedy decoding: always take the highest-scoring token. That removes the sampler as a source of variation and it is worth doing for tests. It does not make the endpoint deterministic, for reasons that sit below the API.
- Floating-point addition is not associative. The reductions inside a forward pass are summed in an order that depends on how the work was split across the hardware, and that split depends on the batch the request landed in. Two runs that group your request with different neighbours can produce logits that differ in the last bits. Where the top two tokens are nearly tied, that is enough to flip the greedy choice — and one flipped token changes everything downstream.
- A tie has to be broken somehow. Greedy decoding assumes a unique maximum. When two tokens score identically the result depends on implementation details you cannot see.
- Seeds are best-effort, and say so. OpenAI documents its
seedparameter as a best-effort attempt at determinism and returns asystem_fingerprintto identify the backend configuration, precisely so you can tell when the thing underneath has changed. Documenting a fingerprint is an admission that the backend changes; treat it as such rather than as a guarantee.
The endpoint under you moves
The remaining two sources are not about decoding at all. An alias like a bare model name resolves to whatever the provider currently points it at, and that pointer moves without your deployment changing — silent model updates are the reason a suite can go red on a morning nobody shipped anything. Pinning a dated or versioned model id removes most of this, at the cost of having to migrate deliberately.
And the request itself may not be what you think. A system prompt assembled from a template, a tool schema serialised with unordered keys, a retrieved document set whose ordering depends on a nearest- neighbour search with ties, a timestamp injected into context: each of these makes today’s prompt differ from yesterday’s while the source code is identical. Before blaming the model, log the exact bytes you sent and diff those.
Telling variance from a regression
The diagnostic is cheap and almost nobody runs it. Take the failing case and call it five times against the current configuration, without touching anything. If the five outputs differ from each other in the same way they differ from the snapshot, the snapshot is testing variance and the test is the bug. If all five agree with each other and disagree with the snapshot, something changed — your prompt, your context assembly, or the endpoint — and the next step is to diff the serialised request against the one recorded when the snapshot was taken.
That distinction is worth automating as a fixture that records the exact request payload alongside every golden file. It converts an argument about whether the model regressed into a two-line diff.
What to assert instead
Nothing here argues against testing model output. It argues against asserting on the one property of that output which carries the least information: its exact bytes. Almost everything else is stable enough to assert. The response parses against its schema. The tool called was lookup_order and it was called once. The answer contains the order id that was in the context and no other order id. finish_reason is stop rather than length. No email address survives into the output. A paraphrase of the question yields the same classification.
Those are properties and invariants, and they fail for reasons you want to hear about. Fuzzy snapshot matching covers the middle ground where you still want a recorded artefact, and invariants for LLM output covers the assertions that hold with no golden file at all.
One more habit is worth building while the mechanisms are fresh. When a snapshot fails, the first question is not “is this acceptable” but “which of the five did this”, and the five leave different fingerprints. Sampler variance moves the prose and leaves the structure intact. A context-assembly bug usually moves the structure too, because the model is answering a subtly different question. An endpoint change tends to move every case in the suite at once rather than one, and often shifts the served model id or the fingerprint alongside it. A genuine prompt regression moves a coherent subset — the cases that exercise the clause you edited — and leaves the rest alone. Reading the failure count and its distribution before reading any individual diff answers the question faster than reading diffs will.
Two of the five mechanisms above are about which endpoint actually served the request. If your suite runs against more than one provider, it is worth asserting on the resolved model identifier that came back rather than the one you asked for — a gateway that reports the served model and the route it took, as Multigrid does, turns “the tests went red overnight” into a single field in the response you can snapshot.
Top comments (0)