Once exact matching is off the table, three techniques do the work. Each detects a different class of change and is blind to a different one, and picking without knowing which blindness you are accepting is how a suite ends up green through a real regression.
Structural diff
The cheapest and by far the most under-used. Do not compare the text; compare a structure derived from it, and compare that structure exactly. For a chat response the structure is the response metadata plus whatever your code parses out: the finish reason, the tool call names and their argument keys, the number of citations, the presence of a refusal, the section headings if the format is fixed, the count of list items.
It detects format drift, tool-selection changes, truncation and refusals with no threshold to tune and no second model in the loop. It is completely blind to content: a response with the right shape and the wrong number passes. That is acceptable exactly when the content is checked somewhere else — which is the argument for combining it with the third technique rather than treating it as a rival.
Structural diffing is also the only one of the three that stays readable in a pull request, because the artefact is a small object rather than a similarity score. That matters more than it sounds: a diff a human can review is the whole value of keeping a golden file at all.
Embedding similarity thresholds
Embed the stored answer and the new answer, take the cosine similarity of the two vectors, and fail below a threshold. This catches the case the other two miss: the answer that is still well-formed and still contains plausible facts but has wandered off topic.
It has three specific weaknesses and all three bite in practice. First, embeddings are poor at negation and at small numeric differences. “You may return items within 30 days” and “You may not return items within 30 days” are lexically almost identical and land close together in embedding space; so do “30 days” and “60 days”. The exact failures that matter most commercially are the ones this technique is worst at seeing.
Second, the threshold is a free parameter with no principled value. Setting it is an empirical exercise on your own data — collect pairs you consider acceptable and pairs you consider regressions, and pick the cut that separates them — and any number carried over from another project or another embedding model means nothing. Third, the threshold is only valid for the embedding model that produced it. Change the embedding model, even to a newer version from the same vendor, and every stored vector and the threshold itself have to be regenerated. Store the embedding model id next to the golden file so this is detectable rather than mysterious.
Key-fact extraction
Decide, per case, the handful of facts the answer must contain, and assert on those. For a policy question that is a number of days and a currency amount. For a lookup it is the order id from the context. For a classification it is the label. Extraction can be a regex, a parse of a structured field, or a schema-constrained second call.
This is the only technique of the three that asserts on correctness rather than on similarity to a previous run, which makes it the one worth investing in. Its cost is per-case authoring: somebody has to decide what the required facts are, and that work does not amortise across cases the way a threshold does. Its failure mode is silent under-specification — a case with one required fact passes on an answer that contains that fact and three fabrications.
Two guards make it much stronger. Assert negative facts as well as positive ones: the answer must contain the order id from the context and no other order id. And assert on the extractor itself — if the regex finds nothing, fail loudly rather than comparing null to null and passing.
The judge, and its recursion problem
The fourth option people reach for is asking a model whether the new answer is equivalent to the approved one. It is genuinely more capable than the other three at the “same meaning, different words” question, and it imports every problem this cluster exists to solve. The judge is itself non-deterministic, its verdict drifts when its model is updated underneath you, and it costs a call per assertion, which changes what your suite costs to run per branch.
If you use one, constrain it to a structured verdict rather than prose, pin the judge model to a dated version, and hold out a small set of pairs whose correct verdict you have decided by hand so you can detect the judge drifting. Treat the judge as a component with its own tests, not as an oracle.
Choosing between them
The layering that holds up is: structural diff on every case because it is nearly free and catches the failures that break clients; key-fact extraction on the cases that carry a commercial consequence; embedding similarity as a broad net over the long tail where nobody will author per-case facts; and a judge only where you have a reason and a budget.
Whatever mix you pick, record which technique failed in the failure message. “Similarity 0.71 below threshold 0.85” and “expected fact refundWindowDays=30, extracted 60” send a reviewer to completely different places, and a suite that reports only “snapshot mismatch” wastes the difference.
Finally, be explicit about what none of them do. Every technique here compares a new answer to an old one, so all of them inherit whatever was wrong with the old one. They detect change. Detecting error needs a case whose correct answer somebody decided independently, which is a golden dataset, not a golden file, and the two are worth keeping separate in your head even when they live in the same directory.
Top comments (0)