Somebody is about to add one sentence to an extraction prompt to fix one field. The sentence will fix that field. Whether it breaks two others is not knowable without a test that was written before the edit, and aggregate accuracy is not that test.
What you are actually protecting
Extraction prompts accrete. A field comes back wrong on a particular layout, somebody adds a clarifying line, and the prompt grows a paragraph a month. Every one of those additions is a global change to a system that produces twenty fields, and the connection between the added sentence and the fields it perturbs is not visible in the diff.
The specific failure this test exists to catch: a hint added to disambiguate invoice_date from due_date improves invoice_date from 91% to 98% and drags due_date from 96% to 88%. Across twenty fields, overall accuracy moves by roughly 0.05 percentage points — well inside run-to-run variation. A test asserting “overall accuracy did not decrease” passes, the change ships, and the due-date regression surfaces weeks later as a billing complaint. Per-field scoring makes it a red build instead.
This is distinct from testing that a schema change is backwards compatible, which the library covers under schema backward compatibility and schema versioning. The schema here is fixed. What is changing is the prompt, and the thing at risk is accuracy rather than shape.
Build the golden set
- Sample forty to sixty documents from production traffic. Not the ones somebody used while developing the prompt — those are already fitted. Stratify deliberately: include the clean digital PDFs that dominate volume, the phone photographs, the faxes, the multi-page ones, at least one of every layout variant you know of, and two or three that are genuinely marginal. Forty is enough that a single field’s accuracy has a usable resolution; six hundred is a cost you pay on every run forever.
- Redact them, then freeze them. These files will sit in a repository and be read by CI, so real customer documents cannot be among them. Mask account numbers, names and addresses at ingestion and keep the masked artefact as the fixture — the same practice as redacting secrets from recorded fixtures. Store them by content hash so that a fixture cannot be edited without the change being visible.
- Label every field by hand, including the nulls. A golden record must state that a field is absent, not omit it. Absence and unlabelled are different, and a scorer that cannot distinguish them will award credit for a hallucinated value on a document where the field does not exist. This is the tedious step and it is the one that determines whether anything downstream means anything; the golden dataset page covers the wider practice.
- Record who labelled each document and when. Golden data is wrong sometimes, and when a test fails you need to be able to ask whether the extraction or the label is at fault without that question being unanswerable.
golden/
0a3f1c...json { "doc": "0a3f1c....pdf", "labelled_by": "rk", "labelled_at": "2026-07-02",
"fields": { "invoice_number": "INV-88213",
"invoice_date": "2026-03-04",
"due_date": "2026-04-03",
"po_number": null,
"total": "4820.15" } }
Normalise, or measure noise
Before comparing anything, decide per field what counts as equal. Without this the harness reports failures that are formatting differences, people learn to ignore the output, and the test stops working as a test.
function normalise(field, value) {
if (value === null || value === undefined) return null;
const s = String(value).trim();
if (s === "") return null;
switch (kindOf(field)) {
case "money":
// strip currency symbols, thousands separators and spaces; keep sign
return Number(s.replace(/[^0-9.-]/g, "")).toFixed(2);
case "date":
// parse with an explicit, per-source format list. Never guess between
// 03/04/2026 and 04/03/2026 -- carry the source locale in the fixture.
return toIsoDate(s, formatsFor(field));
case "identifier":
return s.toUpperCase().replace(/[^A-Z0-9]/g, "");
case "text":
return s.toLowerCase().replace(/\s+/g, " ");
default:
return s;
}
}
Three of those deserve comment. Money must compare as a fixed-precision string, so that 4820.1 and 4820.10 agree and floating point never enters it. Identifiers should compare loosely on punctuation and strictly on characters, because an invoice number rendered with and without a hyphen is the same identifier and one rendered with an O instead of a zero is not. And dates must never be normalised by guessing: an ambiguous numeric date is the single most common source of a fake pass, because the extractor and the labeller can both be wrong in the same direction. Carry the source’s date order in the fixture and parse against it. The library’s date field validation rule page goes further into the ambiguity.
Free text is the field kind where exact match is the wrong test. “ACME Corp.” and “ACME Corporation” are the same vendor. Use a similarity threshold for those fields, record the threshold in the harness configuration, and treat it as part of the test rather than as a detail — changing it changes every result.
Score per field, not per document
The comparator walks the golden fields and produces a per-field tally, never an average over documents:
function scoreRun(golden, extracted) {
const tally = new Map(); // field -> { correct, total, misses: [] }
for (const doc of golden) {
const got = extracted.get(doc.id) ?? {};
for (const [field, want] of Object.entries(doc.fields)) {
const t = tally.get(field) ?? { correct: 0, total: 0, misses: [] };
const a = normalise(field, want);
const b = normalise(field, got[field]);
const ok = fieldKind(field) === "text" ? similar(a, b) : a === b;
t.total += 1;
if (ok) t.correct += 1;
else t.misses.push({ doc: doc.id, want: a, got: b });
tally.set(field, t);
}
}
return tally;
}
Two properties make this useful rather than decorative. It keeps the misses, with the document identifier and both values, so a failure is immediately actionable rather than a number that has gone down. And it never aggregates across fields, so a field with four hundred instances cannot drown a field with forty.
Run each document more than once at the lowest temperature the provider offers. Extraction is not deterministic even at temperature zero, and a field that returns two different answers across three identical calls is telling you something a single run cannot: that field is unstable, and its score in any single run is partly luck. Record the per-field disagreement rate alongside the accuracy, and treat a field with high instability as untrustworthy even when its accuracy looks fine.
Record a baseline and gate the change
- Run the harness on the unmodified prompt and commit the result. This is the step people skip and it is the entire point of the page: a baseline recorded after the edit is not a baseline. Commit it as data next to the prompt, pinned to a specific model version string, because a baseline compared against a different model measures the model rather than the prompt.
- Make the prompt change. One change. A commit that edits three instructions produces a result you cannot attribute.
- Re-run and diff per field. The gate is not on the aggregate. It is: no field may drop by more than a stated tolerance, and any field that drops at all must be acknowledged in the commit.
- Update the baseline in the same commit as the prompt so that the two can never drift apart, and so the diff shows both the instruction that changed and what it did.
$ node extraction-regress.mjs --baseline baseline.json
field baseline now delta
-----------------------------------------------
invoice_number 0.980 0.980 +0.000
invoice_date 0.910 0.980 +0.070 improved
due_date 0.960 0.880 -0.080 REGRESSION (tolerance 0.02)
po_number 0.870 0.870 +0.000
total 0.995 0.995 +0.000
-----------------------------------------------
overall 0.943 0.941 -0.002 within noise
FAIL: due_date regressed by 0.080
doc 4c81ba want 2026-04-03 got 2026-03-04
doc 91e7d2 want 2026-05-19 got 2026-04-19
doc a70c33 want null got 2026-03-04
That output is the argument for the whole approach in one screen. The overall figure moved two thousandths and would have passed any aggregate gate ever written. The per-field view names the broken field, quantifies it, and hands over three documents that reproduce it — including the third, where the model invented a due date on a document that has none, which is a different bug from the first two and would not have been visible as a percentage. Set the tolerance deliberately: zero tolerance on a forty-document set means one flipped document is a failed build, so something in the range of one to two documents’ worth is usually right, and the tolerance belongs in the committed configuration where it can be argued about.
Two operational details decide whether this harness stays trustworthy. The baseline must be pinned to an exact model version, or a provider updating a model behind a stable alias silently invalidates every recorded number; and a run of sixty documents times three repeats is a real cost that grows every time somebody adds a fixture, so it wants to be visible per run rather than at month end. A gateway that pins model versions, records cost per request and lets the same harness run unchanged against a second provider covers both — and comparing a candidate model against the same baseline is then a configuration change rather than an integration. That is what Multigrid provides behind one API and one key.
What the gate cannot catch
- Layouts not in the set. The harness measures the documents it holds and says nothing about the variant that arrives next month. Add a fixture every time a novel layout causes an incident; that is how the set earns its coverage over time, and it is the same instinct as turning a support ticket into a test.
- Wrong golden data. A mislabelled fixture makes a correct extraction fail forever, and the usual resolution — adjusting the prompt until the test passes — is exactly backwards. When a failure looks wrong, re-read the document before re-reading the prompt, and record the correction against the label with its own provenance.
- Downstream damage. A field that is 98% correct and a field that is 98% correct do not have equal consequences if one is a reference number and the other is an amount that gets paid. Weight the review and the alerting by impact rather than by accuracy, which is the argument in prioritising review by financial impact.
- Calibration. Accuracy is not confidence. A prompt change can leave accuracy flat and make the model’s stated confidence much less informative, which quietly changes how much work reaches human review; calibrating confidence against actual error rate is a separate measurement and it needs its own baseline.
Top comments (0)