An accuracy problem surfaces on a field, on some documents, some of the time. The first question is which version of the pipeline produced the bad ones. If the answer has to be reconstructed from deployment dates and a git log, it will be approximate, and an approximate cohort cannot tell you whether the change you are about to make is the one that broke it.
The investigation that needs it
Extraction records are long-lived. A field extracted in March is still being paid against, reported on and disputed in November, by which time the pipeline has moved several times: a model upgrade, four prompt edits, a schema addition, a change of rasterisation DPI.
The investigation that arrives is always the same shape. Some population of records is wrong. You need to partition them by what produced them, compare error rates across the partitions, and identify the change that coincides with the increase. Every step of that requires the version to be an attribute of the record rather than an inference from time. Timestamps are a poor substitute: deployments are gradual, batches are re-run, some documents were processed weeks after they arrived, and a provider can change what a model does without you deploying anything at all.
The second thing it buys is scoped remediation. When you do find the bad version, you need the exact list of affected fields to re-extract — not the whole corpus, which may be tens of millions of fields, and not a date range, which is both too wide and, at the edges, too narrow.
A model alias is not a version
Recording that a field was extracted by a model name ending in -latest, or by an alias that a provider maps to whatever is current, records nothing. The alias is a pointer, and the thing it points at is precisely what changed. Two records with the same alias can have been produced by two different models, which makes the column worse than useless: it looks like a control variable and is not one.
The rule is to record the model the provider says it actually used — most APIs return a resolved model identifier in the response body, and that string is the one to store, not the one you sent. Where the two differ, store both; the difference is itself diagnostic, and it is how you notice that an alias moved under you.
Even a pinned identifier is not a complete guarantee: serving stacks, quantisation and default parameters can change behind a stable name, which is the phenomenon behind regressions after a silent model update. The stamp does not prevent that. What it does is let you detect it, because when a version-stamped population’s error rate changes while its version string did not, you have isolated the cause to something outside your own deployment — which is a far stronger conclusion than a suspicion.
Exactly what a provider returns, and whether an alias is documented as stable, differs between providers and changes over time. Check the current API reference for the providers you use rather than relying on a remembered field name; this page is on a refresh cycle for that reason.
Fallback routing makes this sharper: when a request fails over, the model that produced the field is not the model you asked for, and the proportion affected moves with somebody else’s error rate rather than with a decision you made. If you route across providers, take the resolved model from the response — a gateway such as Multigrid reports the model that actually served each request, along with the per-request cost — and stamp that, not the model in your configuration.
Five things that make up a version
The model is one input among several, and a version stamp that names only the model will fail to explain most of the changes you investigate. The identity you want is a hash over:
- The resolved model identifier, as returned by the provider.
- The decoding parameters that affect output — temperature, top-p, max tokens, seed where supported, and whether constrained decoding was on. A max-tokens change that starts truncating long tables is a real defect with no code change to point at.
- A hash of the fully rendered prompt template, not its filename or its human-readable version number. The hash is computed from the template text after any include or partial has been resolved, so that an edit to a shared fragment changes it.
- A hash of the extraction schema, since a schema change alters what the model is asked for and how the output is constrained. This is also what makes the stamp useful across a schema version boundary.
- The preprocessing identity: rasterisation DPI, colour handling, deskew and denoise settings, OCR engine and its version, page-splitting rules. Image preprocessing changes output as reliably as prompt changes do and is the input people most often forget, because it lives in a different service owned by a different team.
Combine them into one short digest — a truncated hash of the canonicalised tuple — and store the digest on the record with the full tuple in a separate lookup table keyed by that digest. You get a cheap column to group by and a way to answer “what exactly was px-2026.07.3” without duplicating five fields onto a hundred million rows.
Stamping the field, not the batch
The natural place to put the stamp is the extraction run, and for many pipelines that is sufficient. It stops being sufficient the moment one document’s fields come from more than one source, which happens more often than it sounds:
- A first pass extracts most fields and a second, targeted pass — different prompt, sometimes different model — handles the ones that came back low-confidence.
- A schema addition backfills one new field across historical documents, months after the rest of the record was produced.
- Some fields come from a deterministic rule or a lookup rather than from the model at all, and those should be marked as such rather than attributed to a model that did not produce them.
- A retry after a partial failure re-extracts part of a document under whatever version was current at retry time.
So the stamp belongs on the field row, defaulting to the run’s stamp. It costs one small column — a digest, not a string — on a table that already has a row per field because per-field confidence and field-level corrections need one. Human-corrected fields carry an origin of human and no model stamp, which keeps them out of every model accuracy statistic automatically rather than by a filter somebody has to remember.
What you can do once you have it
The stamp turns four otherwise impossible things into queries.
Cohort comparison. Correction rate per field per pipeline version, over documents processed after each version went live. This is the number that tells you whether a prompt change helped, and it is the honest complement to a pre-release regression run, which is measured on a set of documents chosen for being hard.
Scoped re-extraction. When a version is found to be defective, the affected fields are a selection on one column. Without it you either re-run everything or draw a date range and hope.
Correctly scoped calibration. A confidence curve is fitted for a specific pipeline, and applying an old curve to a new version silently mis-states probabilities in exactly the region where auto-acceptance happens. Keying the calibration by the version digest makes that mistake impossible rather than unlikely — see calibrating extraction confidence.
Answering the external question. When a counterparty or an auditor asks how a particular figure on a particular document was produced, the answer is a digest that resolves to a model, a prompt, a schema and a preprocessing configuration, alongside the field-level record of who, if anyone, changed it afterwards. That is a complete answer, and assembling it after the fact from deployment history is not.
Keep the version lookup table for as long as you keep the records that reference it, and treat deleting a row from it as deleting the meaning of every field stamped with it. It is small, it never changes, and it is the part of this that is easy to lose in a migration.
Top comments (0)