A reviewer corrects an extracted date on one supplier’s invoice. Somebody adds a line to the prompt about that supplier’s date format. Repeat forty times and the prompt is two thousand tokens of accumulated special cases, each of which was justified, and the extraction is worse than it was.
A correction is not a data point about a document
The useful unit is not the corrected document. It is the error signature: what kind of mistake was made, on what kind of field, caused by what property of the input. One correction is an anecdote. Twenty corrections sharing a signature are a defect, and the difference is not the count alone — it is whether they came from independent sources.
Ten corrections of the same field on ten invoices from one supplier are one observation, not ten, because they share a cause: that supplier’s template. Ten corrections of the same field across nine unrelated suppliers are a real prompt-level or schema-level defect. Counting corrections without collapsing by source is the single most common way a feedback loop ends up chasing whichever vendor sends the most post.
This is why the correction record needs a reason_code and not only a before-and-after value. Free text is unclusterable at any volume. A short controlled vocabulary — wrong field read, digit misread, wrong date order, wrong currency, sign inverted, value from the wrong page, field was present but reported absent, hallucinated value not on the page — costs the reviewer one click and turns the correction log into something you can actually aggregate.
Clustering by error signature
Group the correction log along four axes and look at the cells, not the totals:
- Field. Which field is being corrected. A field with an outsized share of corrections is either genuinely hard or specified wrongly, and the two look identical from the total.
- Reason code. The class of error. “Sign inverted” and “digit misread” on the same field have entirely different fixes.
- Source. Supplier, template, originating office, scanner. This is the axis that tells you whether the signature is general or local.
- Pipeline version. Which model, prompt and schema produced it. Without this, a signature that was fixed three weeks ago still shows up in the totals and looks live — the reason a per-field version stamp is a prerequisite for this analysis rather than an adjacent nicety.
A signature worth acting on is one with a meaningful correction rate within its cell, drawn from several independent sources, that is still occurring under the current pipeline version. Everything else is either already fixed, or local, or noise.
The rule for earning a prompt change
A workable, defensible rule, stated as three conditions that all have to hold:
- Independence. The signature appears across at least three independent sources — different suppliers, different templates, different originating systems. This is the condition that rules out template-specific quirks, and it is the one people skip.
- Rate. Within the population where the signature could occur, its rate is above the level you would tolerate for that field given its cost of being wrong. A 0.4% error rate on a delivery reference is not worth a prompt change; the same rate on a bank account number is.
- Currency. It is still happening on the current pipeline version, on documents processed after the last change.
When a signature fails the independence test but the corrections keep coming, that is not a reason to relax the rule. It is a signal that you have a template-specific problem, and template-specific problems are better solved by a template-specific route — a per-source configuration, a preprocessing step, a different schema variant — than by a global instruction that every other document also has to pay for, in tokens and in the chance that it misfires somewhere else.
Most corrections do not belong in the prompt
Before editing the prompt, ask which layer the defect is actually in. A large share of corrections in real systems are not model errors at all, and fixing them in the prompt is both ineffective and permanent.
- Normalisation bugs. The model returned exactly what the page said and your parser mangled it — the decimal comma, the accounting negative in parentheses, the ambiguous date order, the currency symbol that was not stripped. Fix the parser. A prompt instruction cannot fix a parser and will make the model produce a second format for the parser to also get wrong.
- Schema defects. The field genuinely occurs twice on some documents, or is a list and was typed as a scalar, or has a state your enum does not have. The corrections are the reviewer forcing a real-world shape into a schema that cannot hold it — which is the subject of designing a schema for variants you have not seen.
- Input quality. The corrections cluster on documents from one office, at one resolution, or with one scanner’s compression artefacts. The fix is upstream of the model entirely.
- Specification ambiguity. Two reviewers correct the same field in opposite directions. Nothing is wrong with the model; nobody has decided what the field means. This one shows up as corrections of corrections, and it is worth alerting on.
Only what survives that filter — the model consistently reading the wrong thing off a page it can see clearly, across independent sources — is a prompt problem.
The few-shot trap
The obvious move once you have a corrected document is to paste it into the prompt as an example. This is the most direct route to the overfitting the row title warns about, and it fails in a specific way worth understanding.
The documents that generate corrections are by construction the unusual ones. Selecting examples by “it was corrected” therefore selects for atypical layouts, and a set of examples drawn from the tail teaches the model that the tail is the norm. The characteristic symptom is a fix that works on the pathological document and degrades a class of ordinary documents that nobody was watching — a regression you only find because the correction rate on some other field went up a week later.
Two mitigations. First, when an example is warranted, add it as a minimal illustration of the rule rather than as the whole document: the smallest fragment that shows the distinction, not fourteen fields of context the model will also generalise from. Second, keep the example set balanced — for every awkward case you add, make sure ordinary cases are still represented, so the distribution the model sees in the prompt is not entirely composed of exceptions. The general mechanics are covered in few-shot prompting; what is specific here is the selection bias built into using corrections as your example source.
Prefer a stated rule to an example wherever a rule exists. “Dates in this corpus are day-month-year unless the document declares otherwise” is a general instruction that transfers. A worked example of one supplier’s date block is a pattern the model may match on the supplier rather than on the format.
Corrections are a regression set, first
The highest-value use of a correction log is not as prompt material at all. It is as a frozen evaluation set. Every corrected field is a labelled example with a verified answer, produced by exactly the distribution you run on, and it costs nothing extra to collect.
So the discipline is: before any prompt change ships, run it against the accumulated corrected set and compare per-field accuracy against the current version. A change that fixes its target signature and degrades two other fields is very common and completely invisible without this step. The general shape is the same as any model regression suite, with one extraction-specific caution: the corrected set is biased towards hard documents, so its absolute accuracy is not your production accuracy. Use it for comparison between versions, never as a headline number, and keep a separate randomly sampled set — the audit stratum from calibration serves this — for anything you want to state as a rate.
Freeze the set at a version boundary and grow it deliberately rather than continuously. A regression set that silently absorbs every new correction cannot be compared against itself over time, which is the only thing it was for.
Top comments (0)