Correcting One Extracted Field Without Re-Running the Whole Document

#ai #machinelearning #tutorial #llm

A reviewer changes one digit of an invoice number. The interface sends the whole record back. Somewhere between the read and the write, another reviewer corrected the payment date on the same record, and it is now gone — replaced by the stale value the first reviewer’s browser loaded four minutes earlier. Nothing errored, nothing logged, and the correction is simply undone.

Two ways a whole-record write loses data

The natural implementation of a review interface is a form over the extracted record: load the JSON, let the reviewer edit, PUT it back. It fails in two distinct ways and both are silent.

Reviewer against reviewer. This is the classic lost update. Two people load the same record, each changes a different field, and the second write overwrites the first person’s change with the value it was loaded with. The record ends up internally consistent and wrong, and the only trace is in the audit trail — if the trail records the whole record, which is itself a symptom of the same design.

Re-extraction against reviewer. Much more damaging and much easier to ship by accident. Someone improves the prompt, or a document is re-processed because a later page arrived, or a nightly job re-runs a batch that previously failed. The pipeline writes a fresh extracted record over the old one, and every human correction on that document evaporates. The reviewers do not find out, because nobody re-opens a document they already signed off. You discover it when the same field is corrected twice by two different people three weeks apart, which is a genuinely confusing bug report.

Both failures have the same root: the unit of storage is the document, and the unit of change is the field. Once those match, both problems become arithmetic rather than luck.

Corrections as events

Store the extraction as a set of field-level facts and a correction as an append-only event that references one field. The record a consumer reads is then a fold over those events rather than a mutable blob.

-- what the pipeline produced, one row per field per extraction run
create table extracted_field (
  document_id   uuid    not null,
  field_path    text    not null,   -- RFC 6901 JSON Pointer, e.g. /lines/3/amount
  run_id        uuid    not null,
  value         jsonb,
  status        text    not null,   -- present | absent | illegible | not_extracted
  confidence    real,
  source        jsonb,              -- page + bbox + verbatim
  primary key (document_id, field_path, run_id)
);

-- what a human decided, append-only, never updated
create table field_correction (
  correction_id uuid    primary key,
  document_id   uuid    not null,
  field_path    text    not null,
  based_on_run  uuid    not null,   -- which extraction the reviewer was looking at
  prior_value   jsonb,              -- what they saw
  new_value     jsonb,              -- what they set
  new_status    text    not null,
  reviewer_id   text    not null,
  reason_code   text    not null,
  created_at    timestamptz not null default now()
);

field_path as a JSON Pointer is worth the small amount of discipline it costs. Pointers are specified in RFC 6901, they address nested and repeated structures unambiguously — /lines/3/amount is one line item’s amount and nothing else — and they are the addressing scheme the JSON Patch format in RFC 6902 already uses, so a correction serialises directly into a patch document if you ever need to ship one over an API.

prior_value looks redundant — it is derivable from the run — and it is not. It is what the reviewer actually had on screen, which is the only honest record of what they were agreeing or disagreeing with, and it is what makes the correction interpretable after the extraction it was based on has been superseded.

The precedence rule

With two sources of values, you need one stated rule for which wins. The rule that works in practice, in order:

A human correction beats any machine value for that field, whatever their timestamps. A reviewer looked at the page; the model did not, in any sense that counts.
Among human corrections for the same field, the most recent wins.
Among machine values, the most recent successful run wins — with one exception below.
A later machine value never replaces a human correction. It is recorded, and if it disagrees with the correction, the field is flagged rather than changed.

That last clause is doing something subtle and valuable. A re-extraction that produces a different value from a human correction is one of the highest-signal events available: either the model has improved and the human was wrong, or the model has regressed, or the document changed. All three are worth knowing, and all three are invisible if the pipeline either overwrites or skips silently. Emit it as a metric — machine-versus-human disagreement rate on re-extraction — and it becomes a leading indicator for which prompt changes are actually working.

Concurrent reviewers

Field-level events remove most of the conflict surface: two reviewers editing different fields never touch the same row, so there is nothing to lose. Two reviewers editing the same field is still a real conflict, and it needs a check rather than a hope.

Optimistic concurrency at field granularity is enough. The interface sends the correction_id of the latest correction it saw for that field — or null if there was none — and the write is rejected if that is no longer the latest. The reviewer is shown the other person’s change and decides. This is cheap because the conflict window is one field and the collision rate is tiny; the point is not to prevent conflicts but to make sure that when one happens a human resolves it instead of a clock.

Do not use a document-level version number for this. It makes every field edit conflict with every other field edit on the same document, which is exactly the coupling the field-level model existed to remove, and it produces spurious conflict prompts that train reviewers to click through them.

What happens when you do re-extract

Re-extraction is legitimate and common: a better model, a fixed preprocessing bug, a page that arrived late. The model above makes it safe, but there are three cases to handle explicitly.

Fields never touched by a human take the new value. Straightforward.
Fields with a correction that the new run agrees with need nothing, but the agreement is worth counting — it is evidence that the change you made was the right one.
Fields with a correction that the new run contradicts keep the human value and raise a low-priority review item. Do not auto-resolve in either direction.

The structural change also fixes a case that has nothing to do with reviewers: a schema change that adds a field. With a document-blob model, adding one field to the schema means re-extracting the whole document and rewriting everything. With field rows, a targeted run can extract only the new field and insert only those rows, leaving everything else — including every correction — untouched. On a large corpus that is the difference between a schema addition being a routine change and being a project.

Building it

Address every field with a JSON Pointer, generated from the schema rather than written by hand, so that array indices and nested objects are consistent between the extractor, the interface and the correction store.
Make the write endpoint accept a single field correction, not a record. If the interface needs to submit several at once, submit several corrections in one transaction — the unit stays the field.
Materialise the current view with a query that applies the precedence rule, and expose only that view to consumers. Nothing downstream should be joining the two tables itself, or the precedence rule will end up implemented three times with two different answers.
Never update or delete a correction row. A reviewer who changes their mind appends another correction. This is what makes the same table serve as the field-level audit trail without a second write path that could disagree with it.
Stamp every extraction run with its full pipeline identity so that a re-extraction is comparable to the one before it — see keeping a per-field model version stamp.

create view field_current as
select distinct on (document_id, field_path)
       document_id,
       field_path,
       coalesce(c.new_value, e.value)   as value,
       coalesce(c.new_status, e.status) as status,
       case when c.correction_id is not null then 'human' else 'machine' end as origin,
       e.confidence,
       e.source
from extracted_field e
left join lateral (
  select * from field_correction fc
  where fc.document_id = e.document_id
    and fc.field_path  = e.field_path
  order by fc.created_at desc
  limit 1
) c on true
where e.run_id = (
  select run_id from extraction_run r
  where r.document_id = e.document_id and r.status = 'succeeded'
  order by r.finished_at desc limit 1
)
order by document_id, field_path;

One property of that view is worth naming: origin is exposed to consumers. A downstream system that knows a value was typed by a human can treat it differently from one the model produced — it should not be re-scored, it should not be counted in an accuracy metric as a model output, and it should not be fed back as training data without somebody deciding to.