Eight months after a payment went to the wrong account, somebody asks what the extraction originally said, who changed it, when, and on the basis of what. If the answer is a row in a table that has since been updated in place, there is no answer.
The questions the trail has to answer
Design it backwards from the questions, because a trail that captures everything except the one join you need is the usual outcome. The questions that arrive, in roughly the order they arrive:
- For this field on this document, what did the pipeline produce, what did each human do to it, and in what order?
- Which human, identified in a way that survives them leaving the company and their account being deleted?
- What were they looking at when they decided — which extraction run, which page region, which prior value?
- Was this value ever different from what it is now, and if the current value came from a machine, did it overwrite a human decision?
- Across the corpus: which reviewer, which field, which reason code, over which period. The aggregate query is what turns the trail from a compliance artefact into the input to the correction feedback loop.
This is a narrower object than a general AI decision log. That page — the AI audit log schema — is about recording that a system made a decision affecting someone. This trail is about a specific value on a specific document changing hands between a model and a person. They overlap and they should not be the same table: their retention periods differ, their access controls differ, and the volumes differ by orders of magnitude.
The record
{
"correction_id": "9c1f...", // stable, generated by the writer
"document_id": "d-2026-114872",
"field_path": "/payment/bank_account_number", // RFC 6901 pointer
"based_on_run": "run-8841", // the extraction the reviewer saw
"pipeline_version": "px-2026.07.3", // resolved model + prompt + schema
"prior_value": "40218317",
"prior_status": "present",
"prior_confidence": 0.71,
"prior_origin": "machine", // machine | human | rule | recovered
"new_value": "40213817",
"new_status": "present",
"reason_code": "digit_transposed",
"note": null, // optional free text, never parsed
"reviewer_id": "u-4471", // stable internal id, not an email
"reviewer_role": "ap_reviewer",
"decided_at": "2026-08-11T09:42:17Z", // RFC 3339, UTC
"recorded_at": "2026-08-11T09:42:17Z",
"evidence": { "page": 2, "bbox": [0.61, 0.44, 0.83, 0.47] },
"prev_hash": "sha256:1f9b...",
"hash": "sha256:7ad0..."
}
Several of those fields are the ones that get left out and then needed.
Two timestamps. decided_at is when the human acted; recorded_at is when the row was written. They differ when a client buffered offline, when a queue retried, or when a backfill imported history, and a trail with one timestamp cannot tell a late write from a backdated one. Store both in UTC in RFC 3339 form with an explicit offset; a local timestamp without a zone is a value nobody can interpret two years later.
A stable reviewer id, not an email address. Emails get reassigned, people change names, and an identifier that is also personal data is one you may be obliged to erase — which is a bad property for the field that answers “who”. Keep an internal id in the trail and resolve it to a person through a directory that can be maintained separately.
prior_origin is what lets you distinguish a reviewer correcting the model from a reviewer overriding another reviewer, which are very different events and are otherwise indistinguishable in the log.
evidence — the page and region the reviewer was shown. A correction made while looking at a specific crop is a materially stronger record than one made while looking at a document, which is a second reason to build source highlighting.
Append-only, and why an UPDATE is the bug
The trail is insert-only. A reviewer who changes their mind produces a second row. A mistake in a row is superseded by a compensating row, never edited. This is not fastidiousness — a trail that can be modified answers no question about the past, because the answer it gives is indistinguishable from an answer somebody wrote yesterday.
Enforce it below the application. Grant the application role INSERT and SELECT on the table and not UPDATE or DELETE. A convention that the code does not update the trail survives exactly until somebody writes a migration to fix a typo. A revoked privilege survives that.
Append-only interacts badly with a single-row-per-field cache if the two are allowed to disagree. Derive the current value from the trail — the materialised view in correcting one field without a re-run is exactly that — rather than writing both and hoping. Two write paths to one fact is how a trail ends up not matching the record it is the trail of.
Reason codes beat free text
A free-text reason field produces a corpus of “fixed”, “typo”, “wrong” and empty strings. A closed enum, chosen by a click, produces something you can group by. Keep the vocabulary short enough that a reviewer reads all of it — eight to twelve codes is about the limit — and make one of them explicitly “other” with a required note, so that the pressure to misuse a nearby code is relieved rather than hidden.
Track how often “other” is chosen. A rising rate is the vocabulary telling you it is missing a category, which is usually the earliest signal that a new document variant has entered the corpus. Version the vocabulary and record which version was in force, or a code that is retired makes every historical row containing it ambiguous.
The old value is often the sensitive one
This is the part that is specific to extraction and gets missed.
Your record store may be carefully designed — the account number is encrypted, the patient identifier is tokenised, the retention policy deletes the document after the statutory period. The audit trail then quietly holds prior_value and new_value in plain text, forever, because it is append-only and nobody thought of it as a data store. For extraction, the corrected values are the sensitive content: bank accounts, national identifiers, medical record numbers, names and dates of birth.
Three things follow. Apply the same protection to the trail as to the record — the same encryption, the same access control, and access logging on the trail itself, since it is now a target. Classify per field, so that a trail row for a high-sensitivity field can hold a token or a hash of the value rather than the value, where the audit question can be answered by “it changed, from something that hashed to X”. And reconcile append-only with erasure obligations deliberately: the usual resolution is crypto-shredding — hold the values under a per-subject key and destroy the key, which leaves the structure and the sequence of events intact while the content becomes unrecoverable. The general handling of sensitive values in logs is covered in PII in LLM logs. Decide this before the trail has two years of history, because the migration afterwards is unpleasant and, on an append-only table, philosophically awkward.
Making it verifiable
For most internal purposes an append-only table with revoked update privileges is enough. Where the trail may be challenged — a dispute, a regulator, a counterparty — you want it to be tamper-evident, which is a small amount of extra work at write time.
- Canonicalise the row: a deterministic serialisation with sorted keys and a fixed number format, so the same logical row always produces the same bytes.
- Hash it together with the previous row’s hash for the same document, giving a per-document chain. Store both
prev_hashandhash. - Periodically — daily is usually enough — publish the head hash of each chain somewhere outside the same trust boundary: a separate account, a write-once store, a countersigned log. This is what makes a chain evidence rather than an assertion, since a chain held entirely by the party it exonerates can be rebuilt end to end.
- Verify on a schedule and alert on failure. A tamper-evident structure that nobody checks detects nothing.
Keep the verification job’s output in the ordinary monitoring system rather than in the trail. A trail that records its own verification is verifying itself, which is the property you were trying to avoid.
Top comments (0)