Your AI Audit Log Proves Nothing About Where the Answer Came From

Two different logs, and teams keep shipping the first while believing it's the second.

The first records that at 14:02:11 UTC, decision d_8f21 ran through model v2.3.1, took this input, produced this output, and a human approved it. Complete, timestamped, retained for six months. It satisfies most audit-trail checklists.

The second proves the answer came from paragraph 4 of manual_A2131.pdf, retrieved at rank 2 with a score of 0.81, and that the sentence in the response is supported by that span. That's a different artifact entirely, and it's the one a regulator asks for when the question stops being "what did the system do" and becomes "was any of it true."

Your log can be flawless and sit on top of a model built from a dataset nobody can trace. The log answers what happened, not whether it was trustworthy to begin with.

Why this is now a delivery constraint

EU AI Act Article 50 and the Annex III high-risk obligations apply from August 2, 2026, with full rollout by August 2, 2027. Article 19 wants six-month log retention. Article 99 puts the penalty at EUR 15 million or 3% of global turnover. FDA and EMA published joint AI guidance in January 2026. US banking supervisors are already asking about AI touchpoints in lending, KYC, and sanctions screening, without waiting for a finished rulebook.

Meanwhile the supply chain underneath is worse than most teams assume. A June 2026 analysis of 908,449 Hugging Face models found 55.46% carrying compliance risks or missing metadata, a 56.67% license omission rate in adapter derivations, and 8.05% license drift in fine-tuned models. A separate audit of 29,000 dataset descriptions found complete provenance chains to be rare.

So the model you pulled last sprint may have terms you can't reconstruct, and the fix is not more logging.

Four things that have to be true

• Source-linked. Every answer names the document, dataset row, or record it drew from.
• Path-logged. The retrieval and reasoning steps are captured, not just the final output.
• Proof-scored. Explainability and fairness metrics ride with the answer instead of living in a quarterly PDF.
• Mark-disclosed. Machine-readable markers flag AI-generated content.

What source-linking looks like in the schema

The design decision is whether citations are a first-class part of the response contract or a string the model was asked to append. Only the first is verifiable.

POST /answer  →

{
  "decision_id": "d_8f21",
  "answer_spans": [
    { "text": "Maximum discharge pressure is 175 psi.",
      "supported_by": ["ret_02"] },
    { "text": "Duty cycle is rated at 75%.",
      "supported_by": ["ret_05"] }
  ],
  "retrieved": [
    { "id": "ret_02", "doc_id": "manual_A2131", "doc_version": "2024-11",
      "chunk_id": 417, "char_range": [1180, 1372],
      "rank": 2, "score": 0.81 },
    { "id": "ret_05", "doc_id": "spec_A2131_rev3", "doc_version": "2025-03",
      "chunk_id": 88,  "char_range": [402, 559],
      "rank": 5, "score": 0.64 }
  ],
  "model": { "id": "...", "version": "v2.3.1", "prompt_hash": "sha256:..." },
  "unsupported_spans": []
}

Three properties make this hold up under review.

Chunks are addressable and versioned. doc_id plus doc_version plus char_range means the cited span can be re-fetched and re-read a year later. A doc_id alone is useless once the manual is revised, and revisions are exactly what a dispute is about.

Support is asserted per span, not per response. A response-level citation list lets a hallucinated sentence hide between two grounded ones. Per-span attribution is what makes unsupported_spans a computable field rather than an aspiration, and a non-empty one is a signal to refuse or escalate rather than ship.

The retrieval set is logged whole, including what lost. Rank and score for everything retrieved, not just what got cited. When an answer is wrong, the question is almost always whether retrieval missed the right document or the model ignored it, and you cannot tell those apart from the citation list alone.

Verification is then a job you can run, not a promise: re-fetch each cited span, check it still exists at that version, and run an entailment check on the span against its support. Failures become a queue, and the queue is your evidence.

Accuracy is not the compliance metric

A 2026 clinical study is the clean illustration. Logistic regression hit 75.2% accuracy against a random forest at 70.1%. On equal opportunity difference, the numbers ran 0.256 against 0.055, meaning the less accurate model carried roughly 57% less bias.

If your model selection criteria are accuracy plus auditability, you pick the first one and log the decision immaculately. The five accuracy points cost less than the bias does when someone asks why a class of applicants was treated differently.

Which means fairness metrics belong in the selection gate and in the response payload, not in a report generated after the fact.

Reconstructing provenance you never had

For models already in production with unclear lineage, Cisco released a Model Provenance Kit in April 2026 that fingerprints model weights and rebuilds derivation chains without the original documentation. It reported 96.4% accuracy and 98.1% precision across 111 test pairs. Useful when the honest answer to "where did this checkpoint come from" is currently a shrug.

What this looks like shipped

AskAC.ai, built for Compressor World, answers technical questions across 4,000+ product manuals and spec sheets, with every answer tracing to a specific source document. That constraint is what stops it inventing part numbers and fabricating pressure ratings, which in an industrial parts context is not a quality problem but a liability one. HeyPractice, a university learning platform, carries transcript analysis for compliance and engagement reporting alongside automated examination.

In both, source-linking was an architectural requirement from the first sprint rather than a retrofit. The division that makes this work in practice: AI tooling generates implementation speed, senior architects own data sourcing and answer traceability. Volume and verifiability are different jobs.

Checklist

• Chunk identity: doc_id + doc_version + char_range, stable and re-fetchable
• Per-span attribution in the response schema, with unsupported_spans computed, not narrated
• Full retrieval set logged with rank and score, including candidates that were not cited
• Prompt and template hashed per call, so a silent prompt change is visible in the log
• Entailment check as a scheduled job over sampled answers, with failures queued
• Fairness metrics in the model selection gate and attached to responses, not filed quarterly
• Model and dataset licenses verified at ingestion, including adapters and fine-tunes
• Retention at six months minimum, with the cited document versions retained too, since a citation to a deleted revision proves nothing
• AI-content markers machine-readable, per Article 50
• Refusal path when retrieval scores fall below threshold, because "no grounded answer" is a valid output

Before your next AI feature ships

Audit what's underneath it: model lineage, dataset licenses, and whether a single answer can be traced to a span you can re-open today. That's a day of work now and a much longer conversation later. brocoders.com