A media team answering questions over a folder of developer changelog PDFs has one constraint that changes the whole retrieval architecture: every claim must lead back to the exact source passage, even after a PDF is replaced or re-parsed. Short answer: treat citation tracing as a set of executable audit tests, then choose lexical, vector, or hybrid retrieval to satisfy those tests. The language model comes last.
I prefer this framing because a fluent answer can conceal a broken evidence trail. A tracker that returns the right sentence from the wrong revision is still wrong. The seven tests below are small enough to run in CI and concrete enough to expose that failure.
1. Hash the bytes before extracting text
The first test asserts that ingestion keeps the original PDF, a SHA-256 hash, a stable document ID, and the parser version. A filename is not identity; release-notes.pdf may be overwritten tomorrow. Keep each page as a versioned record with both the printed page label and the parser's zero-based page number, since scanned media documents often make those differ.
Publication date, product area, and a supersedes link belong in metadata for a developer changelog tracker. Never update an old record in place. A question answered against a historical snapshot must be reproducible byte for byte.
from dataclasses import dataclass
from hashlib import sha256
@dataclass(frozen=True)
class PageEvidence:
document_id: str
version_hash: str
parser_version: str
page_index: int
page_label: str
text: str
def version_hash(pdf_bytes: bytes) -> str:
return sha256(pdf_bytes).hexdigest()
The assertion is boring. Good. Storage layers should be boring.
2. Can a changelog tracker preserve citation tracing through retrieval?
Make the retrieval contract return evidence objects instead of anonymous strings. Each object needs the document ID, version hash, page, character span, extracted text, and the score components; alongside the answer, persist the normalized query, filters, retriever version, candidate IDs, and final IDs. That trace lets an auditor replay why a passage was selected.
from dataclasses import dataclass
@dataclass(frozen=True)
class Evidence:
document_id: str
version_hash: str
page: int
start: int
end: int
text: str
score: float
retriever: str
def citation_key(item: Evidence) -> str:
return f"{item.document_id}@{item.version_hash}:p{item.page}:{item.start}-{item.end}"
Page-only citations are weak: two similar paragraphs can share a page, and a renderer can change line wrapping. Offsets tied to the stored extracted text make the reference precise. I once assumed parser upgrades would leave offsets alone. A font substitution shifted a heading and every later offset; recording the parser version turned a silent mismatch into a failing test.
3. How should a developer changelog tracker test exact-token and semantic recall?
Changelog questions contain version strings, flag names, method signatures, and error codes. A lexical index should handle those exact tokens; a vector index can recover a paraphrase such as “what changed in authentication?” but may blur v2.10 and v2.1. Run both candidate searches, merge by document version, and rerank only after the merge.
The useful comparison is failure behavior:
| Concern | Lexical retrieval | Vector retrieval | Hybrid test target |
|---|---|---|---|
| Version and error-code match | Usually strong | Can be uneven | Exact hit in top-k |
| Paraphrased question | Limited | Usually stronger | Relevant passage in top-k |
| Citation replay | Needs stored span | Needs stored span | Same immutable evidence key |
| Drift risk | Analyzer changes | Embedding changes | Detect either in CI |
Chunk on headings and page boundaries. A large chunk may improve recall while joining two pages and making a citation ambiguous; a tiny chunk can split the sentence that qualifies a breaking change. Measure recall at k and citation completeness on a labeled question set. Answer fluency is not a substitute.
4. Reject citations the generator did not retrieve
The generator should emit only citation keys already present in the retrieval trace. A deterministic validator can reject an unknown key, a span outside the stored text, or a claim with no supporting evidence unit. This does not prove a claim is logically true; it proves the pointer is real.
Use near-duplicate fixtures: an original bulletin, a corrected revision, and an OCR-noisy scan. Ask the same question against each snapshot. The expected result includes the version hash and page, not merely a matching sentence. For a sample, compare the extracted span with a rendered page image because OCR can join columns and create words absent from the PDF.
Three words matter: “I couldn't verify.” Abstention is a valid answer when no evidence clears the documented threshold or the top results disagree about the version. It is preferable to a confident migration instruction with a decorative citation.
5. What storage limits should shape this retrieval architecture?
Object storage supplies durable bytes, but it does not choose retention, legal hold, or access policy. Test deletion attempts, retention expiry, duplicate uploads, and a restored snapshot. The retrieval layer should fail closed when the evidence object is unavailable rather than silently substituting the newest file.
Permission checks deserve their own fixture. A media archive can contain embargoed release notes; filtering after retrieval risks leaking titles or snippets through logs. Apply authorization before ranking and store only the IDs permitted for that run.
Elasticsearch, Vespa, and Postgres with pgvector represent different operating choices, not automatic citation guarantees. Elasticsearch keeps term and vector search in one cluster but brings cluster tuning. Vespa exposes ranking pipelines and asks for ranking expertise. Postgres with pgvector keeps vectors beside transactional metadata and can simplify a small deployment, while competing for database resources. Pick the system your team can observe, back up, and restore under its real workload.
The catch is scope: this approach is not suitable for a live, permission-sensitive corpus that changes every minute and requires row-level authorization during retrieval. Use a search system with native authorization-aware ranking there, accepting a heavier control plane. Stick with a plain keyword index when the corpus is tiny, terminology is exact, and document-level links satisfy the auditors.
6. Roll out with replay and explicit budgets.
Start with a shadow index and replay a fixed question set for two weeks. Log disagreements between old and new retrieval, then sample the cases where the version hash, page, or citation coverage changes. Track recall at k, citation coverage, stale-version rate, abstention rate, and parser-failure rate by document family.
Keep parsing time, embedding calls, retrieval latency, and validation work inside explicit budgets. A budget is an engineering limit, not a promise of a fixed bill. Store metric inputs under the same run ID as the answer so a dashboard can explain a regression instead of merely displaying one.
For the media PDF tracker, the migration order is ingestion hashes, page-level evidence records, hybrid retrieval, then generation and validation. Reversing that order produces a persuasive demo without a durable audit trail.
Top comments (0)