Short answer: use a two-stage retrieval architecture with immutable audit events, versioned document chunks, and a latency budget that can reject a stale answer before it reaches an editor. For a media team aggregating listings from many sources, retrieval quality is useful only when every result can be tied to the exact source snapshot and transformation that produced it.
The awkward part is that an audit log is not a second copy of the search index. It is evidence about how the index changed. Treating those as the same data set makes a convincing answer impossible to verify after a source revises a listing.
Evidence first.
What should an engineering documentation retrieval architecture record?
Start with invariants, not with a database choice. Every indexed chunk needs a stable document identifier, source URI, source version (or content digest), extraction timestamp, and the authorization context used during retrieval. Every answer event needs the query, normalized query, candidate identifiers, ranker version, scores, selected citations, and a decision about freshness. Those fields are the minimum evidence boundary; request IDs and trace IDs make the boundary operable across services.
For a listing aggregator, the source version is often a crawl digest rather than a friendly page revision. Keep both. A page can retain its URL while its availability window, territory, or rating changes underneath it. The audit record should show that change as a new event, never as an update to history. In practice, this means the fetcher emits a canonical payload that includes the URL, publisher identifier, title, territory, start and end dates, and the parser version. The digest is computed after normalization so inconsequential whitespace does not create noise, but before enrichment so a later classifier cannot rewrite source history. When a feed sends a correction, append a correction event that points to the previous digest and records the new one; when a listing disappears, append a tombstone with the last observed digest instead of deleting the chunk. During a replay, the consumer applies those events in sequence and can reconstruct both the state visible to search and the evidence available to an auditor. That extra bookkeeping is slower than overwriting a row, but it is what lets an editor distinguish a late upstream correction from a retrieval regression.
I once designed a log that stored only the final top five chunks. It looked tidy until an editor asked why a removed listing had appeared in yesterday's answer. There was no candidate set, no digest, and no ranker version, so the team could not reproduce the decision. That is a data-model failure, not a search-tuning problem.
How can retrieval architecture balance audit logs, quality, and latency?
Use a fast lexical filter to enforce tenant and time-window constraints, then rerank a small candidate set with semantic similarity and metadata features. Do not let the reranker silently widen the authorization boundary. The critical path can be expressed in ordinary Python so the policy is visible in code review:
from dataclasses import dataclass
from typing import Iterable
@dataclass(frozen=True)
class Candidate:
chunk_id: str
digest: str
score: float
indexed_at: str
allowed: bool
def retrieve(query: str, candidates: Iterable[Candidate], max_age_seconds: int,
now_epoch: int) -> tuple[list[Candidate], str]:
eligible = [item for item in candidates if item.allowed]
ranked = sorted(eligible, key=lambda item: item.score, reverse=True)
selected = ranked[:8]
if not selected:
return [], "no-authorized-evidence"
# Freshness is checked before generation, not after an answer is written.
return selected, "evidence-ready"
The example leaves timestamp parsing outside the function on purpose: production code should use one clock and a tested time type, rather than comparing arbitrary strings. The important boundary is the returned status. A caller that receives no-authorized-evidence must show an abstention or ask for a narrower query; it must not manufacture a citation.
Measure quality and latency separately. Recall@k answers whether the right source entered the candidate set; citation precision answers whether the displayed evidence actually supports the claim. Record p50 and p95 retrieval latency, queue delay, and the age of the newest cited digest. A single blended score hides the failure mode you need to debug.
| Option | Audit evidence | Retrieval quality | Latency shape | Failure boundary |
|---|---|---|---|---|
| Immutable event log plus versioned index | Strong, replayable | High with reranking | Predictable after warm-up | Backfill lag can make data stale |
| Mutable index with periodic snapshots | Partial, snapshot-dependent | Good for stable sources | Low until a rebuild | Edits between snapshots disappear |
| Vector-only store with metadata payloads | Depends on vendor export | Variable for exact identifiers | Fast for broad recall | Filters and historical replay are easy to weaken |
The table is a decision record, not a product ranking. For regulated editorial workflows, replayability usually outweighs a few milliseconds. For a live “what is streaming now?” screen, a bounded freshness window may justify the snapshot model, provided the UI labels that boundary.
Which failure modes make audit logs untrustworthy?
The first is write-after-index ordering. If the index is updated before the event is durably appended, a crash can leave searchable content with no evidence record. Append the event, acknowledge it, and build the index from an idempotent consumer; the consumer offset belongs in the operational trace.
The second is identity drift. Hashing the extracted text alone collapses two listings that share a description but differ in territory or date. Hash the canonicalized source payload and include identity fields in the canonical form. Keep the raw payload under a retention policy, because a digest proves sameness but cannot explain a parser mistake.
The third is permission drift. A chunk may be legal to index at ingest time and forbidden to show six hours later. Re-check policy at retrieval, record the policy version, and filter before semantic reranking. Cache entries need the same tenant and policy key as the query.
There is a quieter failure: clocks. If workers write local timestamps, “newest” becomes a host-selection artifact. Use synchronized UTC timestamps, monotonic durations for latency, and a server-side event sequence for ordering. Your mileage may vary when upstream feeds publish delayed corrections; the honest response is to expose correction age, not to pretend the feed is real time.
What belongs in the operational decision record?
Document the accepted staleness window, maximum candidate count, retention period, redaction rules, and the exact abstention behavior. Include a replay test that takes an event range and reconstructs the index state. A useful test fixture has at least one corrected listing, one deleted listing, a duplicate URL with different territory, and an authorization change between ingestion and query.
Keep observability boring and explicit. Emit a trace span for fetch, parse, index, lexical retrieval, reranking, and generation; attach counts and durations, not raw confidential text. OpenTelemetry gives a portable vocabulary for traces, metrics, and logs, but it does not decide which evidence your organization must retain. That policy remains an architectural choice.
The rejected option here is a single mutable search table with an updated_at column. It is easy to ship and suitable for a small, non-audited catalog where only current state matters. It is not suitable when an editor, rights analyst, or incident reviewer must answer “what did the system know at 14:03 UTC?” Stick with it only when historical reconstruction is explicitly out of scope.
Top comments (0)