Short answer: keep the product index focused on serving queries, and write a separate, append-only audit record for every retrieval decision; a three-tier record (request, candidates, answer evidence) controls index cost without making search behavior impossible to explain.
That boundary matters in a catalog. A customer searches for a “waterproof 20L hiking pack,” while the index is changing prices, stock, translations, and seller eligibility. If the audit trail is stored as copies of every indexed document, storage and re-index work grow together. If it stores only a final product ID, support cannot explain why a result was promoted or filtered. The useful middle is an event that captures the decision inputs and references the data versions, not another product database.
The constraint: explain a result without cloning the catalog
Treat each search as a trace with a stable trace_id. The request record holds the normalized query, tenant or storefront, policy version, timestamp, and the index snapshot identifier. It should not contain a full customer profile or an unredacted free-text query when that text may contain personal data. Hash sensitive values with a keyed digest when an operator needs correlation but not recovery.
The candidate record is deliberately narrower. Store the top K identifiers, rank scores, retrieval channel (lexical, vector, or both), and the filter decisions that removed candidates. K should be a measured design input; 20 candidates may be enough for a product grid, while an offline investigation might sample 100. Do not silently change K between deployments. Put the retrieval configuration revision beside it so a replay can distinguish a model change from a catalog change.
The final tier stores the evidence actually shown to the user: product IDs, field names, and content hashes or document revisions. A hash proves which text was cited without duplicating a large description in every event. Keep the rendered response separately if policy requires it, encrypted and with a shorter retention period than the decision metadata.
This is an audit log, not a queue. Queues are for delivery and retries; an audit stream needs ordered offsets, durable retention policy, and an export path for review. The choice of implementation (relational append table, object files, or a log service) can follow the team’s operations. The contract must remain stable.
What should an audit log capture in product search retrieval?
Start with fields that answer five questions: who requested the search, what data was eligible, which algorithm ran, what was returned, and what policy changed the outcome. A compact event can look like this:
from dataclasses import dataclass
from typing import Any
@dataclass
class RetrievalAudit:
trace_id: str
query_digest: str
catalog_revision: str
retriever_revision: str
filter_policy: str
candidates: list[dict[str, Any]]
evidence: list[dict[str, str]]
created_at: str
def make_audit(trace_id: str, query_digest: str, catalog_revision: str,
retriever_revision: str, candidates: list[dict[str, Any]],
evidence: list[dict[str, str]], created_at: str) -> RetrievalAudit:
return RetrievalAudit(
trace_id=trace_id,
query_digest=query_digest,
catalog_revision=catalog_revision,
retriever_revision=retriever_revision,
filter_policy="storefront-eligibility-v3",
candidates=candidates,
evidence=evidence,
created_at=created_at,
)
The example leaves out raw descriptions on purpose. Keep a separate, access-controlled evidence store if an investigation needs the exact bytes. Record a content hash, locale, and field name in the audit event, then resolve that hash against the catalog revision. This prevents a later edit to “waterproof” copy from rewriting history.
A failed search deserves an event too. Log zero-result reason codes such as NO_ELIGIBLE_STOCK, FILTERED_POLICY, or INDEX_LAG, plus the query class used for fallback. Never turn a zero result into an invented recommendation just to make a dashboard look healthy. For privacy reviews, an event’s deletion or redaction operation should itself produce a new audit event; deleting history without an explanation weakens the trail.
Spend the index budget where recall needs it
Index cost is usually driven by vectors, replicas, and update frequency, not by the few bytes in an audit row. Separate hot retrieval data from cold evidence. Keep searchable fields and embeddings in the serving index; put immutable audit events in cheaper, partitioned storage keyed by day and storefront. A seven-day hot window can support incident response, while older partitions move to archival storage under the retention policy. Your mileage may vary: legal hold and chargeback workflows may require a longer window.
Do not embed every attribute. SKU, availability, numeric price, and policy flags are better modeled as structured metadata filters. Descriptions, compatibility notes, and category text are candidates for semantic vectors. If a seller changes stock every few minutes, updating its embedding on every stock tick wastes index writes and can create inconsistent snapshots. Version the descriptive document independently from fast-changing eligibility fields, then join them at query time under one catalog revision.
Measure before choosing a larger index. Build a labeled set of real queries with expected products and policy exclusions. Track recall@K, zero-result rate, filter violations, p95 latency, and bytes written per catalog update. A small weekly sample of 1,000 queries can reveal that a supposedly expensive vector tier is used by only a narrow query class; the number is a test fixture, not a production promise. Keep the measurement job out of the customer path and attach its run ID to audit exports.
| Decision | Lean default | Move up when |
|---|---|---|
| Candidate storage | IDs, scores, channel, reason codes | Investigations require feature-level attribution |
| Evidence storage | Hashes plus catalog revision | Regulated review needs recoverable text |
| Vector coverage | Long-form and compatibility text | Labeled recall shows lexical misses |
| Retention | Hot partitions plus archive | Legal hold or dispute policy says otherwise |
The table is a starting point, not a pricing promise. Recalculate after a schema, language, or ranking change.
Failure drills expose the real architecture
I start testing with the awkward cases, because clean “boots” queries hide the expensive mistakes. In one fixture, a product is in stock when retrieval begins and out of stock when ranking finishes. The audit event should point to the eligibility snapshot used for the decision, while the response layer applies the final availability check. In another, a seller edits a title between candidate retrieval and evidence rendering. The hash mismatch must produce a visible STALE_EVIDENCE outcome and a retryable trace, not a silently altered citation.
Then test retries. A client timeout can leave the search response unknown even though the server completed it. Reusing the same trace_id for a replay lets the audit writer deduplicate the event while preserving an attempt counter. At-least-once delivery is normal; idempotent writes are the control. Also test a late catalog update, a deleted SKU, a malformed embedding, and a policy rollout that changes exclusions. Each should result in a reasoned event with enough context to reproduce the decision. For a serious drill, freeze a catalog revision, submit the same query twice, delay the evidence lookup, and then release a price update while the second attempt is waiting. Verify that both attempts point to the same retrieval inputs, that only one response is marked customer-visible, and that the price change appears as a later catalog event rather than a rewritten explanation. Repeat with a redaction request and a failed archive upload; the expected result is a durable tombstone and a retry record, not a missing span that an operator has to guess around.
I once assumed that retaining the whole candidate payload would make investigations easier. It made compaction and access reviews harder, and it encouraged people to treat the log as a second index. The correction was simple: retain identifiers and hashes by default, and require an explicit, time-bound lookup for protected evidence.
Three words: trace the boundary.
There is a catch.
This design is not suitable when the business needs sub-second, cross-region forensic queries over years of raw payloads; use a dedicated analytics or log platform for that workload and keep the serving index separate. Stick with a simple database table when traffic is low and the audit volume fits one operational region. The extra event pipeline only pays for itself when replay, policy review, or index-cost attribution is a real requirement.
A rollout that can be reversed
Ship the audit schema behind a version field and write a shadow event before changing ranking. Compare event completeness and index-write cost for a fixed evaluation window. Alert on missing catalog revisions, duplicate trace IDs, rising INDEX_LAG, and evidence hashes that cannot be resolved. Sample payloads in observability systems, but redact query text and product attributes there; the audit store is the controlled source for investigations.
During migration, dual-write only the decision metadata. Rebuilding historical vectors just to populate old audit rows increases cost without improving explainability. Backfill hashes from immutable catalog snapshots where available, mark the rest as LEGACY_UNKNOWN, and document that boundary. Roll back by stopping the new writer and leaving the serving index untouched. The audit consumer can catch up independently.
The right retrieval architecture is the one that keeps search fast, evidence addressable, and policy decisions inspectable. It may use a hosted vector service, a database extension, or a self-managed engine; those choices are secondary to the contract, snapshot discipline, and failure drills above. Pick the implementation that your operators can export, redact, replay, and retire without copying the entire catalog into an ever-growing index.
Top comments (0)