Short answer: make every restaurant menu assistant response carry an immutable evidence record that binds the user query, retrieved candidates, exact menu revisions, authorization decision, generated claims, citations, and final disposition under one trace ID. Logging only the final answer proves almost nothing; a useful audit log must let an engineer reconstruct what the retriever was allowed to see and why each factual claim was shown.
The hard constraint is grounding, not log volume. A diner can ask whether a dish contains peanuts while an operator is publishing a revised recipe and an access rule hides an internal preparation note. The assistant may retrieve plausible text and still cite the wrong revision. Preserve the retrieval boundary first. Everything else follows.
What should restaurant menu assistant retrieval architecture audit logs prove?
An audit event should answer one narrow question: given the data and policy visible at decision time, can another engineer explain the response without rerunning a changed system? This is stricter than storing a prompt and answer. Rerunning retrieval tomorrow can select a different chunk after an embedding refresh, a menu edit, or an authorization change. The event therefore needs stable identifiers and revisions for evidence, plus the decision that admitted or rejected each candidate.
For a menu assistant, I use seven logical fields. They are logical because a field such as retrieval contains a small structured object rather than an unsearchable prose blob.
-
trace_id: the correlation key shared by request handling, retrieval, generation, and delivery. -
request: normalized query, locale, channel, and authenticated policy subject. -
retrieval: index revision, retrieval configuration identifier, and ordered candidate records. -
sources: immutable menu document IDs, revisions, chunk IDs, and content digests. -
claims: the answer split into factual claims with source references. -
policy: authorization and publication-state decisions applied before generation. -
outcome: emitted, refused, or suppressed, with a reason code and timestamp.
No mystery fields.
The content digest detects a mismatch between logged evidence and later storage, but it is not a substitute for retaining the referenced revision. A digest can tell you that two byte sequences differ; it cannot recover the old allergen statement. Retention for the audit event and retention for the evidence object need a shared policy, otherwise an apparently complete record becomes unverifiable halfway through its life.
The trace identifier should follow an interoperable propagation format when requests cross services. W3C Trace Context defines traceparent and tracestate headers for that job. Treat the trace ID as correlation data, not authorization: knowing it must never grant access to a menu record or an audit event.
Define the evidence contract before adding storage
Start with a serialization contract that rejects ambiguous records. The example below uses only the Python standard library, produces canonical JSON for hashing, and makes the source revision explicit. It deliberately doesn't record hidden reasoning. Audit the inputs, observable decisions, claims, and citations instead.
from __future__ import annotations
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from hashlib import sha256
import json
from typing import Literal
@dataclass(frozen=True)
class SourceRef:
document_id: str
revision: str
chunk_id: str
content_sha256: str
@dataclass(frozen=True)
class Candidate:
source: SourceRef
rank: int
score_text: str
decision: Literal["accepted", "policy_rejected", "stale_rejected"]
@dataclass(frozen=True)
class Claim:
text: str
source_indexes: tuple[int, ...]
@dataclass(frozen=True)
class AuditEvent:
schema_version: int
trace_id: str
occurred_at: str
request: dict[str, str]
retrieval: dict[str, object]
sources: tuple[SourceRef, ...]
claims: tuple[Claim, ...]
policy: dict[str, str]
outcome: dict[str, str]
def canonical_bytes(value: object) -> bytes:
return json.dumps(
value, sort_keys=True, separators=(",", ":"), ensure_ascii=False
).encode("utf-8")
def seal(event: AuditEvent) -> dict[str, object]:
payload = asdict(event)
return {
"event": payload,
"event_sha256": sha256(canonical_bytes(payload)).hexdigest(),
}
def utc_now() -> str:
return datetime.now(timezone.utc).isoformat()
Why store a score as text? Floating-point rendering can vary across producers, while the audit contract needs a stable serialized value. A decimal string preserves what the retriever reported without pretending that scores from different retrieval configurations are comparable. Store the configuration identifier beside it. Don't use a score threshold from one index build to interpret another.
There is also a privacy edge here. OWASP's logging guidance warns against recording data such as access tokens, passwords, and sensitive personal data directly. A raw user query can contain a phone number, allergy information, or an order reference. The useful design is a classified request field with deterministic redaction before persistence, not a later cleanup job that assumes nobody read the original event. Your exact classification will depend on jurisdiction and business policy; I'm not sure a generic retention number can be defensible without those inputs.
Bind claims to menu revisions, not URLs
A citation URL is presentation. Evidence is identity plus content. If /menus/dinner changes in place, a log that stores only that URL cannot establish which ingredient list grounded yesterday's response. Assign an immutable revision to every published menu object and keep chunk identity stable within that revision. Then make generated factual claims refer to indexes in the event's sources array. Consider the question, "Does the green curry contain peanuts?" A candidate about green curry may rank first, while a separate allergen matrix is the authoritative source. The generator must not turn lexical proximity into authority. Retrieval should bring back both candidates; policy and source-type rules should determine whether the claim has acceptable evidence. If the allergen record isn't present and current, refusal is the auditable result.
from collections.abc import Sequence
def validate_claims(
claims: Sequence[Claim], sources: Sequence[SourceRef]
) -> list[str]:
errors: list[str] = []
for claim_number, claim in enumerate(claims):
if not claim.source_indexes:
errors.append(f"claim_{claim_number}:missing_source")
continue
for source_index in claim.source_indexes:
if source_index < 0 or source_index >= len(sources):
errors.append(f"claim_{claim_number}:unknown_source")
return errors
def decide_outcome(errors: Sequence[str]) -> dict[str, str]:
if errors:
return {"status": "suppressed", "reason": errors[0]}
return {"status": "emitted", "reason": "claims_grounded"}
This validator checks referential integrity, not semantic entailment. That limit matters. A claim can cite an existing chunk that does not support it, so an offline evaluation set still needs examples where reviewers decide whether the cited revision entails the claim. Keep that evaluation result separate from the production event; retroactively editing an audit event to attach a later judgment destroys the distinction between what happened and what was learned afterward. Retries are another trap. One user request can produce several attempts, each with different candidates, but only one delivered response. Give each attempt its own event ID, retain the shared trace ID, and record a parent attempt when applicable. The delivery event should point to the emitted attempt. Otherwise two individually valid records can make it appear that two answers reached the user.
How can teams expose failure modes without leaking sensitive data?
The storage architecture must support append-only writes, restricted reads, retention enforcement, and integrity checks. Those are capabilities, not a claim that one database engine magically makes records trustworthy. An administrator who can replace both an event and its digest defeats a digest stored beside the event. Stronger tamper evidence needs separation of duties or a hash chain whose checkpoints are held in a different trust domain. The catch is operational weight: key custody, checkpoint recovery, and access review become part of the system.
| Failure mode | Evidence symptom | Control | Remaining limit |
|---|---|---|---|
| Menu changed after retrieval | Logged revision differs from current revision | Retain immutable source revisions | Retention expiry can remove old evidence |
| Unauthorized chunk entered context | Candidate has no allow decision | Filter before generation and log policy result | Policy configuration can still be wrong |
| Citation points to a related dish | Claim cites a valid but non-entailing chunk | Run claim-level entailment review | Automated checks can misclassify nuance |
| Retry delivered twice | Multiple attempts marked emitted | Separate attempt and delivery events | Downstream channels need idempotency too |
| Query contains sensitive data | Raw identifiers appear in request fields | Classify and redact before persistence | Over-redaction can reduce diagnostic value |
| Log was altered | Event digest or chain verification fails | Isolate checkpoints and restrict mutation | Compromised trust domains weaken assurance |
Never log credentials. Keep public menu facts and restricted operational records in different authorization classes even if retrieval uses one logical interface. A restaurant's unpublished recipe note is not made safe by sitting next to a public dish description in a vector index. Access filtering has to happen before candidate text reaches the generation context; post-generation redaction can't prove the model did not use restricted material.
Observability needs a narrower dataset than forensics. Export counters for refusal reasons, missing citations, stale-revision rejection, and audit-write latency, but avoid copying full queries and chunks into every telemetry backend. The primary audit store remains the controlled evidence record. This split reduces the number of systems holding sensitive text and makes operational dashboards cheaper to scan, although it means an on-call engineer follows a trace ID into a separately authorized system for detail.
Choose storage from the verification workload
Don't begin with a brand name. Write down the reads an investigation must perform: fetch one trace and all attempts, fetch the exact source revisions, verify event integrity, enumerate access to an event, and prove retention or deletion according to policy. Then test candidate storage designs against those reads and their failure domains.
An append-oriented object store can fit large event bodies and retained evidence revisions, while a transactional index can serve trace lookup and retention queues. A single transactional store is simpler at modest volume. A streaming log can distribute events to independent controls, but replay, compaction, and consumer authorization add policy surfaces. None is universally correct. The decisive question is which system owns truth when an index entry exists but its evidence object does not. Define that state transition before deployment.
For strict investigations, use a write sequence with an explicit pending state: persist the immutable evidence object, append and seal the audit event that references it, then publish the lookup index. A reconciler can find pending objects that never received an event and events not yet indexed. It must not invent success; it reports a typed incomplete transaction for internal repair while the assistant suppresses an answer unless its evidence commit completed. This is a capability boundary, not an outage story.
Some teams won't need cryptographic chaining. If the threat model is accidental mutation and access is already tightly separated, immutable revisions, ordinary digests, and access logs may be proportionate. Stick with a simpler transactional design when the same small team operates every component and cannot sustain independent key custody. Choose separated checkpoints when audit administrators themselves are inside the threat model. Your mileage may vary because the right boundary comes from the threat model, not the retrieval framework.
Roll out with shadow evidence and refusal tests
Start compactly. In the first release, generate audit events in shadow mode and compare them with delivered answers, but don't let shadow logging alter response behavior. Sample traces across menu edits, locale variants, access classes, and retries. Verify that every emitted factual claim names a retained source revision and that no rejected candidate text reaches generation.
Next, enforce suppression for missing or invalid source references. Add contract tests for schema evolution: an old reader must either understand a new optional field or reject a new major schema version explicitly. Finally, rehearse an investigation by selecting a delivered answer and reconstructing its claims from retained bytes without querying the live retrieval index. If that exercise requires today's index, the audit design is recording clues rather than evidence.
Ship the invariant, not the dashboard: no factual menu claim is delivered unless its authorized, immutable source revision is committed and addressable from the same trace.
Top comments (0)