Here's an illustrative support question. A customer says your help-center assistant told them refunds are allowed within 30 days. Your policy says 14. So which was it: did the model make it up, or did retrieval hand it an old version of the refund page?
If you logged only the answer, you can't tell. And if you logged the full retrieved text, you now have a second copy of your knowledge base, including internal or customer documents, sitting in logs with weaker access controls than the source.
There's a middle path: record references to what was retrieved, plus enough to check later whether those references still point at the same text.
What you need to answer later
When someone questions a RAG answer, the useful questions are narrow:
- Which documents and chunks came back, in what order, with what scores?
- Which of those actually went into the prompt? (Retrieval often returns more than you send.)
- Which version of each document was it?
- Which index was searched, and with which filters, such as tenant or permission scope?
- Has any of that text changed since?
None of these needs the text itself, only stable IDs, versions and a fingerprint of each chunk.
question -> retriever -> top_k chunks -> prompt -> model -> answer
| |
v v
index version per chunk: doc id, version, chunk id,
filters, top_k rank, score, sha256, sent to model?
| |
+------+-------+
v
retrieval record (no text)
A small retrieval record
import hashlib
import hmac
INDEX_VERSION = "kb-2026-11-10" # bump when you re-chunk or re-embed
def chunk_hash(text: str) -> str:
# Plain SHA-256 of the exact text sent to the model.
return hashlib.sha256(text.encode("utf-8")).hexdigest()
def query_ref(query: str, key: bytes) -> str:
# Keyed, so a short query can't be guessed by hashing candidates.
return hmac.new(key, query.encode("utf-8"), "sha256").hexdigest()
def retrieval_record(answer_id, query, hits, filters, top_k, key):
return {
"answer_id": answer_id,
"index_version": INDEX_VERSION,
"query_ref": query_ref(query, key),
"filters": filters, # e.g. {"tenant": "acme", "acl": "support"}
"top_k": top_k,
"chunks": [
{
"doc_id": h["doc_id"],
"doc_version": h["doc_version"],
"chunk_id": h["chunk_id"],
"rank": rank,
"score": round(h["score"], 4),
"sha256": chunk_hash(h["text"]),
"sent_to_model": h["sent"],
}
for rank, h in enumerate(hits, start=1)
],
}
def check_chunk(entry, load_chunk):
# Later: did this chunk change since the answer was produced?
text = load_chunk(entry["doc_id"], entry["doc_version"], entry["chunk_id"])
if text is None:
return "missing"
return "same" if chunk_hash(text) == entry["sha256"] else "changed"
The chunk hash covers the exact text you sent, after whatever cleanup your pipeline does. Hash the raw document instead and you can't tell whether a change in chunking or cleaning altered what the model saw.
The query is keyed, not plain-hashed. User questions are often short and predictable, so a plain SHA-256 can be matched by hashing likely questions. An HMAC with a server-side key stops that, and Python's hmac docs recommend compare_digest() over == when you verify one later. The same reasoning applies to very short chunks, such as a single policy line. If guessing them would matter, key those hashes too.
sent_to_model is separate from rank. If you rerank, truncate to fit a context window or drop near-duplicates, the top-ranked chunk may never reach the prompt. Recording both lets you tell "retrieval missed it" from "we dropped it".
index_version changes when you re-chunk or re-embed, not only when documents change. Otherwise last month's chunk IDs quietly point at different boundaries.
Two tests pin this down:
def test_record_holds_no_text():
rec = retrieval_record("a-1", "can I get a refund?", HITS,
{"tenant": "acme"}, 5, KEY)
blob = json.dumps(rec)
assert "Refunds within" not in blob
assert "can I get a refund" not in blob
def test_detects_edited_chunk():
rec = retrieval_record("a-1", "q", HITS, {}, 5, KEY)
store = {("refund-policy", "v7", "c3"): "Refunds within 14 days."}
assert check_chunk(rec["chunks"][0], lambda *k: store.get(k)) == "changed"
assert check_chunk(rec["chunks"][1], lambda *k: store.get(k)) == "missing"
Why not store the text, or the embeddings?
Storing retrieved text in logs makes the logs a copy of the knowledge base, with the same sensitivity and usually looser controls. OWASP's LLM08:2025 Vector and Embedding Weaknesses lists unauthorized access and cross-tenant leakage among the main RAG risks. Its mitigations include permission-aware stores and detailed logs of retrieval activity. A record like this one is a lightweight form of that log.
Embeddings are no safer as a stand-in. The same OWASP entry describes embedding inversion attacks that recover significant source information from vectors. A chunk ID and hash give a reviewer what they need without carrying the content.
Making the reference checkable later
A reference is only useful if you can resolve it. That puts a requirement on the source system: keep old document versions for as long as you keep the records that point at them. If your knowledge base overwrites pages in place, doc_version is a label you can't follow, and check_chunk can only tell you the text is different or gone.
In the refund example, the review becomes a lookup. Find the answer's record. Check whether refund-policy v7, chunk c3, was sent. Load that version and compare hashes. If v7 said 30 days, retrieval served stale content. If no sent chunk mentioned refunds at all, look harder at the model.
Limits
- A hash tells you text changed, not how. You still need versioned sources to see the old wording.
- Filters are recorded as your code applied them. If permissions were wrong at query time, the record shows the wrong scope faithfully; it doesn't catch the mistake.
- Scores aren't comparable across index versions or embedding models.
- This shows what the model was given, not why it answered the way it did.
Where this fits
We work on Obligra Verify, which is designed to retain the evidence your application selects, including references to retrieved context. Our page on the structure of a decision record shows where retrieved context sits alongside the rest of a record.
This article was generated by an AI agent (Claude) for the Obligra Verify team and checked against the linked sources.
Top comments (0)