Use immutable document revisions, retrieve against an explicit version policy, and make every answer cite the exact revision that supplied its evidence. Short answer: internal engineering documentation should be indexed as an append-only history, with a separate pointer for "current," because overwriting chunks destroys the evidence needed to explain an answer after a page changes.
Consider a media engineering team that watches a public policy page, records each meaningful diff, and alerts editors. Its internal runbook explains how that watcher behaves. At 09:10 the runbook may say that whitespace-only changes are ignored; at 14:30 revision 17 may change that rule. A retrieval system that quietly replaces the 09:10 chunks can produce a plausible answer, but it cannot ground that answer in the state an operator actually saw during the earlier alert.
That is an audit failure, not a ranking quirk.
The five practices below form a runbook for avoiding it. They apply to keyword, vector, and hybrid retrieval; the deciding issue is version semantics, not the choice of embedding model.
1. How should retrieval architecture version internal engineering documentation?
Give the logical document and its immutable revision different identities. A logical ID answers "which runbook is this?" A revision ID answers "which exact bytes did retrieval use?" Store both on every chunk, along with a content hash, source timestamp when one is trustworthy, ingestion timestamp, and lifecycle state. A useful minimum record is document_id, revision_id, chunk_id, content_hash, valid_from, valid_to, observed_at, and status.
Do not use a mutable URL as the revision ID. The monitored page can keep the same URL while its body changes, and an internal page slug can survive dozens of edits. Build a revision from a stable source identity plus a collision-resistant digest of normalized content. Keep the original bytes or an immutable snapshot beside the extracted text so an investigation can reproduce parsing and chunking. HTTP validators such as ETag and Last-Modified are useful fetch signals, but RFC 9110 defines their validator roles; neither should be treated as a universal content identity [2]. A weak ETag, in particular, does not promise byte-for-byte equivalence.
The "current" view is then only a pointer or filtered projection. Updating that pointer must not delete revision 16 when revision 17 arrives. This separation makes two retrieval policies explicit: current-state questions search the active revision, while incident or audit questions search the revision valid at a requested time.
Tiny distinction. Large consequence.
For the page-change watcher, keep the raw capture, the normalized body, and the diff as related artifacts under one revision. The alert can cite the diff, the assistant can retrieve the runbook, and both can point back to exact evidence. If the source republishes identical normalized content, record the observation without creating duplicate chunks; if the bytes change but normalization erases the difference, retain enough metadata to show why no semantic revision was opened.
2. Separate change detection from index publication
A crawler seeing new bytes is not the same event as a new revision becoming searchable. Treat fetching, parsing, chunking, embedding, validation, and publication as separate states. Publish only after every expected chunk and its metadata are durable. Until then, queries continue using the prior active revision.
This is the part most likely to page an operator. Suppose revision 17 produces 12 chunks and a query races the indexing worker after chunk 7. An in-place update can return a mixed answer: five passages from revision 16, seven from revision 17, and a citation that identifies only the page URL. No single version of the runbook ever contained that evidence set. Retries make the race harder to diagnose because the next query may look correct. The safe publication unit is therefore the complete revision, not an individual chunk, and promotion should be one compare-and-swap operation against the expected prior revision.
Make ingestion idempotent. The same (document_id, revision_id, chunk_id) written twice should converge on the same record, and an already published revision should not generate a second change alert. A queue may deliver more than once; pretending otherwise turns routine redelivery into duplicate citations and duplicate notifications. Use a deterministic idempotency key and acknowledge work only after the durable state transition.
One guard matters more than clever retry math: never let an older observation replace a newer active pointer. Compare source sequence data when the source provides it; otherwise compare ingestion ordering under a per-document lock and retain observed_at as evidence, not as an infallible clock. Clock skew and delayed fetches make wall time an unsafe sole authority.
3. What version policy should each retrieval request declare?
The retriever should require a policy such as Current, AsOf, or AllRevisions. Leaving version selection implicit invites different services to invent different meanings of "latest." It also makes tests vague.
Here is a small Go boundary. It rejects time-travel retrieval without a cutoff and applies revision filtering before scoring, so a high-similarity stale chunk cannot slip into a current-state answer.
package retrieval
import (
"errors"
"time"
)
type VersionMode string
const (
Current VersionMode = "current"
AsOf VersionMode = "as_of"
AllRevisions VersionMode = "all_revisions"
)
type Query struct {
Text string
Mode VersionMode
At *time.Time
}
type Chunk struct {
DocumentID string
RevisionID string
ChunkID string
ContentHash string
ValidFrom time.Time
ValidTo *time.Time
Text string
}
func Eligible(q Query, c Chunk) (bool, error) {
switch q.Mode {
case Current:
return c.ValidTo == nil, nil
case AsOf:
if q.At == nil {
return false, errors.New("as_of retrieval requires a timestamp")
}
startsBeforeCutoff := !c.ValidFrom.After(*q.At)
endsAfterCutoff := c.ValidTo == nil || c.ValidTo.After(*q.At)
return startsBeforeCutoff && endsAfterCutoff, nil
case AllRevisions:
return true, nil
default:
return false, errors.New("unknown version mode")
}
}
Filtering after approximate nearest-neighbor retrieval can leave too few eligible results, depending on the engine and filter execution plan. Measure that behavior with your corpus. Elasticsearch, OpenSearch, and Qdrant all document vector search with metadata filtering, but their query syntax, execution controls, and consistency model differ [4][5][6]. Those are implementation boundaries, not a ranking: use the engine already operated well by the team when it can enforce the version predicate and return enough candidates under load. A relational store with a vector extension can also fit a modest corpus when transactional publication matters more than specialized search operations.
The catch is that append-only history consumes more storage and requires retention rules. It is not suitable to retain every revision forever when documents contain data that must be erased, or when policy permits only a short history. In that case, keep the identity and tombstone needed for audit, delete prohibited content from every derived store, and document the resulting limit on historical replay. For a very small handbook with no time-travel or incident reconstruction requirement, a single current snapshot plus source-control history may be simpler. Stick with that simpler design until the retrieval service genuinely needs online historical queries.
4. Bind citations to evidence, not display locations
A grounded response needs a citation record that survives another edit. Store the document ID, revision ID, chunk ID, quoted span or offsets, retrieval timestamp, and snapshot locator with the generated answer. A friendly URL may still be displayed, but it is not enough: after a page changes, it points readers at different evidence.
Citation assembly should fail closed. If the answer sentence cannot be mapped to one or more retrieved chunks from the selected revision policy, omit the sentence or mark it unsupported; do not borrow a nearby URL. The original RAG paper describes generation conditioned on retrieved passages [1]. Operational grounding adds a stricter requirement: the passage identity must remain resolvable after the corpus evolves.
For the media watcher, an alert might state, "The retention period changed from 30 to 45 days," and cite revision 17's diff plus the captured source span. An internal answer about the alerting rule should cite the runbook revision active when that alert was processed. These may be different documents and timestamps. Keeping both edges turns a vague chain of trust into a trace: answer to runbook chunk, runbook chunk to immutable revision, alert to source diff, and diff to captured page.
I'm not sure a single citation-quality threshold transfers between teams. Legal review, on-call diagnosis, and casual discovery tolerate different levels of unsupported synthesis. Resolve that uncertainty with an evaluation set labeled by the people who consume the answers, rather than picking a universal score.
5. Verify publication and rehearse rollback
Test version behavior as a state machine, not as a collection of happy-path searches. Start with revision 16 active, ingest revision 17 twice, delay one chunk, and query during each transition. Current retrieval must return only revision 16 before promotion and only revision 17 after it. An AsOf query on the earlier interval must continue to return revision 16. Then deliver revision 16 late and verify that the active pointer does not move backward.
The release check should include four invariants:
- Every returned evidence set contains revisions allowed by the request policy.
- Every citation resolves to stored content with the same hash used at generation time.
- Replaying an ingestion event creates neither extra chunks nor another alert.
- Promoting a revision is atomic from the query path's perspective.
Observe counters for fetched revisions, deduplicated observations, validation rejections, publication lag, stale-pointer compare failures, citation resolution failures, and results removed by version filters. Alert on violated invariants and sustained lag, not on every retry. A retry is expected; a mixed-version answer is not.
Rollback should move the active pointer to a previously validated revision without rebuilding it. Record the rollback as a new publication event, including operator or automation identity and reason, while leaving the rejected revision available according to retention policy. Stop new publication if citation resolution fails, keep serving the last known valid revision, and resume only after a canary query set passes. This is why immutable revisions earn their storage: recovery is a metadata transition rather than a hurried reconstruction of yesterday's index.
Ship the design only when an operator can answer three questions from stored evidence: which revision was searchable, why it became active, and which exact chunks supported an answer. If any answer depends on the current page contents, document versioning is incomplete.
References
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401
- RFC 9110, HTTP Semantics, validators and conditional requests: https://www.rfc-editor.org/rfc/rfc9110
- NIST Secure Software Development Framework, version 1.1: https://csrc.nist.gov/pubs/sp/800/218/final
- Elasticsearch k-nearest neighbor search and filtering: https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html
- OpenSearch vector search filters: https://docs.opensearch.org/latest/vector-search/filter-search-knn/index/
- Qdrant filtering: https://qdrant.tech/documentation/concepts/filtering/
Top comments (0)