Short answer: assign each document class a freshness window, verify the source on that schedule, and keep expired chunks out of retrieval until one complete new index generation is ready. For a logistics team watching an operating-status page, that hybrid design keeps normal queries fast without letting yesterday's instructions masquerade as current guidance.
The least complex useful pipeline has five stages: fetch the page, extract its meaningful text, compare a stable content hash, build a new chunk generation only when the hash changes, and apply a freshness check before retrieval. Freshness describes source verification, not merely the time a chunk entered the index. A nightly ingest timestamp cannot prove that a page was still current at noon.
How should internal engineering documentation retrieval enforce freshness windows?
Treat the window as an eligibility rule attached to a document class. A short window belongs to a page whose operational instructions can change during the working day. A longer window can fit a design record that changes through review. Those durations are local policy choices, not universal constants; they should come from the cost of serving an obsolete answer and the source's realistic change rate.
For one illustrative logistics deployment, I would start with a 5-minute window for a carrier disruption page, 6 hours for a warehouse runbook, and 7 days for an approved architecture record. These are configuration examples, not measured recommendations. I'm not sure which cadence your pages need until update history and stale-answer impact are visible together.
Keep four times or versions separate:
-
observedAt: the last successful check of the source -
contentVersion: a digest of canonical extracted content -
indexedAt: when that exact version became queryable -
generation: the complete set of chunks created from one version
A document is fresh when now - observedAt is inside its configured window and its active generation was built from the observed content version. An unchanged check advances observedAt; it does not require embedding the same text again. A changed check creates a new generation and promotes it only after every chunk is ready.
That distinction does real work. It avoids paying to reprocess an unchanged page, and it prevents a half-rebuilt document from mixing old instructions with new ones. Fast queries remain a lookup against already prepared chunks. Freshness is enforced by a small metadata test.
One minute.
If that test fails, don't quietly retrieve the expired chunks. Return an explicit stale-source state, use an approved live-source path if the application has one, or tell the caller that current evidence is unavailable. The right fallback depends on the workflow: a drafting assistant may tolerate a labeled older source, while an operational alert should usually wait for a verified update.
Build the watcher before tuning retrieval
The following TypeScript models the core after page-specific extraction. It is runnable as-is with the included sample snapshots, so the freshness and promotion behavior can be tested without a network dependency. In production, feed ingestObservation the main-content text from the logistics page; navigation, clocks, and rotating banners should be removed by that page's extractor before hashing.
import { createHash } from "node:crypto";
type ActiveDocument = {
id: string;
observedAt: number;
indexedAt: number;
contentVersion: string;
generation: number;
chunks: string[];
};
const active = new Map<string, ActiveDocument>();
function canonicalize(text: string): string {
return text.replace(/\s+/g, " ").trim();
}
function digest(text: string): string {
return createHash("sha256").update(text).digest("hex");
}
function chunk(text: string, wordsPerChunk = 80): string[] {
const words = text.split(" ");
const chunks: string[] = [];
for (let start = 0; start < words.length; start += wordsPerChunk) {
chunks.push(words.slice(start, start + wordsPerChunk).join(" "));
}
return chunks;
}
function ingestObservation(id: string, extractedText: string, now: number): string {
const text = canonicalize(extractedText);
const contentVersion = digest(text);
const current = active.get(id);
if (current?.contentVersion === contentVersion) {
active.set(id, { ...current, observedAt: now });
return "verified-unchanged";
}
const candidate: ActiveDocument = {
id,
observedAt: now,
indexedAt: now,
contentVersion,
generation: (current?.generation ?? 0) + 1,
chunks: chunk(text)
};
// A real index writes all candidate chunks before switching this pointer.
active.set(id, candidate);
return "promoted-new-generation";
}
function retrieve(id: string, now: number, freshnessWindowMs: number): string[] {
const document = active.get(id);
if (!document || now - document.observedAt > freshnessWindowMs) {
return [];
}
return document.chunks;
}
const first = Date.parse("2026-09-02T08:00:00Z");
ingestObservation("carrier-status", "Terminal A accepts refrigerated freight.", first);
ingestObservation("carrier-status", "Terminal A accepts refrigerated freight.", first + 60_000);
console.log(retrieve("carrier-status", first + 120_000, 300_000));
The Map stands in for two durable stores: an index that holds immutable generations and a metadata record that points to the active one. The comment marks the transaction boundary. A production implementation must not move that pointer until chunking, embedding, and index writes have all completed for the candidate. If any step stops early, the prior generation stays active and its original observedAt is not extended by an unverified result.
The code returns an empty candidate set when freshness expires. That makes stale evidence observable instead of blending it into a plausible answer. It also separates two latency budgets: source checks happen off the query path, while query-time work is a timestamp comparison plus normal retrieval.
Diff quality sets retrieval quality
A raw HTML diff is usually the wrong signal for a page watcher. Logistics pages often contain layout chrome, timestamps, menus, or counters that are irrelevant to the operating instruction. Hashing that noise causes pointless generations; stripping too much can hide the sentence that changed a dock cutoff. The extractor is therefore part of the retrieval system, even though it runs before chunking.
Use a fixture made from a saved page and assert which text reaches the canonical form. Then make three mutations: change only navigation, change one operational sentence, and remove an entire instruction block. The first should preserve the content version. The other two should create a new one. This is where I spend test effort first — a better similarity function cannot recover text that the extractor discarded. Deletion deserves its own test as part of the same fixture suite. When a source paragraph disappears, the promoted generation must omit its chunks; appending changed chunks to an old collection leaves revoked guidance searchable. Stable chunk identifiers can reduce reprocessing, but generation membership still needs to define exactly what is live. Don't use semantic similarity to decide whether a source changed, either. A content digest answers the binary indexing question without a threshold, while semantic methods belong later, where retrieval ranks candidate chunks for a query. Keeping those decisions separate makes both easier to inspect and gives a failed test one clear owner.
Hash the content.
The latency trade-off is a policy decision
There are three reasonable places to pay for freshness. Scheduled indexing pays before queries and bounds staleness by the check interval, but frequent checks consume fetch capacity even when nothing changes. Query-time fetching sees the latest source, yet adds network and extraction delay to every affected request. A watcher with query-time eligibility checks sits between them: it refreshes off-path, skips unchanged indexing work, and refuses to treat an overdue verification as fresh.
The catch is that the hybrid is not suitable when every answer must reflect a source mutation immediately. In that case, use a source event to trigger ingestion or perform an authoritative live lookup in the request path. Stick with simple scheduled indexing when documents change rarely, a missed window has low impact, and the extra watcher state would cost more to operate than it saves.
For retrieval quality, measure the outcome at the version boundary. Build evaluation cases whose expected evidence belongs to a known content version, run them before promotion, and keep the previous generation addressable for rollback. For latency, split source-check duration, rebuild duration, promotion lag, and query duration. One aggregate percentile can hide a fast retriever waiting behind a slow live fetch.
Cost follows the same decomposition. Count source checks, changed documents, chunks rebuilt, embedding inputs, and retrieval requests separately. I don't use price as the deciding argument here; the useful optimization is structural: unchanged verification should not trigger chunking and embedding. Your mileage may vary when fetching the source dominates the bill.
Operate the window, not just the index
Deployment needs a small state machine rather than a cron expression and hope. Record each scheduled check, whether canonical content changed, the candidate generation, promotion time, and current freshness status. Alert before high-impact documents expire so the system has time to retry verification. Apply backoff to repeated source-check failures, but never advance observedAt until the source was actually verified.
Watch the ratios between checks, changes, and promotions. Many changes with no meaningful source edits point to a noisy extractor. Changes without promotions point to work accumulating between detection and indexing. Promotions without the expected retrieval evidence point to chunking, metadata filters, or evaluation coverage. These are diagnostic relationships, not universal alarm thresholds; set thresholds from the update history of each document class.
The operational checklist is short in prose. Give every document an owner and a freshness class. Test extraction against saved fixtures. Keep candidate generations isolated, switch them atomically, and retain the prior generation long enough to reverse a bad content update. Exercise expiry and deletion in deployment tests. Finally, make the answer layer report stale evidence distinctly from no matching evidence, because those states require different action.
The resulting architecture is deliberately boring: verify, hash, rebuild only on change, promote as a unit, and gate retrieval by age. That is a good fit for internal engineering documentation where query latency matters but an old logistics instruction can be worse than no answer.
References
- Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks": https://arxiv.org/abs/2005.11401
Top comments (0)