Property Operations Search: Freshness Windows for Cited Engineering Documentation

#rag #semanticsearch #python #documentation

Short answer: use a change-aware retrieval window, attach a source revision to every chunk, and refuse an answer when the cited revision is older than the policy allows. That is the least complex design that keeps property-management engineers from acting on stale runbooks.

In a property platform, the documents are ordinary engineering material: lease-import runbooks, maintenance webhook contracts, and incident notes. The risk is not that a model cannot find a similar paragraph. The risk is a confident answer citing a page whose queue name changed last night.

The failure starts before generation

Most teams put a updated_at field on a document and call the problem solved. It is not. A page can change while its embedding job waits in a queue, or a crawler can re-fetch an unchanged URL and make the index look fresh. Retrieval needs two clocks: source freshness and index freshness.

I model each chunk with source_revision, observed_at, and indexed_at. The revision can be a Git commit, a Confluence version, or a content hash. A freshness window is then a policy, not a guess: operational procedures may allow 24 hours, while an API contract may require 15 minutes. The answer service compares the age of the cited revision with that policy before it writes prose.

The small but important detail is deletion. A removed runbook must produce a tombstone that the retriever sees immediately; otherwise an old vector remains eligible and looks like a valid citation.

For example, when the maintenance team changes a webhook field from unit_id to property_id, the change event should carry that revision even if the page title stays the same. The citation then points to the exact contract the caller tested against.

How should a retrieval architecture keep engineering documentation fresh and cited?

Treat ingestion as a state machine. A watcher emits a document change, a normalizer creates stable sections, and an indexer replaces chunks for one revision atomically. The query path filters by freshness policy, retrieves candidates, and passes both text and citation metadata to the generator. If no candidate satisfies the window, return a review-needed result instead of silently widening the window.

Here is a compact Python sketch. It uses plain interfaces so the same policy can sit in front of a hosted vector store, an in-house search engine, or a test double.

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from typing import Iterable, Protocol


@dataclass(frozen=True)
class Chunk:
    text: str
    url: str
    source_revision: str
    observed_at: datetime
    indexed_at: datetime
    deleted: bool = False


class Retriever(Protocol):
    def search(self, query: str, limit: int) -> Iterable[Chunk]: ...


def eligible_chunks(
    retriever: Retriever,
    query: str,
    now: datetime,
    max_age: timedelta,
    limit: int = 8,
) -> list[Chunk]:
    """Return only live chunks observed inside the freshness window."""
    cutoff = now - max_age
    return [
        chunk
        for chunk in retriever.search(query, limit=limit)
        if not chunk.deleted and chunk.observed_at >= cutoff
    ]


def answer_context(chunks: list[Chunk]) -> str:
    if not chunks:
        return "REVIEW_REQUIRED: no source satisfies the freshness policy."
    return "\n\n".join(
        f"[{i}] {c.text}\nSource: {c.url}\nRevision: {c.source_revision}"
        for i, c in enumerate(chunks, start=1)
    )

The generator should be constrained to cite the numbered sources it receives. Store the selected revision alongside the response, too. That makes a later audit possible even after the document is edited again.

Choosing a window without hiding risk

There is no universal number. A practical starting table looks like this:

Document class	Example	Initial window	Reason to tighten
API contract	tenant webhook schema	15 minutes	breaking deployments
Operational runbook	failed rent-payment replay	24 hours	frequent procedure edits
Architecture note	search tenancy boundaries	7 days	design review cadence
Historical incident	2024 queue outage	no freshness filter	explicitly historical

The catch is that a tight window increases abstentions during an indexing delay. That is preferable to inventing certainty, but it needs a visible queue metric and an escalation path. Teams with a mostly static corpus can use a longer window and schedule a nightly verification job. Teams shipping daily should use event-driven invalidation and keep a short grace period only for documents marked informational.

I initially expected one global window to be easier to explain. It made our test set look tidy and hid the important distinction between a contract and a postmortem. Per-class policies add a field and a little test setup; they also make the decision reviewable.

Evaluate grounding, not just retrieval scores

Recall@k cannot tell you whether a citation is current. Build an eval set from real edits: rename a queue, revoke a field, delete a runbook, and publish a corrected procedure. For each question, record the expected revision and whether abstention is acceptable.

Track citation precision (does the cited chunk support the claim?), citation freshness (is its revision inside the policy?), and stale-answer rate. A useful failure assertion is simple: after a tombstone event, the deleted URL must never appear in an answer context. Run these checks in CI against a deterministic index fixture, then sample production traces for drift. I'm not sure a single threshold will fit every organization; your mileage may vary when edits are infrequent or approval workflows are slow.

Emit an event for create, update, and delete. Persist the revision before embedding. Replace a document's chunk set atomically, and expose the age of the oldest eligible revision as a dashboard metric. Log query, selected URLs, revisions, and the freshness policy, but redact tenant data. When the index falls behind, show “review required” with the source links; do not silently expand the window. Re-run the edit-based eval whenever chunking, embedding models, or ranking changes.

References

The retrieval-augmented generation formulation is described in the original research paper:

https://arxiv.org/abs/2005.11401

DEV Community