Short answer: treat crawl boundaries as a product policy, then make freshness an explicit field in retrieval. For a property-management support bot, index approved help articles, lease-policy explanations, and maintenance playbooks while excluding resident records, private tickets, and pages whose meaning depends on a live account. A small Python pipeline can enforce that boundary before embeddings are created, and an eval set can prove that updates become visible without leaking tenant context.
The important choice is deciding what the bot is allowed to know, how quickly that knowledge should change, and what evidence it must show when an answer is challenged. Similarity search is only one part of that contract.
Why crawl scope is a retrieval control
A crawler is often treated as plumbing: fetch URLs, parse text, chunk it, and hand chunks to a retriever. In support, every fetched page becomes a candidate fact, even when the page was written for an administrator, contains a stale PDF, or exposes a resident-specific URL. Retrieval architecture starts with authorization and provenance, not similarity search.
For a property-management team, write a crawl policy that someone outside the search team can review. public_help may include approved articles such as “How to report a leaking faucet.” internal_playbook may include technician escalation rules. resident_portal and ticket_detail should be out of the general corpus because their answers require identity and current account state. A page in the wrong class is not merely noisy; it can make a fluent answer unsafe.
I keep four boundary signals with each document: source class, canonical URL, last-modified timestamp, and an owner. The owner matters when an article says “call the emergency line” but the number has changed. A crawler that cannot identify who updates a page should put it in a review queue instead of silently treating it as current.
Short metadata beats clever prompts.
The RAG paper by Lewis et al. describes retrieval as a way to augment generation with external evidence; in production, that evidence needs an accountable scope and date. The model cannot infer either one from prose reliably.
How can retrieval architecture keep a customer-support knowledge base inside crawl boundaries?
The implementation below uses generic interfaces so the policy can sit in front of any crawler and vector index. It filters URLs, rejects disallowed source classes, and carries a freshness deadline into the record. The example is intentionally boring. Boring gates are easy to test.
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from urllib.parse import urlparse
ALLOWED_HOSTS = {"help.example-property.com", "ops.example-property.com"}
ALLOWED_PATH_PREFIXES = ("/articles/", "/maintenance/", "/policies/")
BLOCKED_PATH_PARTS = ("/resident/", "/ticket/", "/account/")
@dataclass(frozen=True)
class Page:
url: str
text: str
source_class: str
last_modified: datetime | None
owner: str | None
def in_scope(url: str) -> bool:
parsed = urlparse(url)
if parsed.scheme != "https" or parsed.hostname not in ALLOWED_HOSTS:
return False
if any(part in parsed.path for part in BLOCKED_PATH_PARTS):
return False
return parsed.path.startswith(ALLOWED_PATH_PREFIXES)
def freshness_deadline(source_class: str, now: datetime) -> datetime:
windows = {
"public_help": timedelta(days=30),
"internal_playbook": timedelta(days=7),
"policy": timedelta(days=14),
}
return now - windows.get(source_class, timedelta(days=0))
def admit(page: Page, now: datetime) -> dict | None:
if not in_scope(page.url):
return None
if page.source_class not in {"public_help", "internal_playbook", "policy"}:
return None
if not page.owner or not page.text.strip():
return None
cutoff = freshness_deadline(page.source_class, now)
if page.last_modified is not None and page.last_modified < cutoff:
return None
return {"text": page.text, "metadata": {
"canonical_url": page.url,
"source_class": page.source_class,
"last_modified": page.last_modified.isoformat() if page.last_modified else None,
"owner": page.owner,
}}
now = datetime.now(timezone.utc)
There are two deliberate details here. First, the URL check is an allow-list with a small deny-list for paths that can carry identity. Second, freshness is class-specific: a maintenance playbook may need a seven-day window, while a stable how-to article can tolerate thirty days. Those are policy defaults, not universal truths. Your mileage may vary; measure unanswered questions and policy changes before tightening the windows.
Ship it only after the boundary has a paper trail. In a real property-management corpus, one “maintenance” page can contain a general boiler reset, a contractor phone number, and a copied resident note in the same HTML. The crawler should record the URL and hash before parsing, the parser should preserve headings and warning labels, and an admission test should show why the page was accepted. When the owner edits the phone number, the hash changes, the old chunks are retired, and the next eval asks both the new and old question. This makes freshness observable instead of a hope hidden inside a vector score. It also gives support leads a useful answer to “why did the bot cite this?” without opening a notebook or guessing which job ran overnight.
Small gates compound.
Do not embed a page before admit runs. If a document later becomes stale, mark it inactive in the index or filter it at query time using last_modified. Keeping stale vectors around is useful for audit, but they should not compete with current evidence.
What fails when boundaries are left implicit?
The first failure mode is scope creep through navigation. A help article links to a resident portal, the crawler follows it, and an otherwise public index now contains names, balances, or ticket text. Robots exclusions are useful operational hints, but they are not an authorization model. Enforce scope in code and at the fetcher’s credentials.
The second is freshness drift. A chunk can be semantically close to a question while describing an old fee or emergency number. Similarity scores do not know that a policy changed yesterday. Store update timestamps, boost newer approved records, and return the date in citations so a support agent can challenge the answer quickly.
The third is chunk-boundary damage. Splitting a maintenance procedure every 300 tokens may separate the safety warning from the action it qualifies. I prefer structure-aware chunks: keep a heading, prerequisites, numbered steps, and exception text together, then cap size only after those units are formed. A little redundancy is cheaper than an answer that omits “turn off the water main.”
A fourth failure appears in evaluation. Teams test “Where is the mailbox?” and celebrate a high retrieval score, but skip adversarial questions such as “Show me my neighbor’s open ticket” or “What was the fee before the 2026 policy update?” Your eval set should contain boundary violations, stale-document traps, and citations that point to the exact canonical URL.
Which architecture keeps updates visible without widening access?
Use two paths with one policy. A scheduled crawl discovers and admits documents; an event path handles urgent policy changes. Both paths produce the same record shape and pass the same scope checks. The query service then applies tenant or staff authorization separately from corpus admission. This separation prevents a broad internal playbook from becoming visible to every resident simply because it was indexed.
For each admitted record, retain a content hash, fetch time, parser version, and source owner. A changed hash triggers re-chunking and re-embedding; an unchanged hash only refreshes crawl metadata. That distinction keeps token and embedding work predictable while preserving a clear audit trail.
At query time, retrieve a modest candidate set, filter by authorization and freshness, and ask the generator to quote or link the surviving evidence. If no candidate meets the boundary policy, return an explicit “I do not have an approved current source” response. That is a successful control outcome, not a retrieval failure.
The trade-offs I would document before shipping
Short freshness windows reduce stale answers but increase crawl, parsing, and embedding activity. Long windows lower operational work but require stronger warnings and human review. A strict host allow-list improves privacy and makes audits easier, yet it can miss a vendor-maintained article until someone adds its domain. Structure-aware chunking improves procedural answers, while fixed-size chunking is simpler to implement and benchmark.
This design is not suitable when the bot must answer live lease balances, appointment availability, or individualized case status. Those facts belong behind authenticated, transactional tools, with a response assembled at request time. Stick with a conventional document index for static guidance; use a system-of-record integration when the answer changes per resident.
Before release, I run a small gate on every deploy: every indexed URL is in scope, every record has an owner and timestamp, stale fixtures are rejected, and citations resolve to canonical pages. Then I replay a fixed Python eval suite after each crawl. A 403 or a parser change is visible in the crawl report, while an answer that cites an out-of-scope page fails the release check.
That workflow keeps the architecture honest. Crawl boundaries define the bot’s knowledge contract, and freshness defines how long a fact remains eligible. The model is the last step, not the place to repair either decision.
Top comments (0)