Customer-Support Corpus Fences: Retrieval Architecture for Knowledge-Base Crawl Limits
Short answer: use a staged retrieval design with explicit collections, bounded queries, and traceable source context. For a B2B support knowledge base, the least complex option that holds up is a narrow crawler feeding a versioned index, with tenant and permission metadata attached before chunking.
The expensive mistake is crawling everything first and deciding what matters later. That creates a large vector index, muddy recall, and a support agent that cites a retired refund policy with total confidence. Crawl boundaries are a data contract, not a robots.txt afterthought. I've watched teams spend a week tuning retrieval scores while the real problem was 40 near-identical copies of the same article, each with a different tracking query string. The fix was less glamorous: normalize the URL, hash the meaningful body, and make the crawler explain every exclusion. Once those counters were visible, the index stopped growing for reasons nobody could name.
The choice matrix I use before writing an ingester
| Boundary strategy | Recall | Index cost at scale | Failure mode | Best fit |
|---|---|---|---|---|
| Allow-list of support surfaces | Medium | Predictable | New articles stay invisible | Stable, owned documentation |
| Sitemap plus path rules | High | Moderate | Marketing pages leak into answers | Large help centers with clean URL taxonomy |
| Whole-domain crawl | Highest initial recall | Unbounded | Duplicates, stale policy, access leaks | Temporary discovery only |
| Event-driven document feed | Depends on source | Most predictable | Missed or out-of-order events | Teams that control publishing |
My default is an allow-list plus sitemap rules, followed by a small event feed for urgent policy changes. Whole-domain crawling belongs in an audit job, never in the production refresh path. The matrix is intentionally boring. Boring is cheap to operate.
The first design decision is the retrieval contract. Write down what an answer must contain: an article identifier, a section heading, an effective date, a tenant scope, and an access decision. A top-k array without that context is not a contract; it is a future incident.
What should a customer-support knowledge base crawl boundary include?
Start with ownership. A URL is eligible only when a named team owns its content and its audience is known. Include canonical help-center pages, internal runbooks that the answering service is authorized to see, and structured product notes. Exclude navigation shells, search-result pages, user profiles, changelogs that have no support authority, and any page whose permissions cannot be represented in metadata.
Then define time. Store published_at, updated_at, and an effective_from value when policy changes are scheduled. During retrieval, filter by the answer's evaluation time. A document that is current today can still be wrong for a ticket created last week.
Finally define identity. Canonicalize URLs, strip tracking parameters, and hash normalized text. Keep the source URL and document revision beside every chunk. If two URLs produce the same hash, index one logical document and retain the aliases for citation. This cuts duplicate vectors without hiding where the article came from.
I once treated a locale path as cosmetic. It was not. /en-us/returns and /de-de/returns carried different legal windows, and the language detector did not know that. The fix was a boundary rule: locale, tenant, product edition, and permission scope are part of the document key. A missing field is a reject, not a guess.
Three words: reject unknown scope.
Don't infer it later.
A staged pipeline keeps crawl, chunk, and query costs visible
Separate the system into crawl, normalize, chunk, index, and answer stages. Each stage emits a small record with a revision, counters, and a reason for skipping. That lets you measure where index cost is coming from instead of blaming the embedding model for every oversized bill.
For chunking, split on headings and list boundaries before applying a token limit. Keep a short overlap only when a definition crosses the boundary. Store the parent section and ordinal, so a citation can expand to the surrounding paragraph without re-crawling the page. Never concatenate two permission scopes into one chunk; filtering after embedding is too late if the chunk already contains restricted text.
The query path should be bounded in two dimensions: candidate count and wall-clock budget. Apply tenant and access filters in the vector query itself, request a modest candidate set, then rerank only that set. A second lexical check for product names and error codes is often cheaper than increasing k. Log the candidate IDs, filter values, and final citations. Do not log raw customer text by default.
Here is the shape of a provider-neutral ingestion record and query. It is TypeScript on purpose: the types make missing boundaries hard to ignore.
type Scope = {
tenantId: string;
locale: string;
productEdition: string;
accessGroup: string;
};
type Chunk = {
id: string;
text: string;
sourceUrl: string;
revision: string;
section: string;
effectiveFrom: string;
scope: Scope;
};
type VectorIndex = {
upsert(chunks: Chunk[]): Promise<void>;
query(input: {
vector: number[];
limit: number;
filter: Record<string, string>;
}): Promise<Array<Chunk & { score: number }>>;
};
async function retrieve(
index: VectorIndex,
vector: number[],
scope: Scope,
): Promise<Array<Chunk & { score: number }>> {
return index.query({
vector,
limit: 24,
filter: {
tenantId: scope.tenantId,
locale: scope.locale,
productEdition: scope.productEdition,
accessGroup: scope.accessGroup,
},
});
}
The number 24 is not a universal optimum. It is a cap to benchmark. Measure recall against a labeled support set, p95 latency, chunks per answer, and bytes stored per active tenant. I am not sure your mileage will match mine because document length and permission cardinality dominate these curves; publish the test corpus and rerun it whenever chunking or filters change.
How do retrieval architecture and crawl boundaries affect index cost?
Cost is mostly a multiplication problem: eligible documents times revisions times chunks per document, plus replicas and retention. A boundary that removes 30% of irrelevant pages can matter more than a small change in embedding price because it reduces ingestion, storage, and reranking work together. Track those terms separately.
Use a dry-run crawler that emits counts without embedding. Review the top sources by byte size, duplicate hash, and revision churn. A page that changes its footer every hour should not trigger a full re-embed; extract the content region or require a meaningful-text hash. Keep a dead-letter queue for parse failures, with the URL and parser version, so a malformed page does not silently disappear. The useful report is uneven by design: one source may account for thousands of tiny navigation fragments, while another contributes a few long policy pages. I want to see both extremes, the exact revision that created them, and the projected chunk count before an embedding job starts. That preflight catches cost spikes while they are still a spreadsheet problem rather than an incident.
Freshness also needs a budget. Poll high-risk policy paths frequently and low-risk reference pages slowly. An event can schedule an immediate refresh, but the worker still checks revision ordering before replacing a chunk. This prevents an older webhook from overwriting newer policy.
Evaluation must include failure cases: a tenant asking about another tenant's product, a retired policy, a translated article with a different deadline, and a query containing an exact error code. Record whether the expected source was retrieved, whether the answer cited it, and whether the access filter excluded plausible but forbidden matches. Retrieval-Augmented Generation research describes the broader retrieve-then-generate pattern; the boundary and authorization work remains your system's responsibility.
When a wider crawler is the right trade-off
The catch is that an allow-list is not suitable when your documentation taxonomy is unknown or changing daily. For an initial inventory, a bounded whole-domain crawl can reveal orphaned support pages and duplicate paths. Keep it isolated, cap depth and bytes, and send its findings through human review before promotion to the production collection.
Stick with an event-driven feed when your publishing system can guarantee ordered revisions and complete deletion events. It gives the cleanest cost model, but it is a poor fit for third-party portals you cannot instrument. In that case, sitemap polling with explicit path and content rules is easier to audit.
Do not choose a boundary strategy by vector-database brand. Choose it by the evidence you can observe: source eligibility, revision lineage, authorization filters, recall, latency, and bytes per tenant. A simple collection interface is useful only when those invariants are enforced around it.
Top comments (0)