Short answer: put a hard query budget around each retrieval request, then spend that budget on fresh, diverse chunks instead of making the model read everything. For a customer-support knowledge base that also watches a gaming web page, the useful limit is the one that preserves the changed passage and its evidence, not a single global top-k number.
I learned this the annoying way while sketching a watcher for patch notes. A page changed by one sentence, but a broad retrieval pass returned ten old chunks about unrelated balance updates. The answer sounded plausible. The diff was wrong. A query limit is therefore an accuracy control and an observability signal, not just a latency knob.
Start with a budget, not a magic top-k
Think of the pipeline as a funnel: fetch candidates, remove near-duplicates, rerank, then fit the survivors into a context budget. Set limits at each stage. For example, retrieve up to 40 candidates, keep 12 after diversity filtering, and pass 6 evidence chunks to generation. Those are starting values, not facts about every corpus; your mileage may vary.
The numbers are policy, not truth.
The budget should be expressed in tokens as well as documents. A 6,000-token ceiling prevents one verbose FAQ from crowding out the short change notice that answers the question. Track candidate count, selected count, token count, and the age of each selected chunk. A sudden jump from 6 to 40 selected chunks is an operational event worth investigating.
| Signal | What it tells you | Action |
|---|---|---|
| Candidate count | Whether the retriever is broadening | Inspect filters and query class |
| Trimmed candidates | Whether the budget is discarding evidence | Review recall labels |
| Chunk age | Whether stale text wins | Tighten freshness weighting |
Chunk by meaning and by change boundary. A heading, paragraph, or table row is a better unit than an arbitrary 1,000-character slice for release notes. Store the source URL, heading path, content hash, and observed timestamp with every chunk. When the watcher sees a new hash, it can retrieve the previous and current chunk together and show a compact diff.
How should query limits shape a support knowledge base retrieval architecture?
Use two query paths. The support path answers a user question; the freshness path asks, “what changed since the last observation?” Both can share an index, but they should not share limits blindly. Freshness queries need a narrow time window and exact source filters. Support queries need semantic recall plus a diversity rule so five copies of the same article do not consume the budget.
Here is a small TypeScript policy object. It is deliberately boring: the values are visible, testable, and easy to tune from telemetry.
type RetrievalPolicy = {
candidateLimit: number;
evidenceLimit: number;
tokenLimit: number;
freshnessWindowHours: number;
};
const supportPolicy: RetrievalPolicy = {
candidateLimit: 40,
evidenceLimit: 6,
tokenLimit: 6000,
freshnessWindowHours: 24 * 30,
};
function shouldAlert(previousHash: string, currentHash: string): boolean {
return previousHash !== currentHash;
}
The alert should carry the changed source, old and new hashes, and the evidence chunks used for the summary. Do not silently retry with an unlimited query when the first pass is empty. Emit a reason such as no_fresh_evidence and let a human or a scheduled backfill handle it. That keeps the failure visible.
What fails when limits are missing?
The first failure mode is context dilution: relevant text is present somewhere in the index but is pushed out by duplicates. The second is stale dominance, where a popular old article wins because it has more embedding neighbors. The third is pagination drift: a crawler observes a page halfway through an update and stores a mixture of old and new sections.
Consider a support question about a changed tournament rule. The retriever may find the current rule, an archived rule, a moderation note, and a forum excerpt that quotes the old wording. If all four chunks have similar scores, a raw top-k cut is arbitrary. A better pass groups results by canonical source, applies a recency weight only to sources where recency matters, and reserves one slot for a conflicting or historical result. The answer generator then receives the current rule plus the archived wording with explicit dates, so it can explain the change instead of blending them. That extra bookkeeping also makes an alert reviewable: an operator can click the source, inspect the hash transition, and see why a chunk was selected. It costs a few fields in the index and a little ranking code. It saves a much larger debugging session when a player reports that the bot quoted last season's policy.
Use stable snapshots for crawling. Record the fetch timestamp and hash the normalized text, not the raw HTML noise. Compare headings before paragraphs so a renamed section is treated as a meaningful event. If a page is not suitable for incremental observation, say so in the run metadata and use a full snapshot interval instead; forcing a diff creates false confidence.
I also watch the tail, not only the average. A p95 candidate count of 40 with a p99 of 400 usually means one query class is bypassing the policy. The dashboard needs a counter for rejected or trimmed candidates, plus an alert when the trim rate crosses the team’s agreed threshold.
Build a small replay set from real support questions and known page edits. Label the chunk that contains the answer and the chunk that contains the change. Then vary candidate and evidence limits while measuring recall of those labels, citation coverage, latency, and token use. Keep the smallest limits that meet the recall target, and rerun the set after every chunking change.
The catch is that a single limit is not suitable when users ask for long procedural answers or when pages publish bursty release notes. Use separate policies for those workloads, and stick with a simpler full-snapshot search when provenance cannot be preserved. I’m not sure a universal threshold exists; the corpus shape and the cost of a missed alert decide that trade-off.
Top comments (0)