The expensive mistake in a customer-support RAG system is treating top_k as the whole query limit. A useful design budgets work across candidate retrieval, metadata filtering, reranking, and the answer context. For a B2B SaaS knowledge base that also detects semantically near-duplicate records, I use a staged contract: retrieve a bounded candidate set, filter by tenant and status, rerank only the survivors, then cite the records that actually informed the response. The index stays predictable, and an ambiguous question can fail closed instead of consuming an unbounded slice of the corpus.
Short answer: set separate limits for candidates, reranker input, evidence tokens, and latency; enforce them in one retrieval function and measure recall against a small labeled set before shipping.
Why one limit is not enough
Support content has an awkward shape. A single incident can produce a ticket, an internal note, a release entry, and three nearly identical macros. Dense retrieval may return all six. That is useful for finding duplicates, but wasteful when a language model receives every copy as context. A lexical search may do the opposite: it finds the exact error code and misses a paraphrase from a newer ticket.
The contract should name four independent ceilings:
| Stage | Example ceiling | What it protects |
|---|---|---|
| Initial candidates | 40 records | Vector and lexical index work |
| Reranker input | 12 records | CPU and model latency |
| Evidence context | 4 records or 2,400 tokens | Prompt cost and answer focus |
| End-to-end time | 800 ms | User-facing response budget |
Those numbers are starting points, not universal constants. Near-duplicate detection may need 40 candidates to find a matching pair, while answer generation may need only four. Keep the two objectives separate. I learned this while moving a notebook prototype into a service: raising top_k improved a recall chart, then quietly pushed repeated ticket text into the prompt. The retrieval score looked better; the answer got harder to inspect.
A limit is also a security boundary. Apply tenant, product, locale, and publication-state filters before text is handed to a reranker or a model. Every indexed item should carry those fields, plus a stable document identifier and source URL. A missing filter is not a quality issue; it is an isolation defect.
The filter comes first.
How should Python retrieval enforce query limits in a customer-support knowledge base?
The implementation below uses ordinary HTTP calls so the retrieval boundary remains portable. It assumes an embedding has already been produced and that the vector service accepts the documented query and upsert shapes. The important part is the contract around those calls, not a particular database.
from dataclasses import dataclass
from typing import Any
import requests
@dataclass(frozen=True)
class RetrievalBudget:
candidate_k: int = 40
rerank_k: int = 12
evidence_k: int = 4
timeout_s: float = 0.8
def retrieve_support(
base_url: str,
embedding: list[float],
tenant_id: str,
budget: RetrievalBudget = RetrievalBudget(),
) -> list[dict[str, Any]]:
"""Return at most evidence_k records with traceable source metadata."""
query = {
"vector": embedding,
"top_k": budget.candidate_k,
"filter": {
"tenant_id": tenant_id,
"status": "published",
},
}
response = requests.post(
f"{base_url}/v1/vector/query",
json=query,
timeout=budget.timeout_s,
)
response.raise_for_status()
candidates = response.json().get("matches", [])[: budget.rerank_k]
# Replace this deterministic score with your evaluated reranker.
ranked = sorted(candidates, key=lambda item: item.get("score", 0), reverse=True)
return ranked[: budget.evidence_k]
The slice after the response is deliberate defense in depth. The service should honor top_k, but the caller still protects downstream components if a proxy, adapter, or future API change returns more records than expected. A timeout is part of correctness, too. On timeout, return a typed retrieval miss to the answer layer and ask for clarification or use a narrower lexical path; do not retry indefinitely.
Ingestion has its own boundary. Split a support article into stable chunks, preserve the source URL and access metadata, and upsert one record per chunk. A record that changes should be re-indexed deliberately, and a deleted record should be removed from the collection. The query path cannot compensate for stale permissions. For fresh external status pages, keep web retrieval as a separate stage with a small page and byte budget; it should not become an excuse to crawl the open web for every ticket.
Choosing budgets for near-duplicate records
Near-duplicate detection is where support teams tend to overfit. Two records can share an error code yet describe different product versions. Conversely, a rewritten macro can be nearly identical in meaning while sharing few words. Use a two-pass decision:
- Retrieve 30–50 candidates with tenant and product filters.
- Compare normalized text and embedding similarity among those candidates.
- Require a human-review threshold for merges, and retain the original identifiers.
Do not merge on cosine similarity alone. Include version, locale, and resolution status in the decision. A duplicate pair should point to the canonical source, while an answer citation can still show the specific ticket that supplied the evidence. This separation keeps deduplication reversible.
For the answer path, cap evidence by both count and tokens. Four long troubleshooting articles can exceed a model's useful context even when evidence_k=4. Truncate at section boundaries, keep the title and source URL, and log how many tokens were dropped. Prompt-cost awareness belongs in the retrieval design, not in a last-minute prompt template.
Evaluation and observability before production
Create a small evaluation file from real support intents: exact error lookups, paraphrased questions, stale articles, permission-filtered requests, and known duplicate pairs. Label the expected source IDs. Then track recall@40 for candidate retrieval, precision@4 for answer evidence, duplicate-pair precision, and p95 latency. A single aggregate score hides the failure mode you need to fix.
Log a request ID through ingestion, query, rerank, and generation. Record the applied tenant filter, each limit, candidate counts, timeout reason, and cited source IDs. Never log full customer text by default; hash identifiers and sample sanitized snippets for debugging. When a user reports a wrong answer, you should be able to tell whether the source was absent, filtered out, outranked, or discarded by the token budget.
I am not sure one fixed budget will survive every support catalog. Your mileage may vary with document length, language mix, and how many product versions you retain. That uncertainty is exactly why the limits belong in configuration and why the evaluation set should be rerun after each index or embedding change.
The trade-offs that matter
A generous candidate limit increases the chance of finding a paraphrase and increases index work. A tight evidence limit makes answers easier to audit but can omit a crucial exception. Hybrid retrieval improves exact-term coverage, yet it adds a merge policy and another set of metrics. Hosted search reduces operational burden, while a self-managed index gives more control over placement and retention. None of these choices removes the need for an explicit contract.
The catch is that bounded retrieval is not suitable when your product promises exhaustive legal or compliance discovery. In that case, run a separate export or audit workflow with pagination and durable snapshots, then keep the conversational path bounded. Stick with a broad batch search for investigations; use the small, cited context path for interactive support. That split is easy to miss when a prototype has one function doing everything: it may pass a happy-path demo, but it cannot explain which records were intentionally excluded, which were filtered by tenant, or which were dropped after reranking. Give the batch path its own retention and review policy, and keep the interactive path accountable to its latency and evidence budgets.
My operational checklist is short: define limits per stage, attach access metadata to every chunk, bound retries and bytes, measure labeled recall and citation precision, and make every miss explainable. Ship the notebook example only after those counters appear in a service dashboard. That is the point where a RAG demo becomes an accountable support system.
Top comments (0)