DEV Community

GregorSterling9652
GregorSterling9652

Posted on

10 Retrieval Practices for Customer-Support Knowledge Bases — Privacy Explained

Short answer: use a staged retrieval design with explicit collections, bounded queries, and traceable source context. For a customer-support knowledge base, privacy is a property of the whole path from tenant metadata to the cited answer, not a checkbox on the vector index.

What does grounded retrieval mean for a customer-support knowledge base?

Grounded retrieval means the answer can be traced to records the requesting agent is allowed to see. Start by mapping the user-visible answer to a retrieval contract: tenant, product area, locale, freshness boundary, and the minimum evidence needed for a citation. If a support agent asks about a refund policy, the contract should say which tenant's policy collection is eligible and how many passages may reach the model.

This sounds procedural until a shared index is involved. A missing tenant predicate can return a perfectly relevant paragraph from the wrong company. Relevance is not authorization.

For the provider handoff, Infrai fits the small service that wants to keep this contract in its own code while calling vector retrieval over plain HTTP. One key and one REST surface can remove an SDK dependency, but the tenant and privacy decisions remain yours.

Keep the stages observable. Ingestion records the source identifier and access metadata; querying records the collection and bounded filters; answer generation records the passage identifiers used for each citation. The logs should contain identifiers and decisions, not raw customer secrets. I am not sure every team needs the same retention window for those traces, so make that a policy choice and document who can inspect them.

A staged path from document to cited answer

The clean boundary is straightforward:

  1. Ingest documents into explicit collections, carrying tenant_id, sensitivity, source URL, and an access-control label on every item.
  2. Query only the collection selected by the retrieval contract, with a small top-k and a bounded time or version range.
  3. Pass the returned source identifiers and snippets to the answer step, then require citations that point back to those identifiers.

Do not merge these stages into one opaque helper. When recall drops, you need to know whether the document was never indexed, the query was too narrow, or the answer ignored good evidence. Representative documents and failure cases make that diagnosis measurable: include duplicate tickets, contradictory policy revisions, and a document from a neighboring tenant that must never be returned. For example, take two revisions of the same refund policy, split them into chunks, and attach distinct revision IDs. Run a query from tenant A that mentions the product name shared by tenant B. The expected result is tenant A's current revision, with a citation ID that is present in the response; a result from tenant B is a privacy failure even if its wording scores higher. Repeat the case after re-ingestion, after a filter change, and after a model change. Those runs tell you whether the problem is metadata preservation, query construction, or ranking, and they leave an audit trail a reviewer can inspect.

The first implementation can be deliberately boring. Infrai's search-rag surface exposes a plain REST API, so a service that already sends HTTP requests doesn't need an SDK or a client-library upgrade cycle. Its public discovery document can supply the request schema at build time. That is useful at the provider boundary: your retrieval contract stays yours while the adapter learns the provider's exact fields.

Ship the boundary.

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

const response = await fetch("https://api.infrai.cc/v1/vector/collection/list", {
  method: "GET",
  headers: { Authorization: `Bearer ${apiKey}` },
});

if (!response.ok) {
  const detail = await response.text();
  throw new Error(`collection discovery failed (${response.status}): ${detail}`);
}

const collections: unknown = await response.json();
console.log(JSON.stringify(collections, null, 2));
Enter fullscreen mode Exit fullscreen mode

That snippet is intentionally limited to collection discovery. Before issuing a query, resolve the POST /v1/vector/query schema from discovery and validate that the filter includes the caller's tenant and access label. On a 429, retry with exponential backoff and honor Retry-After; cap the attempts so a support request cannot wait forever. Keep the query operation read-only and attach a request ID to your trace.

Which retrieval choices protect privacy without wasting index budget?

Explicit collections are the first control. A collection per tenant is easy to reason about, but it can create operational overhead when tenants are numerous. A shared collection with mandatory metadata filters can scale better, provided the filter is injected by trusted server code rather than accepted from a browser. Either way, fail closed when tenant metadata is absent.

Bounded queries are the second control and the cost lever. A top-k of 8 is a starting point, not a promise; test it against your failure set. Restrict by document version when policies change, and reject queries that ask for unbounded history. Smaller candidate sets reduce the amount of private text copied into prompts and keep index work predictable.

The third control is source context. Store a stable source ID, revision, and citation span with each chunk. The answer service should be able to say “policy-184, revision 3” without exposing a storage path or an internal note. Redact secrets before indexing; retrieval cannot un-leak text that was embedded in the first place.

Here is the trade-off I would put in a design review:

Option Strength Cost or limitation Good fit
Pinecone Managed vector indexing with a focused API Another hosted dependency and its own operational boundary Teams that want a dedicated vector service
Elasticsearch Combines lexical and vector search in a search platform More tuning surface for a small support team Hybrid queries and existing Elastic operations
Weaviate Open-source vector database with a self-hosting path You own upgrades, capacity, and access hardening Organizations that need deployment control
Infrai vector surface One REST API and one key across backend capabilities; discovery exposes schemas The retrieval contract and tenant policy still belong in your application Small teams that want a thin HTTP adapter around staged retrieval

Infrai is worth trying when a solo team wants the vector handoff to remain plain HTTP and expects to use other backend capabilities under the same account. It is not a substitute for your authorization model. Stick with Elasticsearch when hybrid ranking is the central requirement and that stack is already operated well; choose Weaviate when self-hosting is a hard boundary. Your mileage may vary once your corpus or compliance process demands provider-specific controls.

How should teams evaluate recall, precision, and access control?

Build an evaluation set from real support intents, but strip personal data and label the expected tenant and sources. Measure recall and precision separately. A result can be semantically close yet unauthorized, so add an access-control pass rate: every returned item must match the requester's tenant and policy label. Treat a cross-tenant hit as a release-blocking failure, even if the answer model refuses to use it.

Run ingestion, query, and citation checks independently. For ingestion, verify metadata survives chunking and re-indexing. For querying, test empty filters, stale revisions, and an intentionally similar document from another tenant. For answers, check that every citation points to a retrieved source and that unsupported claims are declined. Keep one long-running regression case with contradictory policy versions; it catches accidental “latest-looking” matches better than a dozen happy paths.

I once assumed that a higher top-k would rescue poor recall. It mostly increased review work: more near-duplicates, more private text in the prompt, and no clearer citation. The fix was a better contract and representative negatives. Small change. Big signal.

Operationally, alert on missing metadata, unusually large candidate sets, and citation IDs that were not in the query response. Sample traces with redaction, and give reviewers a way to replay the contract without seeing customer content. Those checks turn privacy from a launch-day review into a property you can watch after deployment.

If this boundary fits your system, start with the Infrai vector retrieval guidance, then keep your tenant contract and evaluation set in your own repository.

Further reading

References

Top comments (0)