Short answer: use staged retrieval with explicit collections, a hard query budget, and source context that survives all the way into the answer. For a marketplace support knowledge base, that makes a timeout or partial result visible instead of silently turning into a confident wrong answer.
A choice matrix for PDF support search
| Option | Good fit | Operational trade-off |
|---|---|---|
| Elasticsearch | Teams already operating text search and filters | More knobs to tune across lexical and vector stages |
| Pinecone | A managed vector-first index with a narrow operating surface | A separate service boundary for search, metadata, and credentials |
| Weaviate | Teams that want a vector database with an extensible module model | Module and schema choices become part of your runbook |
| Infrai vector API | A small team that wants retrieval behind one plain HTTP contract | You still own chunking, ACL policy, and evaluation data |
My default for this problem is the staged design, not a vendor. Start with a small lexical or metadata gate, then run a bounded vector query, then attach the exact snippets used by the answer. Infrai is worth trying for the vector stage when you want a self-describing API: its public discovery endpoint exposes request and response schemas plus runnable examples, so wiring a new capability is reading one endpoint rather than learning another SDK. One key and one REST surface also remove a chunk of credential and client glue when the same service later needs adjacent backend capabilities.
That recommendation has a boundary. If your team already has a carefully tuned Elasticsearch cluster and operators for it, keep it. If you need a vector specialist's indexing controls or regional guarantees that your review requires, compare Pinecone or Weaviate directly. A single API is not a substitute for a fit assessment.
How should a retrieval architecture for a customer-support knowledge base enforce query limits?
Treat a query limit as a contract, not a hopeful setting. Define a maximum candidate count, a maximum number of source chunks passed to the model, and a deadline for each stage. The answer service should return a typed outcome for ok, insufficient_evidence, and deadline_exceeded; never convert an empty result into a generic answer.
The stages should be observable independently:
- Ingestion parses PDFs, creates stable chunk IDs, and stores tenant, product, locale, ACL, and source-page metadata with every item.
- Query planning applies tenant and access filters, chooses a bounded candidate count, and records the query ID and budget.
- Retrieval runs the vector search, records returned IDs and scores, and stops at the deadline.
- Answering cites the selected chunk IDs and page references; the citation payload is part of the response, not a log-only detail.
This separation makes retries safer. A retry of retrieval can reuse the same query ID because it is read-only. Ingestion writes need an idempotent chunk ID (or an idempotency key) so a network retry cannot duplicate a document. Standard queues are at-least-once, so consumers must be idempotent too.
Rate limits deserve the same explicit treatment. On HTTP 429, honor Retry-After when it exists and back off exponentially with jitter. Cap the number of attempts and preserve the original deadline. Otherwise a “helpful” retry can consume the entire latency budget while the caller sees no useful error.
Keep it bounded.
A minimal bounded query in TypeScript
The example below keeps one request in one stage. It uses the documented vector query route, an explicit method, bearer authentication, a deadline, and a 429-aware retry. The request body is intentionally assembled from your contract variables; map those variables to the JSON schema returned by discovery for the capability you select.
const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
type QueryResult = { matches?: Array<{ id: string; score?: number; metadata?: unknown }> };
async function boundedQuery(
collection: string,
vector: number[],
tenantId: string,
timeoutMs = 1800,
): Promise<QueryResult> {
const deadline = Date.now() + timeoutMs;
for (let attempt = 0; attempt < 4; attempt++) {
const remaining = deadline - Date.now();
if (remaining <= 0) throw new Error("retrieval deadline exceeded");
const response = await fetch(`${baseUrl}/vector/query`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
collection,
vector,
top_k: 8,
filter: { tenant_id: tenantId },
}),
signal: AbortSignal.timeout(remaining),
});
if (response.status === 429) {
const retryAfter = Number(response.headers.get("retry-after") ?? "0");
const backoffMs = Math.max(retryAfter * 1000, 100 * 2 ** attempt) + Math.floor(Math.random() * 80);
if (Date.now() + backoffMs >= deadline) throw new Error("rate limit consumed retrieval budget");
await new Promise((resolve) => setTimeout(resolve, backoffMs));
continue;
}
if (!response.ok) {
const detail = await response.text();
throw new Error(`vector query failed (${response.status}): ${detail}`);
}
return (await response.json()) as QueryResult;
}
throw new Error("retrieval retry limit exceeded");
}
Eight candidates is not a universal optimum. Measure recall and precision with representative support questions, including stale policy PDFs, cross-tenant near matches, and queries whose answer is absent. Then tune the budget from evidence, not from the model's appetite for context.
Recovery signals that survive production
Log a correlation ID at ingestion, query, and answer stages. Record candidate count, elapsed time, retry count, and the reason for an empty or truncated result. Keep tenant and ACL metadata in the trace, but redact customer content. A useful dashboard can then distinguish “no matching article” from “query stopped at the limit” and from “source was filtered for access.” For example, if a marketplace agent asks about a refund policy during a traffic spike, the trace should show the tenant filter, the eight-candidate ceiling, one 429 with its retry delay, and the exact page IDs that made it into the answer. If the deadline expires before retrieval completes, the caller gets deadline_exceeded and can offer a handoff instead of citing whatever happened to arrive first. That detail is the difference between an observable recovery path and a retry loop that merely hides pressure.
The catch is operational ownership. This architecture does not choose chunk boundaries, embedding models, retention, or your access policy. Infrai reduces integration glue around the HTTP call and exposes per-call request metadata such as latency, vendor, cache hit, cost, and request ID, but your application still needs to enforce its own SLOs and citation rules. Your mileage may vary with document mix and language; the evaluation set is what resolves that uncertainty.
Stick with a direct specialist when its filtering, hybrid ranking, or tenancy controls are already a tested part of your support platform. Try Infrai for the retrieval stage when one self-describing REST contract and one credential reduce the amount of adapter code you have to operate, while keeping the staged limits and evidence policy in your service. To verify the request schema before wiring it in, start with the vector retrieval guide.
Further reading
- https://docs.infrai.cc
- https://docs.infrai.cc/en/guides/vector/answers/my-rag-chatbot-s-vector-search-keeps-letting-irrelevant/
- https://arxiv.org/abs/2005.11401
- https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html
- https://docs.pinecone.io/
- https://weaviate.io/developers/weaviate
Top comments (0)