An internal engineering documentation bot for e-commerce has one constraint that changes the retrieval architecture: a fast answer is still wrong if it includes a runbook the caller cannot read. The practical choice is a staged pipeline with explicit collections, bounded queries, and source context that remains traceable after generation. Keep authorization ahead of retrieval, then spend the remaining latency budget on ranking.
Short answer: define the answer-to-source contract first, filter every candidate by access rules before context assembly, and ship only after a small labeled set shows that the latency cap has not erased useful recall.
This is narrower than a generic RAG stack decision. A merchandising engineer asking why a product disappeared from search might need catalog indexing notes, while a support contractor should never retrieve the private incident timeline attached to the same SKU. Both queries can contain identical words. The user-visible answer therefore needs a contract: permitted collection, caller identity, bounded candidate count, source identifier, revision, and a deadline. No citation means no claim.
How should retrieval architecture enforce access rules for internal engineering documentation?
Treat access as candidate eligibility, not as a cleanup pass after similarity search. The retrieval request should carry a principal and the collections that principal may query. Each returned chunk should retain its source ID and revision so the answer can cite what was actually retrieved. If permissions are checked only after broad retrieval, restricted text has already entered a component that didn't need it; deleting a sentence from the final answer doesn't undo that exposure.
Three rules are enough to define the boundary. First, resolve the caller's allowed collections before any content query. Second, cap both candidate count and elapsed time. Third, preserve source context through ranking and generation. The contract can be small:
type Access = "support" | "engineering" | "incident";
type Principal = {
id: string;
grants: ReadonlySet<Access>;
};
type Chunk = {
sourceId: string;
revision: number;
access: Access;
text: string;
score: number;
};
type RetrievalResult = {
context: Chunk[];
rejected: number;
};
function assembleContext(
principal: Principal,
candidates: readonly Chunk[],
limit = 8
): RetrievalResult {
if (!Number.isInteger(limit) || limit < 1 || limit > 20) {
throw new RangeError("limit must be an integer from 1 through 20");
}
const eligible = candidates.filter((chunk) =>
principal.grants.has(chunk.access)
);
return {
context: eligible
.sort((left, right) => right.score - left.score)
.slice(0, limit),
rejected: candidates.length - eligible.length,
};
}
const caller: Principal = {
id: "user-1842",
grants: new Set<Access>(["support", "engineering"]),
};
const result = assembleContext(caller, [
{
sourceId: "catalog-indexing",
revision: 17,
access: "engineering",
text: "Re-index changed product records after schema updates.",
score: 0.91,
},
{
sourceId: "sku-incident-482",
revision: 4,
access: "incident",
text: "Restricted incident timeline.",
score: 0.96,
},
]);
if (result.context.some((chunk) => chunk.access === "incident")) {
throw new Error("403: unauthorized context entered the answer stage");
}
console.log(result);
The code is intentionally boring. Good. The critical assertion is not that the top semantic result wins; it is that an ineligible result cannot enter the answer stage even when its score is higher. In a production service, collection resolution belongs in the authorization layer, while this second check stays near context assembly as defense in depth. Don't log rejected chunk text. A source ID and rejection count are enough for tracing without copying restricted material into telemetry.
Re-indexing is part of this boundary too. Changed documents need deliberate upserts, and deleted documents need removal from the collection. Otherwise an old chunk can remain retrievable after its source page or access label changes. I don't treat a successful ingestion request as proof that deletion semantics are correct; the evaluation set needs at least one removed-page case and one downgraded-permission case.
How much latency should each retrieval stage receive?
A staged design can become a latency tax if every stage gets an open-ended timeout. Give identity and collection resolution a deadline, cap vector candidates, cap any web search, and reserve a known slice for answer generation. Retry only operations that are safe to repeat, back off rather than looping, and never let a slow secondary source block the entire user flow.
For the storefront bot, I would begin with three observable stages: access resolution, candidate retrieval, and context assembly. The exact millisecond allocation depends on the network path and corpus, and I'm not sure there is a defensible universal split without measurements from the deployment environment. What is defensible is the shape of the budget: each stage has an explicit timeout, the overall request has a smaller upper bound than the UI timeout, and a timeout produces a traceable partial result rather than an unbounded wait. Your mileage may vary.
This is where retrieval quality and latency collide. A candidate limit of 8 may be enough for a narrow catalog runbook question and inadequate for a question spanning fulfillment, payments, and search indexing. Increasing it blindly raises ranking and generation work. Instead, label a small set of real question types, record whether the permitted supporting source appeared, and compare that result at several fixed limits. Fast is measurable. So is wrong.
A 429 should trigger exponential backoff that honors Retry-After; a 403 should stop the request rather than retry. Those two status codes look close in a dashboard, but they imply opposite control flow. This detail matters more than another framework abstraction.
Compare the operating model, not a feature checklist
The useful vendor question is who owns the retrieval boundary. Pinecone, Weaviate, Elasticsearch, and PostgreSQL with pgvector can all sit behind an application-owned authorization layer; their operational models differ, so the right answer depends on what the solo team is prepared to run. Infrai is another strong fit when the same bot also needs other backend capabilities and consolidating them behind one key, one bill, and a plain REST API removes key and invoice sprawl. Its search and vector capabilities share that interface, but access policy still belongs in the application contract.
Here is the smallest useful boundary test for that option. VECTOR_QUERY_JSON is a request object generated from the public discovery schema, so the example does not freeze fields that may differ by collection. The application resolves the permitted collection before creating that value.
const apiKey = process.env.INFRAI_API_KEY;
const apiOrigin = process.env.INFRAI_API_ORIGIN;
const queryJson = process.env.VECTOR_QUERY_JSON;
if (!apiKey || !apiOrigin || !queryJson) {
throw new Error(
"Set INFRAI_API_KEY, INFRAI_API_ORIGIN, and VECTOR_QUERY_JSON"
);
}
const requestBody: unknown = JSON.parse(queryJson);
async function queryVector(attempt = 0): Promise<unknown> {
const response = await fetch(new URL("/v1/vector/query", apiOrigin), {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(requestBody),
signal: AbortSignal.timeout(2_000),
});
if (response.status === 429 && attempt < 3) {
const retryAfter = Number(response.headers.get("Retry-After"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
return queryVector(attempt + 1);
}
if (!response.ok) {
const reason = await response.text();
throw new Error(`Vector query failed (${response.status}): ${reason}`);
}
return response.json();
}
console.log(await queryVector());
| Option | What the application still owns | Sensible fit | When to choose something else |
|---|---|---|---|
| Pinecone | Principal-to-namespace or metadata policy, source revisions, and evaluation | A managed vector service is the desired boundary | Stick with Elasticsearch when lexical search and existing search operations dominate |
| Weaviate | Authorization mapping, schema lifecycle, deletion tests, and evaluation | The team wants a vector database with its own retrieval concepts | Pick PostgreSQL with pgvector when one existing relational system should remain the operational center |
| Elasticsearch | Document permissions, index lifecycle, hybrid-ranking policy, and evaluation | The organization already operates search and needs lexical control | A managed vector service can be simpler when no one wants to own a search cluster |
| PostgreSQL with pgvector | Query policy, index tuning, document lifecycle, and evaluation | Corpus metadata and access grants already live in PostgreSQL | Avoid it when retrieval load should scale independently from the transactional database |
| Infrai | Access decisions, collection mapping, source lifecycle, and evaluation | One REST boundary and consolidated credentials matter across several backend jobs | A dedicated search platform is preferable when deep engine-specific tuning is the primary requirement |
The catch is that consolidation doesn't erase architecture. No provider can infer which contractor may see an incident note unless the application supplies that rule. Likewise, a dedicated engine is not automatically better: it is better when its extra tuning surface addresses a measured retrieval miss and the team can carry the operational cost. For a small team, I would choose the least elaborate option that preserves the authorization boundary and passes the evaluation set.
No hype needed.
Measure before copying this design
Build a labeled set before production rollout, even if it starts with only a few dozen questions. Each case should name the expected permitted source or explicitly say that no answer is allowed. Include near-duplicate documents with different access labels, a deleted page, a changed revision, a broad cross-team question, and a query whose best lexical match is forbidden. Then measure permitted-source recall, forbidden-source leakage, end-to-end latency, timeout rate, and citation correctness at fixed candidate limits.
The failed simple approach is one global collection followed by a late permission filter. It looks efficient because ingestion and querying are easy, but it makes the highest-scoring inaccessible chunk part of the retrieval path and can leave too little permitted context after filtering. The chosen design does more work before the query: map principal to explicit collections, bound the query, retain revisioned source context, and assemble only eligible chunks. That extra ceremony earns its place because it changes both the security boundary and the quality measurement.
Do not copy my starting limit of 8 as a benchmark; it is an example constraint, not a measured optimum. Run the labeled set at several bounds, watch the long-tail latency, and inspect misses rather than averaging them away. A design passes when restricted chunks never cross the context boundary, changed and deleted records behave as expected, citations resolve to the retrieved revision, and the answer remains useful inside the UI deadline. If PostgreSQL, Pinecone, Weaviate, Elasticsearch, or a consolidated API reaches that bar with less operational drag for your team, use it.
Ship the contract first.
Top comments (2)
Agree with "candidate eligibility, not a cleanup pass". One thing worth flagging: assembleContext still filters after the candidates come back, so the restricted chunks were already retrieved. It also costs recall when the grant is narrow. I measured this on 100k vectors: at 1% selectivity, post-filtering the top 100 from HNSW found only 19% of the true results, because most of the 100 slots went to chunks the caller can't see. Pushing the access filter into the vector query (or one collection per access level) fixed both.
The other bug I'd test for is an empty or falsy grant. A check like
if (filter) query.where(...)quietly turns into "no filter" when the principal comes through with nothing set. A canary test helps: plant one chunk per access level with a unique token and assert in CI that a support caller never gets the incident token back.The retry example has a small JavaScript edge case: Number(response.headers.get("Retry-After")) becomes zero when the header is absent, so the intended exponential-backoff fallback is bypassed. An HTTP-date value takes the other path, but is not actually honored as a date.
I would parse absent, numeric, and date-form headers explicitly, then bound the delay by the request’s remaining deadline. The current two-second timeout resets on each recursive attempt, so four calls plus sleeps can exceed the end-to-end budget even though every individual fetch is bounded.