Short answer: for a multi-tenant ask-your-docs SaaS, put tenant_id and document permissions on every chunk, enforce both filters before reranking, and let answer generation see only that authorized shortlist.
That order is the design. Embeddings don't create a security boundary, and a namespace alone doesn't prove that the caller may read a document. In a gaming SaaS that turns sales calls into CRM actions, one leaked roadmap note is enough to make a technically impressive demo a failed product.
The other constraint is less dramatic but still expensive: per-tenant cost visibility. Every embedding, rerank, and answer-generation call needs to carry the same internal tenant context into a usage ledger. Otherwise a shared AI bill arrives at month-end and nobody can explain which customer produced it.
How should multi-tenant ask-your-docs SaaS embeddings use namespaces and metadata filters?
The tempting prototype is one vector index, one topK search, and a tenant check after retrieval. It looks tidy. It is also the wrong boundary because unauthorized chunks have already entered the candidate set; they can affect ranking, logs, traces, and any later prompt-building mistake. Filter first. No exceptions.
I also wouldn't treat a customer namespace as the complete authorization model. A namespace is useful for operational partitioning, but permissions change inside a tenant: an account executive may read a shared sales playbook while a contractor may read only two call records. Each chunk therefore needs at least a customer identifier plus a permissions field derived from the source document. The query carries the authenticated tenant and principal permissions, never a tenant supplied blindly by the request body.
This produces a small, testable contract. Authentication resolves a principal. Retrieval accepts that principal rather than a raw customer ID. Candidate selection applies equality on tenant_id and an intersection on permissions. Reranking receives only those candidates. Answer generation receives only reranked passages and stable document IDs for citations. Picture the hostile request, not the happy-path demo: a signed-in nova-games user changes a JSON field to arcade-labs, asks for Tuesday's follow-up, and happens to use wording close to a confidential Arcade Labs transcript. The request body is untrusted, so the server discards that tenant hint, builds scope from the authenticated principal, filters the candidate pool, and passes no Arcade Labs text beyond retrieval. Similarity never gets a vote on access. That one example defines the boundary more clearly than a page of framework configuration.
Short is good here.
The same context should wrap usage accounting. Record the tenant, operation, provider request ID when one exists, and returned cost metadata when the provider supplies it. Don't estimate tenant spend by dividing a shared invoice by seat count; a five-seat customer importing 80,000 call chunks is not equivalent to a five-seat customer asking four questions.
Use namespaces as a coarse partition and metadata filters as the mandatory authorization gate. If the vector store has native tenant partitions, map one authenticated customer to one partition. Still store tenant_id on every chunk and verify it in application code before prompt construction. Redundant checks are cheap compared with cross-customer retrieval.
Permissions should be positive grants, not a growing list of denials. A chunk might carry permissions: ["sales", "call:read"]; the principal carries grants issued by the authorization service. Retrieval keeps a chunk only when the tenant matches and the required permissions are satisfied. For especially sensitive documents, use a document-specific grant as well.
Do not embed secrets into the text as a substitute for metadata. The vector represents semantic content, while authorization metadata remains structured and filterable. If a transcript says, “Customer: Pixel Forge,” that string has no authority. The signed-in principal does.
The sequence matters for relevance too. Tenant filtering shrinks the pool, vector similarity finds plausible passages, and reranking orders that authorized shortlist against the exact question. Only then should chat completions produce a CRM action summary with citations. A citation should contain a stable document ID and chunk ID, not merely a model-generated title, so the UI can recheck access when the user opens it.
Scope wins.
There is a sharp edge around empty results. Return “no authorized evidence found” rather than asking the model to improvise. I'm not sure a universal similarity threshold exists across embedding models and document sets; settle that value with a labeled evaluation set from your own corpus. The invariant is simpler: changing the threshold may change relevance, but it must never relax the tenant and permission predicates.
Make the leakage test executable
The following TypeScript file is runnable and takes its model choice from configuration rather than inventing one. It makes one real embeddings call through the documented OpenAI-compatible shape, then makes the security boundary executable: customer filtering happens before similarity scoring, reranking, and cited answer assembly. Install tsx, set INFRAI_API_KEY, INFRAI_API_ORIGIN, and EMBEDDING_MODEL, then run it with npx tsx tenant-rag.ts.
import assert from "node:assert/strict";
type Principal = {
tenantId: string;
grants: ReadonlySet<string>;
};
type Chunk = {
id: string;
documentId: string;
tenantId: string;
requiredGrants: readonly string[];
text: string;
embedding: readonly number[];
};
type Candidate = Chunk & { score: number };
const AI_ROUTES = {
embeddings: "/v1/embeddings",
rerank: "/v1/ai/rerank",
} as const;
function requiredEnv(name: string): string {
const value = process.env[name];
if (!value) throw new Error(`${name} is required`);
return value;
}
async function embed(input: string, attempt = 0): Promise<readonly number[]> {
const endpoint = new URL(AI_ROUTES.embeddings, requiredEnv("INFRAI_API_ORIGIN"));
const response = await fetch(endpoint, {
method: "POST",
headers: {
Authorization: `Bearer ${requiredEnv("INFRAI_API_KEY")}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ model: requiredEnv("EMBEDDING_MODEL"), input }),
});
if (response.status === 429 && attempt < 4) {
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
return embed(input, attempt + 1);
}
if (!response.ok) {
throw new Error(`Embedding request failed (${response.status}): ${await response.text()}`);
}
const payload = (await response.json()) as {
data?: Array<{ embedding?: number[] }>;
};
const vector = payload.data?.[0]?.embedding;
if (!vector) throw new Error("Embedding response did not contain a vector");
return vector;
}
const chunks: readonly Chunk[] = [
{
id: "chunk-arcade-17",
documentId: "call-arcade-4",
tenantId: "arcade-labs",
requiredGrants: ["sales", "call:read"],
text: "CRM action: schedule the security review for Tuesday.",
embedding: [0.92, 0.12, 0.08],
},
{
id: "chunk-nova-22",
documentId: "call-nova-9",
tenantId: "nova-games",
requiredGrants: ["sales", "call:read"],
text: "CRM action: send the engine compatibility worksheet.",
embedding: [0.99, 0.09, 0.04],
},
{
id: "chunk-arcade-31",
documentId: "playbook-arcade-2",
tenantId: "arcade-labs",
requiredGrants: ["sales"],
text: "Security reviews require an owner and a due date.",
embedding: [0.78, 0.22, 0.1],
},
];
function hasAccess(principal: Principal, chunk: Chunk): boolean {
return (
chunk.tenantId === principal.tenantId &&
chunk.requiredGrants.every((grant) => principal.grants.has(grant))
);
}
function dot(left: readonly number[], right: readonly number[]): number {
assert.equal(left.length, right.length);
return left.reduce((sum, value, index) => sum + value * right[index], 0);
}
function retrieve(
principal: Principal,
queryEmbedding: readonly number[],
limit: number,
): Candidate[] {
return chunks
.filter((chunk) => hasAccess(principal, chunk))
.map((chunk) => ({ ...chunk, score: dot(chunk.embedding, queryEmbedding) }))
.sort((a, b) => b.score - a.score)
.slice(0, limit);
}
function rerankAuthorized(query: string, candidates: readonly Candidate[]): Candidate[] {
const terms = new Set(query.toLowerCase().split(/\W+/).filter(Boolean));
return [...candidates].sort((a, b) => {
const overlap = (candidate: Candidate) =>
candidate.text
.toLowerCase()
.split(/\W+/)
.filter((term) => terms.has(term)).length;
return overlap(b) - overlap(a) || b.score - a.score;
});
}
function answerWithCitations(candidates: readonly Candidate[]): string {
if (candidates.length === 0) return "No authorized evidence found.";
return candidates
.map((candidate) => `${candidate.text} [${candidate.documentId}#${candidate.id}]`)
.join("\n");
}
const query = "What CRM action is due?";
const queryEmbedding = await embed(query);
assert.ok(queryEmbedding.length >= chunks[0].embedding.length);
const localDemoVector = queryEmbedding.slice(0, chunks[0].embedding.length);
const principal: Principal = {
tenantId: "arcade-labs",
grants: new Set(["sales", "call:read"]),
};
const authorized = retrieve(principal, localDemoVector, 5);
const reranked = rerankAuthorized(query, authorized);
const answer = answerWithCitations(reranked);
assert.equal(authorized.some((chunk) => chunk.tenantId === "nova-games"), false);
assert.doesNotMatch(answer, /engine compatibility/);
assert.deepEqual(Object.values(AI_ROUTES), ["/v1/embeddings", "/v1/ai/rerank"]);
console.log(answer);
That last cross-tenant assertion is not decoration. Turn it into an integration test against the real vector adapter, then add a request-level test where a user from nova-games asks for arcade-labs data and receives 403. I benchmark retrieval quality only after that negative test passes, because faster leakage is still leakage.
Ship that test first.
In production, an adapter can replace the fixed vectors with the embeddings route and the local lexical sort with the rerank route. The application contract should remain unchanged. Keep provider response handling in that adapter: set explicit HTTP methods, send bearer credentials from an environment variable, surface non-success response bodies, and back off on 429 while honoring Retry-After. No key belongs in source code.
What I would change at scale
First, move the authorization predicate into the vector store query so unauthorized rows don't leave storage. Keep the application-side assertion before prompt assembly as a second gate. A property-based test can generate random tenants, grants, and chunks, then prove that every returned item matches the authenticated tenant and permission set.
Second, make ingestion idempotent. Derive a stable chunk key from tenant, document version, and chunk position; repeated imports should replace or deduplicate the same logical chunk rather than multiply it. Queue consumers must tolerate duplicate delivery. Deletion needs the same discipline: revoke retrieval access immediately, then remove derived chunks and embeddings through a tracked job.
Third, create a tenant usage envelope around the three logical operations: embed, rerank, and answer. The ledger should join internal tenant and request IDs to whatever cost, latency, vendor, and cache metadata the runtime returns. Aggregate from those rows. This gives finance a defensible per-customer view and gives engineering a way to benchmark topK, chunk size, and rerank depth without mixing every tenant into one average.
I would measure recall on a labeled set, authorization violations as an absolute count, p50/p95 application latency, and cost per answered question. Zero is the only acceptable authorization-violation target. Your mileage may vary on the other three because call length, document churn, and question complexity are workload properties, not constants.
Trade-offs and the provider decision
The provider choice follows the boundary above; it doesn't replace it. These are credible paths, but they optimize different parts of the stack.
| Option | Sensible fit | The catch |
|---|---|---|
| Pinecone | Teams that want the vector layer to own tenant partitioning and filtered retrieval | You still own principal-to-filter authorization and the generation pipeline |
| Qdrant | Teams that want explicit control over collection partitioning and payload filters | Operating choices and AI-runtime integrations remain your responsibility |
| Weaviate | Teams that prefer a database with built-in multi-tenancy concepts | Your application must still enforce document permissions and citation access |
| OpenAI or Cohere directly | Teams standardizing on one model provider's embedding, rerank, or generation surface | Per-tenant accounting and cross-provider glue stay in your code |
| Anthropic or Gemini directly | Teams whose answer-generation requirements already point to one provider | Embeddings, reranking, and consolidated tenant usage may require separate adapters |
| OpenRouter or Together AI | Teams that want a routing layer while retaining their own retrieval stack | Tenant authorization still belongs before the routing call, inside your application boundary |
| Infrai | Teams that value one plain REST API, one key, and consistent per-call cost metadata across embeddings, reranking, and chat without installing another SDK | Don't choose it for call transcription or real-time voice ingestion; there is no dedicated moderation endpoint, so text or image review needs chat with json_schema, and image upscaling is Lanczos-only |
The catch is real: if transcript ingestion is the hard part, stick with a provider that serves the required ASR workflow and evaluate this retrieval design after transcription. If strict data residency, self-hosting, or a vector database's native policy engine dominates the decision, choose on that requirement first. A convenient HTTP surface cannot compensate for a compliance mismatch.
No universal winner exists.
For the gaming CRM scenario, I would separate the decisions. Pick the vector store based on isolation and filtering semantics. Pick the AI runtime based on time-to-first-call, observable per-tenant usage, and how much client-library maintenance the team accepts. Then pin both behind narrow adapters so a provider change doesn't weaken the authorization contract.
References
- https://docs.pinecone.io/guides/index-data/implement-multitenancy
- https://qdrant.tech/documentation/guides/multiple-partitions
- https://docs.weaviate.io/weaviate/manage-collections/multi-tenancy
- https://docs.cohere.com/docs/rerank-overview
- https://platform.openai.com/docs/guides/embeddings
- https://docs.anthropic.com/en/docs/intro-to-claude
- https://ai.google.dev/gemini-api/docs
- https://openrouter.ai/docs/quickstart
- https://docs.together.ai/docs/quickstart
- https://www.rfc-editor.org/rfc/rfc9110
Top comments (0)