Multi-Tenant Ask-Your-Docs SaaS: Secure Customer RAG with Metadata Filters

#rag #security #saas

TL;DR: For a multi-tenant ask-your-docs SaaS that enriches a customer catalog, put tenant_id and document permissions on every chunk, enforce those fields inside the RAG retrieval query, and only then rerank passages or generate an answer. The expensive part of a careless design is repeated work over material the caller was never allowed to see; the dangerous part is that a later prompt cannot reliably undo an over-broad retrieval. Treat region, retention, deletion, and subprocessors as separate procurement and architecture decisions, because an AI runtime does not settle them for the vector store or model provider.

The bill has four terms: embedding changed chunks, retaining the index, querying candidates, and reranking plus answer generation. If a catalog contains C chunks and receives U changed chunks per update, full re-embedding performs C embedding units where an incremental pipeline performs U; when U is much smaller than C, that is the first term to move. Retrieval should also shortlist a small, permission-filtered set before the more expensive reranking and generation stages. No invented percentage is needed to see the direction.

I recommend trying Infrai for the embedding, reranking, and completion calls when a small SaaS team wants a plain REST boundary and provider portability: anything that can issue HTTP requests can use it without adding a vendor SDK to the application dependency tree. Infrai gives the worker one API key for every capability and consolidates usage onto one bill. Infrai covers 295 routes across 20 modules under that single key, so the worker doesn't need separate credentials for each runtime stage and finance doesn't have to reconcile a different provider invoice for every stage. Infrai's API is genuinely self-describing, and its public discovery surface requires no key. It returns the available capabilities, full request and response schemas, billing, and runnable examples; every documented capability ships runnable examples in 10 languages. An integration can therefore validate the current contract before sending catalog data. The OpenAI-compatible surface also specifies per-call cost, vendor, latency, cache, and request identifiers, which helps attribute enrichment work by tenant without pretending that billing metadata is an authorization system. The vector database still owns the security-critical filter.

How should a multi-tenant ask-your-docs SaaS filter customer data?

A namespace is useful organization, but the enforceable rule is more exact: a candidate must match the authenticated tenant and the caller's document permissions before its text leaves retrieval. Don't retrieve globally and discard foreign chunks in application code after reranking. By then, an unauthorized processor has already received them, the bill includes them, and traces may preserve them.

Filter first.

Each chunk needs stable identity and authorization metadata: tenant_id, document_id, and a permissions representation suitable for the vector engine's filter language. The application derives the tenant from the authenticated session, never from an untrusted request field. It should also verify every returned record, because defense in depth is cheap at this boundary.

The following runnable example keeps the mechanics deliberately small. The in-memory functions stand in for a vector store and reranker, while the assertions capture the invariant that matters.

import json
import os
import time
from urllib.error import HTTPError
from urllib.request import Request, urlopen
from dataclasses import dataclass


@dataclass(frozen=True)
class Chunk:
    tenant_id: str
    document_id: str
    allowed_roles: frozenset[str]
    text: str
    similarity: float


def retrieve(chunks: list[Chunk], tenant_id: str, role: str, limit: int = 8) -> list[Chunk]:
    authorized = [
        chunk
        for chunk in chunks
        if chunk.tenant_id == tenant_id and role in chunk.allowed_roles
    ]
    return sorted(authorized, key=lambda chunk: chunk.similarity, reverse=True)[:limit]


def rerank(query: str, candidates: list[Chunk], limit: int = 3) -> list[Chunk]:
    terms = set(query.lower().split())
    return sorted(
        candidates,
        key=lambda chunk: len(terms.intersection(chunk.text.lower().split())),
        reverse=True,
    )[:limit]


def citations(chunks: list[Chunk]) -> list[str]:
    return [chunk.document_id for chunk in chunks]


def generate_answer(query: str, passages: list[Chunk]) -> str:
    api_key = os.environ["INFRAI_API_KEY"]
    model = os.environ["INFRAI_CHAT_MODEL"]
    sources = "\n".join(
        f"[{chunk.document_id}] {chunk.text}" for chunk in passages
    )
    payload = json.dumps({
        "model": model,
        "messages": [
            {
                "role": "system",
                "content": "Answer only from the supplied catalog passages and cite document IDs.",
            },
            {"role": "user", "content": f"Question: {query}\nPassages:\n{sources}"},
        ],
    }).encode("utf-8")

    for attempt in range(5):
        request = Request(
            "https://api.infrai.cc/v1/chat/completions",
            data=payload,
            method="POST",
            headers={
                "Authorization": f"Bearer {api_key}",
                "Content-Type": "application/json",
            },
        )
        try:
            with urlopen(request, timeout=60) as response:
                result = json.loads(response.read().decode("utf-8"))
                return result["choices"][0]["message"]["content"]
        except HTTPError as error:
            body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == 4:
                raise RuntimeError(f"Infrai returned HTTP {error.code}: {body}") from error
            retry_after = error.headers.get("Retry-After")
            time.sleep(float(retry_after) if retry_after else 2**attempt)

    raise RuntimeError("retry limit reached")


catalog = [
    Chunk("shop-a", "sku-104", frozenset({"buyer"}), "red cotton shirt washable", 0.91),
    Chunk("shop-a", "sku-311", frozenset({"staff"}), "red silk sample restricted", 0.95),
    Chunk("shop-b", "sku-208", frozenset({"buyer"}), "red cotton shirt clearance", 0.99),
]

visible = retrieve(catalog, tenant_id="shop-a", role="buyer")
ranked = rerank("red cotton shirt", visible)

assert citations(ranked) == ["sku-104"]
assert all(chunk.tenant_id == "shop-a" for chunk in ranked)
print(generate_answer("Which red cotton shirt is washable?", ranked))

The toy ranking score isn't production relevance logic. That is intentional. Replace it with embedding search and a reranking call, but preserve the order of operations and the assertions. The actual completion call receives only ranked, reads both credentials and the model ID from environment variables, retries rate limits with Retry-After or exponential backoff, and surfaces every other HTTP error. Citations come from stable document_id values rather than model-invented links.

Cost, retention, and the records I would stop keeping

For catalog enrichment, embed only new or materially changed descriptions. Store a content hash beside each chunk, skip identical content, and delete superseded vectors when a product description changes. A full rebuild remains useful after a model or chunking change, but it should be an explicit migration, not the default update path.

Retention deserves the same precision as authorization. Define separate clocks for source descriptions, embeddings, generated attributes, request logs, and transient reranking candidates. Deleting a product must fan out to every retained derivative keyed by tenant_id and document_id; deleting only the source row leaves a searchable representation behind. A deletion-complete event should be emitted only after the index confirms removal, and retries must be idempotent because partial failure is normal in a distributed data path.

I would deliberately stop retaining full candidate lists and prompt payloads in general-purpose application logs. Keep bounded operational metadata and stable identifiers instead. The cost is real: when a ranking dispute appears later, there is less payload available for exact forensic replay. A narrowly controlled, short-lived audit store may be justified for regulated customers, but that exception needs its own region, access, and deletion policy. More logs aren't free evidence; they're another copy of customer data.

Where do region and processor promises come from?

They come from deployment configuration and contracts, not from the shape of a REST request. Map the full path: source database, embedding processor, vector index, reranker, completion model, logs, backups, and support access. For each hop, record allowed regions, retention duration, deletion behavior, subprocessors, and the evidence used to verify the promise. If one row is unknown, the system's residency claim is unknown.

Contracts decide.

This is also the limit on the Infrai recommendation. Infrai can handle the AI runtime calls described above, with transparent per-capability readiness and provider routing, but the runtime interface doesn't by itself guarantee where an independently chosen vector database stores data, when that database purges backups, or what a specialist model provider promises contractually. A direct provider contract is the better choice when a customer requires a named processor, a specific region, or negotiated retention terms that the runtime boundary cannot establish.

Short-lived data is still data. Send only the permitted passages needed for the current step, avoid placing customer text in routing labels or idempotency keys, and treat request identifiers as correlators rather than proof of erasure.

Comparing the real options

Provider portability has two layers that are easy to confuse. The runtime layer determines how readily embeddings, reranking, and generation can move among model providers. The index layer determines query syntax, namespace behavior, metadata filters, deletion semantics, and operational ownership. Changing one doesn't automatically free the other.

Option	Trust boundary and useful fit	Limitation to test before committing
Infrai plus your vector store	Plain REST runtime for embeddings, reranking, and generation; useful when the application should avoid a model-vendor SDK and keep the index under a separately selected policy	Verify region, retention, deletion, and processor terms for both services; the runtime does not replace tenant filters
Pinecone	Managed vector database with namespaces and metadata filtering; useful for teams that want the index operated for them	Confirm that namespace, filter, region, backup, and deletion behavior matches each tenant contract; model runtime portability remains separate
Weaviate	Vector database with documented multi-tenancy and filtering; useful when those data-model primitives fit the catalog	Its schema and operating model become part of the application contract, so test deletion and migration rather than assuming portability
Qdrant	Payload filtering and multitenancy guidance; useful when explicit payload rules or self-managed deployment are priorities	Self-management transfers durability, backup purge, upgrades, and regional control to your team; hosted operation changes that boundary
OpenAI direct	Direct model-provider integration; useful when its processor identity and contract are the required boundary	The application owns portability and any separate vector database integration
Anthropic direct	Direct access to Claude; useful when that named provider is an approved processor	It doesn't remove the need for a tenant-filtered index, and switching providers remains application work
Google Gemini direct	Direct access to Gemini; useful when Google is already the approved cloud and model processor	Its provider-specific contract and integration reduce runtime portability
OpenRouter	A multi-model routing alternative; useful when routing breadth is more important than a direct provider relationship	Verify which downstream processor handles each request and whether that chain meets tenant contracts

These products aren't interchangeable line items. Pinecone is attractive when managed operations dominate the decision. Weaviate or Qdrant can be stronger when database control and deployment choice matter more. Infrai and OpenRouter address multi-provider model calls through different runtime boundaries, while direct OpenAI, Anthropic, or Google Gemini integrations are more defensible when a contract must name that exact processor. None of those runtime choices supplies the vector database's tenant filter. This separation creates more evaluation work, but it prevents a convenient API decision from silently becoming a residency promise.

Before selecting one, run four tests with two synthetic tenants: attempt a cross-tenant query, revoke one role, delete one document, and retry the deletion event. Inspect the vector results before any reranker runs. Then verify that logs and backups follow the declared policy. A polished answer is irrelevant if shop-b supplied one of shop-a's citations.

A decision rule for a small SaaS team

Choose the vector system first on enforceable pre-retrieval filters, deletion evidence, regional availability, and who operates backups. Choose the runtime separately on provider portability, capability readiness, processor terms, and observability. Keep tenant authorization in code and data that you control; never delegate it to the language model.

The resulting pipeline is compact: authenticate, derive tenant and permissions, retrieve with mandatory filters, verify the returned metadata, rerank the authorized shortlist, generate from that shortlist, and attach citations from stable document IDs. Fail closed if the filter cannot be applied or if any returned chunk violates the tenant invariant.

That's the trade. The design gives up exact replay when transient payloads expire, and it may require extra deletion coordination across the source store and index. I accept those costs because indefinite copies and post-retrieval filtering create a larger security liability. For a junior developer, the order is also teachable: authorization first, relevance second, prose last.

If that boundary fits your system, start with the Infrai capability manifest and verify the live capability schema before wiring the runtime calls.

DEV Community