A Security Contract for Multi-Tenant SaaS Embeddings and Customer Filters

#security #architecture #embeddings

Use a server-derived tenant context as the authorization boundary for every document operation, then enforce that context independently in the vector namespace and in exact-match metadata filters. An ask-your-docs system should retrieve nothing when either constraint is missing, and it should reject the whole result set if any returned record belongs to a different customer. Similarity is ranking, not permission.

That is the decision. The rest of this ADR is about making it survive queues, retries, migrations, deletes, and the ordinary pressure to add one “temporary” search path.

What must multi-tenant ask-your-docs SaaS security enforce around embeddings and metadata filters?

The trusted input is an authenticated principal, not a tenant_id supplied by a browser, prompt, upload filename, or query parameter. The application resolves that principal to an internal tenant and a set of allowed collections before embedding a query or reading a source object. Display names and customer slugs are poor storage boundaries because they can change and may contain separators; use an opaque internal identifier and one canonical namespace derivation function instead.

Five invariants define the contract:

Authentication resolves a server-known tenant before ingestion or retrieval.
Every chunk carries immutable tenant, collection, document, and chunk identifiers.
Both write and search paths derive the namespace from the resolved tenant.
Every search also includes exact tenant and collection metadata predicates.
Answer assembly accepts only hits that pass a post-query tenant check.

The duplicate checks are intentional. A namespace narrows the search domain, while metadata provides a record-level assertion that can be inspected after retrieval. Neither check excuses the other, and neither should be constructed at a call site. Put both behind one adapter whose public methods require a TenantContext; don't expose a lower-level search method to feature code.

Search is untrusted.

If authorization fails, return the same denial regardless of whether the requested collection exists. If a result lacks tenant metadata or carries another tenant's identifier, discard the complete batch, emit a security event containing opaque identifiers and a correlation ID, and produce no answer. Logging the offending document text would create another disclosure path, so traces should record decisions and timings rather than source content or vectors.

Invariants, failure boundaries, and the isolation choice

An ask-your-docs data path usually crosses more boundaries than its request handler suggests: object ingestion, chunking, embedding, queue delivery, vector writes, retrieval, context assembly, deletion, reindexing, backup, and restore. Tenant context has to cross every one. Administrative jobs deserve particular suspicion because they commonly bypass the request-time adapter while touching a much larger data set.

The practical choices differ in blast radius and operational load:

Isolation design	Property enforced	Main operational burden	Failure mode to test	Appropriate use
Shared index with metadata predicate	Per-record query constraint	One mandatory policy-aware query path	Omitted or widened predicate	Small systems whose store has no namespace primitive
Namespace plus metadata predicate	Search-domain and record checks	Namespace lifecycle and migration discipline	Wrong namespace, missing metadata, partial reindex	General SaaS isolation without separate infrastructure
Index per tenant	Separate index administration	Provisioning, quotas, backup and deletion fan-out	Routing drift or incomplete teardown	Modest tenant counts with stronger administrative separation
Dedicated data plane	Separate infrastructure boundary	Credentials, upgrades, regions, restore drills	Cross-environment routing or credentials	Contractual or regulatory separation requirements

Namespace plus metadata filtering is my default decision here because it makes two independent assertions without multiplying deployments. The catch is important: it is not suitable when a customer controls encryption administration, backup custody, regional placement, or a dedicated data plane. Choose an index per tenant or dedicated infrastructure for those requirements. At the other end, metadata-only retrieval can be defensible when namespaces don't exist, provided all queries pass through one unskippable policy adapter and negative isolation tests cover every entry point.

I'm not sure any universal tenant-count threshold would be honest; index limits, provisioning latency, restore behavior, and the team's operational capacity decide where the model stops working. Measure those properties in a capacity review. Marketing tier names won't answer them.

Failure boundaries should be explicit. Authentication and membership checks happen before semantic work. An ingestion worker verifies authorization before reading an object. Retrieval verifies collection permission before vector search, and answer construction verifies returned records after it. Delete, reindex, and restore jobs use the same tenant-scoped interface. If a queue retries after a successful write but before acknowledgement, deterministic record identity turns the second delivery into the same logical write rather than a duplicate chunk. RFC 9110's treatment of idempotent methods is useful framing for retry reasoning, although a worker still has to define application-level identity for its own operations.

The critical path as an executable contract

The service may have a Node.js edge, but the isolation policy should be portable and covered by conformance tests rather than buried in framework middleware. The Python below expresses the contract: callers provide resolved authorization context, never a namespace; record IDs are stable across retries; and the vector client receives tenant constraints in two places.

from dataclasses import dataclass
from hashlib import sha256
from typing import Protocol


@dataclass(frozen=True)
class TenantContext:
    tenant_id: str
    allowed_collections: frozenset[str]


class VectorStore(Protocol):
    def upsert(self, *, namespace: str, record: dict) -> None: ...

    def search(
        self,
        *,
        namespace: str,
        vector: list[float],
        metadata_filter: dict,
        limit: int,
    ) -> list[dict]: ...


def namespace_for(tenant_id: str) -> str:
    digest = sha256(tenant_id.encode("utf-8")).hexdigest()
    return f"tenant_{digest}"


def require_collection(ctx: TenantContext, collection_id: str) -> None:
    if collection_id not in ctx.allowed_collections:
        raise PermissionError("collection access denied")


def ingest_chunk(
    store: VectorStore,
    ctx: TenantContext,
    *,
    collection_id: str,
    document_id: str,
    chunk_number: int,
    text: str,
    vector: list[float],
) -> None:
    require_collection(ctx, collection_id)
    logical_key = f"{ctx.tenant_id}:{document_id}:{chunk_number}"
    chunk_id = sha256(logical_key.encode("utf-8")).hexdigest()

    store.upsert(
        namespace=namespace_for(ctx.tenant_id),
        record={
            "id": chunk_id,
            "vector": vector,
            "text": text,
            "metadata": {
                "tenant_id": ctx.tenant_id,
                "collection_id": collection_id,
                "document_id": document_id,
                "chunk_number": chunk_number,
            },
        },
    )


def retrieve(
    store: VectorStore,
    ctx: TenantContext,
    *,
    collection_id: str,
    query_vector: list[float],
    limit: int = 8,
) -> list[dict]:
    require_collection(ctx, collection_id)
    hits = store.search(
        namespace=namespace_for(ctx.tenant_id),
        vector=query_vector,
        metadata_filter={
            "tenant_id": {"eq": ctx.tenant_id},
            "collection_id": {"eq": collection_id},
        },
        limit=limit,
    )

    tenant_ids = {
        hit.get("metadata", {}).get("tenant_id") for hit in hits
    }
    if tenant_ids - {ctx.tenant_id} or None in tenant_ids:
        raise PermissionError("retrieval isolation check failed")
    return hits

This is deliberately boring code. Its value lies in the shapes it refuses to represent: there is no method that accepts a user-selected namespace, no retrieval without a collection authorization check, and no successful return containing unmarked records. A production adapter also needs bounded timeouts and retry policy, but retries must be classified by operation semantics; “retry every error” can repeat non-idempotent work or amplify load.

The test suite should create two tenants whose documents contain nearly identical sentences, then verify that each side gets zero records from the other. It should remove the namespace in a fake client, remove each metadata predicate, inject a record with missing metadata, revoke collection membership, deliver the same ingestion job twice, delete a document, reindex it, and restore a snapshot. For the retry case, interrupt processing after the vector write but before queue acknowledgement and assert one logical record per chunk. A happy-path relevance test can't establish isolation.

Observe the boundary rather than the content. Count denied collection checks, missing tenant markers, cross-tenant hit rejection, retry attempts, and duplicate logical identities. Alert immediately on any returned-hit mismatch. Break latency down by object read, embedding, vector search, and answer generation so a slow stage can be located without putting source passages into traces.

Rejected boundaries and when they remain valid

Prompt-only isolation is rejected. An instruction such as “use only this customer's documents” runs after retrieval; unauthorized text may already have entered model context or logs. A client-supplied metadata filter fails for the same reason that a client-supplied role does: the requester can edit it. Both mechanisms may influence relevance, but neither establishes authorization.

Metadata-only isolation is also rejected as the default, though it has a valid use case. Stick with it when the vector store has no namespace feature, per-tenant indexes are operationally impractical, and a single typed adapter makes an exact tenant predicate impossible to omit. Its limitation is concentrated risk: one bypassed predicate broadens the search domain. Compensating controls include construction-time tenant requirements, integration tests with deliberately similar cross-customer records, migration checks that reject unmarked chunks, and an audit signal for every search lacking an exact tenant predicate.

An index per user is the wrong default boundary because users can belong to several organizations while documents usually follow workspace or tenant policy. An index per tenant becomes the better choice when administrative separation is itself a requirement; dedicated infrastructure goes further when contracts require separate credentials, backups, regions, or data planes. Those designs buy stronger boundaries at the cost of provisioning, upgrades, restore exercises, deletion evidence, and routing complexity. There is no free isolation layer.

Content format does not change this decision. If audio is transcribed before chunking, the transcript inherits the source object's tenant and document identifiers; transcription does not create a new authorization domain. A worker should receive resolved tenant context in a protected job envelope or look it up from an opaque job ID, rather than trust tags derived from a filename, and transcript chunks should use the same deterministic identity and scoped write path as text chunks.

The decision is therefore narrow: carry a trusted tenant context end to end, constrain search twice, verify after retrieval, and test the ugly paths. Change the physical isolation tier when custody or regulatory requirements demand it, not because a similarity engine happens to make index creation easy.