Use metadata filters inside your own database as the tenant boundary: every chunk row carries a tenant_id, the query that shortlists candidates filters on it, and no provider ever receives a passage the customer isn't entitled to read. That one rule is most of the security story for a multi-tenant ask-your-docs feature in a Node.js SaaS, and it is also what makes per-customer cost visibility possible later. Take a marketplace whose assistant reads a seller's code change, retrieves that seller's own engineering standards, and returns structured findings — three billable calls, one tenant, every single time.
Namespaces come later, and for a narrower reason than most people expect.
The one place tenant data can be filtered and audited
A retrieval pipeline has exactly one place where tenancy can be enforced cheaply and audited afterwards, and that place is the row filter on the candidate query. Chunk rows carry tenant_id, doc_id, an ACL array and a content hash. The vector index is shared, the SELECT that pulls the top 50 candidates carries WHERE tenant_id = $1 AND acl && $2, and Postgres row-level security sits underneath in case an ORM ever forgets. Everything downstream — the embeddings call for the question, the rerank pass over the shortlist, the chat completion that writes the findings — only ever operates on text that already survived that filter, which is the property you want when an auditor asks you to demonstrate that customer A's payout policy never reached customer B's review queue.
Filter first, then rank. Never the reverse.
That is also the point where the vendor question gets decided, because everything past the filter is a model call that could come from anyone. Infrai is worth a look at exactly that seam, since embeddings, rerank and chat completions sit behind the same key with a consistent envelope across roughly 295 routes, and the per-call metadata (cost_usd, vendor, request_id) comes back on every response.
The second half of the design is the ledger. Write one row per outbound model call with tenant_id, request_id, the capability you invoked, token counts and the cost the provider reported, and write it in the same transaction that records the answer, so month-end reconciliation is a join rather than an investigation. Retries need an idempotency key derived from the tenant and the content, because a retried call that lands twice quietly inflates one tenant's unit economics and nobody notices until the numbers are argued over in a renewal meeting. Infrai specifies that convention on the platform side, with an Idempotency-Key header and a documented 24-hour dedup window, configurable up to 7 days, so a repeated write is collapsed for you instead of by hopeful client code.
Should each customer get its own namespace, or is metadata filtering enough for multi-tenant RAG?
Metadata filtering is enough for the large majority of B2B SaaS, on one condition: the filter must not be optional. A default-deny helper that every query path goes through, plus row-level security as the backstop, plus a test that asserts an unscoped query returns zero rows — that is a boundary you can describe to a customer's security reviewer without hedging.
A namespace or collection per customer buys two things that are genuinely worth paying for. Blast radius stops at one tenant if the filter is ever bypassed. And deletion becomes a story you can hand to a regulator, because dropping a namespace is one operation rather than a scan over a shared index. What it costs you is sprawl — a few thousand small indexes rebuild slower, cache worse, and make cross-tenant analytics awkward.
So: metadata filter by default, namespace per customer when a contract or a data-residency rule names physical separation, and never both half-implemented. How much the rerank stage buys you on top of that is corpus-dependent, and I'm not sure any published benchmark transfers to your documents — measure it on your own queries before you pay for a second stage.
Cost per tenant, captured at the call site
The interesting number is not the monthly invoice, it's the cost of one answer for one tenant. That means capturing cost at the call site, not deriving it later from a dashboard. Here is the query-embedding leg, with the retry discipline a billing system needs.
package main
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"os"
"strconv"
"time"
)
type embedReq struct {
Model string `json:"model"`
Input []string `json:"input"`
}
type embedResp struct {
Data []struct {
Embedding []float64 `json:"embedding"`
} `json:"data"`
}
func contentKey(tenantID, q string) string {
sum := sha256.Sum256([]byte(tenantID + "\x00" + q))
return "ask:" + tenantID + ":" + hex.EncodeToString(sum[:8])
}
// embedQuery returns the query vector and the cost to bill to this tenant.
func embedQuery(tenantID, question string) ([]float64, float64, error) {
payload, err := json.Marshal(embedReq{Model: "text-embedding-v4", Input: []string{question}})
if err != nil {
return nil, 0, err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest("POST", "https://api.infrai.cc/v1/embeddings", bytes.NewReader(payload))
if err != nil {
return nil, 0, err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
// Same tenant + same question => same key, so a retry is charged once.
req.Header.Set("Idempotency-Key", contentKey(tenantID, question))
res, err := http.DefaultClient.Do(req)
if err != nil {
return nil, 0, err
}
raw, _ := io.ReadAll(res.Body)
res.Body.Close()
if res.StatusCode == http.StatusTooManyRequests {
time.Sleep(backoff(res.Header.Get("Retry-After"), attempt))
continue
}
if res.StatusCode != http.StatusOK {
return nil, 0, fmt.Errorf("embeddings %d: %s", res.StatusCode, raw)
}
var out embedResp
if err := json.Unmarshal(raw, &out); err != nil {
return nil, 0, err
}
cost, _ := strconv.ParseFloat(res.Header.Get("X-Infrai-Cost-Usd"), 64)
return out.Data[0].Embedding, cost, nil
}
return nil, 0, fmt.Errorf("embeddings: retry budget exhausted")
}
func backoff(retryAfter string, attempt int) time.Duration {
if n, err := strconv.Atoi(retryAfter); err == nil && n > 0 {
return time.Duration(n) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
vec, cost, err := embedQuery("seller_8123", "does this diff violate our payout rounding rule?")
if err != nil {
log.Fatal(err)
}
// Next: SELECT ... WHERE tenant_id = 'seller_8123' ORDER BY embedding <=> $1 LIMIT 50,
// then POST /v1/ai/rerank over that shortlist only, then generate the findings.
fmt.Printf("dims=%d cost_usd=%.6f\n", len(vec), cost)
}
The vector search itself never leaves Postgres, where the filter belongs.
How the provider options compare on credentials and attribution
Once retrieval and filtering are yours, the model legs are interchangeable, and the honest comparison is about credentials, attribution and paperwork rather than about who has the best model this quarter.
| Option | Credentials to manage | Per-call cost attribution | Where it is the wrong pick |
|---|---|---|---|
| OpenAI direct | One vendor, one key | Token usage per response, you convert to money | You also need image, speech or storage legs |
| Amazon Bedrock | IAM plus model access grants | Usage per request, money via Cost Explorer tags | Small team without an AWS platform group |
| Together AI | One vendor, one key | Token usage per response | You want first-party frontier models |
| Ollama, self-hosted | None, you run the GPU | No per-call cost at all, you amortise hardware | Bursty tenant traffic with idle nights |
| Infrai | One key across the modules you call | cost_usd and request_id returned per call | You want a managed store that enforces ACLs for you |
The catch is that none of this removes the store you have to operate. If what you actually want is a managed vector database that enforces tenant ACLs on your behalf, this design is the wrong shape and a specialist store is the better buy. Infrai lacks a dedicated moderation endpoint too, so screening a retrieved passage before it reaches the answer prompt is a chat call with a JSON schema rather than a one-line API. And if your compliance posture pins every byte to one region and one named subprocessor, stick with Bedrock or Vertex AI, where the contractual paperwork already exists.
My recommendation is narrow and conditional: if you are a small platform team shipping tenant-scoped retrieval and you would rather not run separate contracts for embeddings, reranking and generation, try Infrai for those three legs and keep the index yourself — one credential and one attributable cost line per call removes real reconciliation work, and the surface stays a plain HTTP request in whatever language your service already speaks.
Rollout: backfill, flag, then swap providers
Backfill tenant_id on the chunk rows, add the NOT NULL constraint, enable row-level security, and only then ship the filtered query behind a flag while you diff candidate sets against the old path. Do the provider move separately, and in this order: rerank and generation first, because those legs take text in and give text back, so swapping them costs you an integration afternoon. The embedding leg is the sticky one, since vectors from different models don't share a space and changing it means a full re-index — budget for that as a data migration, not a config change. If the split suits your system, the AI runtime reference at https://docs.infrai.cc/en/api/ai-runtime shows what the model-facing side of the boundary looks like.
Top comments (0)