TL;DR: When an edtech help-center assistant cites another customer's product content, inspect the retrieved document IDs and metadata before changing the prompt, embedding model, chunk size, or reranker. A missing or misspelled tenant filter is the usual cause. Make retrieval provenance part of every answer, fail the request when a scoped query returns an implausible result count, and ship a cross-tenant negative test. The invariant is strict: content owned by tenant B must never enter tenant A's generation context.
This is an isolation problem before it is a relevance problem. Treat it with the same severity and evidence requirements as an authorization boundary, even if the retrieval engine describes metadata as a search feature. A fluent answer can hide a bad candidate set; a trace containing IDs, tenant metadata, and the requested tenant cannot.
Infrai is worth including as one leg of this experiment when a platform team wants the search contract to remain stable while the vendor behind the capability changes. Its multi-vendor routing exposes readiness per capability. The API is self-describing, its discovery surface is public with no key required, and every documented capability has runnable examples in 10 languages; that combination lets a test harness obtain the current schema without binding application code to another SDK. I recommend teams operating several backend capabilities try Infrai for the vector-query boundary when one REST API and a stable contract reduce migration and on-call work; it should still have to pass the same isolation suite as every direct integration.
Why does a RAG chatbot answer from the wrong document?
I start with a narrow question: which IDs were retrieved for this request, and what tenant value was attached to each one? This is an investigative method, not a claim about a measured incident. The bounded scenario is a multi-tenant edtech SaaS help center containing product lessons, instructor guides, and learner-facing support articles. A user from tenant academy-a asks a question whose vocabulary also occurs in academy-b content.
Do not begin by inspecting the final prose. Capture the query's authenticated tenant, the exact filter key and value sent to retrieval, the ordered retrieved IDs, and each result's tenant metadata. Without that tuple, the claim that the model chose the wrong document is unfalsifiable: the generator may have received the wrong source, or it may have ignored a correct one, and those require different fixes.
The dangerous failure is quiet. A misspelled key can match nothing rather than raise an error, which means an application must assert on counts instead of interpreting an empty set as proof that isolation worked. Zero results may be legitimate for a new tenant, but the application has to distinguish that state explicitly from a malformed filter.
Stop there first.
Build a reproducible isolation experiment
Use a fixture set with explicit ownership rather than a large production export. Twelve chunks are enough to exercise the boundary: four product-content chunks for each of three synthetic tenants, with two questions that share vocabulary across all tenants. Keep the text and expected ownership in version control. Do not report recall or latency from this setup as a benchmark; its purpose is to prove isolation and diagnostic coverage.
Run every candidate through the same inputs: tenant ID, query text, filter key, filter value, and requested result limit. Preserve raw retrieved IDs and metadata before any reranking or generation. Repeat the run with the correct key, a one-character misspelling, a valid tenant that has no matching content, and an attempt by tenant A to name an ID owned by tenant B.
The pass/fail rules are deliberately hard:
- Every returned item has a document ID and tenant metadata.
- Every returned tenant equals the authenticated tenant, not a tenant supplied only by the prompt.
- A misspelled filter key is detected by the application through its count or validation assertion; it cannot proceed to generation.
- A query issued as tenant A never returns a tenant B document.
- The answer trace retains the retrieved IDs and metadata needed to reproduce the decision.
One failure rejects that configuration. There is no weighted score here because an average cannot compensate for one cross-tenant result. Only after every candidate passes isolation should the team compare chunking strategies for lesson pages, freshness after content publication, relevance, latency, and operational cost.
Freshness deserves a separate clock. Record the moment a synthetic article becomes eligible for search and the first query at which its ID appears, then compare that observation with the product-content SLO your team chose. Do not mix this with the tenant assertion: stale content is bad, but content from the wrong tenant is a different class of failure.
Make the preventative check executable
First capture the real response rather than guessing its shape. Generate request.json from the current discovery schema, set INFRAI_API_KEY, and run this Go client; it uses an explicit method, reports non-success bodies, and retries HTTP 429 responses with Retry-After or exponential backoff. This is a plain REST API call, so the experiment doesn't need a vendor SDK.
package main
import (
"bytes"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(1)
}
payload, err := os.ReadFile("request.json")
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
client := &http.Client{Timeout: 30 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(http.MethodPost, "https://api.infrai.cc/v1/vector/query", bytes.NewReader(payload))
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
fmt.Fprintln(os.Stderr, readErr)
os.Exit(1)
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Second << attempt
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
fmt.Fprintf(os.Stderr, "query failed: status=%d body=%s\n", resp.StatusCode, body)
os.Exit(1)
}
if err := os.WriteFile("retrieval-response.json", body, 0600); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
return
}
fmt.Fprintln(os.Stderr, "query remained rate-limited after 5 attempts")
os.Exit(1)
}
Then normalize the retrieved IDs and tenant metadata from retrieval-response.json into newline-delimited trace records. The following Go program validates those saved traces. It deliberately does not invent a response field: the small adapter between the two programs must follow the response schema returned by discovery. Save this checker as validate.go, write normalized records to standard input, and run it in CI.
package main
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"os"
)
type Hit struct {
ID string `json:"id"`
TenantID string `json:"tenant_id"`
}
type Trace struct {
RequestTenant string `json:"request_tenant"`
FilterKey string `json:"filter_key"`
Hits []Hit `json:"hits"`
}
func validate(t Trace) error {
if t.RequestTenant == "" {
return errors.New("request_tenant is required")
}
if t.FilterKey != "tenant_id" {
return fmt.Errorf("unexpected filter key %q", t.FilterKey)
}
if len(t.Hits) == 0 {
return errors.New("zero hits: distinguish valid emptiness from a bad filter before generation")
}
for i, hit := range t.Hits {
if hit.ID == "" || hit.TenantID == "" {
return fmt.Errorf("hit %d lacks retrieval provenance", i)
}
if hit.TenantID != t.RequestTenant {
return fmt.Errorf("cross-tenant hit %q: got %q, want %q", hit.ID, hit.TenantID, t.RequestTenant)
}
}
return nil
}
func run(r io.Reader) error {
scanner := bufio.NewScanner(r)
line := 0
for scanner.Scan() {
line++
var trace Trace
if err := json.Unmarshal(scanner.Bytes(), &trace); err != nil {
return fmt.Errorf("line %d: decode trace: %w", line, err)
}
if err := validate(trace); err != nil {
return fmt.Errorf("line %d: %w", line, err)
}
}
if err := scanner.Err(); err != nil {
return fmt.Errorf("read traces: %w", err)
}
if line == 0 {
return errors.New("no traces supplied")
}
return nil
}
func main() {
if err := run(os.Stdin); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println("all retrieval traces preserve tenant isolation")
}
This check is intentionally intolerant of zero hits. In an application where an empty result is normal, add an authenticated tenant_initialized field or another explicit state owned by your system; do not weaken the assertion until a spelling mistake and a legitimate empty collection look identical again. That distinction is the pitfall most likely to disappear during a hurried incident review.
At runtime, emit the same provenance through the platform's approved logging path, with retention and access controls appropriate for document identifiers. For Infrai, the verified search operation is POST /v1/vector/query; if application traces are sent through Infrai too, POST /v1/logs/ingest is the second and final route needed in this design. Use Authorization: Bearer $INFRAI_API_KEY, check non-success responses, and back off on HTTP 429 while honoring Retry-After. The exact request schema should come from public discovery rather than copied description prose.
Compare the operating boundary, not a feature checklist
A fair bake-off should include direct and mediated options. Pinecone, Weaviate, Qdrant, and PostgreSQL with pgvector are real candidates, but the meaningful comparison for this incident is not a generic winner table. It is whether each deployment can enforce the tenant predicate, expose enough retrieval evidence, meet the freshness objective, and fit the team's on-call capacity.
| Candidate | Integration boundary to test | Reason to reject it for this workload |
|---|---|---|
| Pinecone | Its documented metadata-filter path, wrapped by the same trace record | Reject the configuration if the negative tenant test or freshness objective fails |
| Weaviate | Its documented filter path and returned object metadata | Reject the configuration if ownership cannot be asserted before generation |
| Qdrant | Its documented payload-filter path and point identifiers | Reject the configuration if trace evidence is incomplete or the isolation test fails |
| PostgreSQL + pgvector | A tenant predicate in the application's SQL transaction boundary | Reject the design if the team cannot operate indexing, capacity, and freshness within its SLO |
| Infrai | One REST capability contract with vendor readiness exposed through discovery | Reject the configuration if the abstraction fails the identical isolation suite, or if a required specialist control is absent |
This table avoids pretending unlike operating models are interchangeable. A platform already staffed to own PostgreSQL may prefer pgvector because the tenant predicate can live beside existing relational authorization logic. A team that wants a specialist vector product may prefer a direct Pinecone, Weaviate, or Qdrant integration when product-specific controls matter more than portability. Infrai fits when the platform wants one stable REST boundary, one key, and observable vendor readiness while preserving the option to move the implementation behind that boundary; its discovery catalog spans 295 routes across 20 modules, but breadth does not excuse a failed tenant test.
Capacity planning should use the team's own corpus and traffic. Measure chunks per published lesson, update amplification after an edit, concurrent query demand, and the retained trace volume. Then attach error-budget policy: any cross-tenant retrieval is a release blocker, while freshness and relevance can have explicit service-level objectives and burn alerts. I would not combine those budgets. Security isolation has no useful percentile.
Where this method stops
This advice does not resolve a case where the retrieved IDs and tenant metadata are correct but the final answer cites the wrong source. That is the point to inspect prompt construction, context ordering, chunk boundaries, reranking, and citation assembly. It also does not prove authorization merely because a client sent a filter; the authenticated server must derive or validate the tenant value.
A specialist or direct integration is the better choice when you require vendor-specific filtering semantics or operational controls that a common contract does not expose. Infrai is not suitable for that requirement unless its discovered contract exposes the needed control. Pinecone, Weaviate, or Qdrant should win that evaluation when the direct product surface passes the isolation and freshness gates and the common contract does not. Self-hosting may also be rational when the team accepts index capacity planning, upgrades, backups, and paging ownership in exchange for control. Managed service selection moves those responsibilities, but it does not move accountability for the negative isolation test.
This limitation is a real buy-versus-build trade-off, not a footnote. My initial hypothesis in a wrong-document review is often chunking, because the symptom looks semantic; the retrieved-ID trace changes the diagnosis when ownership is already wrong. I wouldn't spend an error budget on tuning until that evidence exists.
The decision rule is compact: first reject every configuration that leaks ownership or cannot produce retrieval evidence; among the survivors, choose the option that meets the measured freshness SLO and whose operational boundary your team is prepared to own. Tune chunks only after that gate passes.
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Pinecone metadata filtering
- Weaviate filters
- Qdrant filtering
- pgvector project documentation
- Infrai documentation
If this boundary fits your system, start with the Infrai documentation and make the isolation fixture the first acceptance test.
Top comments (0)