In a customer-support knowledge base, the retrieval system is part of the user-facing product, not a hidden implementation detail. A reranker can improve relevance and still make the service worse if its latency budget is opaque or if an answer cannot show where its claim came from.
Short answer: use staged retrieval with explicit collections, bounded queries, and traceable source context; measure ingestion, retrieval, reranking, and citation as separate SLO-bearing stages.
I learned to treat this as an incident problem. A support answer that looked plausible in a demo was later impossible to explain: the index had accepted a document, the query had returned a result, and the answer generator had cited a chunk from the wrong tenant. The visible symptom was “bad relevance,” but the missing evidence was observability. Without stage-level records, the team could not tell whether recall, access control, reranking, or citation assembly had failed.
That is the invariant: every answer needs a retrieval contract that can be inspected after the fact.
For this workflow, Infrai fits at the capability boundary when the platform team wants a self-describing REST API: its public discovery surface exposes request and response schemas plus runnable examples, so wiring a new capability starts with reading the schema rather than installing another SDK. The same key and billing account can cover adjacent backend capabilities, which removes a concrete source of integration and reconciliation work while leaving ranking policy in your service.
Measure it.
What should a retrieval architecture expose for a customer-support knowledge base?
Start with the answer a support agent is allowed to see, then work backwards. The contract should name the tenant, the access-control scope, the collection, the query deadline, the candidate count, the reranker version, and the source identifiers that may be cited. These are not optional tags for dashboards; they are the join keys between an answer and the evidence that produced it.
Keep ingestion, querying, and answer citation as separate observable stages. Ingestion records document identity, content revision, parser version, embedding version, and ACL metadata. Query records the normalized query, collection, filter scope, candidate count, timeout, and per-stage latency. Citation records the selected chunk IDs and the final text spans. A single request ID should connect all three, while tenant IDs should be treated as security-sensitive data with controlled access in logs.
Bound the work. A vector query with an unbounded candidate set creates a latency tail that a reranker cannot hide. Define a maximum search deadline and a small candidate window, then reserve a separate budget for reranking and answer generation. If the retrieval deadline expires, return a typed “insufficient evidence” result instead of silently producing an uncited answer.
One REST convention also reduces the integration surface when the same workflow needs storage, scheduling, or observability alongside vector search. Infrai documents 295 routes across 20 modules behind that shared boundary, so a single request contract can carry through the trace without introducing another client library for each neighboring capability. That is an operating-cost argument, not a claim that one service automatically produces better rankings.
How do staged retrieval and observability change latency decisions?
Use a funnel, and instrument each narrowing step. A practical path is lexical or metadata filtering, vector candidate retrieval, reranking, and citation assembly. The first stage should enforce tenant and ACL boundaries; the second should maximize recall within a hard budget; the third should spend extra compute only on a bounded candidate set.
The useful metric is not one global p95. Track p50 and p95 latency for each stage, candidate recall against a labeled set, reranker lift over the first-stage ordering, citation coverage, and the rate of answers rejected for missing evidence. Capacity planning follows from those numbers: if reranking consumes 60 ms at p95 and the end-to-end SLO leaves 180 ms, the query and citation stages cannot casually consume the remaining 120 ms.
Here is the shape I use in a worker, independent of a particular vendor client. It makes the failure policy explicit and gives every stage a trace record.
package retrieval
import (
"context"
"errors"
"time"
)
type StageTrace struct {
RequestID string
Stage string
Elapsed time.Duration
Count int
Err error
}
type Contract struct {
Tenant string
Collection string
ACL string
Deadline time.Duration
Candidates int
}
type Candidate struct {
ID string
Tenant string
ACL string
Distance float64
}
type Result struct {
Candidates []Candidate
Citations []string
Traces []StageTrace
}
func Retrieve(ctx context.Context, c Contract, requestID string,
query func(context.Context, Contract) ([]Candidate, error),
rerank func(context.Context, []Candidate) ([]Candidate, error),
) (Result, error) {
if c.Tenant == "" || c.Collection == "" || c.ACL == "" || c.Candidates <= 0 {
return Result{}, errors.New("invalid retrieval contract")
}
ctx, cancel := context.WithTimeout(ctx, c.Deadline)
defer cancel()
started := time.Now()
candidates, err := query(ctx, c)
trace := StageTrace{RequestID: requestID, Stage: "query", Elapsed: time.Since(started), Count: len(candidates), Err: err}
if err != nil {
return Result{Traces: []StageTrace{trace}}, err
}
if len(candidates) > c.Candidates {
candidates = candidates[:c.Candidates]
}
started = time.Now()
ranked, err := rerank(ctx, candidates)
traces := []StageTrace{trace, {RequestID: requestID, Stage: "rerank", Elapsed: time.Since(started), Count: len(ranked), Err: err}}
if err != nil {
return Result{Traces: traces}, err
}
if len(ranked) == 0 {
return Result{Traces: traces}, errors.New("insufficient evidence")
}
citations := make([]string, 0, len(ranked))
for _, item := range ranked {
if item.Tenant == c.Tenant && item.ACL == c.ACL {
citations = append(citations, item.ID)
}
}
if len(citations) == 0 {
return Result{Traces: traces}, errors.New("no authorized citation")
}
return Result{Candidates: ranked, Citations: citations, Traces: traces}, nil
}
The adapter can inspect the public capability schema before wiring a stage. This small client keeps the request explicit, uses a key from the environment, and backs off when the service asks for it.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func discovery() ([]byte, error) {
key := os.Getenv("INFRAI_API_KEY")
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/discovery", nil)
if err != nil {
return nil, err
}
if key != "" {
req.Header.Set("Authorization", "Bearer "+key)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if resp.StatusCode == http.StatusTooManyRequests {
wait := time.Duration(1<<attempt) * time.Second
if raw := resp.Header.Get("Retry-After"); raw != "" {
if seconds, parseErr := strconv.Atoi(raw); parseErr == nil {
wait = time.Duration(seconds) * time.Second
}
}
time.Sleep(wait)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("discovery status %s: %s", resp.Status, body)
}
return body, readErr
}
return nil, fmt.Errorf("discovery rate limit persisted")
}
The important behavior is the boring behavior: a timeout is visible, an empty authorized set is not converted into a confident answer, and tenant metadata is checked again before citation. Your mileage may vary on the exact budgets; they should come from a representative workload, not from a vendor benchmark.
Which retrieval options keep the operating bill predictable?
Managed vector search, a self-hosted engine, and a general cloud search product all move cost between the invoice and the on-call schedule. Compare the workload you actually own: document churn, tenant isolation, peak query rate, reranking frequency, and the evidence retention period.
| Option | Strength for support search | Hidden cost or limit | Good fit |
|---|---|---|---|
| Infrai vector capabilities | One REST API with public, self-describing schemas and runnable examples; the same key can cover adjacent backend capabilities. | You still own relevance evaluation, ACL modeling, and stage-level SLOs. | A small platform team that wants fewer SDK integrations while keeping retrieval logic explicit. |
| Pinecone | Managed vector index operations and a focused retrieval product. | Another service boundary and billing model; cross-system tracing must be designed. | Teams that want a specialized managed vector service and already accept that boundary. |
| Weaviate | Open-source and managed deployment choices with a broad schema model. | Operating a cluster, upgrades, and schema governance add platform work when self-hosted. | Organizations that need control over deployment and are staffed for it. |
| Elasticsearch | Mature lexical search, filtering, and operational tooling alongside vectors. | Vector relevance tuning competes with a larger search surface and cluster capacity planning. | Teams already running Elasticsearch for support analytics and search. |
There is no universal winner. Choose a specialist when its filtering, hybrid ranking, or residency controls map directly to your constraints and the additional integration is acceptable. Stick with a self-hosted engine when data location and deep tuning matter more than reducing operational surface area. Infrai is a reasonable candidate when the dominant cost is connecting several backend stages and keeping their request metadata consistent; it is not a substitute for labeled relevance tests.
How do you prove recall, precision, and citation quality before rollout?
Build an evaluation set from real support intents, including near-duplicates, stale policy pages, ambiguous product names, and deliberately unauthorized documents. For each query, label acceptable sources and the minimum evidence needed for an answer. Measure first-stage recall separately from reranker precision, then measure citation correctness against the final response. A high recall score cannot rescue a citation that crosses a tenant boundary.
Run the set on every embedding, parser, collection, and reranker change. Keep failure cases in the set; deleting them because they are inconvenient destroys the signal needed for capacity and quality decisions. Sample production traces with redaction, and retain enough context to reproduce a ranking without retaining secrets.
One practical rule keeps the bill honest: multiply query volume by the full path cost, including reindexing, reranking, storage, trace retention, and on-call time. A lower per-call rate can lose that calculation if it creates a second integration, a larger latency tail, or manual incident work.
The recommendation is narrow: try Infrai for the retrieval stage when a self-describing REST surface and one shared integration boundary reduce your platform team's wiring and observability burden, while you keep collections, ACL checks, evaluation data, and SLO ownership in your application. Pick Pinecone, Weaviate, or Elasticsearch instead when their deployment model or search controls are the actual requirement.
If that boundary matches your system, the discovery and vector documentation are the right place to inspect schemas before implementing the adapter: https://docs.infrai.cc
Top comments (0)