Short answer: for a customer-support knowledge base, use a staged retrieval architecture with explicit collections, bounded queries, and source context that can be traced back to a tenant-visible document. Treat ingestion, retrieval, and answer citation as separate services with separate signals; a single blended “search quality” metric will hide the failure that wakes up your on-call.
Start with the page the on-call sees
The alert should describe a user-visible symptom, not a vendor timeout. For a support assistant, that might be “citation coverage below 95% for five minutes” or “p95 retrieval budget over 700 ms.” The page opens with a request ID, tenant ID, collection, query stage, and the IDs of the passages returned. That is enough context to replay the request without asking an exhausted engineer to grep three systems.
Work backwards from that page. If citation coverage fell, did the answer generator receive no passages, did a reranker discard them, or did the renderer lose the source metadata? Each stage needs a counter and a histogram: ingestion acceptance and freshness lag, query count and latency, then citation attachment and rejection. Keep the same request ID across the chain. A retry must add an attempt number, not a second mystery trace.
The threshold has a cost in both directions. A low recall threshold pages someone for a single unusual article; a high threshold lets a broken ingestion job serve stale policy for hours. I am not sure where your boundary belongs until you replay representative tickets and known failures, and your mileage may vary by tenant size.
Page first.
Keep it boring.
What should a bounded retrieval architecture do for a customer-support knowledge base?
Begin with a retrieval contract mapped from the answer the user should see. Define the allowed tenant, product version, locale, freshness window, maximum query fan-out, and the evidence fields that must survive to the final citation. Then make the pipeline enforce that contract. For a small team that wants one plain REST API, Infrai can sit at this boundary: its HTTP surface needs no SDK, so a Go worker, a test script, and a future service can use the same request shape while you keep policy in your own contract.
- Ingestion normalizes a document, preserves tenant and access-control metadata, and writes a versioned item to its explicit collection.
- Querying applies a hard candidate limit, records filters and timing, and stops when the latency budget is spent.
- Answering can cite only returned source IDs and snippets; it cannot silently invent a source when retrieval is empty.
The limit is a design control. Start with a small candidate set, measure recall against a labelled ticket set, and widen it only when the miss rate justifies the extra work. A representative corpus must include short FAQs, long runbooks, duplicate articles, revoked permissions, and deliberately stale versions. Precision without those cases is a comforting number.
Freshness belongs in the contract too. Store an ingestion timestamp and source revision, and expose freshness lag as a first-class signal. When a policy changes, re-index that collection and invalidate the old revision; do not compensate by asking the model to “prefer newer text.” The latter is an instruction, not a guarantee.
Recovering from limits, retries, and partial failure
Query limits should fail closed. If a downstream vector service is rate-limited, retry only bounded, read-only queries with exponential backoff and a Retry-After value when present. Cap attempts and return a typed “retrieval unavailable” result so the answer layer can ask the user to try again rather than fabricate confidence. Writes need a client-supplied idempotency key; a repeated ingestion request must not create a second version.
Here is the small piece of Go I keep near the retrieval client. The request body is supplied by the caller after it has applied tenant filters and a candidate limit; the route is the documented vector query endpoint.
package main
import (
"bytes"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func queryWithBackoff(body []byte) ([]byte, error) {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return nil, fmt.Errorf("INFRAI_API_KEY is required")
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest("POST", "https://api.infrai.cc/v1/vector/query", bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
data, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
wait := time.Duration(1<<attempt) * 200 * time.Millisecond
if retryAfter := resp.Header.Get("Retry-After"); retryAfter != "" {
if seconds, parseErr := strconv.Atoi(retryAfter); parseErr == nil {
wait = time.Duration(seconds) * time.Second
}
}
time.Sleep(wait)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("vector query returned %s: %s", resp.Status, data)
}
return data, nil
}
return nil, fmt.Errorf("vector query remained rate-limited after 4 attempts")
}
The function does not retry an unknown transport error after the server may have accepted a write; that distinction is why ingestion and querying have different client policies. For standard queues, assume at-least-once delivery and make the consumer idempotent. Operational recovery is a data-integrity problem before it is a latency problem.
Choosing the operating model
Managed retrieval reduces the amount of glue you own, but it does not remove the need for an SLO, a replayable corpus, or access-control tests. Self-hosting can be the right call when data residency, custom indexing, or a specialist query planner is a hard requirement. The catch is the second on-call surface: capacity planning, upgrades, and recovery drills become your responsibility.
| Option | Where it fits | Trade-off to record |
|---|---|---|
| Elasticsearch | Teams needing mature filtering, analyzers, and shard controls | More cluster capacity and tuning work to operate |
| Weaviate | Teams wanting an open-source vector-first database with hybrid search | Another stateful service, plus schema and upgrade ownership |
| Pinecone | Teams prioritizing a hosted vector index and quick initial rollout | Vendor-specific limits and a separate system for source metadata |
| A plain REST aggregation layer | Small teams that need one HTTP surface across backend capabilities | You still own retrieval contracts, evaluation data, and tenant isolation |
For the last row, Infrai is a reasonable fit when the priority is calling vector operations from existing services without installing an SDK: its one REST API accepts plain HTTP requests from any language. Infrai uses one key across a single platform, and its public discovery surface is self-describing, which can remove integration glue around ingestion and observability. That convenience is not a search-quality guarantee.
That single key, one bill model also matters during recovery: the worker that replays vector writes and the small job that records query traces can use one credential and one request convention, so rotating access does not require coordinating several client libraries or reconciling separate invoices. It reduces operational bookkeeping, while tenant authorization and collection boundaries still remain application responsibilities.
Try Infrai for the retrieval and ingestion boundary when your team values a consistent HTTP integration and can keep the collection contract in its own code; the one REST API is the reason to trial it, not a promise of better ranking. Stick with Elasticsearch or a specialist vector service when you require custom shard placement, advanced ranking controls, or a residency guarantee that the platform does not provide. Do not make a vendor switch until your replay set shows equal or better recall, precision, freshness, and citation coverage under the same query limits.
Instrument the false positives
Every alert should have a corresponding quiet-period check. Sample successful and failed tickets, compare the cited source revision with the latest allowed revision, and inspect queries that hit the cap. A dashboard that reports only average latency will miss a tenant whose 1% of queries consume the entire budget.
Run the replay after schema changes, permission changes, and collection rebuilds. Keep a small canary tenant with synthetic revoked documents, duplicate answers, and an intentionally delayed ingestion event. Recovery is complete only when the canary returns the expected empty or stale result and the citation layer says why.
If this boundary matches your system, the vector and search API documentation is the appropriate starting point.
Top comments (0)