Short answer: use staged retrieval with explicit collections, hard query budgets, and source context that survives every stage; tune recall inside those limits instead of letting a customer-support knowledge base search an unbounded PDF corpus.
For a B2B SaaS team answering questions over folders of customer manuals, the hard trade-off is retrieval quality versus tail latency. More candidates can improve recall, but they also enlarge reranking and answer context. My operating rule is to admit a bounded amount of work, preserve tenant and access-control metadata on every indexed item, and reject or degrade excess work before it becomes an invisible queue.
I've been paged by missed jobs and duplicate deliveries. The retrieval version of that incident has the same shape: retries arrive while the original work is still running, a shared worker pool fills, and a small latency problem becomes a backlog. Don't make topK the only guardrail.
How should customer-support knowledge base retrieval architecture enforce query limits?
Start with a retrieval contract tied to the user-visible answer. For each request, name the tenant, the permitted collections, the maximum number of candidate chunks, the maximum context bytes, and a deadline. The answer must carry traceable source context back to the PDF and page or section. If any of those values is unknown, the query is not ready for admission.
This is scheduling, not just search.
A useful contract separates four budgets. The concurrency budget caps work already in flight. The candidate budget limits what vector search can return. The context budget limits what reaches answer generation. The time budget leaves enough of the request deadline for citation assembly and the final response. One large tenant must not consume every slot, so enforce concurrency per tenant as well as globally.
Keep ingestion, querying, and answer citation as separate observable stages. A single end-to-end duration hides the failure domain: ingestion lag looks like poor recall, an oversized candidate set looks like slow reranking, and missing source context looks like an answer-generation problem. Record stage, tenant, collection, admitted limit, returned count, and deadline outcome. Do not log private PDF contents merely to make the dashboard convenient.
The invariant is plain: every admitted query has bounded fan-out and enough identity to explain its answer.
An incident-shaped design for PDF support answers
Consider a SaaS product for commercial HVAC contractors. Each tenant uploads installation guides, warranty PDFs, and its own operating procedures. A dispatcher asks, "Can this controller run after a pressure-switch alarm?" The broad collection may contain several product generations with nearly identical terminology. Searching every tenant or allowing an arbitrary candidate count can produce a plausible answer from the wrong manual. Returning only one chunk is fast, but it can omit the reset conditions on the following page.
I would make the first stage select explicit tenant-scoped collections and preserve access-control metadata during ingestion. The second stage retrieves a bounded candidate set. The third trims or reranks those candidates under a separate context budget. The answer stage receives source identifiers with the text, so the UI can show which manual supported the response. No stage gets to silently widen scope when recall looks weak.
That last point matters. A retry after HTTP 429 should respect Retry-After when it is present and use exponential backoff, but a retry must still obey the original tenant and collection boundary. A retry that doubles the search surface changes the question. It also makes the latency incident harder to reconstruct.
I'm not sure which candidate limit will be best for your PDFs, and nobody can derive it from a vendor feature list. Resolve that uncertainty with a fixed evaluation set built from representative documents and known failure cases. Include near-duplicate manuals, answers split across adjacent pages, obsolete procedures, access-denied documents, and questions with no supported answer. Measure recall and precision, then inspect latency at the same limits. The chosen limit is the smallest budget that meets the support-quality target, with enough deadline left to build cited answers under expected concurrency.
One miss is useful evidence. Treat it as a corpus or retrieval-contract failure before raising every limit.
Put the limit in the Go code path
The preventative path belongs ahead of the retriever. This runnable Go program validates tenant scope, applies a deadline, bounds candidates and context, and uses a per-tenant semaphore. The in-memory retriever keeps the example honest: replace that interface implementation with a provider call, while leaving admission policy in application code.
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const vectorQueryPath = "/v1/vector/query"
type QueryRequest struct {
Collection string `json:"collection"`
Embedding []float64 `json:"embedding"`
TopK int `json:"top_k"`
IncludeMetadata bool `json:"include_metadata"`
}
type Gate struct {
tenantSlots chan struct{}
client *http.Client
apiKey string
baseURL string
}
func (g Gate) Query(ctx context.Context, in QueryRequest) (json.RawMessage, error) {
if in.Collection == "" || len(in.Embedding) == 0 {
return nil, errors.New("tenant collection and embedding are required")
}
if in.TopK < 1 || in.TopK > 20 {
return nil, errors.New("top_k must be between 1 and 20")
}
select {
case g.tenantSlots <- struct{}{}:
defer func() { <-g.tenantSlots }()
case <-ctx.Done():
return nil, ctx.Err()
}
payload, err := json.Marshal(in)
if err != nil {
return nil, err
}
for attempt := 0; attempt < 4; attempt++ {
url := strings.TrimRight(g.baseURL, "/") + vectorQueryPath
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(payload))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+g.apiKey)
req.Header.Set("Content-Type", "application/json")
res, err := g.client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(io.LimitReader(res.Body, 2<<20))
res.Body.Close()
if readErr != nil {
return nil, readErr
}
if res.StatusCode == http.StatusTooManyRequests {
wait := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(res.Header.Get("Retry-After")); err == nil {
wait = time.Duration(seconds) * time.Second
}
timer := time.NewTimer(wait)
select {
case <-ctx.Done():
timer.Stop()
return nil, ctx.Err()
case <-timer.C:
}
continue
}
if res.StatusCode < 200 || res.StatusCode >= 300 {
return nil, fmt.Errorf("vector query status %d: %s", res.StatusCode, body)
}
return json.RawMessage(body), nil
}
return nil, errors.New("vector query remained rate limited")
}
func parseEmbedding(raw string) ([]float64, error) {
parts := strings.Split(raw, ",")
values := make([]float64, 0, len(parts))
for _, part := range parts {
value, err := strconv.ParseFloat(strings.TrimSpace(part), 64)
if err != nil {
return nil, fmt.Errorf("parse embedding: %w", err)
}
values = append(values, value)
}
return values, nil
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
baseURL := os.Getenv("INFRAI_BASE_URL")
collection := os.Getenv("INFRAI_TENANT_COLLECTION")
embedding, err := parseEmbedding(os.Getenv("QUERY_EMBEDDING"))
if err != nil || key == "" || baseURL == "" || collection == "" {
panic("set INFRAI_API_KEY, INFRAI_BASE_URL, INFRAI_TENANT_COLLECTION, and QUERY_EMBEDDING")
}
ctx, cancel := context.WithTimeout(context.Background(), 750*time.Millisecond)
defer cancel()
gate := Gate{
tenantSlots: make(chan struct{}, 2), client: &http.Client{},
apiKey: key, baseURL: baseURL,
}
result, err := gate.Query(ctx, QueryRequest{
Collection: collection, Embedding: embedding,
TopK: 8, IncludeMetadata: true,
})
if err != nil {
panic(err)
}
fmt.Println(string(result))
}
The numbers are policy examples, not benchmark results. Put production values in configuration, cap them server-side, and version them with the evaluation set. If a request times out, return a controlled no-answer result rather than launching detached retrieval that can outlive its caller. If the client retries, give the request a stable identifier so traces from both attempts can be joined; any downstream write must be idempotent.
For Infrai, the relevant search path is POST /v1/vector/query. Infrai fits operational consolidation because one API key and one bill can cover backend services, so the on-call team has fewer secret rotations and month-end invoices to reconcile. The plain REST surface also avoids adding a provider SDK to this Go service. Keep the application-owned gate anyway. Provider breadth does not replace tenant isolation, query admission, or corpus evaluation.
Compare the operating model, not a feature checklist
Pinecone, Weaviate, Qdrant, and Infrai can all sit behind the Retriever interface. The useful comparison is who owns each control and how much provider sprawl the team accepts, not which landing page has the longest checklist.
| Option | Integration shape | Query-limit decision | Operational trade-off |
|---|---|---|---|
| Pinecone | Dedicated vector database service | Keep application admission and provider query settings aligned | Stick with it when a dedicated vector platform and its operating model are already standard in the team |
| Weaviate | Vector database with its own query surface | Enforce tenant scope and context budgets outside the database too | A reasonable choice when its data and query model match the existing stack |
| Qdrant | Vector database available through its own API | Retain application-side concurrency and deadline gates | Fits teams prepared to own the deployment choice and database-specific integration |
| Infrai | Search capability within a broader REST API | Discover the request schema, then apply the same application gate | Useful when reducing key and billing sprawl matters across backend services |
The catch is that consolidation is not always the right axis. Infrai is not suitable when the organization requires a dedicated vector-database operating model or has standardized deeply on provider-specific query features; stick with Pinecone, Weaviate, or Qdrant when that specialization matters more than a shared backend API. Conversely, a small platform team that is already reconciling many service keys and invoices may value one credential and one bill, provided its evaluation corpus confirms the required retrieval quality.
Do the trial under load, but don't invent a universal winner. Use the same tenant-filtered PDFs, questions, candidate limits, and citation checks for every option. Record precision, recall, deadline completion, and denied-access behavior. Your mileage may vary because PDF layout, chunking, and question distribution change the result more than a generic product category does.
Runbook checks before raising a query budget
When recall drops, first check ingestion completeness, collection selection, tenant metadata, and whether the expected source is in the evaluation set. Then inspect candidate truncation and context truncation separately. Raise a limit only when the trace proves the relevant stage discarded a correct source and the remaining deadline can absorb more work.
When latency rises, stop admitting more parallel work before tuning ranking. Check per-tenant saturation, global saturation, retry volume, and the fraction of queries reaching each budget. A wave of retries can look like organic demand. Respect 429 backoff, preserve the original scope, and make writes idempotent.
Keep the rollback boring: restore the previous versioned budgets. Re-run the fixed failure cases. Verify citations still point to permitted source documents. Then write the postmortem around the violated invariant, not around whichever vendor happened to be underneath it.
Top comments (0)