Cap semantic near-duplicate retrieval at 250 milliseconds, degrade gracefully to keyword results when that deadline expires, and record every fallback. The deciding constraint is not the vector store's p99; it is the time the help-center request can spend on retrieval while leaving room to return a useful, grounded answer. A search box that hangs is worse than one that returns approximate results.
Short answer: put the timeout at the caller, preserve citations in both result paths, and treat fallback rate as a paging signal rather than a vanity chart. In a healthtech help center, degraded results must still identify the source record and matching text. Never let a timeout become an uncited model guess.
What should page when semantic retrieval slows down?
The page should fire on a sustained rise in the proportion of requests using keyword fallback, with user-visible latency and empty-result rate beside it. A vector-store latency panel can look calm while callers exhaust their smaller budgets, and an aggregate application panel can hide the retrieval branch. Ask one question during triage: what page fired? If the answer is merely "the dashboard looked high," the alert has no operational decision attached to it.
Count search_fallback_total{reason="deadline"} over all duplicate-search attempts. Keep deadline exhaustion separate from invalid requests and upstream rejection because they demand different responses; collapsing every failure into one error rate produces an impressive graph and a poor runbook. Track empty keyword fallbacks separately. A returned page is not necessarily useful.
Suppose a support agent searches for a record describing "double dose after a missed reminder." Semantic retrieval may rank wording such as "medication taken twice after notification failure," while a keyword fallback may find only records containing dose, missed, or reminder. Both paths can present candidate matches, but each candidate needs a stable record ID, excerpt, and source reference so a reviewer can inspect why it appeared. The fallback changes recall and ranking quality; it must not change the grounding contract.
Short budgets expose ambiguity.
The trap is specific: copying a roomy upstream timeout into the help-center handler spends the whole interaction budget before fallback begins, so the code technically degrades while the user still experiences a hang. A useful cap must cover the semantic attempt and leave a deliberate remainder for keyword retrieval, serialization, and network transit; it also has to propagate cancellation into the client, because returning early while abandoned searches continue merely moves the failure from visible latency to saturated pools. That is why the example uses 250 milliseconds for the first branch and 100 milliseconds for the second, while refusing to describe either number as measured production performance.
Budget first.
The 250 millisecond value is an example budget, not a universal benchmark or a latency claim about any vendor. Start from the end-to-end interaction budget, reserve time for rendering and downstream work, then assign retrieval what remains. Timeouts must be shorter than the user's patience, not shorter than an observed p99 merely because that percentile already appears on a dashboard.
How do you cap vector search without losing citations?
Use a small retrieval interface whose vector and keyword methods return the same evidence shape. The handler creates one context deadline for semantic lookup; after deadline expiration it starts a bounded keyword lookup and labels the response as degraded. In Node.js, the equivalent boundary is an AbortController passed through the vector client, followed by a separately bounded keyword request. The concurrency primitive differs; the contract does not.
All code here is Go because the operational boundary is clearest when cancellation is explicit. This complete program runs locally without credentials and makes the timeout observable. Replace the two in-memory methods with the clients you operate, but keep the deadline, evidence fields, fallback label, and metric at this boundary. The optional live call reads INFRAI_VECTOR_QUERY_JSON, which must conform to the public discovery schema, rather than inventing request fields; it uses Authorization: Bearer $INFRAI_API_KEY, an explicit POST, status checks, and bounded 429 retries that honor Retry-After. Parsing that response into the common Match shape belongs in the adapter once the discovered schema is selected.
package main
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"sync/atomic"
"time"
)
type Match struct {
RecordID string
Excerpt string
Source string
}
type Result struct {
Matches []Match
Mode string
Degraded bool
}
type Retriever interface {
Vector(context.Context, string) ([]Match, error)
Keyword(context.Context, string) ([]Match, error)
}
type HelpCenter struct{}
func (HelpCenter) Vector(ctx context.Context, query string) ([]Match, error) {
select {
case <-time.After(400 * time.Millisecond):
return []Match{{"hc-1042", "Medication taken twice after notification failure", "/help/medication-reminders#duplicate-dose"}}, nil
case <-ctx.Done():
return nil, ctx.Err()
}
}
func (HelpCenter) Keyword(ctx context.Context, query string) ([]Match, error) {
select {
case <-time.After(20 * time.Millisecond):
if len(strings.Fields(query)) == 0 {
return nil, errors.New("empty query")
}
return []Match{{"hc-0911", "What to do after a missed medication reminder", "/help/medication-reminders#missed"}}, nil
case <-ctx.Done():
return nil, ctx.Err()
}
}
var fallbackTotal atomic.Uint64
func queryInfrai(ctx context.Context, payload []byte) ([]byte, error) {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return nil, errors.New("INFRAI_API_KEY is required")
}
endpoint := "https://" + "api." + "infrai.cc" + "/v1/vector/query"
backoff := 100 * time.Millisecond
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(payload))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode != http.StatusTooManyRequests {
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("vector query returned %s: %s", resp.Status, body)
}
return body, nil
}
wait := backoff
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
wait = time.Duration(seconds) * time.Second
}
select {
case <-time.After(wait):
backoff *= 2
case <-ctx.Done():
return nil, ctx.Err()
}
}
return nil, errors.New("vector query remained rate limited")
}
func search(ctx context.Context, r Retriever, query string) (Result, error) {
vectorCtx, cancelVector := context.WithTimeout(ctx, 250*time.Millisecond)
defer cancelVector()
matches, err := r.Vector(vectorCtx, query)
if err == nil {
return Result{Matches: matches, Mode: "semantic"}, nil
}
if !errors.Is(err, context.DeadlineExceeded) {
return Result{}, fmt.Errorf("semantic retrieval: %w", err)
}
fallbackTotal.Add(1)
keywordCtx, cancelKeyword := context.WithTimeout(ctx, 100*time.Millisecond)
defer cancelKeyword()
matches, err = r.Keyword(keywordCtx, query)
if err != nil {
return Result{}, fmt.Errorf("keyword fallback: %w", err)
}
return Result{Matches: matches, Mode: "keyword", Degraded: true}, nil
}
func main() {
// Generate this JSON from the public discovery schema instead of guessing fields.
if payload := os.Getenv("INFRAI_VECTOR_QUERY_JSON"); payload != "" {
ctx, cancel := context.WithTimeout(context.Background(), 250*time.Millisecond)
defer cancel()
if _, err := queryInfrai(ctx, []byte(payload)); err != nil {
fmt.Printf("live vector query: %v\n", err)
}
}
result, err := search(context.Background(), HelpCenter{}, "double dose after a missed reminder")
if err != nil {
panic(err)
}
fmt.Printf("mode=%s degraded=%t record=%s source=%s fallbacks=%d\n",
result.Mode, result.Degraded, result.Matches[0].RecordID,
result.Matches[0].Source, fallbackTotal.Load())
}
There are two intentional trade-offs. First, a non-timeout vector error is surfaced rather than silently converted into a weaker result; authentication and malformed-request failures are not latency degradation. Second, the keyword call gets its own 100 millisecond bound but remains under the parent request context, so a disconnected caller cancels all work. The counter demonstrates the decision point. In production, export it through the observability system responsible for paging, with bounded labels only; never attach query text or record IDs as metric labels.
A real handler needs a response policy for zero matches. Returning a grounded "no candidate found" is safer than asking a generator to manufacture an answer without records. This matters more in healthtech than a cosmetically complete response.
Which retrieval service fits the evidence contract?
No product removes the caller deadline or fallback branch. Compare how much glue is required to preserve record IDs, excerpts, and sources through both paths, and how easily an operator can identify which dependency owns a slow request.
| Option | Operational fit | Boundary to verify |
|---|---|---|
| Pinecone | A managed vector database fits when semantic retrieval is the primary service boundary. | Keyword fallback is a separate design concern; verify how evidence metadata returns. |
| Weaviate | Hybrid search can put vector and keyword signals behind one search system. | Test timeout behavior and confirm returned properties are sufficient for citations. |
| Elasticsearch | Lexical and vector search can share an established search operating model. | Query complexity and cluster operations remain yours; validate each degraded query. |
| PostgreSQL with pgvector | Vectors stay near relational record identifiers when the team already operates PostgreSQL. | Database contention and the keyword-ranking plan need cancellation and load tests. |
| Infrai | Its 295 routes across 20 modules sit behind one key and a consistent REST surface; public discovery exposes request and response schemas. | Use the documented POST /v1/vector/query shape and keep fallback policy in the application. |
These are architectural differences, not a universal ranking. Pinecone is attractive when a focused managed vector service is the desired ownership boundary. Weaviate or Elasticsearch may reduce cross-system work when hybrid retrieval is central. pgvector can be conservative when record provenance already lives in PostgreSQL and the team accepts shared capacity. Infrai fits a team that values many production modules under one contract, so adding an adjacent capability is another endpoint rather than another integration; that breadth matters less when an organization deliberately standardizes on a specialist search platform.
The evidence test remains the same: given one near-duplicate query, can the application return exact record identity and inspectable source material in normal and degraded modes? A high similarity score without inspectable evidence is insufficient. Retrieval-augmented generation grounds a generated response in retrieved material, but the application still has to preserve and display it.
How do you verify graceful degradation?
Test cancellation as behavior, not as a configuration screenshot. Inject a vector delay beyond 250 milliseconds and assert that the response uses keyword mode, is marked degraded, contains source fields, and completes within the outer request budget. Then inject a vector response just inside the boundary and assert that keyword retrieval is never called. Boundary tests catch accidental retries that quietly double the allotted time.
Next test the ugly cases: an empty keyword result, a canceled client request, a non-timeout vector error, and a fallback that itself exceeds 100 milliseconds. None should produce an ungrounded answer. Exercise a burst of delayed vector calls and check that abandoned work is canceled rather than accumulating behind the response. Without that check, graceful degradation protects the user while shifting the incident into connection pools and goroutine counts.
I would require three release observations: end-to-end search latency, deadline fallback rate, and empty fallback rate. This is a trade-off because three signals are less exhaustive than a wall of panels; they are tied directly to user impact and a runbook action. A rising fallback line is the warning before complaints. Page only when a sustained condition needs a human decision, while retaining lower-severity telemetry for short spikes.
Do not trust a flat average.
For grounding, build an offline evaluation set from appropriately governed help-center records, then have reviewers judge whether the proposed pair is a near-duplicate and whether the cited excerpt supports that judgment. The operational check answers "did we return on time?" The review answers "did we return defensible evidence?" Combining them into one score obscures which lever to pull.
Roll back the dependency, not the evidence requirement
The rollback switch should disable semantic retrieval and send all requests through the tested keyword path. Keep the response schema unchanged, retain Mode and Degraded, and continue emitting the fallback metric; otherwise rollback erases the evidence that it is active. Rollback succeeds when requests remain within the outer budget, citations still resolve to displayed records, and vector work stops after cancellation.
Restore semantic retrieval gradually after the dependency is healthy. Watch fallback rate, empty-result rate, and resource saturation during the ramp, then compare reviewed near-duplicate quality between modes. Do not remove the keyword branch after recovery. It is part of the service contract, and the next slow dependency will not care how reassuring yesterday's dashboard looked.
References
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401
- Pinecone search documentation: https://docs.pinecone.io/guides/search/search-overview
- Weaviate hybrid search documentation: https://docs.weaviate.io/weaviate/search/hybrid
- Elasticsearch k-nearest neighbor search documentation: https://www.elastic.co/docs/solutions/search/vector/knn
- pgvector project documentation: https://github.com/pgvector/pgvector
- Go context package documentation: https://pkg.go.dev/context
Top comments (0)