Retrieval Latency Budgets in RAG Pipelines: Vector Search, Reranking, and the Timeout Cascade Problem

RAG pipelines are not prompt engineering problems. They are distributed systems problems dressed in LLM clothing. The user-facing latency budget is fixed—call it 2 seconds for a synchronous assistant API. Inside that budget you are running vector search, an optional reranking call, context assembly, and an LLM completion that alone can consume 800ms to 1.5 seconds. The retrieval layer has to finish its work in whatever remains, and it rarely gets a dedicated timeout because most teams wire the pipeline together sequentially and measure the whole thing only at the edge.

The result is what I call the timeout cascade: a slow vector index query delays reranking, reranking blows the LLM call's deadline, the LLM call either times out or gets cancelled mid-stream, and the user sees a degraded or failed response. The root cause is not the model. It is the absence of per-stage latency contracts enforced at the code level.

The Latency Stack

A minimal production RAG backend has these sequential stages:

1. Query embedding — 20–80ms depending on embedding model and whether it is local or a remote API call
2. ANN vector search — 10–150ms depending on index type, recall target, and cluster load
3. Reranking — 50–400ms for a cross-encoder, often a remote gRPC or HTTP call
4. Context assembly — sub-millisecond if done in memory
5. LLM completion — 500ms to 2000ms+ for streaming first-token

If you treat these as a linear chain with one outer timeout, any stage that runs slow consumes budget from every subsequent stage. The LLM call, which cannot be sped up, gets what is left—sometimes nothing.

Enforcing Per-Stage Budgets in Go

Go's context package is the right primitive. The pattern is to derive a child context with a deadline for each stage, pass it to the remote call, and make a local decision about whether to continue, degrade, or abort before moving to the next stage.

type RetrievalConfig struct {
    EmbedTimeout   time.Duration // e.g. 100ms
    SearchTimeout  time.Duration // e.g. 200ms
    RerankTimeout  time.Duration // e.g. 300ms
    TotalBudget    time.Duration // e.g. 700ms leaving ~1.3s for LLM
}

func Retrieve(ctx context.Context, query string, cfg RetrievalConfig) ([]Chunk, error) {
    root, rootCancel := context.WithTimeout(ctx, cfg.TotalBudget)
    defer rootCancel()

    // Stage 1: embedding
    eCtx, eCancel := context.WithTimeout(root, cfg.EmbedTimeout)
    embedding, err := embedQuery(eCtx, query)
    eCancel()
    if err != nil {
        return nil, fmt.Errorf("embed: %w", err)
    }

    // Stage 2: ANN search
    sCtx, sCancel := context.WithTimeout(root, cfg.SearchTimeout)
    candidates, err := vectorSearch(sCtx, embedding)
    sCancel()
    if err != nil {
        return nil, fmt.Errorf("search: %w", err)
    }

    // Stage 3: rerank with graceful degradation
    rCtx, rCancel := context.WithTimeout(root, cfg.RerankTimeout)
    ranked, err := rerank(rCtx, query, candidates)
    rCancel()
    if err != nil {
        // Degradation: skip reranking, return ANN order
        return candidates, nil
    }

    return ranked, nil
}

The critical detail: eCancel() is called immediately after use, not deferred to function exit. Deferring all cancels to the function boundary causes the parent context's timer to run the full allocation for each stage even after the call completes. Explicit release is mandatory when multiple staged timeouts share a single parent budget.

The rerank stage explicitly degrades on error rather than propagating it. Reranking improves recall precision but is not load-bearing for correctness. Losing it costs answer quality; losing it and aborting the request costs the user entirely. That is the wrong tradeoff.

The Remaining-Budget Pattern

Fixed per-stage timeouts have a flaw: if embedding finishes in 20ms instead of 80ms, the saved 60ms is not reclaimed for downstream stages—the child context budget is gone but the root budget still has it. To propagate unused budget, derive each child timeout from the root context's remaining deadline:

func stageTimeout(root context.Context, max time.Duration) (context.Context, context.CancelFunc) {
    deadline, ok := root.Deadline()
    if !ok {
        return context.WithTimeout(root, max)
    }
    remaining := time.Until(deadline)
    budget := remaining - 50*time.Millisecond // reserve 50ms for assembly + overhead
    if budget <= 0 {
        // No budget left; return an already-cancelled context
        ctx, cancel := context.WithCancel(root)
        cancel()
        return ctx, cancel
    }
    if budget > max {
        budget = max
    }
    return context.WithTimeout(root, budget)
}

This lets fast stages donate time to slower ones up to the per-stage cap, while ensuring the root deadline is never violated. The 50ms reserve guards against clock skew and serialization overhead between stages.

Vector Search as a Latency Variable

Vector databases (pgvector, Qdrant, Weaviate, Pinecone) expose recall/latency tradeoffs through ANN parameters: HNSW ef at query time, number of probes for IVF indexes, or the equivalent. Higher recall means higher latency. In a RAG context, recall above 90% rarely changes answer quality because the LLM is reading 10–20 chunks anyway—the marginal improvement from chunk 4 versus chunk 5 in the result set is negligible.

The operational pattern: profile your vector search at the 95th percentile, not the median. If p95 is 120ms but p50 is 30ms, you have either query skew (some queries hit cold partitions), cluster load spikes, or a poorly tuned index. Wire your timeout at p95 + 20% headroom. Anything above that is a degraded path, not a normal path. Alerting on retrieval_stage_timeout_total by stage gives you visibility into which stage is eating budget in production.

Reranking: Remote Call or Skip

Cross-encoder reranking is typically a remote HTTP or gRPC call to a model serving endpoint (Cohere Rerank, a self-hosted Hugging Face model, or a sidecar). This means it inherits all the failure modes of a remote dependency: cold starts, queue buildup under load, and tail latency amplification.

Two patterns for production:

Circuit breaker on the rerank call. If the reranker's p95 latency exceeds threshold or error rate spikes, open the circuit and fall back to BM25 score or ANN cosine similarity for ordering. The pipeline keeps running; quality degrades gracefully.

Parallel speculative execution. Fire the vector search and pre-fetch a lightweight BM25 result simultaneously. When reranking completes, merge. If reranking times out, the BM25 result is already available as a fallback without adding latency to the critical path. This requires a local BM25 index or Elasticsearch, but it eliminates the binary fail/succeed on the reranker.

Observability for Staged Retrieval

Instrument at the stage boundary, not the pipeline boundary. Each of embedding, search, and reranking should emit a histogram with labels for outcome (success, timeout, error, degraded). Aggregate these independently:

• rag_embed_duration_seconds{outcome="success"} histogram
• rag_search_duration_seconds{outcome="timeout"} counter
• rag_rerank_duration_seconds{outcome="degraded"} counter

The degraded outcome label is important. If rag_rerank_duration_seconds{outcome="degraded"} is consistently non-zero in production, your reranker's SLO is too slow for your pipeline budget and you need to renegotiate the timeout, horizontally scale the reranker, or drop it from the critical path permanently.

Distributed tracing with propagated trace IDs across the embedding call, vector DB query, and reranker gives you the waterfall view needed to diagnose which stage is the actual bottleneck on a per-request basis—something aggregate metrics cannot provide.

Decision Framework

When designing the retrieval layer's timeout strategy, answer these questions in order:

1. What is your total user-facing latency SLO? Subtract LLM p50 completion time (measure it). That is your retrieval budget.
2. Is reranking load-bearing or optional? If answer quality is acceptable without it (run an offline eval to verify), treat it as a degradable stage with a circuit breaker, not a hard dependency.
3. What is your vector index's p95 latency under production load? Set the search timeout at p95 + 20%, not at the average.
4. Are you using fixed per-stage timeouts or remaining-budget propagation? Fixed is simpler and correct for independent deployments. Remaining-budget is better when embedding is highly variable (remote API vs. local model).
5. Do you have per-stage outcome metrics? If not, you cannot tell whether degradations are happening silently. Add them before you go to production.
6. What is your fallback for total retrieval failure? The LLM should receive a signal that context is absent, not an empty string. Return an explicit no-context marker and prompt accordingly, or return a 503 to the caller rather than a hallucinated response with no grounding.

A RAG pipeline without per-stage timeout contracts is a latency gamble dressed as a feature. The vector database and reranker will eventually be slow on the same request, and without explicit budget enforcement and degradation paths, the failure propagates to the LLM call and surfaces as an opaque timeout to the user. Enforce the contracts in code, measure each stage independently, and build degradation as a first-class behavior rather than an afterthought.