Semantic Retrieval at Scale: Hybrid Search, Reranking, and the Precision-Recall Tradeoff

The first production incident I saw on a RAG system wasn't caused by the LLM. It was caused by retrieval.

The model was fine. The prompt was fine. What wasn't fine: our vector search was confidently returning the third-most-relevant chunk in position one, burying the actual answer around rank 15, well outside the top-k we were feeding into the context window. The LLM did exactly what you'd expect with bad context. It hallucinated a plausible-sounding wrong answer, citing a document that never said what the response claimed it said.

Nobody looked at retrieval first. Everyone looked at the model first. That's the pattern I've seen repeat across every team I've worked with: retrieval gets treated as a solved problem because "we already have a vector database," and then six months later someone's debugging prompt templates trying to fix a problem that lives two layers below the prompt.

This is about what actually happens to retrieval systems once they leave the demo and hit real traffic, real document counts, and real users typing queries nothing like your eval set.

Why Pure Vector Search Plateaus

Vector search is great at capturing semantic similarity. It's bad at a few things that matter a lot in practice:

• Exact match sensitivity. Search for an error code like ETIMEDOUT or an API parameter like max_tokens, and dense embeddings will happily return semantically "close" documents that never mention the literal token. Cosine similarity doesn't care that you needed an exact string match.
• Rare terms and identifiers. Product SKUs, function names, ticket numbers, config keys: embedding models compress these into the same latent space as everything else, and rare tokens get smoothed out by the training distribution the model saw.
• Domain drift. Off-the-shelf embedding models are trained on general web text. Your internal knowledge base full of Kubernetes manifests, Terraform modules, and internal acronyms lives in a distribution the model has barely seen.

None of this shows up in a 50-document demo. It shows up once your index crosses a few hundred thousand chunks and the semantic neighborhoods start getting crowded. Lots of documents cluster near the query vector, and the ranking signal that used to separate "relevant" from "topically adjacent" gets noisy.

Here's the part nobody puts in the demo video: vector search doesn't fail loudly. It fails by quietly returning plausible-but-wrong results, which is the worst kind of failure because it doesn't look broken.

Why Lexical Search Still Matters

BM25 (the ranking function behind Elasticsearch, Lucene, and most "classic" search engines) is often treated as legacy tech now that embeddings exist. It isn't. BM25 is a term-frequency / inverse-document-frequency scoring function, and it's extremely good at exactly what dense retrieval is weak at: exact and near-exact term matches, rare identifiers, structured technical vocabulary.

Search "OAuth2 refresh_token grant_type" against an OAuth documentation corpus, and BM25 will nail it, because those are literal tokens that appear in a small number of highly relevant documents. A dense retriever might return a conceptually related page about authentication flows in general that never uses the phrase refresh_token at all.

The point isn't that BM25 is better than dense retrieval. It's that they fail in different, complementary places. That complementarity is the entire justification for hybrid search, not because it's trendy, but because the failure modes barely overlap.

Hybrid Search: The Architecture

At a basic level, hybrid search runs two retrieval paths in parallel and merges them:

A few things worth calling out that don't come up until you actually build this.

Score fusion is not trivial. BM25 scores and cosine similarity scores live on completely different scales and distributions. You can't just average them. Two common approaches:

• Reciprocal Rank Fusion (RRF). Instead of combining raw scores, you combine ranks. Each document gets 1 / (k + rank) from each retriever, and you sum across retrievers. This sidesteps the scale mismatch problem entirely, which is why RRF shows up in almost every production hybrid search implementation (Elasticsearch, Weaviate, and Vespa all support it natively).
• Weighted linear combination. Normalize both score distributions (min-max or z-score) and combine with a tunable alpha. More flexible, but the weight becomes another hyperparameter you now have to tune and monitor for drift.

Most teams I've seen start with RRF because it requires zero calibration and works reasonably well out of the box, then move to weighted fusion once they have enough query logs to actually tune alpha against a real relevance signal instead of guessing.

Deduplication matters more than people expect. If your chunking strategy has any overlap (sliding window chunking, overlapping paragraphs), the same underlying content can surface from both retrieval paths as near-duplicate chunks, silently eating slots in your candidate set that should have gone to genuinely different content.

Metadata filtering should happen as early as possible. If you know the query is scoped to a specific product version, tenant, or document type, filter before you do the expensive stuff, not after. Filtering post-hoc means you paid the ANN cost and the BM25 cost for a huge portion of candidates that just get thrown away anyway. Pre-filtering (as a filtered ANN query, not a post-filter) is one of the highest-leverage, lowest-effort optimizations in this whole pipeline, and it's the one most commonly skipped in v1 implementations.

Candidate Explosion

Here's a failure mode that's easy to walk into: as your hybrid pipeline matures, you start pulling top-50 or top-100 from each retriever "to be safe," fuse them into a candidate pool of 150-200 documents, then feed all of that into a reranker.

This is candidate explosion, and it's a silent latency killer. The ANN search itself is fast, sub-50ms typically for a well-tuned HNSW or IVF index even at high k. The problem is downstream: every candidate you keep is a candidate the reranker has to score, and reranking cost scales roughly linearly (sometimes worse) with candidate count.

The instinct to "just retrieve more candidates to be safe" is exactly the instinct that turns a 200ms retrieval pipeline into a 2-second one.

Reranking: Why Cross-Encoders Exist

Bi-encoders (the standard embedding model setup, à la Sentence-Transformers) encode the query and document independently into fixed vectors, then compare with cosine similarity. This is what makes vector search fast: you can precompute document embeddings once and just compute a dot product at query time.

Cross-encoders work completely differently. They take the query and a candidate document together, as a single input, and run the whole pair through the transformer jointly. This lets the model attend across query tokens and document tokens simultaneously, which produces a dramatically more accurate relevance judgment, but it means you cannot precompute anything. Every query-document pair requires a fresh forward pass at inference time.

Bi-Encoder (fast, less precise)
Query  → Encoder → Vector Q
Doc    → Encoder → Vector D
Score = cosine(Q, D)

Cross-Encoder (slow, more precise)
[Query + Doc] → Transformer → Score

This is the fundamental tradeoff of reranking: cross-encoders are meaningfully better at judging true relevance because they see the full interaction between query and document, but they don't scale to your entire index. You can't cross-encode 500,000 documents per query. You use them as a second stage: take the top-N from cheap retrieval, then re-score just those with the expensive, accurate model.

User Query
      │
      ▼
Hybrid Retrieval (BM25 + Vector)
      │
      ▼
Candidate Set (~100-200 docs)
      │
      ▼
Cross-Encoder Reranker
      │
      ▼
Top-K (5-10 docs)
      │
      ▼
LLM Context

Models like Cohere Rerank, BGE-reranker, and ColBERT-style late-interaction models (a middle ground between bi-encoder and full cross-encoder) all exist to solve this same problem at different points on the latency/accuracy curve. ColBERT in particular is worth knowing about because it does per-token late interaction instead of full joint attention, giving you most of the accuracy gain of cross-encoders at a fraction of the compute cost, though it comes with a heavier storage footprint since you're indexing token-level embeddings instead of one vector per document.

The Precision-Recall Tradeoff

This is the section that actually determines whether your retrieval pipeline works, and it's the one most people skip past to get to the "cool" architecture diagrams.

Every knob in your retrieval pipeline (top-k at the ANN stage, top-k at BM25, fusion strategy, reranker cutoff) is really just moving you along a single axis: how much you prioritize not missing the right document (recall) versus not including wrong documents (precision).

High Recall path
      │
Retrieve more candidates
      │
      ▼
Broader net, catches edge cases
      │
      ▼
More noise for reranker to sort through
      │
      ▼
Higher reranking latency & cost


High Precision path
      │
Retrieve fewer candidates
      │
      ▼
Tighter, faster, cheaper
      │
      ▼
Risk: the actually-relevant doc
never made it into the candidate set
      │
      ▼
No reranker can recover a document
that was never retrieved

That last line is the one that trips people up: reranking can only reorder what retrieval already found. If your first-stage recall is bad, if the ANN search's top-100 didn't include the document that actually answers the question, no amount of cross-encoder sophistication downstream will fix it. This is why teams obsess over recall at the retrieval stage and precision at the reranking stage. They're different jobs assigned to different parts of the pipeline on purpose.

In practice this plays out as a very concrete engineering decision: how big should your first-stage candidate set be? Too small, and you're capping your ceiling recall before reranking even runs. Too large, and you're paying reranking latency for candidates that had no chance of being relevant. There's no universal number. It depends on your corpus density, your chunk size, and how semantically crowded your embedding space is, which is exactly why you need to actually measure this instead of guessing.

Metrics That Actually Matter Here

You don't need a research-paper-level treatment of these, but you do need to know why engineers track them:

• Recall@K: of all the truly relevant documents, what fraction showed up in your top K? This is your ceiling metric. If Recall@50 is low, no reranker will save you; you have a retrieval problem, not a ranking problem.
• Precision@K: of the K documents you returned, what fraction were actually relevant? This matters most at small K, because small K is what actually reaches the LLM's context window.
• MRR (Mean Reciprocal Rank): how high up was the first relevant result, on average? Useful when there's typically one clearly correct answer (support tickets, docs lookups) rather than several equally valid ones.
• NDCG: like precision, but weighted by position and relevance grade rather than binary relevant/irrelevant. More useful when documents have graded relevance rather than a strict yes/no.
• Hit Rate: simplest possible signal. Did any relevant document appear in top K at all? Good as a coarse health check, bad as your only metric.
• Latency percentiles (P50/P95/P99): because average latency lies. Your P50 might be 150ms and feel great in a demo while your P99 is 2.5 seconds because some queries trigger cold cache misses or hit reranker batching edge cases. P99 is what your angriest user experiences.

The lesson that took me longest to internalize: retrieval quality matters more than model quality. Teams spend weeks tuning prompts and swapping LLMs while their Recall@20 sits at 60%. Fix retrieval first. The best model in the world can't answer a question correctly if the answer was never in its context.

Chunking Feeds All of This

Chunking strategy isn't a separate topic from retrieval quality, it's upstream of everything above. Chunk too large, and you dilute the embedding signal (a 2,000-token chunk covering five subtopics produces a vector that's a mediocre match for all five, and an excellent match for none). Chunk too small, and you fragment context so badly that the "relevant" chunk your retriever finds is missing the surrounding information the LLM actually needs to answer correctly.

This also directly affects candidate explosion: smaller chunks mean more chunks per document, which means more candidates competing for the same top-K slots, which means your reranker sees more near-duplicate content from the same source document instead of diverse, distinct candidates.

There's no universally correct chunk size. It depends on document structure (API docs chunk naturally along endpoint boundaries; Kubernetes docs chunk naturally along resource-type sections), but it's worth treating as a first-class tunable parameter you evaluate against your recall metrics, not a default you set once in week one and forget about.

Production Lessons, Collected

A few things that only became obvious after running these systems against real traffic:

• Vector search alone rarely survives contact with a real, growing knowledge base. It's a great starting point, not an end state.
• Metadata filtering is one of the cheapest wins available. Filtering by tenant, doc type, or recency before the expensive ANN/BM25 stages saves enormous compute, and most teams add it too late.
• Rerankers should be used strategically, not universally. If your first-stage retrieval already has high precision (small, well-scoped corpus), a cross-encoder may add latency without meaningfully improving results.
• Measure retrieval before you touch the prompt. It's tempting to iterate on prompts because the feedback loop is faster, but if retrieval is broken, prompt iteration is polishing the wrong layer.
• Index synchronization is an operational tax nobody budgets for. Embeddings drift as models get updated, BM25 indices need to stay fresh as documents change, and keeping two indices consistent under concurrent writes is genuinely nontrivial distributed systems work, not a footnote.

Where a Semantic Cache Fits In

Once you have hybrid retrieval plus reranking running in production, you've built something accurate, but also something expensive. Every query pays for a BM25 pass, an ANN pass, fusion, deduplication, and a cross-encoder pass over a hundred-plus candidates. At scale, a meaningful chunk of your query traffic is semantically repetitive: users ask the same underlying question in different phrasings, support agents rerun similar lookups, internal tools fire near-identical queries in tight loops.

This is where a semantic execution cache like Remem becomes relevant, as a complementary layer rather than a replacement for anything above. Instead of caching on exact query string match (which misses paraphrases entirely), it caches based on semantic similarity of the query itself, so a previously-executed retrieval-plus-rerank result can be reused for a new query that means the same thing, even if it's worded differently. Done well, this can meaningfully cut the repeated retrieval and reranking cost for query patterns that recur often, without touching the accuracy of your actual retrieval pipeline for genuinely novel queries. It's a caching layer sitting in front of the pipeline described above, not a substitute for building that pipeline correctly in the first place.

Closing Thought

Great RAG systems aren't built by choosing the largest LLM. They're built by designing retrieval systems that consistently deliver the right context, at acceptable latency, at acceptable cost, and by treating that as the primary engineering problem rather than an afterthought bolted on before the "real" LLM work begins.

The model is usually the easiest part to swap. The retrieval pipeline is the part that actually determines whether your system works.

What does your retrieval stack look like in production? Pure vector, hybrid, something with a reranker in the loop? What broke first when you scaled past your initial index size? I'd genuinely like to hear how other people's architectures evolved once real traffic hit them.