How to build vector search systems that outperform keyword matching, and where they fail.
Semantic search with embeddings lets you find what users mean, not just what they type. Instead of matching keywords, you convert queries and documents into high-dimensional vectors that capture meaning, then retrieve neighbors in that space. The result is often better relevance for typos, synonyms, and intent. The cost is latency, complexity, and a pile of new failure modes. This guide walks search and product engineers through the decisions you face when moving from keyword search to semantic search, and how to know if it's actually working.
Why this matters now
Embedding models have crossed a quality threshold. In 2020, semantic search was a research toy. By 2024, open models like BAAI/bge-base-en-v1.5 and Voyage AI deliver strong relevance on diverse domains at reasonable latency. Inference costs have dropped below one cent per million tokens. This means semantic search is now cost-effective for search at scale, not just for chatbots or small collections. Meanwhile, end-users expect search to understand intent, not parse keywords. The pressure to migrate away from BM25 or Elasticsearch keyword-only stacks is real.
But 2026 reality includes vector databases, hybrid retrieval, and freshness problems that didn't exist when you shipped keyword search. Embeddings are expensive to compute, require a separate index, and degrade when documents change. Hybrid retrieval (combining keyword and semantic) has become the default production architecture, not an edge case. Evaluation shifted from "does it rank relevant docs first" to "do users click, stay, and convert," because embeddings can be statistically similar without being actually useful. This guide covers all three layers: why semantic search matters, how to build it without breaking, and how to know you've built it right.
From keyword search to embeddings: what changes
Photo by AS Photography on Pexels.
Keyword search indexes exact terms. When a user types "laptop battery life," the search engine finds documents containing those words or stems (laptop, batter, life). Matching is fast, deterministic, and cheap. Relevance depends entirely on term frequency, field boosts, and link structure. Synonyms and intent are invisible.
Semantic search converts both the query and every indexed document into a dense vector, usually 384 to 1536 dimensions, representing semantic meaning. A query for "laptop battery life" and a document titled "How long does a MacBook Pro last?" are both encoded into vectors. The search engine then finds the k nearest neighbors (kNN) in vector space. A document about snakes won't rank, no matter how many times the word "python" appears, because the vector is far away. A document about "device power duration" will rank higher if it's semantically close, even if it shares no keywords with the query.
The trade-off is immediate: semantic search is slower (kNN search is O(n) without heavy indexing), more expensive (encoding every document costs money), and less predictable (the semantic space can be noisy). But it handles typos, synonyms, and queries that require understanding context, not just term overlap.
Architecture: embedding pipelines and hybrid retrieval
A production semantic search system has three layers: embedding generation, vector indexing, and retrieval ranking.
Embedding generation. Before any search happens, you need vectors for all documents. This is a batch or stream process. For a 10 million document collection, you embed once at scale (using GPUs or a batch API), then update incrementally when documents change. Embedding models are often open (BAAI/bge-base-en-v1.5, Sentence Transformers, MiniLM) or closed (OpenAI text-embedding-3-small, Voyage AI). Open models run on your infrastructure; closed models are APIs. For most teams, an open model on rented GPU or a managed embedding API (like Cohere or Voyage) is the middle ground: no self-hosting complexity, no vendor lock-in to a single closed model. Store vectors alongside documents in your vector database or in a secondary index.
Vector indexing. Raw k-nearest-neighbor search is linear in collection size. At scale, you need an index. Standard options include HNSW (Hierarchical Navigable Small World), IVF (Inverted File), and PQ (Product Quantization). HNSW is the default for most production systems: it trades a small amount of recall (you might miss 5-10 percent of true nearest neighbors) for much faster query time. A vector database like Postgres with pgvector, Weaviate, Qdrant, or Pinecone handles this indexing for you. Choosing between them is mostly about operational burden and feature set, not search quality. Postgres is simplest if you're already using it; managed services like Pinecone reduce operational overhead but lock you into their infrastructure.
Hybrid retrieval. Pure semantic search often ranks off-topic results if they're vector-close to the query. A search for "python programming" might rank content about snakes. Hybrid retrieval merges keyword and semantic results before final ranking, using both. The typical pattern:
Run the query against your keyword index (Elasticsearch, Postgres full-text, or similar) and retrieve the top 100 documents.
Encode the query into an embedding and run kNN search on your vector index, retrieving the top 100 documents.
Merge both result sets, removing duplicates.
Re-rank the merged set using a learned-to-rank model (LTR) or hand-tuned scoring function that weights semantic similarity, keyword match, freshness, and other signals.
Return the top k results.
Hybrid retrieval adds latency (you're making two queries instead of one), but it recovers precision without sacrificing recall. In practice, hybrid systems outperform pure semantic search on most real-world queries. The keyword index catches exact-match queries and filters out off-topic results. The semantic index handles intent and synonyms. The merge step ensures both methods contribute.
Managing freshness and index updates
Photo by Caio on Pexels.
Keyword indexes handle updates easily: add or remove a document from the inverted index, re-index a few postings lists, done. Vector indexes are slower to update. Recomputing an embedding for a changed document is cheap (milliseconds per vector), but rebuilding the HNSW index structure is expensive. Many vector databases support incremental updates (delete old vector, insert new one), but the index can become suboptimal over time, requiring a full reindex every few days or weeks.
In practice, most teams batch re-embedding. If a document changes, you queue it for the next nightly batch job. New documents are indexed within 24 hours. For higher-frequency updates, some options:
Real-time embedding in write path. When a document is created or updated, embed it immediately and insert into the vector database. This works if your write volume is moderate (thousands of documents per day). If you're indexing millions of new documents daily, batch is faster and cheaper.
Index versioning. Maintain two vector indexes: the current one (read-only) and a staging one (being built). Queries read from current. Once staging is ready, swap. This avoids downtime during reindexing, but doubles storage temporarily.
Acceptable staleness. Be explicit about how old your vectors can be. If you're indexing news, 1 hour old might be acceptable. For e-commerce inventory, 30 minutes. For documentation, 24 hours is fine. This lets you batch updates at scale without falling into a real-time trap.
Hybrid coverage. Use hybrid retrieval to mitigate embedding staleness. If a document was updated but the vector is still old, the keyword search catches it. This is a pragmatic trade-off: lose some semantic relevance but gain freshness.
Choosing and evaluating embedding models
Embedding quality directly affects search relevance. A weak model creates a weak index. Fortunately, the gap between a good model and a great model is now small, and good models are free or cheap.
Benchmark your candidate models on your own queries and data, not just public leaderboards. BEIR is a standard benchmark for semantic search, but it's generic. If you're searching e-commerce products, domain-specific benchmarks (like those from LLM4eCommerce) correlate better with real-world performance. Start with BAAI/bge-base-en-v1.5 (384 dims, open source, strong on many domains) or Sentence Transformers MPNet (768 dims, also open). For closed models, OpenAI's text-embedding-3-small is cost-effective; text-embedding-3-large is stronger but slower. Voyage AI's models (voyage-2, voyage-3) are competitive alternatives.
Test on a holdout set of 200-1000 real queries from your application. Encode them and your documents with each candidate model. Measure retrieval@10 (does the first relevant document appear in the top 10?), MRR (mean reciprocal rank), and NDCG (normalized discounted cumulative gain). Run an offline evaluation: "Does model A rank the ground-truth relevant result higher than model B?" Then run a live A/B test with real users to measure engagement (click-through rate, dwell time, conversion). A model that wins offline might lose online if it ranks unpopular results, or if its speed trade-off increases bounce rate.
Once chosen, commit to a version. Embedding models change between releases. If you upgrade from bge-base-1.5 to 2.0, old vectors become incompatible with the new model. You'll need to recompute vectors for the entire collection. This is expensive at scale. Plan quarterly or semi-annual upgrades, not continuous ones, unless you have infrastructure to re-embed on a schedule.
Common pitfalls and when semantic search fails
Semantic search is not magic. It will fail in ways keyword search doesn't, and vice versa.
Off-topic results due to semantic similarity. A search for "apple fruit" might rank results about Apple Inc. if the vectors are close. A query about "python" might rank snake content. Embeddings capture statistical similarity in the training data, not true understanding. Mitigation: use hybrid retrieval with keyword filters, add metadata fields to re-rank by category or domain, or fine-tune the embedding model on your data so it learns your domain-specific semantics.
Typos and misspellings. Semantic search handles some typos (the embedding of "laptpo" might still be close to "laptop"). But don't rely on it. Add a spell-check or fuzzy-match layer in your keyword search pipeline. Hybrid search makes this less critical, but it's still needed for good UX.
Embedding costs at scale. Encoding 100 million documents costs money. If you use a closed API (OpenAI, Cohere), expect hundreds or thousands of dollars. If you self-host, GPU costs mount quickly. Calculate total cost of ownership: embedding compute, storage, query latency, and engineering time. For some use cases (internal docs, small catalogs), the cost is easy to justify. For others (high-volume e-commerce), it might exceed the lift in relevance. Measure ROI, not just relevance.
Latency and P95. kNN search is slower than keyword search. A Postgres full-text query might take 10ms. A vector similarity search might take 100-200ms. This matters on latency-sensitive applications (search autocomplete, instant results). HNSW indexing helps, but doesn't eliminate the gap. Some teams use semantic search for initial ranking and keyword search for re-ranking, or limit semantic search to certain query types (long-form queries, not short keywords).
Stale embeddings. If you batch re-embed daily but serve traffic 24/7, up to 24 hours of documents might have stale vectors. For time-sensitive content (news, events), this is a problem. Hybrid retrieval helps: the keyword index catches new documents. But you'll sacrifice some semantic relevance until the vectors refresh.
Cold start and relevance cliffs. When you first roll out semantic search, relevance is often worse than the keyword search it replaces, because the embedding model has never seen your data and hasn't been tuned for your domain. Expect a 2-4 week period of negative A/B results before relevance stabilizes. Some teams mitigate this by starting with a small holdout (5-10 percent of traffic) and shipping hybrid search, not pure semantic, to avoid a noticeable drop.
Model drift and benchmark gaming. As you fine-tune or switch embedding models, offline metrics and online metrics can diverge. A model that ranks "relevant" documents first offline might rank unclicked results online. Build a continuous feedback loop: measure offline NDCG weekly, but weight online CTR and engagement more heavily. Don't optimize for offline metrics in isolation.
Measuring search quality in production
You have semantic search built and deployed. How do you know it's actually helping?
Offline metrics (MRR, NDCG, retrieval@10) are necessary but not sufficient. A model that ranks the "right" document first but is ignored by users is a false positive. Build a labeling pipeline: sample 500-2000 real queries monthly, have humans label the top-3 results as "relevant," "somewhat relevant," or "not relevant." Compute NDCG@3 and NDCG@10. Track it weekly as a KPI. This grounds relevance in your actual data and intentions.
Online metrics are the truth:
Click-through rate (CTR). Percentage of searches with at least one click. Higher is better, but plateaus quickly.
Clicks per search (CPS). Average number of results clicked per query. Higher means results are useful or users are browsing.
Dwell time. Time spent on a clicked result. If users click a result and bounce in 2 seconds, it wasn't useful. If they stay 30+ seconds, it was. Measure median dwell time by result position.
Query success rate. Percentage of queries that led to a meaningful outcome (purchase, document view, etc.). Domain-specific.
Zero-result rate. Percentage of queries that returned no results. Should be under 5 percent for most applications.
A/B test continuously. Split traffic 50/50 between keyword search and hybrid semantic + keyword search. Run each test for at least 1-2 weeks to stabilize metrics. Measure CTR, dwell time, and conversion as primary metrics. Offline relevance is the secondary check: if online metrics are equal but offline metrics improve, you're likely capturing benefits that take time to show up online.
Track cost per query. If semantic search increases your embedding and vector database costs by 30 percent but increases conversion by 2 percent, the trade-off might be worth it. If it increases cost by 50 percent for no gain, abandon it. Don't optimize for pure relevance without considering operational expense.
Getting started: a practical roadmap
If you're replacing keyword search with semantic search, here's a sequence:
Evaluate models offline on your data. Pick 5 candidate models. Encode a sample of 500 queries and documents. Measure retrieval@10 and NDCG@3. Pick the top 2 models.
Set up a vector database. If you're already using Postgres, add pgvector. If you're starting fresh, try Qdrant (self-hosted) or Pinecone (managed). Load your document collection and generate embeddings (batch or streaming).
Implement hybrid retrieval. Don't replace keyword search. Run both in parallel. Merge and re-rank using a simple formula (0.6 * semantic_score + 0.4 * keyword_score) or a learned-to-rank model. Test on a small holdout (10-20 percent of traffic).
Establish an evaluation baseline. Label 500 queries (or pull relevance data from search logs). Compute offline NDCG@3 for your hybrid system. This is your baseline.
Run an A/B test. Ship hybrid search to 50 percent of users. Measure CTR, dwell time, and conversion for 2 weeks. If metrics improve, roll out to 100 percent. If they degrade, revert and adjust weights or add filters.
Iterate on freshness and cost. Once live, measure embedding staleness (how often vectors lag behind documents). If it's a problem, move to real-time embedding or reduce batch interval. Monitor vector database costs. If they're too high, consider a smaller embedding model or sampling strategy.
Semantic search is not a drop-in replacement for keyword search. It's a complementary technology that works best in hybrid form. Start small, measure rigorously, and iterate. Most teams see relevance improve 5-15 percent after 4-8 weeks, with an operational cost of 20-40 percent higher than keyword search alone. Whether that trade-off is worth it depends on your domain, traffic, and margin. Build the measurement infrastructure first, then commit to a long-term evaluation plan. Semantic search pays off on platforms where relevance directly drives engagement or revenue. On smaller collections or lower-value search, it might not justify the complexity.
This article was originally published on AI Glimpse.
Top comments (0)