The first time I swapped an embedding model in production, the answer quality on our internal eval set jumped by twelve points and the latency went down. I felt very smart for about a week. Then a customer success engineer asked why the assistant had stopped finding documents that contained exact product SKUs, and I spent a Saturday discovering that the new model, which was great at semantic similarity, had gotten worse at lexical matching. The old model carried enough surface-level signal to find the SKU. The new one had been trained out of that and pretended every SKU was a similar SKU. Recall on a specific class of query had collapsed, and our eval set had not covered that class.
That is the standard embedding-model story. The model that wins on benchmarks is not always the model that wins on your data, and the model that wins on your data is not always the model that keeps winning when the queries change shape next quarter. Embeddings are not a commodity. The choice of embedding model and the decision of whether to put a reranker behind it are two of the highest-leverage tuning operations in a retrieval pipeline, and most teams treat both as defaults. The defaults are not bad. They are also not what you ship past year one.
By 2026 the patterns for picking embedding models and adding rerankers have settled into a small set of choices that consistently outperform the defaults. None of them are exotic. All of them are about understanding what each layer does, what it cannot do, and where the failure modes hide. This post is what I would tell my past self after that Saturday.
What An Embedding Model Actually Encodes
The framing that helps most when picking an embedding model is to think about what the model was trained to optimize, because that is what its vectors will encode well. Models trained on web search query-document pairs are good at matching short queries to long documents. Models trained on natural language inference are good at semantic similarity between full sentences. Models trained on code are good at code-to-code or code-to-comment retrieval. Models trained on multilingual corpora are good at cross-language retrieval and often slightly worse at any single language than a dedicated monolingual model.
What this means in practice is that the right model for your corpus depends on what your queries and documents look like. A support knowledge base with short user queries and medium-length policy documents wants a model trained on query-document pairs. A semantic search across blog posts wants a model trained on long-form similarity. A code search wants a code-specific model. A multilingual product wants a multilingual model and accepts the small penalty in any single language. Defaulting to the highest-MTEB-scoring model regardless of corpus is how teams end up with embeddings that are good in general and mediocre on the specific shape of data they actually run.
The other thing the embedding encodes is what it does not encode. Most general-purpose embedding models are trained to be invariant to surface-level details that do not affect meaning. Word order, exact phrasing, specific identifiers, punctuation. That invariance is great for semantic search. It is terrible for any retrieval that depends on those exact details. SKUs, version numbers, function names, error codes. The model has been trained to compress these into a representation where similar identifiers are close to each other, which is exactly the wrong behavior when the user wants the specific identifier and not a similar one.
The fix is not always a different embedding model. The fix is often a hybrid retrieval pipeline that combines dense embeddings with a lexical signal. More on that below. But the framing matters: if you understand what the embedding encodes, you understand which queries it will fail on, and you can plan for those failures instead of being surprised by them in production.
The Embedding Model Choice In Three Tiers
The market in 2026 looks like three tiers, and most teams should pick from one of them based on their constraints.
The frontier tier is the proprietary embedding APIs from the major model providers. These are the models with the highest benchmark scores, the broadest training, and the steepest cost. They are the right default when you do not want to think about it, when latency is not critical, and when sending your data to an external API is acceptable. The capability is real. The trade is the per-token cost and the network round trip on every embed call.
The open-weights tier is the strong open models, the descendants of E5, BGE, GTE, Nomic, and the like. By 2026 these are good enough that the gap with the frontier API tier is small for most use cases, and they can be served on commodity GPUs at a fraction of the cost. The trade is that you now run inference: GPU bills, autoscaling, monitoring. For high-volume retrieval, this is almost always cheaper than the API after a few weeks. For low-volume systems, the operational cost is not worth it. The same calculus I covered in small language models in production applies here, because embedding models are exactly that: small models you can host yourself when the volume justifies it.
The specialized tier is models fine-tuned for a specific domain or task. Code embeddings, scientific paper embeddings, legal document embeddings, product search embeddings. These are not always better than the general models on benchmarks, but they are often better on the specific shape of data they were trained for. For domain-heavy products, this tier is worth the search cost. For general-purpose retrieval, it is not.
The pattern that has worked when I am unsure is to pick a strong open-weights model, run it on a representative eval set, and only escalate to the frontier tier if the open model leaves measurable quality on the table. Start cheap, measure, escalate only when measurement justifies it. The opposite pattern, starting on the frontier API and trying to descend later, almost always stalls because the team gets used to the latency and quality and the migration becomes a project.
Embedding Dimension And The Cost Curve
The other axis on which embedding models differ is dimension. Models output vectors of varying lengths: 384, 512, 768, 1024, 1536, sometimes higher. Higher dimensions can encode more information. They also cost more in storage, more in retrieval, and more in latency, and the cost scales linearly with the number of vectors in the index.
The trade-off is real and the right setting depends on corpus size. For small indexes, up to a few million vectors, dimension does not matter much. The storage and retrieval costs are rounding error, and the quality gain from higher dimensions is worth taking. For larger indexes, tens or hundreds of millions of vectors, dimension becomes a real cost line. Doubling the dimension doubles the storage and roughly doubles the retrieval cost. At those scales, the right move is often the lower-dimension variant of the same model family, accepting a small quality hit for a large cost reduction.
The pattern that has emerged in 2026 is Matryoshka embeddings, where the same model can produce vectors at multiple dimensions and the lower-dimension variant is a meaningful prefix of the higher-dimension one. This lets a single model serve both a fast, low-dimension index for the first retrieval pass and a slower, high-dimension representation for reranking. If your embedding model supports this, use it. If it does not, picking a fixed dimension that fits the corpus size is the right move. Avoid the trap of picking the highest dimension the model offers because it scored slightly higher on the benchmark. The benchmark did not run at your scale.
Hybrid Search Is Not Optional
Pure dense retrieval, where the only signal is embedding similarity, is the default in tutorials and the wrong default in production. By 2026 the consensus pattern is hybrid search: combine dense retrieval with a lexical signal, usually BM25 or its variants, and merge the results. Teams that do this consistently see measurable lifts on real-world queries. Teams that skip it consistently rediscover this lesson when their assistant fails to find the document containing the exact phrase the user typed.
The reason hybrid works is that dense embeddings and lexical search fail in opposite ways. Dense embeddings handle paraphrases, synonyms, and semantic similarity. They miss exact-match queries with rare terms. Lexical search handles exact matches and rare terms. It misses paraphrases. The two signals together cover both failure modes, and the resulting retrieval is more robust than either alone.
The pattern that has worked is to run both retrievers in parallel, take the top-k from each, and merge with a reciprocal rank fusion or a weighted score combination. The simplest weighting is to give each retriever equal weight and fuse by reciprocal rank, which produces solid results without any tuning. The tuned version weights the two signals based on the query type, but the simple version is good enough for most production systems and avoids the complexity of dynamic weighting.
The implementation cost is low. Most modern vector stores support a sparse index alongside the dense one, and the additional storage for the sparse index is small. The latency cost is also low, because the two retrievals run in parallel and the merge is a few milliseconds. The quality lift is real and shows up most clearly on the queries that pure dense retrieval was secretly failing on. If your retrieval pipeline is dense-only, adding a sparse component is the highest-leverage change available, and it is usually a half-day project.
What A Reranker Does, And Why You Probably Need One
A reranker is a model that runs on the top results from the initial retriever and reorders them by relevance to the query. The initial retriever, dense or hybrid, optimizes for recall: getting the right candidates into the top-k. The reranker optimizes for precision: making sure the most relevant candidates are at the top of that list, where the LLM will see them.
The reason rerankers exist is that the initial retriever is doing fast similarity matching against a vector index, and that matching is approximate. A bi-encoder embedding model produces one vector per document and one vector per query, then computes similarity. It is fast and scales to billions of documents. It is also limited, because the document and the query are encoded independently, without the model ever seeing them together. A cross-encoder, which is what most rerankers are, takes the query and a candidate document as a single input and produces a relevance score that takes both into account. It is much slower, because it has to run for each candidate. It is also much more accurate, because the model can attend to specific overlaps and interactions between query and document.
The production pattern is to use the bi-encoder for the first pass, retrieve the top 50 to 200 candidates, and run the cross-encoder reranker on that smaller set to pick the top 5 to 10 that go to the LLM. The bi-encoder handles the scaling problem. The cross-encoder handles the quality problem. Together they get you both, with a latency cost in the tens to low hundreds of milliseconds for typical reranker sizes.
The teams that ship without a reranker usually do so because the demo looked fine and the additional latency felt unnecessary. The teams that add a reranker after the fact almost always see a measurable lift in answer quality, especially on harder queries where the initial retrieval put the right document at rank 5 instead of rank 1. The LLM cannot prioritize a document the retrieval pipeline ranked low, and a reranker is the cheapest way to fix that ordering.
Picking A Reranker
Rerankers come in roughly the same three tiers as embedding models. Frontier APIs from major providers, open-weights cross-encoders, and specialized variants. The cost calculus is similar but the latency story is different. Reranking adds latency on every query, which means it sits in the user-perceived path. The choice of reranker is a tighter trade-off than the choice of embedding model, because embedding latency is paid once at indexing time while reranking latency is paid on every query.
The frontier rerankers are accurate and add real latency. They are the right choice for high-stakes retrieval where the latency budget can absorb a few hundred milliseconds. The open-weights rerankers are nearly as accurate and faster, especially when self-hosted on a GPU close to the application. They are the right choice for most production systems, particularly chat applications where the user is waiting on the response.
The other lever is reranker size. The same family often comes in multiple sizes, and the small variants are dramatically faster than the large ones with a small quality penalty. For most production systems, the small variant is the right starting point, and the upgrade to a larger variant happens only if the quality measurements justify it. The latency budget is real, and a 50-millisecond reranker that is 95 percent as good as a 250-millisecond reranker is the better production choice nine times out of ten.
The pattern that has worked when I am picking a reranker is to evaluate three to five candidates on the same eval set used for the embedding model, look at both the quality lift and the p95 latency, and pick the one that maximizes the quality-per-millisecond. The candidate list is small, the eval is fast, and the answer is almost always clearer than it looks before you measure.
Cost And Latency Budgets
A pipeline with hybrid retrieval and reranking has more moving parts than a pure dense pipeline, and each part has its own cost and latency profile. The discipline is to be honest about the budget at each stage and to allocate it intentionally.
The dense retrieval is the cheapest and fastest stage. It runs in milliseconds against a vector index, and the cost is dominated by the storage of the vectors themselves. The sparse retrieval is similarly cheap, with the storage cost of an inverted index that scales with the number of unique tokens in the corpus. Both run in parallel and contribute milliseconds to the latency budget.
The reranker is the expensive stage. A cross-encoder running on 50 candidates is a meaningful chunk of latency, and on 200 candidates it can dominate. The lever is the candidate count: rerank fewer candidates and the latency drops linearly. The right candidate count is the smallest one that still surfaces the correct document into the top-k after reranking, which is something the eval set can tell you. Most production systems land somewhere between 30 and 100 candidates, and the variance below that range is small.
The LLM call is the slowest and most expensive stage by far, and the retrieval pipeline's job is to keep its input small and relevant. A retrieval that returns five precise chunks lets the LLM run on a small input and produce a fast, focused answer. A retrieval that returns twenty mediocre chunks forces the LLM to read more, costs more in tokens, and dilutes the answer. Investing in retrieval quality is the same as investing in LLM cost reduction, and the LLM cost optimization story I covered earlier is downstream of how good the retrieval is.
Multilingual, Multimodal, And The Rest Of The Long Tail
Most embedding models are trained primarily on English. If your corpus or your queries are in other languages, you need a multilingual model, and you need to be honest about the quality trade. Multilingual models are usually slightly worse at any single language than a dedicated monolingual model, and the gap shrinks every year but does not close. For a single-language product, monolingual is the right choice. For a multilingual product, multilingual is the right choice, and the small quality gap is the price of language coverage.
Multimodal embeddings, where the model encodes both text and images into the same vector space, have matured to the point where they are useful in production for image-text retrieval and visual search. The trade-off is that a model trained on text-image pairs is usually worse at pure text-text retrieval than a dedicated text model. For products where images are central, multimodal embeddings are the right choice. For products where images are incidental, the right move is often two separate indexes, one for text and one for images, with the application deciding which to query based on the input.
The long tail of edge cases is the part where evals matter most. Numeric reasoning, chronological ordering, complex multi-clause queries, queries that mix exact matches with semantic intent. Each of these is a class where embedding-only retrieval can fail in ways that are not obvious until they show up in production. The defense is the eval set, again. Cover the long tail in your evals and the failures show up before the users find them.
How To Tune The Pipeline Without Breaking It
Embedding models and rerankers have a lot of knobs, and the temptation is to tune everything at once. The discipline is to tune one thing at a time, on a fixed eval set, with a measurement loop that takes minutes rather than days.
Start with the embedding model. Pick three candidates, run them on the eval, look at recall at the top-k that the reranker will see. Pick the best one and lock it in.
Move to the reranker. Pick two or three candidates, run them on the locked embedding model, look at the answer quality and the latency. Pick the one that maximizes quality within the latency budget.
Then tune the candidate count for reranking. Sweep from 20 to 200, plot quality versus latency, pick the knee of the curve. The knee is usually obvious. The temptation to rerank everything is rarely justified by the data.
Finally, tune the merge weights for hybrid retrieval, if you are running it. The default of equal weights with reciprocal rank fusion is usually within a percent or two of the optimum, and tuning past that is worth doing only if the gap shows up in evals.
The discipline that ties all of this together is the same one I covered for AI evals for solo developers, and it applies the same way here: build the eval first, run the eval on every change, trust the eval over your intuition. Retrieval is a place where intuition is consistently wrong, because the failure modes are subtle and the wins are often counter-intuitive.
What I Would Build From Scratch
If I were building a retrieval pipeline today, I would start with a strong open-weights embedding model in the bi-encoder tier, hybrid search combining dense and BM25 with reciprocal rank fusion, a small open-weights cross-encoder reranker on the top 50 candidates, and an eval set built from real user queries and corrected answers. The candidate count and the reranker size would be tuned by measurement. The frontier APIs would be in reserve for the case where the open stack hit a quality ceiling I could measure.
That stack is unglamorous. It is also the stack that production teams have converged on by 2026, because it works and because the trade-offs are honest. The interesting work in retrieval is no longer at the embedding model. It is at the chunker, where the unit of retrieval gets decided, and at the reranker, where the order gets fixed. The same chunking discipline I covered in RAG chunking strategies in production is the layer above this one, and the two layers together are most of what determines whether a RAG system is good or just demoable.
If your retrieval is producing the right kind of answer at the wrong rank, the fix is a reranker. If it is failing to find documents that contain the exact phrase the user typed, the fix is hybrid search. If it is finding the wrong documents entirely, the fix is the chunker or the embedding model, in that order. The patterns are mostly known. The work is in measuring carefully and resisting the urge to swap models when the actual problem is one layer up or one layer down.
The pipeline that ships in 2026 and still works in 2027 is the one with an eval set that grows when production surfaces a new failure class, a chunker that respects document structure, an embedding model picked on data and not on benchmarks, hybrid retrieval as a default, and a small fast reranker that earns its latency. None of that is novel. All of it is the thing that turns a retrieval demo into a retrieval product, and most teams are still one or two of these layers short of where they need to be.
Top comments (0)