Most enterprise RAG system design starts with the LLM decision. It should start with the embedding model decision.
When enterprises evaluate AI infrastructure, the conversation almost always centers on the LLM: which model, which provider, what capabilities, what cost per token.
The embedding model — which converts text into vector representations for semantic search — gets treated as a commodity choice. Pick one of the standard options, deploy it, move on.
This ordering is backwards. For enterprise RAG systems, the embedding model choice has more downstream impact on retrieval quality than the LLM choice. A great LLM on poor retrievals produces poor answers. A capable LLM on accurate retrievals produces accurate answers.
Here's the architectural reasoning.
What Embedding Models Actually Do (And Why It Matters)
When you index a document in a RAG pipeline, the embedding model converts each chunk of text into a high-dimensional vector — a mathematical representation of its meaning in the embedding model's semantic space.
When a user submits a query, that query is also converted to a vector using the same embedding model. Retrieval works by finding the document chunks whose vectors are closest to the query vector in that semantic space.
The quality of retrieval is therefore bounded by the quality of the embedding model's semantic representations. If the embedding model maps similar concepts to similar vectors accurately, retrieval will surface the right documents. If it doesn't, no amount of LLM capability will compensate — because the LLM never sees the documents it wasn't given.
The Five Dimensions That Actually Differentiate Embedding Models
1. Domain specificity
General-purpose embedding models are trained on broad web-scale text. They represent general English well and handle common topics accurately.
Enterprise data is not general. It contains domain-specific terminology, proprietary jargon, internal product names, technical specifications, and vocabulary patterns that general training data either doesn't include or includes with different semantic weight than your domain.
A legal firm's documents use "consideration," "material," and "party" in ways that differ from general usage. A biotech company's documents use terminology that appears rarely in general training data. A software company's internal documentation uses product names and technical terms that are either absent from general training data or present with different contextual meaning.
The practical consequence: for high-specificity domains, a general embedding model will produce retrievals that look plausible but miss the precise conceptual matches that matter. The failure mode is subtle — retrievals aren't obviously wrong, they're just not the best available.
2. Asymmetric vs. symmetric retrieval
Some embedding models are trained for symmetric similarity — finding texts that are similar to each other. Others are trained for asymmetric retrieval — finding documents that answer a question, where the query and the answer don't need to look similar at the surface level.
For enterprise knowledge retrieval, asymmetric retrieval is almost always what you want. A query "what is our refund policy" should retrieve the refund policy document even though the document doesn't contain the words "what is our." Symmetric models trained on text similarity will underperform on this task compared to models trained specifically for question-document retrieval.
3. Multilingual coverage
Enterprises operating across geographies often have document corpora in multiple languages. Embedding model performance varies significantly across languages — a model with strong English performance may perform substantially worse on French, German, or Japanese.
If your knowledge base is multilingual, evaluate retrieval quality across all represented languages, not just English. The headline benchmark numbers for most models reflect English performance.
4. Context length handling
Embedding models have maximum input lengths, typically measured in tokens. When a document chunk exceeds this limit, the model either truncates the input or handles it with pooling strategies that often degrade representation quality for longer passages.
For enterprise documents — contracts, technical specifications, research reports — chunk sizes that preserve useful context often exceed the maximum input lengths of standard embedding models. Verify that your embedding model handles your actual chunk size distribution, not just the theoretical case.
5. Embedding stability over model updates
If you update your embedding model — moving to a newer version or a different model — you must re-embed your entire document corpus. The new model's vector space is incompatible with the old model's vector space, and old embeddings will produce incorrect retrievals.
For an enterprise with a large document corpus, re-embedding can be a significant compute and time cost. More importantly, if you're using an external embedding API and the provider updates the model without notice, your retrieval quality can silently degrade without any obvious system error.
This is a strong argument for either pinning your embedding model version explicitly (if using an API) or running your own self-hosted embedding model where you control update timing.
The Case for Self-Hosted Embedding
The embedding model sees everything that goes into your vector store. Every document chunk you index passes through the embedding model's inference endpoint.
If you're using an external embedding API, every document in your knowledge base has been sent to a third party for vectorization. This is worth considering separately from the LLM inference question — even if you've made careful decisions about which data goes to your LLM API, the embedding model may be seeing the same data or more.
Self-hosted embedding models — running on your own infrastructure — eliminate this exposure and provide the additional benefits of version control and consistent behavior regardless of vendor-side changes.
The compute requirements for embedding models are substantially lower than for LLMs. Running a capable self-hosted embedding model (BGE-large, E5-mistral, or similar) requires hardware that most enterprises can provision without significant investment. The operational argument for self-hosted embedding is stronger than for self-hosted inference.
Practical Evaluation Approach
Rather than relying on published benchmarks, evaluate embedding models against your actual data:
Create a test set of 50-100 query-document pairs from your actual corpus. These should be real queries your system will receive and the documents that correctly answer them.
Compute retrieval recall at k (what percentage of the correct documents appear in the top-k retrievals) for each candidate model. Use k=5 and k=10 as the evaluation points.
Run this evaluation against at least three models: the general-purpose default you're currently using or considering, a domain-specific model if one exists for your domain, and an asymmetric retrieval-optimized model.
The results will be more informative than any published benchmark for your specific use case.
The Downstream Effect on LLM Spend
There's a cost argument here that doesn't get made enough.
If your retrieval recall at k=5 is 60%, your LLM is answering based on incomplete information 40% of the time. Improving retrieval recall to 85% doesn't just improve answer quality — it reduces the LLM tokens required, because accurate retrievals typically require fewer retrieved chunks to contain the relevant information.
Better embedding → more accurate retrieval → fewer tokens needed per query → lower LLM inference cost.
The embedding model investment pays back in reduced LLM spend at sufficient query volume. For high-volume enterprise deployments, the payback timeline is typically under six months.
Top comments (0)