In RAG, an embedding model turns text into a list of numbers (a vector) so similar meanings sit near each other. Your question becomes a vector; each document chunk is already a vector; search finds the closest chunks. Pick a weak or mismatched model and retrieval fails before the LLM ever answers.
This is a beginner guide to choosing one—not a deep dive into training theory.
What you are choosing
You need one embedding model for both:
- Indexing: embed every chunk when you build (or refresh) the store.
- Querying: embed the user question the same way at ask time.
If those two steps use different models, distances are meaningless. Treat "which embedding model?" as a single decision for the whole index.
Criteria that matter in practice
Quality on your data. Leaderboards help as a shortlist, not a final answer. Build 20–50 sample questions with the chunks that should be retrieved, and compare recall@k.
Language and domain. General English chat models may underperform on legal, medical, or non-English text. Prefer models trained (or evaluated) on languages you actually use.
Size and cost. Larger vectors and larger models often score better and cost more to store and embed. For many internal knowledge bases, a "small" API model is enough to start.
Latency. Embedding runs on every ingest batch and every user query. If p95 query time matters, measure embed + search together.
Max input length. Chunks must fit the model's context. If you use 800-token chunks, the model must accept them.
A simple comparison for beginners
| Option | Good for | Tradeoff |
|---|---|---|
Small hosted model (for example text-embedding-3-small) |
Getting started, lower cost | May lose edge cases vs larger models |
Large hosted model (for example text-embedding-3-large) |
Higher recall when eval shows a gap | More cost and storage |
| Open models (e5, bge, nomic, and similar) | Self-hosting, air-gapped, custom fine-tunes | You own serving, updates, and ops |
Beginner default: start with a well-supported small hosted model, measure on your test questions, and only move up if recall is clearly weak.
Dimensions and storage (keep it light)
A vector with 1,536 numbers uses more disk and memory than one with 384. Some APIs let you shorten dimensions for a speed/storage trade. If you shorten, keep the same setting for every chunk and every query, and re-test retrieval—do not mix full and shortened vectors in one index.
Switching models later
Changing the embedding model means re-embedding the whole corpus and replacing vectors in the index. Old vectors from model A cannot sit next to new vectors from model B. Plan for that cost before you "just try" a new model in production.
Practical pattern:
- Build a second index (or a new vector field) with the candidate model.
- Run the same eval set against both.
- Cut over only if metrics and spot-checks win.
- Delete or freeze the old vectors after cutover.
Checklist
- Same model (and settings) for documents and queries.
- Eval set of real questions before you commit.
- Language/domain fit checked.
- Chunk size under the model limit.
- Cost and latency acceptable at your expected QPS.
- A plan to re-embed if you switch later.
You do not need the "best" embedding model on a public leaderboard. You need one that retrieves the right chunks on your questions at a cost you can live with—and a habit of measuring before you change it.
Related reading
- Building a RAG Pipeline from Scratch: Embeddings, Vector Store, and Generation: how embeddings sit in the full RAG pipeline.
- Hybrid Search in RAG: Combining BM25 Keyword and Vector Retrieval (companion draft in this series): keyword search covers exact terms when embeddings alone miss them.
About the author
I write about Applied AI, MCP, Azure AI Search, and RAG. Follow me on Dev.to and Hashnode for more.
Top comments (0)