DEV Community

dubleCC
dubleCC

Posted on • Originally published at heycc.cn

How to Choose an Embedding Model in 2026 (RAG & Semantic Search)

#embeddings #rag #semanticsearch #mteb

Originally published at heycc.cn. This is a mirrored copy — the canonical version is kept up to date at the source.

How to Choose an Embedding Model in 2026 (RAG & Semantic Search)

The first thing most people do when picking an embedding model is open the MTEB leaderboard and copy whatever sits at row one. That is the wrong starting point. A leaderboard tells you how a model performs on average across someone else's tasks. It does not tell you whether it will retrieve the right paragraph for your support tickets, your legal clauses, or your code.

The real question has three parts: which model best serves your queries, at your scale, under your privacy constraints. Everything else is detail. Below are the six decision axes that actually move the needle, then a default ladder so you can stop reading and start shipping.

Decision flow for choosing an embedding model in 2026 across six axes leading to a default ladder

Axis 1: Retrieval quality on your data

The official MTEB leaderboard now ranks models by Borda rank across 131 tasks (the multilingual MMTEB suite spans 250+ languages). Borda count rewards a model that is broadly consistent across every task, not one that dominates a handful. That is why the top spot moves more than a raw mean-score column would suggest. NVIDIA's Llama-Embed-Nemotron-8B held the #1 Borda position at its November 2025 release with a mean task score of 69.46, ahead of Qwen3-Embedding-8B's higher mean (70.58) but different overall rank — a clean illustration of Borda-vs-mean divergence. That top spot had already changed hands again by early 2026: Tencent's KaLM-Embedding-Gemma3-12B-2511 leads as of this writing with a mean of 72.32. None of these numbers tell you how any of them handles your domain.

Even vendors admit this. Voyage's own benchmarks claim voyage-3-large beats OpenAI's text-embedding-3-large by 9.74% on average across the domains they tested — at a smaller default dimension. That is a real result on their evaluation set. It does not automatically transfer to your invoices or your Slack archive.

The blunt rule: before you commit, run a small eval on 30–100 of your own queries. Hand-label the correct document(s) for each query, embed your corpus with two or three candidate models, and measure recall@k and MRR. This takes an afternoon and routinely overturns the leaderboard order. (For how to build that harness, see our RAG evaluation guide.)

Axis 2: Dimensions vs storage and search cost

Vector dimension is a direct multiplier on index size and query latency. A 3072-dim float32 vector is 12 KB; a 256-dim vector is 1 KB. Across 10 million chunks that is the difference between ~120 GB and ~10 GB of index, which changes whether your index fits in RAM, and therefore your hosting bill.

The good news in 2026 is Matryoshka Representation Learning (MRL). MRL-trained models pack the most important information into the leading dimensions, so you can truncate the vector and keep most of the accuracy. OpenAI's v3 models expose this through a dimensions parameter; Gemini, nomic, Voyage, and Qwen3 do the same. Truncating Gemini from 3072 → 768, or nomic from 768 → 256, often costs only a point or two of recall while cutting storage 4×. Decide this with your vector database, since some indexes (e.g. binary-quantized HNSW) interact strongly with dimension. See our vector database comparison.

Axis 3: Max input tokens vs chunk size

Your model's max input length caps how big a chunk can be before it gets silently truncated, and truncation is a silent killer of retrieval quality.

  • 512 tokens (Cohere Embed v3, multilingual-e5-large-instruct): fine for sentence/paragraph chunks, painful for long documents. E5 truncates longer text rather than erroring.
  • 8,192 tokens (OpenAI v3, BAAI bge-m3, nomic-embed-text-v1.5): the comfortable default for most RAG; a full section fits in one vector.
  • 32k+ tokens (Voyage general-purpose models, Qwen3-Embedding): enough for whole documents, useful for code files or long contracts.
  • ~128k tokens (Cohere Embed v4) — document-level embedding without aggressive chunking.

Bigger is not automatically better: embedding a 30k-token chunk into one vector averages away fine detail, so a long-context model still benefits from sensible chunking. Match the limit to your real chunk size, not the maximum. Our RAG chunking guide covers how to pick that size.

The 2026 embedding-model lineup

Numbers below are defaults; most models are dimension-reducible via MRL. Prices are per 1M input tokens on the standard API (embeddings bill input tokens only).

Model Default dims Max input tokens Price /1M Open source Multilingual
OpenAI text-embedding-3-small 1,536 (reducible) 8,192 $0.02 No Yes
OpenAI text-embedding-3-large 3,072 (reducible) 8,192 $0.13 No Yes
Google gemini-embedding-001 3,072 (→1536/768) 2,048 $0.15 No 100+ langs
Voyage voyage-3.5 1,024 (256–2048) 32,000 $0.06 No Yes
Voyage voyage-3-large 1,024 (256–2048) 32,000 $0.18 No Yes
Cohere Embed v4 1,536 (256–1536) ~128k API No Yes (multimodal)
BAAI bge-m3 1,024 8,192 Self-host MIT 100+ langs
multilingual-e5-large-instruct 1,024 512 Self-host MIT 94 langs
nomic-embed-text-v1.5 768 (→512/256/128/64) 8,192 Self-host Apache-2.0 English-centric
Qwen3-Embedding-8B 4,096 (32–4096) 32,000 Self-host Apache-2.0 Strong MMTEB (was #1 mid-2025)

Voyage's current generation also includes a voyage-4 family (voyage-4-large $0.12, voyage-4 $0.06, voyage-4-lite $0.02 per 1M, each with a 200M-token free allotment). Treat exact figures as snapshots and confirm on the pricing page.

Axis 4: Cost at scale

For most teams, embedding cost is a one-time-ish bill (index your corpus) plus a small per-query cost. The 6.5× spread between text-embedding-3-small ($0.02) and gemini-embedding-001 ($0.15) only matters once you are embedding hundreds of millions of tokens. Two levers cut it hard: the Batch API (OpenAI, Google, and others give ~50% off non-real-time jobs — $0.065 for 3-large, $0.075 for Gemini), and self-hosting open models where the only cost is GPU time. A free allotment (Voyage's 200M tokens) covers many prototypes outright.

Axis 5: Open-source vs API

This axis is usually decided for you by privacy. If data cannot leave your network, an API is off the table and you self-host an MIT/Apache model: bge-m3 (1024d, 8k context, hybrid dense/sparse/multi-vector retrieval) is the reliable workhorse; Qwen3-Embedding-8B (Apache-2.0, 32k context, top-tier MMTEB) is the quality play if you have the GPU. If privacy is not binding, an API removes ops burden and gives you newer models for free.

Dimension API model Open-source model
Setup effort Minutes (API key) GPU + serving stack
Data residency Leaves your network Stays on-prem
Marginal cost Per token GPU time (amortized)
Upgrades Vendor ships them You re-index
Best when Speed to ship, low volume Privacy, high volume, control

Axis 6: Multilingual coverage

If your corpus or queries are not English-only, the default ladder shifts. gemini-embedding-001 (100+ languages, strong MMTEB multilingual performance), bge-m3 (100+ languages, open), and Qwen3-Embedding are the strongest multilingual picks. Watch the fine print: multilingual-e5-large-instruct covers 94 languages but caps at 512 tokens, and nomic is English-centric. Always include cross-lingual queries in your Axis 1 eval, because average multilingual scores hide large per-language gaps.

The default ladder

When you have no strong opinion, climb this ladder until something works:

  1. Start: OpenAI text-embedding-3-small — 1,536d, $0.02/1M, 8k context. Cheap, good, dimension-reducible.
  2. Need more quality / scale: Voyage voyage-3.5 or Google gemini-embedding-001.
  3. Privacy or on-prem required: self-host bge-m3 (safe default) or Qwen3-Embedding-8B (top quality).

Then validate the winner against your own 30–100 query eval (Axis 1). That single step beats any leaderboard.

Where these specs and prices come from

Dimensions, token limits, and prices above were verified against official docs and pricing pages on 2026-06-28, then re-verified 2026-07-16: OpenAI ($0.13 / $0.02 per 1M, 8,192 tokens, MRL dimensions param, unchanged), Google gemini-embedding-001 ($0.15/1M, 50% batch discount, 3072→1536/768, 2,048 tokens, 100+ languages, unchanged), Voyage (1024 default, 32k context; voyage-4 family now live with 200M free tokens, unchanged; the 9.74%-vs-OpenAI benchmark figure was corrected from an earlier "roughly 7–9%" rounding against Voyage's own blog post), Cohere Embed v4 (1,536d, ~128k tokens, multimodal, dims/tokens confirmed directly, though the $0.12/1M text price is corroborated only by third-party trackers, since cohere.com/pricing's serverless API rate did not render on re-fetch; confirm before budgeting). Note: OpenAI's own pricing page and model cards have at times shown minor discrepancies for embedding rates; confirm against the live pricing page before budgeting. MTEB/MMTEB figures are leaderboard snapshots that shift as models are added: Nemotron's #1 Borda position (69.46) was current at its November 2025 release but had already been overtaken by Tencent's KaLM-Embedding-Gemma3-12B-2511 (72.32) by early 2026; Qwen3-Embedding-8B's "top MMTEB" framing likewise dates to a June 2025 snapshot per its own model card. Treat all rankings and pricing as a point-in-time snapshot.

Sources

Top comments (0)