Retrieval quality on code is more sensitive to your corpus than to the model, and the difference between two reasonable models is usually smaller than the difference chunking makes. Measure on your own repository or you are choosing at random.
Why a leaderboard cannot answer this
Published code-retrieval scores are computed on public open-source corpora with docstring-derived queries. Two things about that make them a poor predictor for an internal codebase.
The queries are not your queries. In the standard construction, the query is the function’s own docstring, held out, and the target is the function it documented. That is a well-posed task, but a developer does not search with a docstring. They search with a symptom (“order stuck in pending”), a half-remembered name, or a description of a behaviour that no docstring in the repository uses the words for.
The corpus is not your corpus. Public code is heavy in the six languages of the classic benchmarks and light in the internal DSLs, config formats and heavily abbreviated domain vocabulary that fill a company codebase. A model that has never seen your abbreviations treats them as near-random tokens, and that effect dominates any two-point difference between models on a public set. The related asymmetry across programming languages is worked through in the cross-language index page.
Building a query set from your repo
A hundred queries is enough to separate models that differ meaningfully, and you can assemble them without a labelling project.
- Take fifty closed issues or pull request titles that describe a change to a specific piece of code. The title is the query; the file or function the PR actually touched is the ground truth. This is free, and it is the closest thing to a real query distribution you will get.
- Add thirty questions from your team’s chat history that begin “where do we” or “how does”. Resolve each answer by hand once.
- Add twenty deliberately hard cases: concepts implemented in more than one language, code with no comments, and a function whose name actively misleads. These are where models separate.
- Freeze the set and store it in the repository as a file. It has to be the same set next quarter or the comparison is meaningless.
What to measure, and at what size
With one relevant target per query, the two numbers worth reporting are Recall@10 — the fraction of queries whose target appears in the top ten — and MRR, the mean of 1/rank of the target. Recall@10 tells you whether the answer is on the first screen. MRR tells you how far down. Both are computed with three lines of code and no statistics library.
Report the query-set size next to every figure, because at n = 100 the resolution is coarse. A difference of two or three queries is well inside what re-running with a different chunker would produce, so treat anything under roughly five points of Recall@10 at that size as a tie and decide on other grounds. Also report the two figures per language and per query category; an average over a mixed set hides the case where a model is excellent on documented Python and useless on your Terraform.
Hold the chunker fixed while comparing models. It is the one confounder that will otherwise swallow the result — swapping function-level for file-level chunks moves Recall@10 far more than swapping models does, for the reasons in the granularity page. Run the chunking comparison separately, and run it first.
Properties that change the index design
- Context length. This is a hard constraint, not a preference: a model that accepts 512 tokens cannot embed a 200-line function, and most clients truncate rather than error. Voyage AI documents voyage-code-3 at a 32,000-token context and voyage-code-2 at 16,000, per its embeddings documentation. Check yours and check what your client library does when a chunk exceeds it.
- Dimension, and whether it is truncatable. Storage and memory scale linearly with dimension. Models trained with Matryoshka representation learning — voyage-code-3 documents outputs at 256, 512, 1024 and 2048 — let you truncate the vector after the fact and lose accuracy gradually rather than catastrophically. That turns dimension into a tuning knob instead of a commitment. See what dimension actually buys.
- Asymmetric query and document encoding. Several code models expect an input-type or prefix distinguishing a query from an indexed document. Forgetting it at query time silently degrades every search, and nothing errors.
- Hosted or local. A local model removes the per-token cost and the requirement to send source code to a third party, which for many organisations is the deciding constraint rather than accuracy. Running an embedding model locally covers what that costs in throughput.
Every model name, dimension, context length and price in this section is what the vendor documented at the time of writing and all of them move. Re-read the model card before you commit a repository-sized index to one.
Comparing two hosted embedding models honestly means running the same hundred-query set through two providers, which is two SDKs, two auth schemes, two batch-size limits and two different error shapes for the same rate-limit condition — before you have measured anything. A gateway such as Multigrid puts both behind one endpoint and one key, and attributes the token spend per model, which is convenient during a bake-off mainly because it makes the two runs differ only in the model string.
The cost of changing your mind
Changing embedding model invalidates every stored vector. There is no partial migration and no way to compare a vector from one model with a vector from another — the spaces are unrelated, and a cosine similarity computed across them is a meaningless number that will not look obviously wrong. Plan for a full re-embed and a period where two indexes exist side by side.
The good news is that the compute is rarely the obstacle: embedding a large monorepo costs less than most teams assume, and the real expense of a switch is the rebuild of the ANN index and the window during which search quality is unpredictable. Make the model id part of the chunk key from day one, as the architecture page sets out, and the migration becomes a cache miss on everything rather than a manual purge — slow, but not error-prone.
Top comments (0)