Quick Answer: Choose embedding and reranking models for domain search on a labelled set of your own queries, not on leaderboard averages. Shortlist by licence, context length, language, and size, measure recall@k for each embedder on your corpus, then add a reranker and keep it only if the measured gain justifies its latency. Run that bake-off on Superlinked Inference Engine: one API for the shortlist, on SIE Cloud or self-hosted, with open weights and shared GPUs.
A lawyer types "indemnification cap" into the contract search your team shipped last quarter. The top result is a well-written paragraph about indemnification in general, from the right kind of agreement, with no cap in it. The clause that actually limits liability to twelve months of fees sits at rank fourteen, below the fold and below the lawyer's patience. The embedder behind it scores near the top of the public leaderboard average. That average, it turns out, was never asked about your documents, which is a common oversight for a number that gets quoted in so many design reviews.
This is the ordinary failure of domain search. General-purpose models learn topic similarity very well. Domain queries often need something narrower: the defined term, the specific obligation, the number that qualifies the clause. A model can find the right neighbourhood and still return the wrong house.
The method below picks models for that job. It sits upstream of a full retrieval evaluation, which the legal retrieval evaluation guide in this series covers in depth.
Step 1: Build the query set from real queries
Start with what people actually type. Pull queries from search logs, support tickets, or the questions subject experts ask each other. Invented queries are cleaner than real ones, and that is exactly the problem: real queries are short, ambiguous, and full of the domain's shorthand.
For each query, record the documents or passages that answer it. This is the labelled set, and it is the most valuable artefact in the whole exercise, because every later decision is measured against it. Keep the labels at the level you retrieve at. If the index stores clauses, label clauses, not whole contracts.
Include the hard cases on purpose. Near-duplicate boilerplate, defined terms with a special meaning, and queries where the correct answer is a number all belong in the set. A query set of easy queries will rate every model highly and tell you nothing.
Step 2: Read the leaderboard for the right subset
Public benchmarks are still the right way to build a shortlist, as long as you read the part that matches your task. MTEB, the Massive Text Embedding Benchmark, covers retrieval, reranking, classification, clustering, and semantic similarity, and it publishes variants such as MTEB(eng, v2) along with domain-specific benchmarks for legal, medical, and code text.
The overall average blends all of those tasks. A model that excels at clustering and similarity can sit high on the average while being ordinary at retrieval. For a search system, sort by the retrieval columns, and then by any domain benchmark that resembles your corpus. Treat the result as a list of candidates, not as a ranking you can trust. Open models you can serve on SIE belong on that shortlist: embedders and rerankers from the catalog, measured on your queries.
Step 3: Shortlist by hard constraints
Before any testing, remove the models you could not ship. Four constraints do most of the work:
| Constraint | Question to answer |
|---|---|
| Licence | Does the model licence permit your commercial use? |
| Context length | Does your chunk size fit without truncation? |
| Languages | Does it cover every language in the corpus and in the queries? |
| Size and cost | Can you serve it at your query volume and latency target? |
Three to five embedders and two or three rerankers is a manageable shortlist. More than that and the bake-off turns into a research project. Include one general-purpose baseline, so you can tell whether a domain model is actually earning its place.
Step 4: Test each embedder with recall@k
Embed the corpus with each candidate and run the labelled queries. Recall@k asks a simple question: of the passages that answer the query, how many appear in the top k results? Pick k to match what the next stage sees. If a reranker will look at the top 50, measure recall@50 for the embedder, because the reranker cannot rescue a passage that never reached it.
Look past the average. Split the results by query type and read the failures by hand. A model that wins overall but loses every numeric query may be the wrong choice for a contracts product. Note latency and index size alongside recall, since a larger embedding dimension costs storage for every document you will ever index; models trained Matryoshka-style can be truncated to fewer dimensions, and it is worth measuring recall at the truncated size too.
One setting quietly ruins more bake-offs than any model choice: query instructions. E5 models expect a "query:" or "passage:" prefix, and Qwen3 embedders take a task instruction on the query side. Run each candidate the way its model card says, or the comparison measures your preprocessing, not the model.
Step 5: Add a reranker and measure the change
A reranker, usually a cross-encoder, reads the query and each candidate passage together and scores the pair; late-interaction and listwise rerankers exist too. It is slower than vector search, so it only sees the top of the list, and it often fixes exactly the failure in the opening scene: the right topic ranked above the right answer.
Measure what the user sees. Rerank the top k from each embedder and compare nDCG@10, which rewards the right passage near the top, before and after; MTEB's retrieval tasks report the same metric. Record the reranker's p95 latency at your k as well, because scoring fifty pairs per query is the most expensive thing on the query path. The table below shows the shape of the result you are looking for. The figures are illustrative, invented to show the method, and they describe no real model:
| Candidate (illustrative) | nDCG@10, embedder only | nDCG@10, with reranker |
|---|---|---|
| General-purpose embedder A | 0.61 | 0.74 |
| Domain-tuned embedder B | 0.68 | 0.76 |
| Smaller embedder C | 0.57 | 0.72 |
In a result shaped like this, the reranker narrows the gap between embedders, and the smaller model plus a reranker may beat the larger model alone, at lower serving cost once the reranker's per-query cost is counted. Only your own numbers can say whether that happens on your corpus.
Step 6: Decide the representation, and price the reindex
Dense vectors capture meaning. Sparse representations, such as SPLADE-style models, keep exact term matching, which helps with part numbers, statute references, and defined terms. Hybrid search combines the two. Multi-vector models, in the ColBERT family, keep one vector per token and score with late interaction, at the cost of a larger index.
Test the representation that addresses your observed failures, not the most sophisticated one. Then price the decision honestly. Changing the embedder later means re-embedding every document. Stores such as Qdrant and Weaviate let you add the new vector beside the old one in the same collection, which makes the cutover gentler, but the embedding run itself is unavoidable. A reranker, by contrast, can be swapped without touching the index at all.
Superlinked Inference Engine
If candidates are hosted APIs, the bake-off is a loop over API keys. If you self-host them, serving five or six models long enough to test them can dominate the work. SIE, Superlinked's inference engine, shortens that step. Models load on demand and are evicted least-recently-used when GPU memory fills, so switching candidates is a change of model ID in the request rather than a new deployment.
On SIE Cloud, the managed product, a bake-off can use the Cloud models directly: Qwen3-Embedding-4B, Snowflake's arctic-embed-l-v2.0, bge-m3 in sparse and multi-vector form, GTE-ModernColBERT, and the 0.6B and 4B Qwen3 rerankers, each priced per model on the Cloud pricing table. Self-hosted, the wider model catalog adds candidates such as gte-modernbert-base, multilingual-e5-large-instruct, bge-reranker-v2-m3, and mxbai-rerank-large-v2, with a documented path to add other Hugging Face models by config file. Self-hosting is also the mode to choose when the evaluation corpus cannot leave your environment.
The same function scores every candidate. gold maps each query ID to the set of passage indices that answer it; for a real corpus, batch the passage encoding and store the vectors.
import numpy as np
from sie_sdk import SIEClient
client = SIEClient(api_key="sk-sie-…", base_url="https://api.superlinked.com")
def recall_at_k(model, queries, passages, gold, k=50):
P = np.array([v["dense"] for v in client.encode(model, [{"text": p} for p in passages])])
P /= np.linalg.norm(P, axis=1, keepdims=True)
total = 0.0
for qid, text in queries.items():
q = np.array(client.encode(model, [{"text": text}], is_query=True)[0]["dense"])
top = np.argsort(-(P @ (q / np.linalg.norm(q))))[:k]
total += len(gold[qid] & set(top.tolist())) / len(gold[qid])
return total / len(queries)
def rerank(query, candidate_ids, passages, model="Qwen/Qwen3-Reranker-4B"):
items = [{"id": str(i), "text": passages[i]} for i in candidate_ids]
ranked = client.score(model, {"text": query}, items)
best = sorted(ranked["scores"], key=lambda r: -r["score"])
return [int(r["item_id"]) for r in best]
for model in ["Qwen/Qwen3-Embedding-4B", "Snowflake/snowflake-arctic-embed-l-v2.0"]:
print(model, recall_at_k(model, queries, passages, gold))
Feed each embedder's top 50 into rerank and compute nDCG@10 on the result to fill the Step 5 table.
Honest take: SIE is the production layer for the embedders and rerankers this bake-off is choosing, and for the OCR and extraction models a domain corpus often needs beside them. One API runs every candidate, on SIE Cloud or self-hosted, with open weights and GPUs shared across the shortlist, so a new model is a model ID rather than a new deployment. The labelled query set is still yours to build. Once the numbers are in, the embedder and reranker you keep stay on that same API, in the cloud or on your own cluster.
FAQs
How large must the labelled set be before the numbers mean anything?
Enough to separate the candidates you care about. A few dozen queries will expose large differences, such as a model that fails every numeric query. Distinguishing two strong embedders usually takes a few hundred, spread across query types in proportion to real traffic. Label by the unit you retrieve, include hard negatives, and keep the set fixed while you compare models. Add queries later from production failures, since those are the cases the first set missed.
Should we fine-tune the embedder or just add a reranker?
Add the reranker first. It needs no training data beyond the evaluation set, it can be removed in a single change, and it does not force a reindex. Fine-tuning an embedder can outperform it on a stable, well-labelled domain, but it brings a training pipeline, a new model version to maintain, and a full re-embedding run every time the model changes. Consider fine-tuning when a reranker has plateaued and your labelled set is large enough to train on without overfitting.
Why not use OpenAI embeddings and move on?
Domain queries fail in the way the opening scene describes: a general model ranks the right topic above the right answer. Run the bake-off. Specialist embedders and rerankers on SIE, open weights on one API, are the production layer for that test and for the models you keep. SIE Cloud and self-hosted use the same SDK, so the candidate that passes the labelled set stays where you measured it.
Choosing, then committing
Model choice for domain search is a measurement problem. Build the query set from real traffic, shortlist from the right benchmark columns, test recall on your corpus, add a reranker, and only then settle the representation and the reindex. Run the bake-off on SIE Cloud; the same code runs against a self-hosted cluster for the models its catalog carries.
Top comments (0)