You can load-test search latency with a script and a graph. Relevance has no such gauge by default, so most teams ship a new ranking rule, eyeball a handful of queries, and hope nothing important regressed. The fix is a small, boring relevance test suite: a fixed set of queries, human-judged expected results, and a metric you compute the same way every time — so "did this ranking change help?" becomes a number you can diff, not an argument you have in Slack.
This post is a build guide. By the end you'll have a judgments file, a scorer that outputs precision@k, MRR, and nDCG, and a before/after comparison you can wire into CI. The examples use Postgres full-text search, but the harness is engine-agnostic — Elasticsearch, Meilisearch, or a vector store all slot into the same shape.
Why can't I just load-test relevance the way I load-test latency?
Latency is a property of the system. Relevance is a property of the match between a query and what a human expected to see — and that judgment lives outside the database. A commenter on an earlier post about running Postgres search in production put it well: latency can be load-tested, but quality needs query sets, expected result buckets, bad-query examples, and a way to compare changes before shipping a new ranking rule. That's the whole job, and none of it comes for free with your index.
The trap is thinking a passing query proves relevance. SELECT ... WHERE tsv @@ query returning rows tells you the index matched. It says nothing about whether the right rows landed in the top 5, which is all a user ever sees.
The takeaway: relevance is measured against human judgments, not row counts — so the first artifact you build is the judgments, not the query.
Building the golden query set
Start with 20–50 real queries. Pull them from your search logs if you have them (the head terms plus a long tail of specific ones), or write them from real user intents if you don't. For each query, mark which documents should come back and how relevant each one is on a graded scale — not just yes/no. Graded judgments matter because "perfect match" and "acceptable match" should score differently.
Store it as flat JSON. This is the file a human curates and code never overwrites:
{
"queries": [
{
"q": "reset password email not arriving",
"judgments": { "doc_412": 3, "doc_88": 2, "doc_17": 1 }
},
{
"q": "export data to csv",
"judgments": { "doc_205": 3, "doc_9": 1 }
}
],
"negatives": [
{ "q": "asdfghjkl", "expect_empty": true },
{ "q": "the a of", "expect_empty": true }
]
}
The grade scale here is 0–3: 3 = exactly what the searcher wanted, 2 = relevant, 1 = loosely related, absent = irrelevant. The negatives block is the part teams skip: gibberish, stop-word-only queries, and known bad inputs that should return nothing (or a graceful empty state), not a pile of low-confidence matches.
The takeaway: a judgment file with graded relevance and explicit negative cases is the contract — everything downstream just measures against it.
Which metrics actually tell you something?
Three metrics cover most of what you need, and each answers a different question.
| Metric | Question it answers | Uses grades? | Good for |
|---|---|---|---|
| Precision@k | Of the top k results, how many are relevant? | No (binary) | "Is the first screen clean?" |
| MRR | How high is the first relevant result? | No (binary) | "Do users find one right answer fast?" |
| nDCG@k | Are the best results ranked above the merely-okay ones? | Yes (graded) | Comparing ranking rules |
Precision@k is the blunt instrument — easy to explain, good for a smoke test. MRR (Mean Reciprocal Rank) rewards getting a single correct answer to the top, which matches navigational queries. nDCG (Normalized Discounted Cumulative Gain) is the one to watch when you're tuning ranking, because it uses the graded scores and penalizes putting a 2 above a 3.
Here's a self-contained scorer. It takes the judgments and a search(query) -> [doc_id, ...] function and prints all three:
import math
def dcg(gains):
return sum(g / math.log2(i + 2) for i, g in enumerate(gains))
def ndcg_at_k(ranked_ids, judgments, k=10):
gains = [judgments.get(d, 0) for d in ranked_ids[:k]]
ideal = sorted(judgments.values(), reverse=True)[:k]
idcg = dcg(ideal)
return dcg(gains) / idcg if idcg > 0 else 0.0
def precision_at_k(ranked_ids, judgments, k=10):
hits = sum(1 for d in ranked_ids[:k] if judgments.get(d, 0) > 0)
return hits / k
def reciprocal_rank(ranked_ids, judgments):
for i, d in enumerate(ranked_ids):
if judgments.get(d, 0) > 0:
return 1 / (i + 1)
return 0.0
def evaluate(spec, search, k=10):
p, rr, ndcg = [], [], []
for item in spec["queries"]:
ranked = search(item["q"])
j = item["judgments"]
p.append(precision_at_k(ranked, j, k))
rr.append(reciprocal_rank(ranked, j))
ndcg.append(ndcg_at_k(ranked, j, k))
neg_fail = []
for n in spec.get("negatives", []):
if search(n["q"]): # expected empty, got results
neg_fail.append(n["q"])
mean = lambda xs: sum(xs) / len(xs) if xs else 0.0
return {
"precision@k": round(mean(p), 4),
"MRR": round(mean(rr), 4),
"nDCG@k": round(mean(ndcg), 4),
"negative_failures": neg_fail,
}
The search function is the only engine-specific part. For Postgres it wraps one query:
import psycopg
def make_pg_search(conn, rank_sql):
def search(q, limit=10):
sql = f"""
SELECT id
FROM articles
WHERE tsv @@ websearch_to_tsquery('english', %s)
ORDER BY {rank_sql} DESC
LIMIT %s
"""
with conn.cursor() as cur:
cur.execute(sql, (q, limit))
return [f"doc_{row[0]}" for row in cur.fetchall()]
return search
Swapping rank_sql between ts_rank(tsv, websearch_to_tsquery('english', %s)) and ts_rank_cd(...) — or adding a recency boost — is exactly the kind of "new ranking rule" you want to measure. Note websearch_to_tsquery is used on both the filter and the rank so the parsed query stays consistent; passing the raw string to ts_rank instead of the parsed tsquery is a common mistake that silently scores the wrong thing.
The takeaway: pick nDCG@k as your headline number for ranking work, and keep precision@k and MRR as cheap sanity checks.
How do I compare before and after without fooling myself?
Run the same suite against both ranking configs and diff the per-query scores. The aggregate can improve while individual important queries regress — a mean that goes up is not permission to ship.
def compare(spec, search_a, search_b, k=10):
print(f"{'query':<38} {'A':>7} {'B':>7} {'Δ':>7}")
regressions = 0
for item in spec["queries"]:
j = item["judgments"]
a = ndcg_at_k(search_a(item["q"]), j, k)
b = ndcg_at_k(search_b(item["q"]), j, k)
delta = b - a
if delta < -0.05:
regressions += 1
flag = " <-- REGRESSED" if delta < -0.05 else ""
print(f"{item['q'][:36]:<38} {a:>7.3f} {b:>7.3f} {delta:>+7.3f}{flag}")
return regressions
# In CI: fail the build if the new ranking regresses more than a threshold
regressions = compare(spec, baseline_search, candidate_search)
assert regressions <= 2, f"{regressions} queries regressed — review before shipping"
The per-query view is what makes this trustworthy. You'll routinely see a change that lifts the mean nDCG by 0.03 while quietly tanking three head queries that make up half your traffic. Weight the queries by real volume if you have it, and treat any regression on a top-traffic query as a blocker regardless of the average.
The takeaway: diff per-query, not just the mean, and gate CI on regressions rather than on the aggregate going up.
FAQ
How many queries do I need for a search relevance test set?
Start with 20–50 hand-judged queries covering your head terms plus a spread of long-tail intents. That is enough to catch obvious ranking regressions. Precision on the metrics improves with more queries, but the first 30 catch most real breakage — expand the set as you find failures it missed.
What is a good nDCG score for full-text search?
There is no universal target; nDCG is only meaningful relative to your own baseline. Measure your current ranking, then require any change to hold or beat that number without regressing individual high-traffic queries. A "good" score is one that doesn't drop when you ship.
How do I test search relevance without a machine-learning pipeline?
Write a JSON file of queries with human-judged relevant document IDs, run each query through your search function, and compute precision@k or nDCG against the judgments. It's a few dozen lines of code and needs no model — just a person deciding, once, which results are correct.
Bottom line
If you change ranking based on how three queries look in a browser tab, you're guessing. Build the judgments file first, score with nDCG@k as the headline and precision@k plus MRR as sanity checks, and always diff per-query before and after so a lifted average can't hide a regressed head term. Teams shipping frequent ranking or dictionary changes should wire the comparison into CI with a regression gate; teams that touch search once a quarter can run it by hand — but either way, the suite is what turns "feels better" into evidence.
Top comments (2)
Strong practical harness. One subtle evaluation trap is that “absent from judgments” is not always the same as irrelevant. A new ranker may surface a genuinely useful document the old pooling process never exposed, and the scorer assigns it zero.
I’d build the judgment pool from the union of top-k results from baseline, candidate, and one deliberately different retriever, then blind and randomize document order for assessors. Track inter-rater agreement and adjudicate only the high-impact disagreements. Version the judgments together with the corpus snapshot, analyzer/dictionary config, and query-set time window so a document or synonym change cannot masquerade as a ranking regression.
For the gate, segment before aggregating: navigational vs exploratory queries, head vs tail, empty/unsafe inputs, and freshness-sensitive intents. Report paired bootstrap intervals for per-query deltas; with 20–50 queries, a few judgments can move the mean noticeably.
That turns the JSON file into a durable evaluation artifact rather than a frozen reflection of whichever system produced the first candidate pool.
The pool-bias point is the one I keep relearning the hard way — a better ranker gets punished precisely because it's better, since it surfaces documents the old pooling never had a chance to judge. Your union-with-a-divergent-retriever setup is the right structural fix, and versioning judgments alongside the analyzer/dictionary config is what stops half these "regressions" from being config drift in disguise. One thing I'd add on top: track the unjudged rate within each system's top-k as a first-class metric, not just as pool hygiene. When a candidate's unjudged fraction spikes on a query, that's your signal to re-pool and re-judge before you trust the per-query delta at all — otherwise the bootstrap interval is honest about noise it can measure but silent about the holes it can't. How are you handling the incremental re-judging cost when the divergent retriever keeps dragging in fresh documents each cycle — batch it, or judge on demand at gate time?