Before you change a single ts_rank weight, freeze your corpus and score the current ranking against a set of graded judgments. Otherwise you are comparing two moving targets — new weights and new documents — and you will never know which one moved the needle. A relevance regression harness turns "search feels worse since Tuesday" into a number you can diff, the same way a test suite turns "the code feels broken" into a red check.
I wrote earlier about load-testing a Postgres full-text index and pinning down relevance. A commenter on that post sharpened the key point: if you tune weights against a live table, you are comparing two systems that are both changing at once. This post is the practical build-out of the fix — how to actually collect judgments, freeze the corpus, and compute the scores.
Why can't I just eyeball the search results?
Because search quality has no single-row assertion. A unit test says assert total == 42. Relevance says "for the query postgres connection pool, the pgbouncer guide should rank above the unrelated changelog entry, and the deep tuning post should sit somewhere in the top five." That is a judgment about an ordering, and orderings degrade quietly. You bump a title weight to fix one complaint, three other queries get subtly worse, and nobody notices until a user does.
The other trap is the moving corpus. Your articles table gains rows every hour. If you score today's ranking against today's table and last week's ranking against last week's table, any difference is contaminated by the documents themselves. You have to hold the corpus still to isolate the ranking change.
The takeaway: relevance is a property of an ordering over a fixed corpus, so you cannot measure it without freezing the corpus first.
How do I freeze the corpus so the comparison is fair?
Snapshot the exact columns your ranker reads into a dated table. This is your evaluation fixture — it never changes, so every ranking variant is scored against identical inputs.
-- One-time: capture the corpus as it was on this date.
CREATE TABLE eval_corpus_2026_08 AS
SELECT id, title, body, search_vector
FROM articles;
-- Run any candidate ranker against the frozen table, not the live one.
SELECT id
FROM eval_corpus_2026_08
WHERE search_vector @@ websearch_to_tsquery('english', :query)
ORDER BY ts_rank_cd(search_vector, websearch_to_tsquery('english', :query)) DESC
LIMIT 10;
When you want to test a new weighting, you only change the ORDER BY, never the FROM. That is the whole discipline: one variable moves per experiment. Keep the snapshot table (or a pg_dump of it) in the repo's fixtures or a durable bucket so the baseline is reproducible on any machine, including CI.
The takeaway: a dated snapshot table makes the corpus a constant, so a score change can only come from the ranking change you are actually testing.
How do I build the judgment set without judging every document?
You need a set of (query, document, grade) rows, where grade is something like 0 = irrelevant, 1 = marginal, 2 = relevant, 3 = perfect. Two practical rules keep this from becoming a research project.
Source queries from your logs, not your imagination. Pull the top queries by volume plus a handful of painful long-tail ones from support tickets. Thirty to fifty real queries beat a thousand invented ones, because they carry the distribution your users actually type.
Judge a pool, not the whole corpus. You cannot label every document for every query. Instead, run every ranker you plan to compare — the current one, and each candidate — take the top 10 from each, and union those into a pool. You only grade documents that appeared in some ranker's top results. This is standard pooling, and it avoids the bias of judging only what your current ranker happens to surface.
judged = {
("postgres connection pool", 101): 3,
("postgres connection pool", 102): 2,
("postgres connection pool", 205): 0,
("full text search stemming", 118): 3,
("full text search stemming", 140): 1,
}
def grades_for(query):
return {doc: g for (q, doc), g in judged.items() if q == query}
One warning: judgments go stale. When you add documents that clearly belong to an existing query, re-pool and grade them, or your metrics will silently punish the new-and-correct result as "unjudged, therefore zero." Treat the judgment set as code that needs maintenance, not a one-time artifact.
The takeaway: pool the top results from every candidate ranker and grade only that union — real queries, judged shallow but fair.
Which metric should I actually report?
Pick the metric that matches how your interface is used. Here is the short version.
| Metric | What it measures | Best when | Blind spot |
|---|---|---|---|
| precision@k | Fraction of top-k that are relevant | A grid/list where users scan k results | Ignores order within the top k |
| MRR | 1 / rank of the first relevant hit | "I feel lucky" — one right answer near the top | Ignores everything after the first hit |
| nDCG@k | Graded relevance discounted by position | Ranked lists where order and degree both matter | Needs graded labels, not just yes/no |
For a typical article or docs search, nDCG@k is the one to gate on, because it rewards putting a 3 above a 2 above a 1, and it discounts gains as you go down the page. Here is a correct, dependency-free implementation:
import math
def dcg(gains):
return sum((2**g - 1) / math.log2(i + 2) for i, g in enumerate(gains))
def ndcg(ranked_doc_ids, query, k=10):
judged = grades_for(query)
gains = [judged.get(doc, 0) for doc in ranked_doc_ids[:k]]
ideal = sorted(judged.values(), reverse=True)[:k]
idcg = dcg(ideal)
return dcg(gains) / idcg if idcg else 0.0
def reciprocal_rank(ranked_doc_ids, query, rel_threshold=2):
judged = grades_for(query)
for rank, doc in enumerate(ranked_doc_ids, start=1):
if judged.get(doc, 0) >= rel_threshold:
return 1.0 / rank
return 0.0
The 2**g - 1 gain and the log2(i + 2) positional discount are the standard nDCG formulation; the ideal DCG is computed from the best possible ordering of that query's judged grades, so the score lands in a clean 0–1 range. Average nDCG across all your queries and you have one number for the whole search experience.
The takeaway: report nDCG@k for ranked lists, and average it over your judged queries to get a single relevance score you can trend.
Wiring it into a regression check
Once you can score one ranking, comparing two is trivial. Capture the baseline once, then fail the check when a candidate drops below it by more than noise.
def mean_ndcg(ranker, queries, k=10):
return sum(ndcg(ranker(q), q, k) for q in queries) / len(queries)
def assert_no_regression(baseline, candidate, tolerance=0.01):
delta = candidate - baseline
if delta < -tolerance:
raise AssertionError(f"nDCG regressed by {-delta:.3f} (>{tolerance})")
return delta
ranker is any function that takes a query string and returns ranked document IDs from the frozen snapshot — so your current ts_rank_cd config and a candidate with different weights are just two ranker callables. Run this in CI on every change to the search query, the weights, or the text-search dictionary. A dictionary swap that quietly re-stems half your corpus will now show up as a red check instead of a support ticket three weeks later.
The takeaway: a baseline score plus a tolerance turns every ranking edit into a pass/fail gate, which is exactly what "grade before you tune" was asking for.
FAQ
How many queries do I need for a relevance test set?
Start with 30–50 real queries pulled from your search logs, weighted toward high-volume terms plus a few painful long-tail ones. That is enough to catch broad regressions; the averaged nDCG stabilizes well before you reach hundreds.
What is a good nDCG score for site search?
There is no universal pass mark — nDCG is only meaningful relative to your own baseline on your own judged set. Treat your current production ranking as the number to beat, and gate on "no worse than baseline minus a small tolerance" rather than any absolute target.
Do I need labeled data to test Postgres full-text search relevance?
Yes, but far less than you think. Pool the top 10 results from each ranking variant you want to compare, grade only that union on a 0–3 scale, and reuse those judgments across every experiment until the corpus meaningfully changes.
Bottom line
If you tune ts_rank weights by staring at live results, you are guessing. Freeze the corpus into a dated snapshot table, collect 30–50 real queries with pooled 0–3 judgments, and score every ranking variant with averaged nDCG@k against that fixture. Then a weight change, a dictionary swap, or a new stemming config becomes a measurable diff with a pass/fail gate — a repeatable regression check instead of a subjective complaint. Build the harness once and every future search change gets cheaper and safer.
Top comments (0)