Is your new search actually better? Label gaps, LLM judge panels, and paired statistics for offline search evaluation

Every search team eventually ships a ranking change on the strength of a table where the new system's nDCG@10 is a little higher than the old one's. Then it goes to production and nothing measurable happens.

Usually nothing went wrong in the code. The evaluation was answering a different question from the one people thought they were asking. I built llm-search-eval to make the evaluation answer the right one: is the new system better for shoppers, and how sure are we? This post is about the design decisions behind it, in the order the problems show up.

All numbers below come from the small bilingual electronics example that ships in the repository (10 queries, 22 products, 30 gold labels). It is far too small to make product decisions with, which is exactly why it is useful for showing where a naive evaluation goes wrong.

Problem 1: labels run out, and the gap is not neutral

Offline evaluation needs relevance labels: for each (query, product) pair, how relevant is it? Teams label a set once and reuse it. A retriever built later surfaces products that were never labeled.

The common default is to count an unlabeled result as not relevant. That sounds conservative, but it is not neutral. The system that is more different from whatever produced the labeled pool gets more unlabeled results and is penalized more. A new dense retriever is often the one being compared, and it is often the one that finds products the old pool never saw.

So the first thing llm-search-eval reports is coverage. Here is the example run with gold labels only and no judge:

llm-search-eval run \
  -q examples/electronics/queries.jsonl \
  -p examples/electronics/products.jsonl \
  --qrels examples/electronics/qrels.csv \
  -s bm25=examples/electronics/runs/bm25.jsonl \
  -s hybrid=examples/electronics/runs/hybrid.trec \
  -o reports/

## Label coverage

| System | Queries answered | Labeled in top 10 | Unknown products |
| :--- | ---: | ---: | ---: |
| `bm25` | 9/10 | 48.3% | 0 |
| `hybrid` | 10/10 | 80.6% | 0 |

Gold labels used: 29, new judgments: 0, from cache: 0, failed: 0, still unlabeled: 18.

Less than half of BM25's top 10 has a label. Any number computed on that is partly a measurement of which system the labelers happened to look at. Whatever else you do, look at this table before the summary table.

Filling gaps without touching gold labels

The fix is to label the gaps, and an LLM judge is the cheapest way to do it at scale. The resolution order for every pair in each system's top depth is strict:

1. Gold label from --qrels, if present. Gold labels are never re-judged.
2. Cached judgment, if the same judge configuration already saw this exact query and product content.
3. Judge, if --judge is set. Otherwise the pair stays unlabeled, and the report shows the gap.

Then every system is scored against the same merged label pool. That detail matters: the ideal ranking used by nDCG and the recall denominator are identical for every system, so a judgment added because of system B also counts when scoring system A.

Finding the gaps is a short function:

def pending_pairs(dataset: Dataset, runs: Sequence[Run], depth: int) -> list[tuple[str, str]]:
    """``(query_id, product_id)`` pairs in the runs' top ``depth`` that have no gold label."""
    pending: dict[tuple[str, str], None] = {}  # ordered set
    for run in runs:
        for qid in dataset.queries:
            for pid in run.ranking(qid, depth):
                if (qid, pid) not in pending and dataset.gold_label(qid, pid) is None:
                    pending[(qid, pid)] = None
    return list(pending)

Judges grade on a 0 to 3 scale (3 exact match, 2 relevant, 1 partially relevant, 0 irrelevant), the same scale as the gold labels, so the two can be compared directly.

Problem 2: one judge is one opinion

An LLM judge has its own biases. Some are lenient, some ignore attributes like size or color, some behave differently on Persian queries than on English ones. Swapping human labels for a single model's opinion just moves the bias somewhere you cannot see it.

llm-search-eval handles this in two ways.

Panels. Pass --judge more than once and the judges form a panel. Grades are combined with --aggregate (majority, mean, median, min, or max), and --min-votes sets how many members must answer, so one failing endpoint does not stop the run.

Agreement and calibration. The report measures how much the panel agrees with itself (Fleiss' kappa, Krippendorff's alpha, pairwise Cohen's kappa plain and quadratic-weighted) and, with --calibrate, how much each judge agrees with your gold labels. Gold labels still decide the scores; calibration pairs are judged only to measure the judge.

To show what that looks like without an API key, here is a panel of three mock judges, two of them with injected noise:

llm-search-eval run ... \
  --judge mock --judge "mock=noise=0.2,seed=1" --judge "mock=noise=0.2,seed=2" \
  --calibrate --bootstrap 2000 -o reports/

### Inter-judge agreement

| Statistic | Value |
| :--- | ---: |
| Items with every vote | 48 |
| Unanimous | 72.9% |
| Fleiss' kappa | 0.732 |
| Krippendorff's alpha (ordinal) | 0.873 |

### Agreement with gold labels

| Rater | n | Exact | Within one | Binary | Weighted kappa | Mean bias |
| :--- | ---: | ---: | ---: | ---: | ---: | ---: |
| `panel(majority: ...)` | 30 | 33.3% | 66.7% | 63.3% | 0.257 | -1.000 |

This output is a good illustration of why calibration exists. The three judges agree with each other quite well (Fleiss' kappa 0.732). They agree with the gold labels badly: 33.3% exact, weighted kappa 0.257, and a mean bias of -1.0, meaning they grade a full point harsher than the humans. The mock judge is deterministic term coverage, meant for tests and CI, not a quality signal, and the calibration table says so. High inter-judge agreement only tells you the judges share a bias. You need the gold comparison to know whether it is the right one.

With a real model the same flags apply:

llm-search-eval run ... --judge openai=gpt-4o-mini --cache .judge-cache.jsonl \
  --price gpt-4o-mini=0.15,0.60 --max-cost 2

The openai judge works with any OpenAI-compatible chat completions endpoint, including local servers such as Ollama or vLLM (--judge "openai=qwen2.5:14b,base_url=http://localhost:11434/v1").

Problem 3: a cache that lies about which prompt produced a label

A judgment cache is essential. Re-running an evaluation on every pull request should not pay for the same (query, product) pair twice. But a cache keyed on ids is subtly wrong: if a product description is edited, the old label is served for the new text.

So keys hash content, not ids:

def cache_key(fingerprint: str, query: Query, product: Product) -> str:
    material = "\x1f".join((fingerprint, query.text, query.lang or "", product.to_text()))
    return hashlib.sha256(material.encode("utf-8")).hexdigest()

The other half is the fingerprint, which identifies the judge. Until v0.4.0 it included the endpoint, model, temperature, and a hand-bumped prompt version. The flaw: if someone edits the prompt text and forgets to bump the version, every cached label from the old prompt keeps being served as if it came from the new one. Nothing fails. The numbers just quietly mix two different judges.

v0.4.0 adds a prompt hash computed from the text itself:

def prompt_hash(*templates: str) -> str:
    """First 12 hex characters of the SHA-256 of the templates, joined by a unit separator."""
    return hashlib.sha256("\x1f".join(templates).encode("utf-8")).hexdigest()[:12]

#: Hash of the rendered system prompt and the user prompt template.
PROMPT_HASH = prompt_hash(system_prompt(), USER_PROMPT_TEMPLATE)

and puts it in the fingerprint, and so in every cache key:

@property
def fingerprint(self) -> str:
    return (
        f"openai:{self.base_url}:{self.model}:t={self.temperature}"
        f":{PROMPT_VERSION}:{PROMPT_HASH}"
    )

Any edit to the prompt changes the hash, and a judgment made with a different prompt is never reused. The version (relevance-v1, pairwise-v1), hash, provider, and model are also written to report.json, the Markdown report, and the dashboard, so a result always says which prompt produced it. The trade-off is that upgrading to v0.4.0 re-judges old cache entries once. I chose a one-time cost over silent contamination.

Problem 4: averages hide noise

With labels fixed, the next mistake is reading a mean difference as a result. Ten or fifty queries is a small sample, and per-query nDCG varies a lot.

Paired randomization test

Every system is compared to a baseline with a two-sided paired sign-flip randomization test on the per-query values of the primary metric. Paired, because both systems ran on the same queries, and pairing removes the variation between easy and hard queries. Randomization, because it makes no normality assumption about nDCG, which is bounded and often skewed.

def paired_randomization_test(
    a: Sequence[float], b: Sequence[float], permutations: int = 10_000, seed: int = 13
) -> float | None:
    """Two-sided paired sign-flip randomization test on the mean difference."""
    if len(a) != len(b):
        raise ValueError("paired samples must have equal length")
    diffs = [x - y for x, y in zip(a, b, strict=True)]
    if not diffs or permutations <= 0:
        return None
    observed = abs(sum(diffs))
    if observed == 0:
        return 1.0
    rng = random.Random(seed)
    extreme = 0
    for _ in range(permutations):
        total = sum(d if rng.random() < 0.5 else -d for d in diffs)
        if abs(total) >= observed - 1e-12:
            extreme += 1
    return (extreme + 1) / (permutations + 1)

The idea: if the two systems were interchangeable, the sign of each per-query difference would be a coin flip. Flip signs at random many times and count how often the total is at least as extreme as what you observed. The +1 on both sides keeps the p-value from ever being exactly zero, which it should not be with a finite number of permutations. The seed is fixed so the same inputs give the same p-value.

Paired bootstrap intervals

A p-value says whether a difference is distinguishable from zero. It does not say how big it plausibly is. So every mean gets a percentile bootstrap interval over queries, and every delta gets a paired bootstrap interval, where queries are resampled as units so both systems see the same queries in each resample:

def paired_bootstrap_ci(
    a: Sequence[float],
    b: Sequence[float],
    *,
    confidence: float = DEFAULT_CONFIDENCE,
    resamples: int = DEFAULT_RESAMPLES,
    seed: int = 13,
) -> Interval | None:
    """Bootstrap interval for the mean paired difference ``a - b``.

    Queries are resampled as units, so both systems see the same queries in
    each resample. This is the interval that matches the paired randomization
    test used for p-values.
    """
    if len(a) != len(b):
        raise ValueError("paired samples must have equal length")
    return bootstrap_ci(
        [x - y for x, y in zip(a, b, strict=True)],
        confidence=confidence,
        resamples=resamples,
        seed=seed,
    )

On the example data, with the mock panel from above filling the gaps:

| System | Δ ndcg@10 | Δ 95% CI | Relative | Win / tie / loss | p-value |
| :--- | ---: | ---: | ---: | ---: | ---: |
| `hybrid` | +0.4470 | [+0.3401, +0.5862] | +84.3% | 10 / 0 / 0 | 0.0020 (significant) |

The hybrid system wins all 10 queries, so even 10 queries are enough. Note the p-value: with 10 queries and all differences positive, the smallest possible two-sided p-value is 2/1024, about 0.002. That is a property of the sample size, not of the systems, and it is worth knowing before you look for p < 0.001 on a small set.

The report also prints a line I think every evaluation tool should print: narrow intervals need more queries, not more judgments. Filling label gaps fixes bias. It does not fix noise. Only more queries do.

That is the honest answer to the question I get most often: "my new system is +2% on 50 queries, what now?" Check coverage first. If many top results are unjudged, fill them and re-run, because the +2% may be a labeling artifact. Then look at the paired interval. If it still crosses zero, you need more queries.

Pairwise judging as a second opinion

Pointwise grades can miss ordering differences when two systems are close. With --pairwise-judge, a judge sees the top --pairwise-k results of both systems as two lists and picks the one a shopper would prefer. Each query is judged in both orders by default, and a system wins only when both orders agree, which cancels position bias. Opposite verdicts are counted as inconsistent, a direct read on how stable the judge is. Results come with a Wilson interval on the win rate and an exact sign test (ties ignored).

Problem 5: cost surprises

Judging thousands of pairs with a large model adds up. Two features exist so that nobody learns the cost from an invoice.

Estimate before you run. llm-search-eval estimate takes the same arguments as run, works out which pairs still need a judge after gold labels and the cache, and predicts calls, tokens, and cost without calling any model:

$ llm-search-eval estimate -q queries.jsonl -p products.jsonl --qrels qrels.csv \
    -s bm25=runs/bm25.jsonl -s hybrid=runs/hybrid.trec \
    --judge openai=gpt-4o-mini --price gpt-4o-mini=0.15,0.60 --max-cost 0.01
queries: 10
result pairs: 47 (29 gold, 0 unknown products)
pairs to judge: 18 (0 already cached)
  gpt-4o-mini: 18 calls, 5624 in + 720 out tokens, $0.0013
total: 18 calls, about 6344 tokens, $0.0013 (assumes 40 output tokens per call)
budget: fits

Input tokens are counted from the real prompts that would be sent, using a rough rule (about four characters per token for Latin text and two for Persian), so the estimate is approximate. It exits with 1 when the budget is too small, which makes it usable as a CI pre-check.

Budgets during the run. --max-cost, --max-tokens, and --max-judgments stop judging when a limit is hit. The run still completes, the remaining pairs stay unlabeled, and the coverage table shows how many were skipped. Prices are never built in, because they change and differ between providers. A model without a price still counts tokens, and the total cost is reported as unknown rather than as a misleading partial number.

Putting it in CI

llm-search-eval gate turns a report into a pass or fail check:

llm-search-eval gate reports/report.json --min ndcg@10=0.6 --max-drop 0.01 --significant-only

--significant-only fails the build only when a drop is statistically significant, which stops noisy small regressions from blocking every pull request. The repository is also a GitHub Action that runs the evaluation, applies the gate, and keeps one pull request comment updated on every push.

Limitations

• LLM labels are still model opinions. Calibration tells you how far off they are on the pairs you have gold labels for. It cannot tell you about query types your gold set does not cover.
• The token estimate is a heuristic, not a tokenizer.
• Small query sets stay small. No amount of statistics turns 10 queries into a decision. The tool reports the uncertainty; it cannot remove it.
• Offline is not online. This measures relevance against labels. It does not replace an A/B test on conversion.

What's next

The roadmap in the README lists what I want to do next: pairwise results as a gate condition, active selection of the queries most worth labeling next (the direct answer to "you need more queries, but which ones?"), a Vespa connector with multi-page fetching for deep runs, and interleaving and click log metrics for online comparison.

If your retrieval is in Persian, I also wrote about the language-specific side of that problem in Why RAG breaks on Persian text (persian-rag).

Install

pip install llm-search-eval

It has zero runtime dependencies and supports Python 3.10 to 3.13. There is also a Docker image:

docker run --rm -v "$PWD:/work" sadrmanesh/llm-search-eval:0.4.0 run \
  -q eval/queries.jsonl -p eval/products.jsonl --qrels eval/qrels.csv \
  -s bm25=eval/runs/bm25.jsonl -s hybrid=eval/runs/hybrid.trec --judge mock -o reports

• Repo: https://github.com/sadrmanesh/llm-search-eval
• Docs: https://sadrmanesh.github.io/llm-search-eval/
• Sample dashboard: https://sadrmanesh.github.io/llm-search-eval/demo/report.html

MIT licensed. If you run it on a real catalog and the calibration table surprises you, I would like to hear about it.