68 questions before a single token: eval-first RAG

Part 2 of the series. Part 1: why I stopped trusting generic LLMs for horticulture.

In Part 1 I promised the decision that changed everything. Here it is:

I don't touch the LLM until a fixed suite of domain questions passes retrieval.

Not "the demo looked good." Not "I asked it five things and it answered." A versioned file of questions with expected evidence, run as a regression suite — the same way you'd treat unit tests. Today that suite is 68 questions: 45 apple, 8 pear, 10 plum, and 5 for an HR-policy sandbox that proves the pipeline isn't hard-coded to orchards.

Why retrieval, not answers

A RAG pipeline fails in two places: the right passage never reaches the prompt, or the model mangles a passage that did. The first failure is cheap to detect and free to test — no API key, no tokens, no flaky LLM in the loop. So the default eval mode stops at retrieval:

{
  "crop_id": "apple",
  "question": "What are signs of scab?",
  "expect_contains": ["scab", "spot"],
  "expect_context": true,
  "expect_out_of_scope": false,
  "category": "disease"
}

The runner sends the question to the retrieval service (POST/rag/context) and checks that every expect_contains substring appears in the combined retrieved context. A few details that turned out to matter more than I expected:

• expect_contains_any for synonyms. The literature writes Marssonina; users write the colloquial disease name. Either counts as evidence.
• Light stemming. rootstock should match rootstocks. Without it you either overfit the expected strings to one article's phrasing or get false failures.
• expect_out_of_scope: true questions. A question about, say, car maintenance must return weak or no context. This catches the embarrassing failure mode where vector search happily returns "closest" chunks for any string whatsoever.

The metrics that survived

pass_rate alone saturates: once you hit 68/68, it can't tell you whether a refactor made ranking worse as long as the evidence still sneaks into position 8. So the runner also reports ranking metrics per suite:

• MRR (mean reciprocal rank) of the first relevant fragment,
• hit_rate@1 / @3 / @5 — did relevant evidence appear in the top-1/3/5 fragments.

A fragment counts as relevant when it contains at least one expected substring. That's a single-relevant proxy — the baselines don't carry ground-truth chunk ids — and I'm fine with it: it's cheap, stable, and moves in the right direction when I break something.

There's also a --full mode that does call the LLM and reports two more numbers:
verify_pass_rate (do the numbers in the answer actually appear in the retrieved context — more on that verifier in Part 5) and answer_contains_rate (does the answer mention the expected terms). Out-of-scope questions skip the LLM entirely, mirroring the production short-circuit: if retrieval finds nothing, we refuse before paying for tokens.

Making it cheap enough to actually run

An eval nobody runs is documentation. The full HTTP run over 68 questions takes about 4 minutes; that was too slow for "run after every change," so the runner grew flags:

Flag	Effect
--in-process	Import the retrieval module directly, skip HTTP
--fast	Disable the cross-encoder reranker (~15× faster; still 68/68 on the current set)
--workers 2	Parallel requests against one retrieval worker

The ~20-second --in-process --fast combination is what I run reflexively. The full run with reranking is for before releases and after reindexing. In CI, unit tests run on every PR and the complete eval is a manual GitHub Actions workflow — model downloads are too heavy to justify on every push.

Results land in eval/results/<timestamp>_<suite>.json, so "did Tuesday's chunking change hurt pear questions?" is a diff, not an argument.

What the suite caught (a sample)

• A chunking change that split experiment tables from their headers — apple pass_rate dropped 7 points, nothing else moved. Reverted in minutes.
• BM25 index not rebuilt after a corpus update — vector search masked it for common questions, but exact-code questions (rootstock "SK-4") failed instantly.
• A glossary entry that expanded a term too aggressively and pushed the right article out of the top-5 for two questions: visible as an MRR drop with pass_rate unchanged.

None of these would have been caught by "chat with the bot for a while." All of them would have shipped.

What I'd tell you to steal

1. Write the eval file before tuning retrieval. Even 20 questions change how you work.
2. Test retrieval separately from generation. It's the cheap 80%.
3. Add out-of-scope questions early. Refusing well is a feature.
4. Make the fast path under 30 seconds, or you'll stop running it.

Part 3 is the payoff: what it actually took to get those 68 questions passing — hybrid search, RRF, and why "just use a better embedding model" wasn't the answer.

Disclaimer: assistant output is informational; field decisions require local experts and compliant product labels.