The least complex release gate is a fixed set of representative questions, relevance judgments for the expected source chunks, and a budget for how many chunks the system may index and retrieve. Do that before exposing a B2B SaaS RAG feature to customers.
TL;DR: retrieval evaluation matters because a generator cannot reliably repair missing evidence, while a relevance-only test can hide an index whose chunk count and query fan-out make it operationally unacceptable. Gate the release on both retrieval quality and resource use: test known questions, inspect misses, measure chunk growth, and set an explicit service-level objective before the first production page.
Imagine the page: indexed vector count has crossed its capacity threshold after a documentation import, query work has climbed with it, and the on-call engineer sees a system that is technically answering requests but consuming its planned headroom. The alert is late. The earlier signal should have been visible in pre-release evaluation: the same source document produced too many low-value chunks, retrieval returned redundant passages, or the correct passage consistently fell below the retrieval cutoff.
The original RAG architecture combines a parametric model with retrieved, non-parametric memory, so retrieval is part of the answer path rather than background plumbing [1]. Treating it as an untested preprocessing step is therefore a category error. Here are five checks I would put between ingestion and release.
1. Why does retrieval evaluation matter before shipping RAG?
A fluent answer is weak evidence that retrieval worked. The generator may produce plausible language even when the needed source passage was absent, and an end-to-end review can then credit the wrong subsystem. Retrieval needs its own test boundary: given a question, did the ranked results contain the judged-relevant chunk within the allowed depth?
It can look fine.
Start with questions that represent actual B2B SaaS documentation work: locating an account retention rule, finding the prerequisite for an administrative setting, or distinguishing two similarly named configuration options. For each question, record the source document and the smallest passage that contains the answer. The judgments should be reviewed when the corpus changes; stale labels are quieter than broken tests and just as misleading.
Use at least two views of the result. Recall at a chosen cutoff asks whether any relevant chunk appeared. Reciprocal rank makes an early hit worth more than a late one. Neither number proves answer correctness, but together they expose two distinct failures: missing evidence and buried evidence.
Keep the test honest. Questions copied verbatim from headings reward lexical overlap, while synthetic paraphrases alone may miss the terse language customers actually use. A mixed set is more defensible, with failures retained as regression cases instead of being edited away until the score looks clean.
2. Make chunking a testable change, not a preprocessing hunch
Chunk size, overlap, document boundaries, and metadata filtering alter both what can be found and how much must be stored. That is the operational hinge. A smaller chunk can isolate a precise fact, yet splitting related conditions across chunks can remove the context needed to interpret it; heavy overlap can restore context while duplicating indexed text.
So compare chunking policies against the same frozen judgments. Do not change the chunker, embedding process, cutoff, and question set in one experiment, because the resulting score cannot tell you which change helped. Version the corpus build inputs and keep an immutable evaluation manifest containing the document revision, chunking policy identifier, retrieval parameters, and judgment-set revision.
A compact evaluator is enough to make the contract executable. This Go example calculates recall at k and mean reciprocal rank from retrieved chunk IDs; it deliberately leaves retrieval behind an interface so the same test can exercise a local implementation or a managed service.
package retrievaleval
type Case struct {
Question string
Relevant map[string]bool
}
type Retriever interface {
Search(question string, k int) ([]string, error)
}
type Result struct {
RecallAtK float64
MRR float64
}
func Evaluate(r Retriever, cases []Case, k int) (Result, error) {
if len(cases) == 0 || k <= 0 {
return Result{}, nil
}
var hits, reciprocalRank float64
for _, c := range cases {
ids, err := r.Search(c.Question, k)
if err != nil {
return Result{}, err
}
for rank, id := range ids {
if c.Relevant[id] {
hits++
reciprocalRank += 1.0 / float64(rank+1)
break
}
}
}
n := float64(len(cases))
return Result{RecallAtK: hits / n, MRR: reciprocalRank / n}, nil
}
The short code is intentional. The hard work is the judgment set and failure review, not a large evaluation framework.
3. Put index growth beside relevance
For this workload, index cost at scale is a primary decision axis, so a retrieval report without ingestion volume is incomplete. Record source bytes, extracted text bytes, chunk count, indexed vector count, and the distribution of chunks per document for every build. Then forecast growth from the tenant and document volumes in the capacity plan rather than extrapolating from a friendly demo corpus. Consider a hypothetical release budget, clearly chosen rather than measured: the candidate chunker must not increase vectors per source document by more than 20% over the approved baseline unless it also clears a separately approved relevance improvement. The exact threshold belongs to the team's economics and risk tolerance. Its value is that it forces a conversation before a 5% relevance gain quietly doubles the index footprint. Long-tail documents deserve their own inspection because a median can stay flat while a few generated manuals or repeated navigation blocks produce thousands of near-duplicate chunks; track a high percentile and the maximum, then sample the documents responsible for both. The same discipline applies at query time. Log the requested cutoff, returned result count, filter selectivity, latency, and stable corpus version, but avoid logging document text or user questions by default because those fields may carry tenant data and are not required to calculate retrieval metrics. Join production aggregates to a corpus version, not to raw content. This longer record matters because capacity surprises rarely come from the tidy document at the center of the distribution; they arrive from the odd export, the repeated footer, or the tenant whose documentation structure does not resemble the test fixture.
One number will not save you.
4. Choose the evaluation boundary before choosing who operates it
Managed and self-hosted retrieval can both satisfy the same evaluation contract. The architectural decision changes who owns capacity, upgrades, failure recovery, and portability, but it should not change the release evidence demanded from the system.
| Decision area | Managed path | Self-hosted path | Evidence required either way |
|---|---|---|---|
| Capacity | Service limits and scaling behavior are external constraints | The platform team provisions and tests headroom | Forecast vector growth and verify it against a representative corpus |
| On-call load | Escalation crosses an organizational boundary | Diagnosis and recovery stay with the team | Define latency, error, and freshness objectives with named owners |
| Lock-in | Export and query semantics may constrain migration | Internal APIs can still create local lock-in | Keep judgments, chunk IDs, and evaluation inputs portable |
| Change control | Provider changes and client upgrades need qualification | Index and dependency upgrades need qualification | Run the same regression suite before promotion |
This is where skepticism pays. A feature matrix cannot tell an infrastructure lead how many pages the choice creates, and a benchmark detached from the team's corpus cannot settle the buy-versus-build question. Run the candidate behind the same Retriever contract, include ingestion and recovery work in the review, and reject any option whose operational boundary cannot be observed.
The method has limitations. Offline judgments age as documents and user vocabulary change, a small probe set underrepresents rare queries, and aggregate retrieval scores can hide one tenant's systematic misses. The trade-off is deliberate: a compact, reviewed gate gives fast release feedback, while sampled production evaluation and periodic relabeling cover drift at higher operational and review cost. Neither substitutes for the other.
The evaluation dataset itself should remain outside the retrieval implementation. That small separation preserves leverage: the team can compare architectures without translating its definition of relevance each time.
5. Turn the failed page into an earlier release signal
Work backward from the opening alert. If vector count crossed a capacity threshold, the pre-production build should have emitted vectors per document and projected index growth. If query work rose because results were redundant, the evaluation report should have shown unique source coverage within the top k. If the relevant passage sat below the cutoff, the ranked test should have failed before any answer-quality review began.
Instrumentation should therefore attach a corpus version and chunking-policy version to ingestion counts and retrieval summaries. A release candidate gets promoted only when it passes the agreed relevance floor, stays inside the index-growth budget, and has no unexplained regression hidden by the aggregate. The gate is a joint reliability and capacity decision. A higher relevance score does not automatically authorize an unplanned storage or on-call obligation.
Define the production objective in user terms, then derive alerts from it. For example, the team might choose to measure the share of an approved probe set whose relevant evidence appears within a fixed cutoff, plus ingestion freshness for newly published documents. Those are candidate service indicators, not universal targets. The target and evaluation window must come from the product's tolerance for stale or missing evidence.
Thresholds carry their own cost. Set them too loose and the page arrives after headroom has disappeared; set them too tight and routine corpus growth wakes the on-call engineer, trains the team to distrust alerts, and consumes the very operational capacity the control was meant to protect. Backtest a proposed threshold against several representative corpus builds, require sustained breach where appropriate, and route non-urgent budget drift to planning rather than paging.
False positives are real work.
That closes the trace: the page becomes a capacity forecast, the forecast becomes build instrumentation, and the instrumentation becomes a release decision. Retrieval evaluation belongs before shipping because it is the earliest place to distinguish an answer problem from an evidence problem and a useful index from an economically uncontrolled one.
Top comments (0)