Evaluation-Driven Development for RAG Systems: Building the Feedback Loop That Makes AI Actually Improve

Most RAG systems get built in one direction: design the architecture, implement it, test it manually, deploy it. The evaluation, if it exists, happens before deployment as a gate. After deployment, quality assessment happens informally through user complaints and anecdotal feedback.

This approach produces systems that get worse over time, not better. The document corpus changes. User query patterns shift. The system drifts without any systematic mechanism for detecting or correcting that drift.

Evaluation-driven development is a different approach that treats systematic quality measurement not as a pre-deployment gate but as a continuous operational practice that drives ongoing improvement. The eval suite is not the last step before you ship. It is the mechanism through which you learn what needs to change and verify that changes worked.

Here is how I build and operate evaluation suites for production RAG systems, including the specific components that most implementations skip.

Building the evaluation dataset

The evaluation dataset is the foundation of everything else. It needs to be representative of real production queries, not queries you invented to test hypothetical capabilities.

The process I use to build the initial dataset:

First two weeks of any new RAG deployment, log every query submitted to the system along with the retrieved documents and generated response. This gives you the actual distribution of what users ask rather than what you thought they would ask. The actual distribution is almost always different from the anticipated distribution in ways that matter.

From this log, sample 200 to 300 queries that represent the spread of query types. For each query, establish a ground truth answer. For factual questions with specific correct answers, this is straightforward. For synthesis questions, it requires human judgment about what constitutes a good response. This human annotation is time-consuming and cannot be automated away. Budget 30 to 45 minutes per query for a careful annotator.

Organize the dataset by query type and difficulty. I typically use four categories: direct factual lookup, policy or procedural questions, synthesis across multiple documents, and edge cases where the answer requires acknowledging uncertainty or where the correct answer is that the information is not in the knowledge base.

from dataclasses import dataclass
from typing import Optional, List
from enum import Enum

class QueryType(Enum):
    FACTUAL_LOOKUP = "factual_lookup"
    POLICY_PROCEDURAL = "policy_procedural"
    SYNTHESIS = "synthesis"
    EDGE_CASE = "edge_case"

class DifficultyLevel(Enum):
    EASY = "easy"      # single document, clear answer
    MEDIUM = "medium"  # multiple documents, some synthesis
    HARD = "hard"      # sparse evidence, requires reasoning, or should decline

@dataclass
class EvalQuery:
    query_id: str
    query_text: str
    query_type: QueryType
    difficulty: DifficultyLevel
    ground_truth_answer: str
    relevant_document_ids: List[str]    # documents that should be retrieved
    acceptable_answer_variants: List[str]  # alternative correct phrasings
    failure_indicators: List[str]          # strings that indicate a bad answer
    should_decline: bool                   # True if AI should say "I don't know"
    annotator_notes: str
    created_at: str
    last_verified: str  # when ground truth was last confirmed as current

The last_verified field is critical and often missing. Ground truth answers go stale as the knowledge base evolves. An evaluation dataset that was accurate when built becomes misleading over time if it is not maintained. Schedule quarterly reviews of the evaluation dataset to confirm that ground truth answers still reflect current organizational truth.

The metrics that matter

Retrieval metrics and generation metrics are both necessary and measure different things.

For retrieval:

def compute_retrieval_metrics(
    retrieved_doc_ids: List[str],
    relevant_doc_ids: List[str],
    k_values: List[int] = [1, 3, 5]
) -> dict:
    metrics = {}

    for k in k_values:
        retrieved_at_k = set(retrieved_doc_ids[:k])
        relevant = set(relevant_doc_ids)

        precision_at_k = len(retrieved_at_k & relevant) / k if k > 0 else 0
        recall_at_k = len(retrieved_at_k & relevant) / len(relevant) if relevant else 0

        metrics[f"precision@{k}"] = precision_at_k
        metrics[f"recall@{k}"] = recall_at_k

    # Mean Reciprocal Rank: where did the first relevant doc appear?
    mrr = 0
    for rank, doc_id in enumerate(retrieved_doc_ids, 1):
        if doc_id in set(relevant_doc_ids):
            mrr = 1 / rank
            break
    metrics["mrr"] = mrr

    return metrics

For generation, automated metrics alone are insufficient. ROUGE and BLEU scores measure lexical overlap, not factual accuracy or usefulness. I use a combination:

def compute_generation_metrics(
    generated_response: str,
    ground_truth_answer: str,
    retrieved_docs: List[str],
    query: EvalQuery,
    llm_judge  # small LLM used for automated quality judgment
) -> dict:
    metrics = {}

    # Groundedness: is the answer supported by retrieved documents?
    groundedness_prompt = f"""
    Query: {query.query_text}
    Retrieved context: {' '.join(retrieved_docs)}
    Generated answer: {generated_response}

    Rate from 0 to 1: Is every factual claim in the generated answer
    directly supported by the retrieved context?
    Respond with only a number between 0 and 1.
    """
    metrics["groundedness"] = float(llm_judge.complete(groundedness_prompt))

    # Correctness: does it match the ground truth?
    correctness_prompt = f"""
    Question: {query.query_text}
    Correct answer: {ground_truth_answer}
    Generated answer: {generated_response}

    Rate from 0 to 1: How factually correct and complete is the generated answer
    compared to the correct answer? Minor wording differences are fine.
    Respond with only a number between 0 and 1.
    """
    metrics["correctness"] = float(llm_judge.complete(correctness_prompt))

    # Failure indicator check: did the answer contain known bad patterns?
    metrics["contains_failure_indicator"] = any(
        indicator.lower() in generated_response.lower()
        for indicator in query.failure_indicators
    )

    # Appropriate decline: did it decline when it should have?
    if query.should_decline:
        decline_indicators = ["don't have", "no information", "not in", "cannot find"]
        metrics["appropriate_decline"] = any(
            indicator in generated_response.lower()
            for indicator in decline_indicators
        )
    else:
        metrics["appropriate_decline"] = True  # N/A for queries that should be answered

    return metrics

The LLM judge approach for groundedness and correctness is an approximation, not a ground truth. It is faster and cheaper than human evaluation and good enough for continuous monitoring, but significant changes in scores should be verified with human review before drawing strong conclusions.

Running evals as a deployment gate

Every change to the retrieval configuration, the prompts, the chunking strategy, or the embedding model should be evaluated against the full eval suite before deployment to production.

class EvalGate:
    def __init__(
        self,
        eval_dataset: List[EvalQuery],
        rag_pipeline,
        thresholds: dict
    ):
        self.eval_dataset = eval_dataset
        self.pipeline = rag_pipeline
        self.thresholds = thresholds  # e.g., {"recall@5": 0.80, "correctness": 0.85}

    def run_evaluation(self) -> dict:
        all_metrics = []

        for query in self.eval_dataset:
            retrieved_docs, response = self.pipeline.run(query.query_text)

            retrieval_metrics = compute_retrieval_metrics(
                retrieved_doc_ids=[doc.id for doc in retrieved_docs],
                relevant_doc_ids=query.relevant_document_ids
            )
            generation_metrics = compute_generation_metrics(
                generated_response=response,
                ground_truth_answer=query.ground_truth_answer,
                retrieved_docs=[doc.content for doc in retrieved_docs],
                query=query,
                llm_judge=self.llm_judge
            )

            all_metrics.append({**retrieval_metrics, **generation_metrics})

        return self.aggregate_metrics(all_metrics)

    def passes_gate(self) -> tuple[bool, dict]:
        results = self.run_evaluation()
        failures = {
            metric: (results[metric], threshold)
            for metric, threshold in self.thresholds.items()
            if results.get(metric, 0) < threshold
        }
        return len(failures) == 0, failures

This gate prevents regressions from reaching production. Before deploying any change, run the gate. If it fails, the failing metrics tell you specifically what degraded and by how much. You can then either fix the issue or make an explicit decision to accept the regression with full awareness of what you are accepting.

The monitoring layer that connects to the eval suite

Running the eval suite before deployment catches regressions from changes you made intentionally. It does not catch regressions from changes that happen without your involvement: document corpus changes, query distribution shifts, or model behavior drift.

For continuous monitoring, run a subset of the eval suite (50 to 100 queries) on a daily schedule against the production system. Track the metrics as time series. Alert when any metric drops more than a defined threshold from its rolling average.

import schedule
import time

def daily_quality_check(eval_subset, rag_pipeline, alert_threshold=0.05):
    results = run_evaluation_subset(eval_subset, rag_pipeline)
    baseline = load_rolling_baseline(window_days=30)

    regressions = {
        metric: {
            "current": results[metric],
            "baseline": baseline[metric],
            "drop": baseline[metric] - results[metric]
        }
        for metric in results
        if baseline[metric] - results[metric] > alert_threshold
    }

    if regressions:
        send_alert(
            subject="RAG Quality Regression Detected",
            body=format_regression_report(regressions)
        )

    store_daily_results(results)

schedule.every().day.at("06:00").do(daily_quality_check, eval_subset, rag_pipeline)

while True:
    schedule.run_pending()
    time.sleep(60)

The combination of pre-deployment gates and continuous monitoring gives you coverage against both types of quality change: the ones you cause and the ones that happen to you.

The part that requires human judgment

I want to be direct about the limits of automated evaluation. Automated metrics measure proxies for quality. They are useful proxies and they are far better than no measurement, but they are not the same as direct human judgment about whether the system is useful.

The evaluation-driven development loop needs a human component. Monthly, someone should manually review a random sample of production queries: what was retrieved, what was generated, and whether the output was genuinely useful. This review catches things the automated metrics do not: outputs that are technically accurate but misleadingly incomplete, responses that are polite and hedged in a way that makes them useless, or answer quality that is systematically different for specific user groups or query types.

The automated metrics tell you when something changed. The human review tells you whether the change matters. Both are necessary. The organizations that rely entirely on automated metrics will miss qualitative degradation that the metrics cannot capture. The organizations that rely entirely on human review will miss systematic patterns that only become visible at scale.

Build both. They are not in tension. They are complementary layers of the same feedback loop.