EvalFlow: A Regression Gate for RAG Systems in CI

EvalFlow: A Regression Gate for RAG Systems in CI

EvalFlow is an open-source tool for checking changes to a RAG pipeline before merge. It compares the current version with a baseline on a golden dataset and produces an explainable CI verdict: confirm a regression, allow the change, or report that the run was not sensitive enough to support a conclusion.

A RAG pipeline for incident investigation

This article uses a representative scenario: an observability system receives an alert and log excerpts, searches runbooks, incident history, and technical documentation for related material, then sends a concise report to Slack. The report should include a likely cause, the affected service, source links, and first actions for the on-call engineer.

This RAG pipeline has two dependent parts. Retrieval selects documents and context fragments from the incident data; generation uses that context to produce an answer. System characteristics remain around both stages: latency, request cost, and error rate.

alert + logs
    -> retrieval over runbooks, incident history, and documentation
    -> selected context
    -> LLM-generated incident report
    -> Slack

The pipeline will change over time. A team may update the index, document set and chunking, retriever, top-k, prompt, or model. These changes affect more than the final report: they change which sources the model sees, how quickly the result arrives, how much it costs, and how often a request fails.

One run is not enough to check such a change. A model can write a similar report from a different selection of context, while sampling variation can create a difference where no product change exists. The opposite case is more dangerous: a retriever stops finding the relevant runbook, but the model still produces coherent and plausible text. An output-only metric then fails to show that the evidence behind the answer has disappeared.

For this scenario, CI should do more than print a set of scores after a run. It should compare the new version with a baseline and show whether retrieval, generation, or the system characteristics changed.

What CI needs to check

EvalFlow is intended as a reusable tool for RAG and LLM pipelines, not as a collection of one-off scripts inside one service. It takes a golden dataset, pipeline configuration, and baseline artifact as input. It produces a saved report and a CI status that can be used in a pull request.

Before merge, that status needs to answer three separate questions:

1. Is there a confirmed regression relative to the baseline, or does the observed difference fit within noise?
2. Which layer changed: retrieval, generation, or system metrics?
3. Was the run sensitive enough to detect a degradation that the team considers important?

The tool cannot decide on its own how much degradation is unacceptable for a particular product. The team therefore configures a practical margin for each metric and, when needed, the effect size the system must be able to detect. EvalFlow applies those rules consistently and saves both the inputs and comparison results in an artifact instead of replacing product requirements with universal thresholds.

The boundary matters as well. EvalFlow does not replace production observability, load testing, red teaming, or the judgment of an on-call engineer. It checks a reproducible change to a RAG pipeline against a predefined set of cases before that change reaches production.

How EvalFlow produces a CI verdict

The check starts with a golden dataset. Each case contains an input request and, where appropriate, an expected answer and identifiers of documents that the retriever should return. A baseline is not stored as one aggregate score. It is an artifact from a previous run that retains results for each case, so the current and previous versions can be compared on the same population of requests.

Non-deterministic stages run more than once. The statistical unit is still the case rather than the individual model call: repeated results are first reduced to one case-level value and only then aggregated across the dataset. Five generations for 60 cases provide 60 observations for quality estimation, not 300 artificially independent points. Deterministic stages, such as BM25 against a fixed index, run once per case because repeating them would not add information.

Each metric belongs to a layer. recall_at_k and MRR show whether retrieval found the expected documents; answer_similarity and exact_match describe the answer; latency, cost, tokens, and error_rate describe the system as a whole. This separation does not infer the cause of a regression automatically, but it prevents a retrieval drop from being hidden inside an aggregate score for final text.

After a run, EvalFlow matches the same case IDs in the current and baseline artifacts. For each bootstrap iteration, it resamples cases with replacement, recomputes the aggregate for both versions, and records their difference. This is a paired bootstrap: it uses the fact that both versions saw the same requests, and it works for more than a mean, including rates and p95.

A confidence interval alone is not enough for a merge decision. A small but reliably measured decline may not matter to the product, so every rule includes a practical margin. A regression is confirmed only when the entire interval for the difference lies beyond that threshold. EvalFlow also calculates the minimum confirmable regression and compares it with detection_effect. The report can therefore make two different statements: a regression is confirmed, and the run was or was not capable of detecting a degradation of the required size.

Errors are part of the comparison as well. A case that fails cannot be silently removed from a quality metric: a new version could fail more often and appear better on the remaining, easier requests. An error_rate rule is therefore required, while excluded cases and error asymmetry between versions remain in the report.

A controlled retrieval-regression example

The repository contains a small support-assistant fixture with six knowledge-base documents and 60 golden cases. Five cases do not require a document and do not participate in retrieval metrics. For the other 55, the baseline returns the expected document IDs. One refund-policy document is then reindexed under a new identifier. Its content remains available, but the retrieval evaluation contract is broken: refund requests no longer receive the expected document ID.

The generator in this example is intentionally an offline mock. It produces an answer from the expected answer rather than from retrieved context. This is not intended to model how a production LLM behaves. The fixture isolates the case under discussion: retrieval changes while the output metric does not. In a production system, the model normally depends more strongly on context, and both layers may change at the same time.

After the change, evalflow run produces the following result:

retrieval
  recall_at_k        0.8091

generation
  answer_similarity  0.9165

gate
  recall_at_k        delta -0.1909, 95% CI [-0.3000, -0.0909]
  answer_similarity  delta +0.0000, 95% CI [0.0000, 0.0000]

The configured margin for recall_at_k is 0.05. The upper end of the interval, -0.0909, is still beyond that threshold, so EvalFlow confirms a retrieval regression and completes the CI run with exit code 1. answer_similarity passes because it did not move within this intentionally constructed fixture.

The report includes one more important qualification. The retrieval rule is configured to detect a drop of 0.10, but the minimum confirmable regression for this run is 0.1546. The status makes two statements at once: the observed drop of 0.1909 is confirmed, but this dataset would not provide the same confidence for every degradation of 0.10. Instead of a green or red status without context, the team sees the operating boundary of its own experiment.

This is the intended integration outcome: a pull request with a changed index does not pass CI because of an absolute threshold in one configuration line, but because comparison with the baseline confirmed a retrieval-layer regression. The run artifacts retain metrics, intervals, case-level data, and the decision, so the result can be inspected after the job ends.

Conclusion

EvalFlow does not define RAG quality for a team. Golden-dataset quality, metric selection, the practical margin, and detection_effect remain product decisions. Production observability and investigation of real incidents are not replaced by an offline check.

Its scope is narrower: to make a version change to a RAG pipeline inspectable before merge. Instead of a one-off score for the current version, CI receives a comparison with a saved baseline, layer separation, uncertainty intervals, and an explicit statement about the sensitivity of the experiment. That makes the job status part of an engineering process while retaining the data on which it was produced.

The source code, configuration, and reproducible support-assistant fixture are available in the EvalFlow repository. You can run the example locally with:

pip install -e .
cd examples/support-assistant
evalflow run --config evalflow.yaml

The run intentionally ends with exit code 1: the fixture contains a confirmed retrieval regression relative to the committed baseline.