Standing up a RAG pipeline is an afternoon's work these days. Chunk the documents, run them through an embedding model, push them into a vector database, staple the search results onto the prompt. The demo works, the answers look reasonable, everyone leaves happy. Then the system goes to production and something strange happens: nobody notices when quality drops. Because nobody is measuring.
My thesis is this: the real engineering in RAG is not building the retrieval layer, it is making accuracy measurable. The only honest sentence you can say about the quality of a pipeline you don't measure is "I don't know." And the bad news is that this sentence sounds a lot like silence.
One night, sixteen poisoned records
I won't keep this abstract, because it happened to me.
Topic selection in this blog's content pipeline runs through a research chain: a search layer (my own SearXNG instance) collects results for a topic, an LLM judge looks at that evidence and either approves or rejects the topic, and the decision is cached for 30 days. A classic retrieve-then-generate pattern — except what it produces is not prose, it's a decision.
On the night of 20-21 August 2026 the search layer was up. It didn't die, it didn't time out, it didn't return HTTP 500. Its engines just started flapping: google_cse hit an "unusual traffic" block, startpage ran into a CAPTCHA wall, and bing began returning completely unrelated results. The results for the query "kubernetes operator pattern" included Treasure Island and Lidl stores.
My code only treated "unreachable" as a fault. A reachable-but-garbage result it treated as evidence. Garbage evidence went to the judge, and the judge — to be fair, behaving correctly given the data in front of it — rejected legitimate topics. The rejection was cached for 30 days. By morning sixteen topics had been poisoned and were sitting in the queue as dead records; cleaning them up took a separate PR.
Not a single component reported an error during this incident. The vector store was healthy, the LLM was answering, the cache was working correctly. The only thing that broke was retrieval relevance, and no metric was watching it. Black boxes have their worst nights without telling anyone.
The fix was embarrassingly simple in hindsight, and it has two steps: first extract the distinctive terms from the query (stopwords removed, at least three characters, Turkish characters folded) and drop every result whose title, snippet and URL text contains none of them; then, if nothing is left, treat that not as editorial evidence but as a search fault. Let the fault propagate upward, and don't cache the decision.
I also built in a deliberate exemption: if the query yields no distinctive terms at all, nothing gets dropped. I don't punish what I can't measure. The easiest way for a cheap gate to break is to make a decision where it cannot be sure — at that point it replaces the very fault it was meant to catch.
It turns out the thing I built without knowing it has a name: a context precision gate that doesn't use an LLM. I wish I had learned it from the literature instead of from reading logs at three in the morning.
Don't lump retrieval and generation together
The most common mistake I see in RAG evaluation is producing a single "accuracy" number and watching that. When the number drops, you have no directional information at all.
Yet the pipeline breaks along two separate axes. Either you failed to find the right document (a retrieval problem: wrong chunk size, weak embeddings, missing index, or in my case a collapsing search engine), or you found the right document and the model didn't use it (a generation problem: ignoring context, hallucinating, over-summarising). The fixes are completely different. For the first you add a reranker; for the second you change the prompt or the model.
Ragas's metric family draws exactly this line:
| Axis | Metric | What it asks |
|---|---|---|
| Retrieval | Context Precision | How many of the retrieved chunks are actually relevant? |
| Retrieval | Context Recall | How much of the relevant information did we manage to retrieve? |
| Retrieval | Noise Sensitivity | Does the answer degrade when an irrelevant chunk slips in? |
| Generation | Faithfulness | Is the answer faithful to the retrieved context? |
| Generation | Answer Relevancy | Does the answer actually address the question asked? |
Internalising this distinction buys you far more than staring at one number. If Faithfulness is high but Context Recall is low, you have a system that "stays loyal to what it has and answers incompletely" — it doesn't lie to the user, but it doesn't do the job either. In the opposite case, high Recall with low Faithfulness, it found the right document and then made things up on top of it. The second is far more dangerous, because the answer cites a source and is therefore convincing.
Ragas v0.4: most examples online are now outdated
I need to raise a warning here, because my own memory was wrong while preparing this article.
With v0.4.0, released in December 2025, Ragas rebuilt its metrics API. Metrics moved from ragas.metrics into the ragas.metrics.collections module, and the old location was put on a deprecation path. The metric documentation states plainly that the legacy API is deprecated in v0.4 and removed in v1.0; the migration guide is more cautious and only says "a future release." At the time of writing the latest release is v0.4.3 (13 January 2026).
It isn't just the import path that changed:
# v0.3 — most examples you'll find online look like this
from ragas.metrics import Faithfulness
sample = SingleTurnSample(
user_input="...", response="...", retrieved_contexts=[...],
)
score = await metric.single_turn_ascore(sample) # returns a float
# v0.4 — the current way
from ragas.metrics.collections import Faithfulness
result = await metric.ascore(
user_input="...", response="...", retrieved_contexts=[...],
)
score = result.value # MetricResult
reason = result.reason # you get the rationale too
Other breaking changes worth noting: the ground_truths field (a list) in SingleTurnSample became reference (a single string); the evaluate() function was replaced by the @experiment() decorator; framework-specific wrappers such as LangchainLLMWrapper and LlamaIndexLLMWrapper were dropped in favour of a single llm_factory() that infers the provider from the model name. The details are in the v0.3 to v0.4 migration guide.
One more small but irritating detail: the project's GitHub organisation changed. The old explodinggradients/ragas address still redirects, but the canonical repository is now vibrantlabsai/ragas.
The reason field that comes with MetricResult is on its own a good enough reason to upgrade. A score of 0.62 tells you nothing; a judge saying "the third claim in the answer does not appear in the context" is directly debuggable output.
A metric that needs a reference is useless in production
This distinction is the backbone of your evaluation architecture, and somehow the least discussed part of it.
Some metrics require a reference (ground truth): ContextRecall and ContextPrecisionWithReference. These cannot be computed without knowing what the correct answer is. Others work without one: Faithfulness checks the answer against the context, while AnswerRelevancy generates questions backwards from the answer (three by default) and measures cosine similarity against the original question — neither needs to know the right answer. Likewise ContextPrecisionWithoutReference compares each retrieved chunk not against a reference but against the generated response.
I've given the class names from the collections API, because this is exactly where the trap is: their equivalents in the old module carry LLM-prefixed names such as LLMContextPrecisionWithReference, and most examples online still use those. One more warning: the LLM-free NonLLMContextPrecisionWithReference has no equivalent in the collections API yet — if you need it, you are stuck on the old module for now.
The practical consequence is clear:
- Offline (CI) evaluation: use the reference-based metrics here. You have a golden set, so you can measure Recall. This is where the release gate belongs.
- Online (production) monitoring: you have no reference on real traffic. Faithfulness and reference-free precision fit here — these are what catch regressions in production.
I keep seeing teams try to run reference-based metrics in production. It ends either in a design that eats its own tail — "another LLM to produce the ground truth" — or in a metric that simply never gets computed.
The golden set: boring, but not negotiable
The real cost of evaluation is not in metric code, it's in the dataset. The approach that works in my own setups:
Start small, but make it real. Fifty to a hundred question-answer pairs is enough to build a serious CI gate. The source of those questions should not be your imagination but real user queries — they are sitting in your logs. Synthetic generation (including Ragas's own test set generator) is good for widening coverage, but if there isn't a single real question in the set, you don't know what you're measuring.
Include the hard cases deliberately. Questions whose answers are not in the documents must be in the set. The most valuable behaviour of a RAG system is being able to say "I don't know," and you can only test that with unanswerable questions. If your set consists only of easy questions, what you're measuring isn't quality, it's your own optimism.
Date the set and feed it from incidents. When an answer comes out wrong in production, your reflex should not be to fix the prompt; first add that question to the golden set. A year later your set becomes a map of every wound your system has ever taken. That is what regression testing actually is.
You can also sweep retrieval parameters such as chunk size with this set; I went into how that sweep works on Turkish documents in the RAG chunking test article. And once the measurement pipeline exists, you can finally prove whether improvements like hybrid search and rerankers actually help — instead of guessing.
Wiring the gate into CI: the LLM judge is noisy
Computing metrics and writing them somewhere is not evaluation. Evaluation is something stopping when a threshold is crossed.
But be careful here: LLM-based metrics are not deterministic. Run the same dataset twice and the scores will not be identical. So a gate like if score < 0.85: fail will wake you with a false alarm three times a week, and within two weeks everyone will have disabled it.
The arrangement I prefer:
- Establish the baseline by measuring it. Don't pick the threshold at a desk; run the current pipeline five times and look at the mean and the spread.
- Use a regression threshold instead of an absolute one. Rather than "Faithfulness must stay above 0.85," use "Faithfulness must not drop more than 0.05 below baseline." The gate then checks the effect of your change, not the absolute level of quality.
- Pin the judge model and record its version. When the evaluator model changes, all your historical scores become incomparable. The judge model's version matters as much as the dataset's version.
- Put the cheap gate first. Checks that call no LLM — query term overlap, empty context, source count — take seconds and already catch the dumbest failures. My sixteen poisoned records could have been prevented by a single term-overlap check.
I learned the fourth point the expensive way, which is why I'm underlining it.
I covered the agent-level counterpart of this setup — tool-use accuracy, rubrics, human annotation — in setting up a test harness for AI agent evals. The difference here is that what's being measured is not the agent's behaviour but the retrieval layer itself: in RAG you have to measure context as a separate axis to find out where the error came from.
The bill for evaluation
Nobody mentions this, but LLM-based evaluation is not cheap, because each metric makes model calls of its own.
Look at the mechanics: Faithfulness decomposes the answer into individual claims and checks each claim against the context. Answer Relevancy generates questions backwards from the answer — three by default — and computes embeddings for each. Context Precision asks for a separate relevance judgement for every retrieved chunk. So running a 100-question golden set with five metrics means hundreds of calls. And these are not your product's calls; they return nothing to any user, they only answer the question "are we okay?"
I manage this with three decisions — plus one item at the end that isn't a decision at all, just the bill.
Separate the judge model from the working model. The evaluator does not have to be the same model your product uses. A smaller, cheaper model is usually more than enough for narrow tasks like claim checking. You also avoid the bias of a model grading its own answer.
Don't measure everything on every run. I run two tiers: cheap checks plus a small subset (20-30 examples) on every PR, and the full set on a nightly run. The PR tier is there to catch coarse regressions, the nightly one to show the real picture.
Sample in production. Auditing every answer on live traffic is both expensive and unnecessary. A one percent sample is enough to see the trend — as long as the sampling is random. If you set it up as "measure only when a user votes negatively," what you're measuring is not quality but the propensity to complain.
There's one more boring but real line item: evaluation runs hit rate limits too. If hundreds of concurrent judge calls share the same quota as your production traffic, your quality measurement will slow down your own product. Use a separate key.
Monitoring in production: where do you put the signal?
The offline gate protects what happens before release, but data changes, user behaviour drifts, and providers update their models quietly. You need a live signal.
The good news is that you no longer have to invent one: OpenTelemetry's GenAI semantic conventions are standardising this space. Among the well-known values of the gen_ai.operation.name attribute, alongside chat and embeddings, sits retrieval — meaning you can mark your RAG pipeline's search step as a standard span. gen_ai.usage.input_tokens / gen_ai.usage.output_tokens are defined for token usage, and gen_ai.provider.name for the provider.
Two caveats: these conventions are still at Development status (not stable), so attribute names may change. And during 2026 the documentation was split out of the main semantic-conventions repository into its own — your old links may now land you on an empty redirect page.
The minimum signals worth wiring into monitoring in practice: retrieved chunk count and score distribution, empty-context rate, query-result term overlap, sampled reference-free faithfulness, and whether the citations in the answer actually correspond to retrieved chunks. None of these requires an LLM judge; all but the last are a single counter.
This is a security topic too
We're used to filing evaluation under "quality," but it has a counterpart on the risk side. Two entries in OWASP's 2025 list for LLM applications target RAG directly: LLM08 — Vector and Embedding Weaknesses (vulnerabilities in the vector store and embedding layer) and LLM09 — Misinformation (the model producing false information convincingly). An unmeasured retrieval layer is a silent carrier for both: you only notice a poisoned index when the relevance metric drops.
Let me not skip the cost of the architecture I've just recommended, either: saying "sample one percent in production and run reference-free faithfulness" means sending real user queries and retrieved document content to a third-party model. Your evaluation pipeline can quietly become the place where your product's most sensitive data leaves the building. I offered the separate-key advice above on quota grounds; in an enterprise setting the real reason is usually the data boundary. Decide what gets masked, which region the judge runs in, and how long sampled records are retained — before you build the dashboard.
If you need to tie this to a framework on the enterprise side, one of the four functions of the NIST AI Risk Management Framework is already Measure, and there is a NIST AI 600-1 profile for generative-AI-specific risks. Don't end up answering "how do you measure its quality?" in an audit with "users aren't complaining."
Checklist
Before you ship, can you answer yes to all of these?
- Are retrieval and generation measured separately, or do you have a single quality number?
- Does your golden set contain questions whose answers are not in the documents?
- Have you split offline and online metrics according to whether they need a reference?
- Does your CI gate measure regression against a baseline rather than an absolute threshold?
- Is the judge model's version recorded?
- Do the cheap, LLM-free health checks run before the expensive judge?
- Are empty-context and zero-term-overlap rates wired to a dashboard in production?
- If the nightly evaluation run fails silently, who notices — is your gate fail-open or fail-closed?
- Is the judge model separate from — and cheaper than — the model it evaluates?
- If you use Ragas, is your code on the v0.4 collections API or on the old path that is being removed?
Conclusion
The most repeated sentence about RAG evaluation is "you can't improve what you can't measure." True, but incomplete. The real issue isn't improvement: when you don't measure, you can't even notice things breaking. Improvement is optional; noticing degradation is not.
What broke in my sixteen poisoned records was not the embedding model or the chunk size; it was the system's inability to recognise that its own input was garbage. Before you build a metrics dashboard, ask your pipeline this: if my retrieval layer started returning completely irrelevant results today, who would notice, and how long would it take? If your answer is "when a user complains," then what you actually know about your system's quality is just the limit of your users' patience.
Top comments (0)