Ragas: how to actually measure your RAG pipeline (faithfulness, context precision/recall, relevancy)

Thursday AI Tool/Workflow, part two. This morning was about parsing PDFs cleanly with Docling. This afternoon: Ragas, the open-source (Apache-2.0) library for evaluating RAG and LLM apps, so you can tell whether a change actually made your pipeline better.

Most RAG tuning goes like this: change the chunk size, ask the bot three questions, decide it "feels better", ship. Then a week later someone finds the question that broke. Ragas replaces the vibe check with numbers you can track: did retrieval find the right context, did the answer stick to that context, did it actually answer the question?

Here's exactly how to use it on a real pipeline.

A quick note on versions

Ragas has changed a lot, and many tutorials online use APIs that are now deprecated. Everything below targets Ragas 0.4.x (0.4.3 is the latest release on PyPI as I write). The main changes from 0.3 to 0.4, per the official migration guide:

• Metrics now live in ragas.metrics.collections and are called with keyword arguments: await metric.ascore(user_input=..., response=...).
• They return a MetricResult (.value plus an optional .reason) instead of a bare float.
• LLMs are created with a single llm_factory("model-name", client=...). LangchainLLMWrapper and LlamaIndexLLMWrapper are deprecated.
• evaluate() still works but is deprecated in favour of the @experiment decorator. The old SingleTurnSample + single_turn_ascore() pattern is legacy.

The project also moved on GitHub: explodinggradients/ragas now points to vibrantlabsai/ragas. Docs are still at docs.ragas.io.

1. Install

pip install ragas openai

The default examples use OpenAI as the judge model, so set OPENAI_API_KEY. llm_factory also supports other providers (Anthropic, Google, Azure, Bedrock and more) by passing that provider's client.

One thing worth knowing: Ragas collects minimal anonymized usage analytics. To opt out, set RAGAS_DO_NOT_TRACK=true.

2. The four RAG metrics, in plain words

Each metric looks at a different part of the pipeline, which is what makes them useful for debugging. All of them score from 0 to 1, higher is better, and all four use an LLM as the judge.

Metric	Plain-words question	Needs	Points at
Context recall	Did we retrieve everything needed to answer?	user_input, retrieved_contexts, reference	Retrieval missing info
Context precision	Are the useful chunks ranked at the top?	user_input, retrieved_contexts, reference	Retrieval noise / ranking
Faithfulness	Is every claim in the answer backed by the retrieved context?	user_input, response, retrieved_contexts	Generation making things up
Answer relevancy	Does the answer address the question that was asked?	user_input, response (+ embeddings)	Off-topic or padded answers

How they're computed, from the docs:

• Faithfulness splits the response into claims and checks each one against the retrieved context: supported claims / total claims. The docs' example: "Einstein was born in Germany on 20th March 1879" against a context saying 14 March gives 1 of 2 claims supported, so 0.5.
• Context recall does the same with your reference answer: how many claims in the reference can be attributed to the retrieved context. That's why it always needs a reference.
• Context precision is rank-aware: it averages precision@k over the positions of relevant chunks, so a relevant chunk at rank 1 counts for more than the same chunk at rank 5. In the docs' example, putting an irrelevant chunk first drops the score from about 1.0 to about 0.5.
• Answer relevancy asks an LLM to generate questions from the answer (3 by default, set with strictness) and averages their embedding cosine similarity to the real question. It deliberately does not check factual accuracy, and the docs note the score isn't strictly guaranteed to stay between 0 and 1.

3. Build a small eval dataset

You don't need a giant benchmark. Start with 20 to 50 real questions your users actually ask, each with a short reference answer written by someone who knows the docs. That's the slow, valuable part.

Example (hypothetical): an HR policy bot, built on the Docling chunks from this morning's post.

{"id": "leave-01", "user_input": "How many unused leave days can I carry over?", "reference": "Up to 10 days of unused annual leave carry over to the next year."}
{"id": "leave-02", "user_input": "Do carried-over days expire?", "reference": "Carried-over days must be used by 31 March or they lapse."}
{"id": "exp-01", "user_input": "What's the limit for client dinners?", "reference": "Client dinners are reimbursed up to the per-person limit in the Expense Policy, with itemised receipts."}

Save that as golden.jsonl. Then run your real pipeline on each question and record what it retrieved and what it answered. Ragas evaluates outputs; it doesn't run your app for you.

# collect.py - run the pipeline under test and save its outputs
import json
from my_rag import answer_question  # your app: returns {"answer": str, "contexts": [str, ...]}

def collect(config: str, path: str = "golden.jsonl") -> list[dict]:
    rows = []
    for line in open(path):
        q = json.loads(line)
        out = answer_question(q["user_input"], config=config)
        rows.append({
            **q,
            "response": out["answer"],
            "retrieved_contexts": out["contexts"],  # the chunk texts, in rank order
        })
    return rows

Keep retrieved_contexts in the order your retriever returned them. Context precision depends on that order.

If you have no questions yet, Ragas also has a TestsetGenerator that builds synthetic questions from your documents. Treat those as a starting point and add real user questions as soon as you have them.

4. Score it with the current (0.4) API

# eval_rag.py
import asyncio
import json
import statistics
import sys

from openai import AsyncOpenAI
from ragas.embeddings import embedding_factory
from ragas.llms import llm_factory
from ragas.metrics.collections import (
    AnswerRelevancy,
    ContextPrecision,
    ContextRecall,
    Faithfulness,
)

from collect import collect

client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)  # the judge model
emb = embedding_factory("openai", model="text-embedding-3-small", client=client)

faithfulness = Faithfulness(llm=llm)
precision = ContextPrecision(llm=llm)
recall = ContextRecall(llm=llm)
relevancy = AnswerRelevancy(llm=llm, embeddings=emb)

METRICS = ["faithfulness", "context_precision", "context_recall", "answer_relevancy"]

async def score_row(row: dict) -> dict:
    q, ctx = row["user_input"], row["retrieved_contexts"]
    f, p, r, a = await asyncio.gather(
        faithfulness.ascore(user_input=q, response=row["response"], retrieved_contexts=ctx),
        precision.ascore(user_input=q, reference=row["reference"], retrieved_contexts=ctx),
        recall.ascore(user_input=q, retrieved_contexts=ctx, reference=row["reference"]),
        relevancy.ascore(user_input=q, response=row["response"]),
    )
    return {"id": row["id"], "faithfulness": f.value, "context_precision": p.value,
            "context_recall": r.value, "answer_relevancy": a.value}

async def main(config: str) -> None:
    rows = collect(config)
    scores = await asyncio.gather(*(score_row(r) for r in rows))
    summary = {m: round(statistics.mean(s[m] for s in scores), 3) for m in METRICS}
    json.dump({"config": config, "summary": summary, "rows": scores},
              open(f"results-{config}.json", "w"), indent=2)
    print(config, summary)

if __name__ == "__main__":
    asyncio.run(main(sys.argv[1]))

Run it: python eval_rag.py baseline. You get a summary per metric plus per-question scores in results-baseline.json. For a single spot check there's also a synchronous metric.score(...), and MetricResult.reason is worth printing when a score surprises you.

The evaluate() way (still works, now deprecated)

Most existing tutorials and the "Evaluate a simple RAG system" guide use evaluate() with an EvaluationDataset. It still runs in 0.4 (with a deprecation warning) but expects the legacy metric classes from ragas.metrics, not the collections ones:

from ragas import EvaluationDataset, evaluate
from ragas.metrics import (
    Faithfulness,
    LLMContextPrecisionWithReference,
    LLMContextRecall,
    ResponseRelevancy,
)

dataset = EvaluationDataset.from_list(collect("baseline"))  # user_input, retrieved_contexts, response, reference
result = evaluate(
    dataset=dataset,
    metrics=[LLMContextRecall(), LLMContextPrecisionWithReference(), Faithfulness(), ResponseRelevancy()],
    llm=llm,
    embeddings=emb,
)
print(result)               # mean score per metric
df = result.to_pandas()     # one row per question

For new code, use the collections metrics above (or wrap them in @experiment, which also saves each run to CSV under experiments/). Keep evaluate() for existing code until you migrate.

5. Reading the scores

The value of four metrics is that the pattern tells you where to look:

• Low context recall → retrieval didn't find the information. Look at chunking, top-k, hybrid search, or the parsing step (hello, Docling).
• Good recall, low context precision → the right chunk is in there but buried under noise. Try a reranker or a smaller k.
• Good recall, low faithfulness → the context was there and the model still went beyond it. Fix the prompt ("answer only from the documents, say when they don't cover it") or the model.
• Low answer relevancy → the answer is padded, evasive, or answering a different question.

Don't stop at the averages. Sort rows by the lowest score and read those questions. That's where the fix usually is. The Ragas "Evaluate and Improve a RAG App" guide follows exactly this loop: its baseline BM25 RAG passed 43 of 66 questions (65.2%), the error analysis showed retrieval was the bottleneck, and an agentic retrieval version passed 58 of 66 (87.9%).

6. Example workflow: did the chunking change help?

Example workflow (hypothetical). You want to compare your current 800-token chunks with Docling's heading-aware chunks. Same golden set, same judge model, one change:

python eval_rag.py chunks-800
python eval_rag.py docling-hybrid

# compare.py
import json
a = json.load(open("results-chunks-800.json"))
b = json.load(open("results-docling-hybrid.json"))

for m in a["summary"]:
    print(f"{m:18} {a['summary'][m]:.3f} -> {b['summary'][m]:.3f}")

# per-question regressions are more useful than the averages
by_id = {r["id"]: r for r in a["rows"]}
for r in b["rows"]:
    drop = by_id[r["id"]]["context_recall"] - r["context_recall"]
    if drop > 0.2:
        print("recall dropped:", r["id"], round(drop, 2))

Change one thing at a time (the Ragas experimentation docs say the same), otherwise you won't know which change moved the score.

7. Turn it into a regression check in CI

Once you have a baseline, make the eval run on every pull request that touches retrieval, chunking or prompts:

# check_regression.py
import json, sys
TOLERANCE = 0.05  # my choice; tune it once you know how noisy your scores are

base = json.load(open("baseline.json"))["summary"]
new = json.load(open(sys.argv[1]))["summary"]
failed = [m for m in base if new[m] < base[m] - TOLERANCE]
for m in base:
    print(f"{m:18} base {base[m]:.3f}  new {new[m]:.3f}")
sys.exit(1 if failed else 0)

# .github/workflows/rag-eval.yml (excerpt)
on:
  pull_request:
    paths: ["rag/**", "prompts/**"]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt   # pin the ragas version here
      - run: python eval_rag.py candidate
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          RAGAS_DO_NOT_TRACK: "true"
      - run: python check_regression.py results-candidate.json

Commit baseline.json and update it on purpose when you accept a new baseline. To save money on reruns, Ragas supports exact-match caching: llm_factory("gpt-4o-mini", client=client, cache=DiskCacheBackend()) with from ragas.cache import DiskCacheBackend.

Pitfalls

• The judge is an LLM. Scores move a little between runs and between judge models. Pin the judge model and the Ragas version, and compare runs made with the same setup.
• Small sets are noisy. With 30 questions, one question flipping moves the average noticeably. Read per-question results before celebrating a small gain.
• References drive recall and precision. A sloppy reference answer gives you a sloppy score. Review them like code.
• Relevancy isn't correctness. A confident, on-topic, wrong answer can score well on answer relevancy. Faithfulness and recall catch different failures; use them together.
• Old tutorials break. If you see ground_truths, instructor_llm_factory or single_turn_ascore, you're reading pre-0.4 code (ground_truths is now reference, a single string).
• It costs tokens. Every metric makes LLM calls per question. Cache, and run the full set on PRs that touch RAG, not on every commit.

Change one thing. Rerun the same questions. Let the numbers argue with your gut.

References

• Ragas on GitHub: https://github.com/vibrantlabsai/ragas
• Ragas documentation: https://docs.ragas.io/
• Available metrics: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/
• Faithfulness: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/
• Context precision: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/
• Context recall: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/
• Answer (response) relevancy: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_relevance/
• Migration from v0.3 to v0.4: https://docs.ragas.io/en/stable/howtos/migrations/migrate_from_v03_to_v04/
• Evaluate a simple RAG system: https://docs.ragas.io/en/stable/getstarted/rag_eval/
• How to evaluate and improve a RAG app: https://docs.ragas.io/en/stable/howtos/applications/evaluate-and-improve-rag/
• Experiments: https://docs.ragas.io/en/stable/concepts/experimentation/
• Caching: https://docs.ragas.io/en/stable/howtos/customizations/_caching/
• Ragas on PyPI: https://pypi.org/project/ragas/
• Ragas paper (arXiv 2309.15217): https://arxiv.org/abs/2309.15217

How do you check today whether a RAG change made things better or worse?