The Four RAGAS Metrics
RAGAS (RAG Assessment) is an evaluation framework designed specifically for RAG systems. Its four metrics each measure a different component of the RAG pipeline:
Faithfulness
→ Does the answer stay within the retrieved context?
→ Low score = hallucination risk; the model is filling in rather than retrieving
→ Computed: decompose answer into claims; check each against context
Answer Relevancy
→ Does the answer actually address the user's question?
→ Low score = off-topic or too vague
→ Computed: generate questions from the answer; measure similarity to original question
Context Precision
→ Of what was retrieved, how much is actually useful?
→ Low score = retrieval pulls in noise, diluting the useful signal
→ Computed: for each retrieved chunk, determine whether it contributed to the answer
Context Recall
→ Was all the information needed to answer correctly retrieved?
→ Low score = retrieval missed critical information; answer is incomplete
→ Computed: decompose ground_truth into claims; check each against context
The four metrics split across RAG's two modules: Retrieval gets Context Precision + Context Recall, Generation gets Faithfulness + Answer Relevancy.
Experiment Design
Knowledge base: Python development standards documentation (naming conventions, exception handling, performance, testing, logging). 8 test questions + ground truth.
Two versions compared:
Version A (raw):
Original enterprise documents: meeting notes, draft markers,
file references, version history notes — the "real state" of
a typical enterprise knowledge base
Version B (cleaned):
Same content, knowledge distillation applied:
→ Removed metadata (meeting dates, author info, version history)
→ Restructured as rule lists
→ Removed filler language and repetition
→ Content focused on core rules only
Results
RAGAS Evaluation Results
──────────────────────────────────────────────────────────────────────
Metric Version A (raw) Version B (clean) Delta
──────────────────────── ──────────────── ───────────────── ──────
Faithfulness 1.000 1.000 +0.000
Answer Relevancy 0.732 0.721 -0.011
Context Precision 1.000 1.000 +0.000
Context Recall 1.000 1.000 +0.000
──────────────────────── ──────────────── ───────────────── ──────
Average 0.933 0.930 -0.003
Expected: cleaned version scores higher — less noise, better precision. Actual: nearly identical. Worth examining.
Why Both Versions Score the Same
Reason 1: What Context Precision = 1.0 Actually Means
A perfect Context Precision score says "every retrieved chunk contributed to generating the answer." But this is judged by an LLM, which tends to find a reason to call any chunk "useful" — even if a raw document contains meeting notes, as long as the final paragraph has the relevant rule, the LLM calls the chunk a contributor.
RAGAS Context Precision measures what the LLM considers useful, not what is objectively noise-free.
Reason 2: Small Knowledge Base, Limited Documents
This experiment has 5 documents. Retrieving top-3 from 5 means the relevant chunk is almost always retrieved, so Context Recall naturally approaches 1.0.
Expanding to 100 documents, where similar-topic documents compete, is when recall starts to separate.
Reason 3: Questions Are Too Direct
All 8 questions ask directly about a specific rule ("what's the naming style," "what does __slots__ do"). These questions can be answered from a single chunk — raw or cleaned makes no difference.
Multi-hop questions ("combining exception handling and logging standards, how would you design a fault diagnosis function") amplify document quality differences.
When RAGAS Can Distinguish Document Quality
In real enterprise RAG systems, these conditions produce noticeable score separation:
| Scenario | Affected metric | Expected gap |
|---|---|---|
| Knowledge base > 1,000 docs, many noisy | Context Precision ↓ | 0.3–0.5 |
| Answers require cross-document reasoning | Context Recall ↓ | 0.2–0.4 |
| Raw docs contain hallucination-style statements | Faithfulness ↓ | 0.1–0.3 |
| Doc language doesn't match question language | Answer Relevancy ↓ | 0.1–0.2 |
| Chunk size is wrong (too large or too small) | Context Precision ↓ | 0.2–0.5 |
The right way to use RAGAS: build a baseline with a small test set (don't expect large gaps), then run sampling evaluation on real production traffic. RAGAS is most valuable as a continuous quality monitor, not as a one-shot comparison tool.
Common RAGAS Misconceptions
Misconception 1: High RAGAS Scores = Good RAG System
Context Precision and Context Recall at 1.0 only means "retrieval quality exceeded the test set's ability to detect problems." Switch to a harder test set and scores may drop immediately.
Actual quality validation comes from L1 metrics (adoption rate, task completion). RAGAS is the L2 middle layer — not the endpoint.
Misconception 2: Low Answer Relevancy = Low Answer Quality
Answer Relevancy is computed by generating questions from the answer and measuring similarity to the original question. If the original question is very specific but the answer is concise, that similarity score can be low — not because the answer is wrong, but because a short answer has narrow coverage.
This experiment's Answer Relevancy (0.732) is low partly because the test questions are specific and the generated answers are concise. The reverse-generated questions don't match the original in phrasing, pulling the score down.
Misconception 3: All Four Metrics Should Be As High As Possible
High Context Precision + low Context Recall: retrieved content is all relevant, but missed important information. Not a good retrieval outcome.
High Context Recall + low Context Precision: retrieved all necessary content, but also introduced a lot of noise. Answer quality suffers.
Track all four metrics separately. The average hides important trade-offs.
Running RAGAS in Code
from ragas import evaluate
from ragas.metrics.collections import (
faithfulness, answer_relevancy, context_precision, context_recall
)
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from datasets import Dataset
data = Dataset.from_dict({
"question": [...],
"answer": [...], # RAG system output
"contexts": [[...]], # list of retrieved chunks per question
"ground_truth": [...], # reference answers
})
ragas_llm = LangchainLLMWrapper(your_llm)
ragas_embeddings = LangchainEmbeddingsWrapper(your_embedder)
result = evaluate(
data,
metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
llm=ragas_llm,
embeddings=ragas_embeddings,
)
print(result)
# {'faithfulness': 0.95, 'answer_relevancy': 0.82, ...}
Note: RAGAS v0.2+ recommends importing from ragas.metrics.collections; the old import path triggers DeprecationWarnings.
Test Set Design Guidelines
RAGAS evaluation quality depends heavily on test set design.
1. Cover three difficulty levels:
Simple (answer in a single paragraph) → validates basic functionality
Medium (answer integrates multiple chunks) → validates chunking strategy
Complex (answer requires cross-doc reasoning) → exposes real bottlenecks
2. Ground truth should be specific:
Bad: "use pytest for tests"
Good: "test function naming: test_<feature>_<expected_behavior>;
fixtures use yield to separate setup from teardown"
3. Include boundary questions (no answer in KB):
Validates rejection behavior, guards against hallucination
4. Minimum 50 samples for stable scores
This experiment uses 8 — minimum for demonstration purposes only
Summary
- Four metrics, four different measurements: Context Precision/Recall measure retrieval, Faithfulness measures hallucination risk, Answer Relevancy measures answer focus — look at all four, don't average them into one number
- Perfect scores don't mean perfect: Context Precision = 1.0 reflects LLM judgment, not objective noise-free retrieval; small knowledge bases and simple questions produce artificially high scores
- RAGAS value is in continuous monitoring: a single comparison may not distinguish versions, but weekly sampling that tracks quality over time will surface drift early
References
- RAGAS Documentation
- RAGAS GitHub
- Full demo code: eval-04-ragas
Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.
Find more useful knowledge and interesting products on my Homepage
Top comments (0)