DEV Community

Cover image for I Built a RAG Evaluation Checklist: 7 Things I Would Test Before Production
Taha hussein
Taha hussein

Posted on

I Built a RAG Evaluation Checklist: 7 Things I Would Test Before Production

#llm #python #rag #testing

One of the biggest mistakes in LLM development is treating a successful demo as proof of reliability.

A RAG chatbot may answer ten questions correctly and still fail when users ask about outdated policies, missing documents, or information spread across multiple sources.

Before deploying a RAG application, I would evaluate these seven areas.

1. Does Retrieval Find the Right Documents?

Start with a set of questions and manually verified relevant document IDs.

Measure Recall@K to understand whether the retriever finds the relevant documents within its top K results.

def recall_at_k(results, relevant_ids, k=5):
    relevant_ids = set(relevant_ids)

    if not relevant_ids:
        return None

    retrieved_ids = {
        item["id"] for item in results[:k]
    }

    return len(retrieved_ids & relevant_ids) / len(relevant_ids)
Enter fullscreen mode Exit fullscreen mode

This simple metric assumes a result structure containing document IDs.

For multi-document questions, record all relevant documents. Retrieving one useful chunk may not be sufficient to answer a question requiring several pieces of evidence.

2. Are Relevant Results Ranked Highly?

Finding the right document is useful, but its position also matters.

If the relevant document appears at rank 20, it may never reach the generation model.

Consider evaluating:

  • Precision@K
  • Mean Reciprocal Rank
  • NDCG@K

These metrics help you understand whether the retriever is ranking useful information effectively.

If ranking is the problem, investigate hybrid retrieval, metadata filtering, and reranking.

3. Are Generated Answers Faithful to the Context?

Consider this retrieved context:

"Customers may request a refund within 30 days of purchase."

Generated answer:

"Customers can request a refund within 90 days."

The answer contradicts the supplied evidence.

Faithfulness evaluation helps detect unsupported claims, but it should not be confused with correctness against the real world. A model can faithfully repeat an outdated or incorrect document.

For important applications, validate both the source material and the generated answer.

4. Does the System Handle Missing Information?

A reliable RAG system should not invent an answer when the knowledge base does not contain the required information.

Include test cases such as:

{
    "question": "Does the company support Product X?",
    "expected_behavior": (
        "State that the available documentation "
        "does not provide enough information."
    ),
}
Enter fullscreen mode Exit fullscreen mode

Test whether the system acknowledges uncertainty and avoids unsupported claims.

This matters especially for internal knowledge assistants, customer support, and policy-related applications.

5. Can It Answer Questions Requiring Multiple Documents?

Simple questions are not enough to test a RAG system.

Create cases where the answer depends on multiple documents or sections.

For example:

  • One document contains a product's eligibility requirements.
  • Another contains the relevant exceptions.
  • A third contains the effective date.

Check whether retrieval captures the necessary evidence and whether the final answer combines it correctly.

These cases often reveal weaknesses hidden by simple demos.

6. Does Performance Remain Stable After Changes?

Suppose you switch embedding models or increase the chunk size.

The new version may improve some questions while breaking others.

Run the same evaluation dataset against both versions and compare:

  • Retrieval metrics
  • Answer correctness
  • Faithfulness
  • Latency
  • Token usage and cost

Save the configuration for every run.

Do not assume that a change is an improvement simply because a handful of manually tested answers look better.

7. Can You Trace and Reproduce Failures?

For each evaluation case, record enough information to investigate what happened:

  • Question and expected answer
  • Retrieved document IDs and chunks
  • Similarity scores, when available
  • Prompt and model configuration
  • Generated answer
  • Metric results
  • Latency and token usage

Avoid logging secrets or sensitive user data unnecessarily.

This record makes debugging much easier because you can distinguish a retrieval problem from a generation problem instead of guessing.

My Minimal RAG Evaluation Checklist

Before considering a system ready for broader testing, I would verify:

  • [ ] A representative test dataset exists.
  • [ ] Relevant documents have been verified.
  • [ ] Retrieval is evaluated independently.
  • [ ] Answers are checked against reference data.
  • [ ] Missing-information questions are included.
  • [ ] Changes are tested against the same dataset.
  • [ ] Failures can be traced and reproduced.

Passing this checklist does not guarantee production readiness. Security, access control, privacy, load testing, and domain-specific requirements need separate validation.

But it gives you a much stronger foundation than simply asking a chatbot a few questions and deciding that it works.

My takeaway: RAG evaluation is not just a scoring exercise. It is an engineering feedback loop that helps you discover failures, understand their causes, and make informed improvements.

What is the first evaluation metric you would add to a RAG application?

Connect with me

RAG #AIEngineering #Python #LLMEvaluation #DevOps

Top comments (1)

Collapse
 
williamsj04 profile image
Jessica Williams •

Good checklist. For the closing question, I would start with Recall@K, because if the right document never reaches the model, nothing downstream can fix it. One case worth adding to the missing-information tests is a permission check: ask a question whose only answer sits in a document the test user is not allowed to see, and confirm the system neither quotes it nor hints at it. That failure is easy to miss when every test runs as an admin.