DeepEval for AI Testing: The Complete End-to-End Engineering Guide

Author: Himanshu Agarwal
Level: Intermediate → Advanced
Focus: Testing, evaluating, and shipping reliable LLM & RAG applications with DeepEval
Reading time: ~30 minutes

---

🎁 Featured Resource — GenAI Engineering Vault (16 Books Bundle)

If you are building production-grade AI systems and want the entire engineering playbook in one place — evaluation, RAG, agents, prompt engineering, LLMOps, and deployment — grab the full bundle here:
👉 GenAI Engineering Vault — 16 Books Bundle
Explore all my playbooks at himanshuai.gumroad.com.

---

Table of Contents

1. Why AI Testing Is a First-Class Engineering Problem
2. What Is DeepEval?
3. Why Choose DeepEval Over Rolling Your Own Evals
4. Prerequisites
5. End-to-End Installation (Step by Step)
6. Recommended Project Folder Structure
7. Core Concepts You Must Understand
8. Your First Evaluation (Single-Turn)
9. Multi-Turn & Conversational Testing
10. The Metrics Deep Dive
11. Component-Level Evals With Tracing
12. Synthetic Data Generation
13. Running DeepEval in CI/CD
14. Framework Integrations
15. A Complete Worked Example: End-to-End RAG Chatbot Evaluation
16. Advanced Patterns & Best Practices
17. Troubleshooting Common Issues
18. Resources
19. Frequently Asked Questions (FAQs)
20. Final Thoughts

---

1. Why AI Testing Is a First-Class Engineering Problem

Traditional software is deterministic. Feed a function the same input twice and you get the same output twice. That predictability is what makes conventional unit tests trustworthy — you assert add(2, 2) == 4 and move on with your life.

Large Language Model (LLM) applications break this assumption completely. The same prompt can produce different phrasing, different reasoning paths, and occasionally, a confidently wrong answer. A model that scored beautifully on Monday can quietly regress on Friday after a prompt tweak, a temperature change, a new retrieval chunking strategy, or a silent upstream model update from your provider.

This is the core reason AI testing deserves to be treated as a first-class engineering discipline rather than a "we'll eyeball the outputs" afterthought. When your chatbot, RAG pipeline, or autonomous agent is in front of real users, "it looked fine in the demo" is not a quality strategy — it's a liability.

The problems you actually need to catch include:

• Hallucinations — the model inventing facts that are not grounded in the provided context.
• Irrelevance — answers that technically respond but miss the user's actual intent.
• Faithfulness failures in RAG — the generation contradicting the retrieved documents.
• Safety issues — toxicity, bias, PII leakage, or jailbreak susceptibility.
• Regressions — a change that improves one scenario while silently breaking three others.
• Agentic failures — wrong tool calls, broken task completion, or reasoning that spirals.

You cannot manually re-check hundreds of these scenarios every time you push a commit. What you need is an evaluation framework that turns these fuzzy quality questions into scored, repeatable, automatable tests. That is exactly the gap DeepEval fills.

---

2. What Is DeepEval?

DeepEval is an open-source LLM evaluation framework (Apache 2.0 licensed) built by the team behind Confident AI. If pytest is the standard way to unit-test Python code, DeepEval is designed to be the standard way to unit-test LLM outputs.

At its heart, DeepEval lets you:

• Write evaluation tests for LLM outputs the same way you write pytest tests.
• Score outputs using 50+ research-backed metrics — including faithfulness, answer relevancy, contextual precision/recall, hallucination, bias, and toxicity.
• Evaluate end-to-end (treating your app as a black box) and component-level (scoring individual tool calls, retrievers, and sub-agents through tracing).
• Generate synthetic datasets for edge cases that are painful to collect by hand.
• Plug evaluations directly into CI/CD so regressions are caught before they reach production.

DeepEval is local-first — your evaluations run in your own environment, and you only need an LLM provider key (like OPENAI_API_KEY) for the metrics that use an LLM as a judge. It is also model-agnostic and framework-agnostic: it works with OpenAI, Anthropic, Gemini, Azure OpenAI, Ollama, and local/custom models, and integrates natively with LangChain, LangGraph, LlamaIndex, CrewAI, Pydantic AI, OpenAI Agents, Google ADK, and more.

The optional cloud companion, Confident AI, sits on top of DeepEval and adds shared dashboards, regression tracking, observability, and production monitoring — but you never need it to run evaluations.

A useful mental model: Observability tools tell you what happened. DeepEval tells you whether what happened was good enough, by running metrics against test cases, traces, spans, and datasets.

A word on design philosophy. DeepEval treats your LLM app as a black box by default — you don't have to expose internals to evaluate the final output. When you do want to look inside (to evaluate an agent's individual steps), tracing is opt-in and non-intrusive, meaning it never changes how your code runs. This "black box first, glass box when you need it" philosophy is deliberate: it lets a beginner get a passing eval in five minutes, while giving an advanced team the depth to score every tool call, retriever, and sub-agent in a complex pipeline. You grow into the complexity rather than being forced to confront all of it on day one.

---

3. Why Choose DeepEval Over Rolling Your Own Evals

Plenty of teams start by writing a quick if "sorry" in output: fail script. It works for a week. Then it collapses under the weight of real-world nuance. Here is why a dedicated framework wins:

• Research-backed metrics out of the box. Metrics like GEval (an LLM-as-a-judge metric with human-like accuracy) and RAG-specific metrics like faithfulness and contextual recall are already implemented and tuned. You do not reinvent them.
• pytest-native ergonomics. DeepEval feels like the testing you already know. deepeval test run slots into any workflow that already uses pytest.
• A first-class regression story. Run more than one test run and you can compare test cases side by side to catch improvements and regressions.
• Tracing for agents. Modern AI apps are multi-step. DeepEval's non-intrusive @observe tracing lets you score individual components without rewriting your architecture.
• Synthetic data + benchmarks. You get tooling to generate edge cases and run standard benchmarks, not just a metric library.
• A serious community and ecosystem. 250+ contributors, 20+ integrations, and active development (DeepEval 4.0 is the current major line).

The bottom line: a home-grown eval script optimizes for today's demo. DeepEval optimizes for the messy, evolving reality of a system you have to maintain for months.

---

4. Prerequisites

Before installing, make sure you have:

• Python 3.9+ installed (python --version to check).
• pip available and reasonably up to date.
• A terminal you are comfortable in (macOS/Linux shell or Windows PowerShell/CMD).
• An LLM provider API key. Most DeepEval metrics are LLM-as-a-judge metrics, so you'll typically want an OPENAI_API_KEY. You can swap in Anthropic, Gemini, Azure, Ollama, or a custom local model later.
• Basic familiarity with pytest concepts (test functions, assertions) — helpful but not mandatory.

Tip: Always work inside a virtual environment. It keeps your evaluation dependencies isolated from your application dependencies and prevents version conflicts.

---

5. End-to-End Installation (Step by Step)

This section takes you from a clean machine to your first passing evaluation. Follow it in order.

Step 1 — Create and activate a virtual environment

macOS / Linux:

mkdir deepeval-testing && cd deepeval-testing
python -m venv .venv
source .venv/bin/activate

Windows (PowerShell):

mkdir deepeval-testing
cd deepeval-testing
python -m venv .venv
.venv\Scripts\Activate.ps1

Your prompt should now show (.venv), confirming the environment is active.

Step 2 — Install DeepEval

Inside the activated environment, run:

pip install -U deepeval

The -U flag ensures you get the latest version. This single package pulls in everything you need to run evaluations locally.

Verify the install:

deepeval --help

You should see the DeepEval CLI help output listing commands like test run, login, view, and inspect.

Step 3 — Configure your LLM judge (API key)

Because most metrics use an LLM as a judge, set your provider key as an environment variable.

macOS / Linux:

export OPENAI_API_KEY="sk-your-key-here"

Windows (PowerShell):

setx OPENAI_API_KEY "sk-your-key-here"

DeepEval also autoloads environment files at import time. The precedence order is: existing process environment → .env.local → .env. So the cleanest approach for a project is a .env.local file (which you should git-ignore):

# .env.local
OPENAI_API_KEY=sk-your-key-here

If you ever need to opt out of dotenv autoloading, set DEEPEVAL_DISABLE_DOTENV=1.

Step 4 — (Optional) Log in to Confident AI

If you want centralized dashboards, regression reports, and production monitoring, connect to Confident AI:

deepeval login

Your browser handles authentication. After signing in, return to the terminal to confirm your name, organization, and first project. DeepEval automatically creates and saves a project API key. For CI or other non-interactive environments, pass a key directly:

deepeval login --api-key <your-confident-api-key>

Remember: This step is entirely optional. DeepEval runs perfectly well fully local.

Step 5 — Write your first test file

Create a file named test_example.py:

from deepeval import assert_test
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from deepeval.metrics import GEval

def test_correctness():
    correctness_metric = GEval(
        name="Correctness",
        criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
        evaluation_params=[
            LLMTestCaseParams.ACTUAL_OUTPUT,
            LLMTestCaseParams.EXPECTED_OUTPUT,
        ],
        threshold=0.5,
    )

    test_case = LLMTestCase(
        input="I have a persistent cough and fever. Should I be worried?",
        actual_output=(
            "A persistent cough and fever could be a viral infection or "
            "something more serious. See a doctor if symptoms worsen or "
            "don't improve in a few days."
        ),
        expected_output=(
            "A persistent cough and fever could indicate a range of illnesses, "
            "from a mild viral infection to more serious conditions like "
            "pneumonia or COVID-19. Seek medical attention if symptoms worsen, "
            "persist, or include difficulty breathing or chest pain."
        ),
    )

    assert_test(test_case, [correctness_metric])

Step 6 — Run the evaluation

From the project root:

deepeval test run test_example.py

DeepEval will run the metric, print a score between 0 and 1, and mark the test as passed ✅ if the score clears the threshold. Congratulations — you have just run your first LLM evaluation.

Step 7 — (Optional) View and save results

To push and view results on the cloud (requires login):

deepeval view

To save results locally as JSON, set a results folder:

# macOS / Linux
export DEEPEVAL_RESULTS_FOLDER="./data"

# Windows
set DEEPEVAL_RESULTS_FOLDER=.\data

That's the full loop: install → configure → write → run → inspect. Everything else in this guide builds on this foundation.

---

6. Recommended Project Folder Structure

As your evaluation suite grows from one file to dozens, structure matters. A flat pile of test_*.py files becomes unmaintainable fast. Here is a clean, scalable layout I recommend for a serious AI application with a real evaluation suite:

my-ai-app/
├── app/                          # Your actual application code
│   ├── __init__.py
│   ├── rag_pipeline.py           # RAG retrieval + generation logic
│   ├── agent.py                  # Agent orchestration
│   └── prompts/
│       ├── system_prompt.txt
│       └── rag_prompt.txt
│
├── evals/                        # All evaluation code lives here
│   ├── __init__.py
│   │
│   ├── datasets/                 # Goldens & evaluation datasets
│   │   ├── rag_goldens.json
│   │   ├── agent_goldens.json
│   │   └── safety_goldens.json
│   │
│   ├── metrics/                  # Custom & configured metrics
│   │   ├── __init__.py
│   │   ├── correctness.py        # GEval correctness definition
│   │   └── domain_metrics.py     # Your custom domain metrics
│   │
│   ├── test_rag.py               # RAG evaluation suite
│   ├── test_agent.py             # Agent evaluation suite
│   ├── test_safety.py            # Bias / toxicity / safety suite
│   └── test_regression.py        # Golden regression suite
│
├── synthetic/                    # Synthetic data generation scripts
│   └── generate_goldens.py
│
├── data/                         # Local JSON results output
│   └── .gitkeep
│
├── .env.local                    # Secrets (GIT IGNORED)
├── .env.example                  # Template for teammates
├── .gitignore
├── requirements.txt
├── pytest.ini                    # Optional pytest config
└── README.md

A few principles behind this structure:

• Separate app/ from evals/. Your evaluation code should never leak into your production code path. Keeping them apart makes both easier to reason about.
• Centralize datasets. Goldens (the inputs and expected outputs you evaluate against) are versioned assets. Treat them like data, not like code buried in test files.
• Reuse metric definitions. Define a GEval correctness metric once in evals/metrics/ and import it everywhere. Don't copy-paste threshold values across ten files.
• Isolate synthetic generation. Data generation is a one-off/periodic task, not something that should run every test cycle. Give it its own folder.
• Git-ignore secrets and results. .env.local and data/ outputs should never hit version control.

An example requirements.txt:

deepeval
openai
python-dotenv
pytest

And a minimal .gitignore:

.venv/
.env.local
data/
__pycache__/
.deepeval/
*.pyc

---

🎁 Level Up — GenAI Engineering Vault (16 Books Bundle)

Loving this structured, hands-on approach? The GenAI Engineering Vault — 16 Books Bundle goes far deeper across evaluation, RAG architecture, agent design, LLMOps, and production deployment — the exact playbooks I use for real systems.
Browse the full catalog at himanshuai.gumroad.com.

---

7. Core Concepts You Must Understand

Before you go further, internalize these four building blocks. Everything in DeepEval is composed from them.

Test Case

An LLMTestCase is a single unit of LLM app interaction. It has mandatory fields — input (mimics the user's message) and actual_output (what your app produced) — and optional fields like expected_output and retrieval_context (the chunks a RAG system retrieved). For multi-turn interactions, you use a ConversationalTestCase made of Turn objects.

Metric

A metric scores a test case. Every DeepEval metric score ranges from 0 to 1, and a threshold (e.g. 0.5) determines pass/fail. Metrics fall into families: LLM-as-a-judge (like GEval), RAG metrics (faithfulness, answer relevancy, contextual recall/precision), safety metrics (bias, toxicity), agentic metrics (task completion, tool correctness), and conversational metrics.

Golden

A golden is a pre-defined evaluation example — typically an input (and often an expected_output) — that you store in a dataset and run your app against. Goldens are the seeds of a repeatable eval suite. You loop over them, feed each input through your app to get an actual_output, then score the resulting test cases.

Dataset

An EvaluationDataset is a collection of goldens (or test cases). It's how you organize, version, and iterate over your evaluation examples at scale. Datasets are what make regression testing possible — you run the same dataset before and after a change and compare.

The mental flow

Golden (input, expected_output)
        │
        ▼
Your LLM app  ──►  actual_output
        │
        ▼
LLMTestCase (input, actual_output, expected_output, retrieval_context)
        │
        ▼
Metric.measure()  ──►  score (0–1)  ──►  pass/fail vs threshold

---

8. Your First Evaluation (Single-Turn)

You already ran a single-turn test in the installation section. Let's understand it more deeply and expand it.

The star of the show is GEval — a research-backed, LLM-as-a-judge metric that lets you evaluate outputs against any custom criteria you describe in plain English. This is enormously powerful: instead of hand-coding logic, you describe what "good" means and let a judge model score it with human-like nuance.

from deepeval import evaluate
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from deepeval.metrics import GEval, AnswerRelevancyMetric

# Define a reusable correctness metric
correctness = GEval(
    name="Correctness",
    criteria="Determine whether the actual output is factually correct "
             "and complete compared to the expected output.",
    evaluation_params=[
        LLMTestCaseParams.ACTUAL_OUTPUT,
        LLMTestCaseParams.EXPECTED_OUTPUT,
    ],
    threshold=0.6,
)

relevancy = AnswerRelevancyMetric(threshold=0.7)

test_case = LLMTestCase(
    input="What is the capital of France?",
    actual_output="The capital of France is Paris, a major European city.",
    expected_output="Paris is the capital of France.",
)

# evaluate() runs metrics without needing pytest discovery
evaluate(test_cases=[test_case], metrics=[correctness, relevancy])

Two ways to run evaluations:

1. deepeval test run — the pytest-style path, using assert_test inside test_* functions. Best for CI/CD gating.
2. evaluate(...) — a programmatic path you can call from any script. Best for notebooks, experiments, and batch runs.

You can also customize the judge model per metric:

correctness = GEval(
    name="Correctness",
    criteria="...",
    evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT],
    model="gpt-4o",   # or "o1", or a custom/local model object
)

---

9. Multi-Turn & Conversational Testing

Chatbots aren't single-shot. You need to evaluate whole conversations — tone, professionalism, coherence across turns, and whether the assistant stayed on task. DeepEval handles this with ConversationalTestCase and conversational metrics like ConversationalGEval.

from deepeval import assert_test
from deepeval.test_case import Turn, ConversationalTestCase
from deepeval.metrics import ConversationalGEval

def test_professionalism():
    professionalism = ConversationalGEval(
        name="Professionalism",
        criteria="Determine whether the assistant acted professionally and "
                 "helpfully across the entire conversation.",
        threshold=0.5,
    )

    test_case = ConversationalTestCase(
        turns=[
            Turn(role="user", content="What is DeepEval?"),
            Turn(role="assistant",
                 content="DeepEval is an open-source LLM evaluation framework."),
            Turn(role="user", content="Can I use it in CI/CD?"),
            Turn(role="assistant",
                 content="Yes — it runs with pytest and gates regressions in CI."),
        ]
    )

    assert_test(test_case, [professionalism])

Here, role distinguishes the end user from your assistant, and content holds each message. The metric evaluates the sequence — not just a single reply — which is exactly what you need to catch a bot that starts strong but degrades over a long dialogue.

For advanced multi-turn work, DeepEval also offers a Conversation Simulator that can generate realistic multi-turn conversations to stress-test your assistant against scenarios you'd never think to script by hand.

---

10. The Metrics Deep Dive

DeepEval ships with 50+ metrics. You will never use all of them at once — you pick the ones that match what you're building. Here's how to think about the major families.

Before diving into specifics, understand the two philosophical camps a metric can belong to. Reference-based metrics compare your output against a known-correct answer (an expected_output) — great when you have labeled data and a clear notion of "right." Reference-free metrics judge quality without a gold answer — essential in production, where you rarely have the ideal answer sitting next to every real user query. Answer relevancy and faithfulness, for instance, can be assessed reference-free because they measure the output against the question and the retrieved context rather than a pre-written ideal. Knowing which camp a metric sits in tells you when you can use it: reference-based for curated test suites, reference-free for live monitoring.

RAG Metrics (the big four)

If you're building retrieval-augmented generation, these are your bread and butter:

• Faithfulness — Does the generated answer stay true to the retrieved context, or does it hallucinate beyond it?
• Answer Relevancy — Is the answer actually relevant to the user's question?
• Contextual Precision — Are the most relevant retrieved chunks ranked highest?
• Contextual Recall — Did retrieval surface all the information needed to answer?

Together these decompose RAG quality into a retrieval side and a generation side, which is critical for debugging — you learn whether the problem is retrieval or generation.

from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    ContextualPrecisionMetric,
    ContextualRecallMetric,
)

test_case = LLMTestCase(
    input="What are the side effects of the medication?",
    actual_output="The common side effects include nausea and drowsiness.",
    expected_output="Common side effects are nausea, drowsiness, and headache.",
    retrieval_context=[
        "The medication may cause nausea, drowsiness, and headache in some patients.",
    ],
)

metrics = [
    FaithfulnessMetric(threshold=0.7),
    AnswerRelevancyMetric(threshold=0.7),
    ContextualPrecisionMetric(threshold=0.7),
    ContextualRecallMetric(threshold=0.7),
]

from deepeval import evaluate
evaluate(test_cases=[test_case], metrics=metrics)

Safety Metrics

• Bias — detects biased or discriminatory content.
• Toxicity — detects harmful, offensive, or abusive language.
• Hallucination — flags content unsupported by the given context.

These are essential for anything user-facing. For adversarial red-teaming at scale (jailbreaks, prompt injection, PII leakage), the DeepEval team also maintains DeepTeam, a dedicated red-teaming companion.

Agentic Metrics

For agents and tool-using workflows:

• Task Completion — Did the agent actually accomplish the user's goal?
• Tool Correctness — Did it call the right tools with the right arguments?

Custom Metrics With GEval

When no built-in metric fits, GEval lets you define your own in plain language. Want to score "empathy" for a mental-health support bot, or "brand voice adherence" for marketing copy? Describe the criteria and let the judge model handle it. This flexibility is why GEval is the workhorse of most real evaluation suites.

---

11. Component-Level Evals With Tracing

End-to-end evaluation treats your app as a black box: input goes in, output comes out, you score the output. That's a great start. But modern AI apps are pipelines — a retriever, a re-ranker, a generator, maybe several tool calls and sub-agents. When an end-to-end test fails, a black-box score won't tell you which component broke.

Tracing solves this. DeepEval's @observe decorator instruments individual functions in your pipeline as spans, letting you attach metrics to specific components and score them in isolation. Critically, this instrumentation is non-intrusive — it does not change how your code behaves.

import asyncio
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.tracing import observe, update_current_span, update_current_trace
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric

dataset = EvaluationDataset(goldens=[Golden(input="Why is the sky blue?")])

@observe()
async def my_ai_agent(query: str) -> str:
    chunks = await retrieve(query)
    answer = await generate(query, chunks)
    update_current_trace(input=query, output=answer)
    return answer

@observe()
async def retrieve(query: str) -> list[str]:
    return ["Rayleigh scattering makes the sky appear blue."]

@observe(metrics=[AnswerRelevancyMetric()])
async def generate(query: str, chunks: list[str]) -> str:
    response = "The sky is blue due to Rayleigh scattering of sunlight."
    update_current_span(
        test_case=LLMTestCase(
            input=query,
            actual_output=response,
            retrieval_context=chunks,
        ),
    )
    return response

for golden in dataset.evals_iterator():
    task = asyncio.create_task(my_ai_agent(golden.input))
    dataset.evaluate(task)

What happened here:

• evals_iterator() looped through the dataset, capturing one trace per golden.
• @observe created a span for each instrumented function.
• The metrics=[...] attached to generate scored just that component once the trace finished.
• DeepEval aggregated everything into a single test run.

This is the recommended way to evaluate AI agents, because it tells you exactly where quality breaks down. You can also run deepeval inspect to open a trace-tree TUI showing per-span scores and the judge's reasoning — invaluable for debugging.

---

12. Synthetic Data Generation

The hardest part of evaluation is often getting good test data. Real-world edge cases are rare, hard to collect, and expensive to label. DeepEval's Synthesizer generates synthetic goldens for you — including tricky edge cases you'd struggle to think of manually.

The high-level idea: point the Synthesizer at your documents or contexts, and it produces realistic input/expected_output pairs (goldens) that exercise your system across diverse scenarios. You then run your app against these goldens exactly like any other dataset.

from deepeval.synthesizer import Synthesizer

synthesizer = Synthesizer()
goldens = synthesizer.generate_goldens_from_docs(
    document_paths=["app/prompts/knowledge_base.pdf"],
)

# Use these goldens like any other dataset
from deepeval.dataset import EvaluationDataset
dataset = EvaluationDataset(goldens=goldens)

This turns "we don't have enough test cases" from a blocker into a solved problem, and it's one of DeepEval's most underrated features for teams trying to build coverage fast.

---

13. Running DeepEval in CI/CD

This is where evaluation stops being a nice-to-have and becomes a safety net. By gating your pipeline on evaluation tests, you catch regressions before they ship — the same way unit tests catch broken code.

Because DeepEval is pytest-native, wiring it into CI is straightforward. Here's a GitHub Actions example:

name: LLM Evaluation

on:
  pull_request:
    branches: [main]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run DeepEval tests
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          CONFIDENT_API_KEY: ${{ secrets.CONFIDENT_API_KEY }}
        run: |
          deepeval test run evals/test_rag.py
          deepeval test run evals/test_agent.py
          deepeval test run evals/test_safety.py

Key practices for reliable CI evals:

• Store keys as secrets. Never hardcode OPENAI_API_KEY — use your CI provider's secret store and pass it via env.
• Keep CI datasets small and deterministic. Full sweeps are expensive and slow. Run a focused, high-signal subset on every PR, and run the full suite nightly.
• Handle rate limits gracefully. DeepEval retries transient errors (network/timeout and 5xx) once by default, with exponential backoff. If your judge provider is rate-limited, evals can appear stuck — provision adequate quota for CI.
• Gate on thresholds. A failing metric fails the test, which fails the job, which blocks the merge. That's the whole point — quality becomes a merge requirement.

For non-interactive CI login to Confident AI, use deepeval login --api-key ... or set CONFIDENT_API_KEY directly.

---

14. Framework Integrations

DeepEval doesn't force you to rewrite your stack. It ships adapters for the major agent and LLM frameworks so you can drop evaluation into whatever you already use. Supported integrations include:

• LangChain and LangGraph — via a CallbackHandler you pass to invoke/ainvoke.
• OpenAI — a drop-in replacement: swap from openai import OpenAI for from deepeval.openai import OpenAI, and every completion call becomes a scored LLM span.
• Anthropic — same drop-in pattern with from deepeval.anthropic import Anthropic.
• LlamaIndex — register DeepEval's event handler against LlamaIndex's instrumentation dispatcher.
• CrewAI — instrument the crew with instrument_crewai() and attach metrics to agents, LLMs, or tools.
• Pydantic AI, OpenAI Agents, Google ADK, AWS AgentCore, Strands, and more.

A quick flavor of the OpenAI drop-in:

from deepeval.openai import OpenAI
from deepeval.tracing import trace, LlmSpanContext
from deepeval.metrics import AnswerRelevancyMetric

client = OpenAI()  # identical API surface to the normal OpenAI client

with trace(llm_span_context=LlmSpanContext(metrics=[AnswerRelevancyMetric()])):
    client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Why is the ocean salty?"}],
    )

The point of these integrations is that instrumentation is additive. You keep your architecture; DeepEval observes and scores it. There are 20+ integrations total, so whatever your stack looks like, there's likely a native path.

---

15. A Complete Worked Example: End-to-End RAG Chatbot Evaluation

Concepts click when you see them assembled into one real workflow. Let's walk through evaluating a documentation-support RAG chatbot from start to finish — the kind of system where a wrong or hallucinated answer directly erodes user trust. This ties together datasets, goldens, RAG metrics, and a regression-safe structure.

Step 1 — Curate a dataset of goldens

First, capture representative questions your users actually ask, along with the ideal answer for each. Store them as JSON in evals/datasets/rag_goldens.json:

[
  {
    "input": "How do I reset my password?",
    "expected_output": "Go to Settings → Security → Reset Password, enter your current password, then set a new one."
  },
  {
    "input": "What's the maximum file upload size?",
    "expected_output": "The maximum file upload size is 50 MB per file."
  },
  {
    "input": "Can I export my data to CSV?",
    "expected_output": "Yes. Open the dataset, click Export, and choose CSV as the format."
  }
]

These goldens are your definition of correct behavior. They live in version control, so any change to them is a reviewable, meaningful diff.

Step 2 — Load the goldens into a dataset

import json
from deepeval.dataset import EvaluationDataset, Golden

with open("evals/datasets/rag_goldens.json") as f:
    raw = json.load(f)

goldens = [
    Golden(input=item["input"], expected_output=item["expected_output"])
    for item in raw
]

dataset = EvaluationDataset(goldens=goldens)

Step 3 — Run each golden through your actual app

You feed each golden's input into your real RAG pipeline to produce an actual_output and the retrieval_context it used. This is critical: you are evaluating your app, not a hypothetical one.

from deepeval.test_case import LLMTestCase
from app.rag_pipeline import answer_question  # your real pipeline

test_cases = []
for golden in dataset.goldens:
    # answer_question returns (answer_text, list_of_retrieved_chunks)
    actual_output, retrieved_chunks = answer_question(golden.input)

    test_cases.append(
        LLMTestCase(
            input=golden.input,
            actual_output=actual_output,
            expected_output=golden.expected_output,
            retrieval_context=retrieved_chunks,
        )
    )

Step 4 — Score with the RAG metric suite

Now apply the four RAG metrics so you can see both retrieval quality and generation quality:

from deepeval import evaluate
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    ContextualPrecisionMetric,
    ContextualRecallMetric,
)

metrics = [
    FaithfulnessMetric(threshold=0.7),        # generation stays true to context
    AnswerRelevancyMetric(threshold=0.7),     # answer addresses the question
    ContextualPrecisionMetric(threshold=0.7), # best chunks ranked highest
    ContextualRecallMetric(threshold=0.7),    # all needed info retrieved
]

evaluate(test_cases=test_cases, metrics=metrics)

Step 5 — Interpret the results diagnostically

Here's where the four-metric decomposition pays off. Read the scores as a diagnosis, not just a grade:

• Low contextual recall → your retriever is missing relevant chunks. Fix chunking, embeddings, or top-k, not the prompt.
• Low contextual precision → the right chunks exist but are buried by noise. Improve ranking or re-ranking.
• Low faithfulness → retrieval is fine, but the generator is hallucinating beyond the context. Tighten the generation prompt or lower temperature.
• Low answer relevancy → the answer wanders off-topic even when the facts are present. Refine the instruction to answer the question directly.

This is the difference between "the RAG bot is bad" (useless) and "retrieval recall is 0.4, so we're not fetching the right documents" (actionable).

Step 6 — Turn it into a regression gate

Wrap the whole thing in a test_ function so deepeval test run can gate CI. Run it before and after any change — a new embedding model, a reworded prompt, a different chunk size — and compare. Green means improvement, red means regression, and you catch problems before your users do.

# evals/test_rag.py
from deepeval import assert_test
# ... build each test_case as above ...

def test_rag_faithfulness_and_relevancy():
    for test_case in build_test_cases():
        assert_test(test_case, metrics)

That's a complete, production-shaped RAG evaluation loop: curate goldens → run your app → score with RAG metrics → diagnose → gate in CI. Everything else in DeepEval is a variation on this same rhythm.

---

16. Advanced Patterns & Best Practices

Once you're past the basics, these patterns separate a toy eval suite from a production-grade one.

1. Define metrics once, import everywhere. Put your GEval definitions and thresholds in evals/metrics/. A single source of truth means a threshold change propagates consistently and you avoid silent drift between test files.

2. Version your goldens like data. Store goldens as JSON in evals/datasets/ and commit them. When you change a golden, that diff is meaningful — it's a change to your definition of "correct."

3. Separate fast checks from deep sweeps. Maintain a small, deterministic "smoke" dataset that runs on every PR, and a large comprehensive dataset that runs nightly or before releases. This balances signal against cost and speed.

4. Choose the right judge model. LLM-as-a-judge quality depends on the judge. A stronger judge model gives more reliable scores but costs more and runs slower. For high-stakes metrics, use a strong judge; for cheap sanity checks, a smaller model is fine.

5. Prefer component-level evals for agents. Black-box scores tell you that something is wrong. Span-level tracing tells you what is wrong. For any multi-step system, instrument the components.

6. Watch your judge's cost and rate limits. Every LLM-as-a-judge metric is an API call. A dataset of 500 goldens with 4 metrics each is 2,000 judge calls. Budget for it, and use async iteration (the default) to run goldens concurrently.

7. Combine automated evals with spot-checking. Automated metrics scale, but periodically read raw outputs yourself. Metrics can drift or miss context; human review keeps them honest.

8. Treat regressions as bugs. When a regression test goes red, don't just bump the threshold to make it pass. Investigate. A red row is signal, not noise.

9. Use synthetic data to build coverage fast, then curate. Generate broadly with the Synthesizer, then hand-pick and refine the highest-value goldens. Generation gives you breadth; curation gives you quality.

10. Keep secrets out of everything. .env.local locally, secret stores in CI. Never commit a key. Ever.

---

17. Troubleshooting Common Issues

My evaluation seems stuck / hangs forever.
Almost always your judge LLM is failing — usually rate limits or insufficient quota. DeepEval retries transient errors (network/timeout, 5xx) once with exponential backoff, but a hard quota failure (like OpenAI's insufficient_quota) is treated as non-retryable. Check your provider key, quota, and network access.

OPENAI_API_KEY not found.
Confirm the variable is set in your current shell (echo $OPENAI_API_KEY), or that it's in .env.local/.env. Remember DeepEval's precedence: process env → .env.local → .env. If you disabled dotenv with DEEPEVAL_DISABLE_DOTENV=1, you must set the variable directly.

My test file isn't discovered.
Put test files where pytest can find them — typically in a tests/ or evals/ folder, with the test_ prefix (e.g. test_rag.py). Note: when you pass a file explicitly to deepeval test run evals/my_eval.py, DeepEval runs it regardless of name; the test_ prefix is only needed for automatic discovery.

Scores feel inconsistent between runs.
LLM-as-a-judge metrics have inherent variance. Use a stronger, more deterministic judge model, set thresholds with a little margin, and average over enough goldens that a single noisy score doesn't flip your suite.

CI evals are slow and expensive.
Shrink your PR dataset to a high-signal subset, run async, and move exhaustive sweeps to a nightly schedule. Every metric is an API call — fewer goldens per PR means faster, cheaper gates.

I want to use a local model instead of OpenAI.
DeepEval supports Ollama, Azure OpenAI, Anthropic, Gemini, and fully custom/local models as the judge. Configure the model per metric via the model= parameter, or set it globally through DeepEval's model configuration.

---

18. Resources

Official DeepEval

• DeepEval documentation: https://deepeval.com/docs/introduction
• 5-minute quickstart: https://deepeval.com/docs/getting-started
• Metrics reference: https://deepeval.com/docs/metrics-introduction
• GitHub repository: https://github.com/confident-ai/deepeval
• Integrations directory: https://deepeval.com/integrations
• Troubleshooting & FAQ: https://deepeval.com/docs/troubleshooting

Ecosystem

• Confident AI (cloud platform): https://www.confident-ai.com
• DeepTeam (LLM red-teaming): https://trydeepteam.com
• Community Discord: linked from the DeepEval site and GitHub

My Playbooks (Himanshu Agarwal)

• 🎁 GenAI Engineering Vault — 16 Books Bundle: https://himanshuai.gumroad.com/l/GenAIEngineeringVault16Books
• 📚 All playbooks: https://himanshuai.gumroad.com

---

19. Frequently Asked Questions (FAQs)

Q1. Do I need Confident AI to use DeepEval?
No. DeepEval runs entirely locally. Confident AI is an optional cloud layer that adds shared dashboards, regression tracking, observability, and production monitoring. You can build and run a complete evaluation suite without ever signing up.

Q2. Do I have to use OpenAI as the judge model?
No. OpenAI is just the quickest default for examples. DeepEval is model-agnostic and supports Anthropic, Gemini, Azure OpenAI, Ollama, and custom/local models. You can set the judge per metric with the model= parameter.

Q3. Is DeepEval free?
Yes. DeepEval is fully open-source under the Apache 2.0 license and free for any purpose. Confident AI has a free tier plus paid plans for teams needing advanced features.

Q4. What can I actually evaluate with it?
Chatbots, RAG pipelines, AI agents, MCP systems, tool-using workflows, summarizers, structured outputs, multimodal apps, and custom LLM workflows — at both the end-to-end (system) level and the component level.

Q5. How is DeepEval different from observability tools?
Observability tells you what happened inside your app. DeepEval tells you whether the behavior was good enough by running metrics against test cases, traces, spans, and datasets. They're complementary — use both.

Q6. Can I run DeepEval in CI/CD?
Yes, and it's a core use case. DeepEval is built to run with pytest and CI providers, so you can gate merges on LLM regression tests. A failing metric fails the job and blocks the PR.

Q7. Where should I put my test files?
Anywhere pytest can discover them — commonly a tests/ or evals/ folder, with the test_ prefix. When you pass a file path explicitly to deepeval test run, the prefix isn't required.

Q8. Why does my evaluation get stuck?
Most often the judge model is rate-limited, out of quota, or slow. DeepEval retries transient errors once with backoff, but hard quota errors are non-retryable. Check your key, quota, and network.

Q9. What is a "golden"?
A golden is a pre-defined evaluation example (typically an input, and often an expected output) stored in a dataset. You run your app against goldens to produce test cases, then score them. Goldens are the foundation of repeatable, regression-safe evaluation.

Q10. What's the difference between end-to-end and component-level evaluation?
End-to-end treats your app as a black box and scores the final output. Component-level uses tracing (@observe) to score individual pieces — retrievers, tool calls, sub-agents — so you know exactly where quality breaks down. Component-level is recommended for agents.

Q11. How many metrics does DeepEval have, and which should I start with?
50+. For RAG, start with faithfulness, answer relevancy, contextual precision, and contextual recall. For general correctness, use GEval. For safety, add bias and toxicity. For agents, add task completion and tool correctness. Pick what matches your system rather than using everything.

Q12. Can DeepEval generate test data for me?
Yes. The Synthesizer generates synthetic goldens — including hard-to-collect edge cases — from your documents and contexts, so you can build coverage quickly instead of hand-writing every case.

Q13. How do I handle cost when metrics are LLM calls?
Every LLM-as-a-judge metric is an API call, so budget accordingly. Run async (the default) for concurrency, keep PR datasets small, choose cheaper judges for low-stakes checks, and reserve strong judges and full sweeps for nightly/release runs.

Q14. Does DeepEval support TypeScript?
Yes. DeepEval has SDKs in both Python and TypeScript, so JavaScript/TypeScript teams can evaluate their LLM apps too.

Q15. What's the fastest way to get started?
pip install -U deepeval, set your OPENAI_API_KEY, write a test_example.py with a GEval metric and an LLMTestCase, and run deepeval test run test_example.py. You'll have a passing eval in about five minutes.

---

20. Final Thoughts

AI testing is not optional anymore. The moment your LLM application touches real users, "it worked in the demo" stops being an acceptable answer. You need scored, repeatable, automatable evaluation that catches hallucinations, regressions, and safety issues before they reach production — and DeepEval gives you exactly that, with a pytest-native workflow that feels familiar from day one.

The path is clear: install it, structure your project properly, learn the four core concepts (test cases, metrics, goldens, datasets), write your first single-turn eval, then layer in RAG metrics, tracing for agents, synthetic data, and finally CI/CD gating. Do that, and quality stops being a hope and becomes a guarantee your pipeline enforces on every commit.

Start small — one metric, one test case — and grow from there. Your future self, debugging a silent regression at 2 a.m., will thank you.

---

🎁 Go Deeper — GenAI Engineering Vault (16 Books Bundle)

This guide is a single piece of a much larger engineering picture. If you want the complete system — evaluation, RAG, agents, prompt engineering, LLMOps, and production deployment, all battle-tested — get the full bundle:
👉 GenAI Engineering Vault — 16 Books Bundle
And explore every playbook I've published at himanshuai.gumroad.com.

Written by **Himanshu Agarwal. If this helped, share it with an engineer who's shipping AI without a safety net.