DEV Community

Cover image for AI Evaluation Series (06): DeepEval in Practice — Enterprise Agent Evaluation Suite
WonderLab
WonderLab

Posted on

AI Evaluation Series (06): DeepEval in Practice — Enterprise Agent Evaluation Suite

#ai #ragas #deepeval #agents

DeepEval and RAGAS Solve Different Problems

Article 04 used RAGAS to batch-evaluate a RAG system, producing four metric scores. This article uses DeepEval on the same Agent — with a completely different paradigm.

RAGAS paradigm: batch evaluation, DataFrame input, outputs per-metric averages. Good for analyzing quality trends and comparing system versions.

DeepEval paradigm: test-case-first, each case has an explicit Pass/Fail verdict, native pytest integration. Good for CI quality gates — giving a clear go/no-go before each merge.

RAGAS answers "how was system quality this week?" DeepEval answers "can this commit go to production?" Different questions, different tools.

Connecting a Custom LLM

DeepEval defaults to OpenAI as its Judge LLM. To use glm-4-flash, subclass DeepEvalBaseLLM:

from deepeval.models.base_model import DeepEvalBaseLLM

class GlmFlashEval(DeepEvalBaseLLM):
    def __init__(self):
        self._llm = ChatOpenAI(
            model="glm-4-flash",
            api_key=os.environ["LLM_API_KEY"],
            base_url="https://open.bigmodel.cn/api/paas/v4",
            temperature=0.0,
        )

    def load_model(self): return self._llm

    def generate(self, prompt: str, *args, **kwargs) -> str:
        return str(self._llm.invoke([HumanMessage(content=prompt)]).content)

    async def a_generate(self, prompt: str, *args, **kwargs) -> str:
        return self.generate(prompt)

    def get_model_name(self) -> str:
        return "glm-4-flash"

judge_llm = GlmFlashEval()
Enter fullscreen mode Exit fullscreen mode

Pass judge_llm when instantiating each metric:

AnswerRelevancyMetric(threshold=0.7, model=judge_llm)
FaithfulnessMetric(threshold=0.7, model=judge_llm)
ToolCorrectnessMetric(model=judge_llm)
Enter fullscreen mode Exit fullscreen mode

Building Test Cases

DeepEval's unit of work is LLMTestCase. Each case holds:

from deepeval.test_case import LLMTestCase, ToolCall

case = LLMTestCase(
    input="What's your refund policy?",
    actual_output=answer,                           # Agent's actual response
    expected_tools=[ToolCall(name="search_faq")],   # expected tool sequence
    tools_called=[ToolCall(name="search_faq")],     # actual tool sequence
    retrieval_context=["Refund policy: full refund within 7 days..."],
)
Enter fullscreen mode Exit fullscreen mode

tools_called and expected_tools require ToolCall objects — not plain strings.

Results

Raw Results (5 test cases) 

Question                                AnsRel   Faith   ToolOK
────────────────────────────────────── ───────  ──────  ───────
What's your refund policy?              1.00 ✓   0.50 ✗   ✗
Did order ORD-001 ship?                 0.33 ✗   0.50 ✗   ✗
How much refund for ORD-004?            1.00 ✓   1.00 ✓   ✗
What payment methods do you support?    0.50 ✗   0.00 ✗   ✗
Bought 299¥ item 3 days ago, refund?    1.00 ✓   1.00 ✓   ✓

Aggregate:
  AnswerRelevancy   avg=0.767  pass_rate=60%
  Faithfulness      avg=0.600  pass_rate=40%
  ToolCorrectness   avg=0.200  pass_rate=20%
Enter fullscreen mode Exit fullscreen mode

Reading the Three Metrics

AnswerRelevancy (avg 0.767, 60% pass)

Q2 ("Did order ORD-001 ship?") scored 0.33. The Agent skipped get_order_status and produced a vague "Regarding your order..." non-answer. Low Answer Relevancy is downstream of tool triggering failure, not LLM generation quality.

Faithfulness (avg 0.600, 40% pass)

Q1 scored 0.50: the Agent answered from its own knowledge without calling a tool, and some details differed from what's in the FAQ database.

Q4 scored 0.00 — an extreme case: the Agent answered "we support WeChat Pay, Alipay, bank cards..." but retrieval_context was "No context retrieved" (no tool was called). The framework considers an answer completely unsupported when context is empty.

This 0.0 exposes an evaluation trap. When an Agent skips tool calls and answers directly, the context is empty, and Faithfulness scores zero — even if the factual content is correct. Faithfulness measures "does the answer go beyond the context," but that only makes sense when context exists.

ToolCorrectness (avg 0.200, 20% pass)

Only the last case passed — the one where the user provided amount and days directly, triggering calculate_refund. The other 4 cases: Agent didn't call the expected tools. ToolCorrectness is DeepEval's most distinctive advantage over RAGAS: it directly evaluates whether the tool call sequence matched what was expected.

CI Integration

DeepEval's strength is native pytest integration. In CI:

# tests/test_agent_quality.py
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase, ToolCall
from deepeval.metrics import AnswerRelevancyMetric, ToolCorrectnessMetric

@pytest.mark.parametrize("case", build_test_cases())
def test_agent_response(case):
    assert_test(case, metrics=[
        AnswerRelevancyMetric(threshold=0.7, model=judge_llm),
        ToolCorrectnessMetric(model=judge_llm),
    ])
Enter fullscreen mode Exit fullscreen mode 
# .github/workflows/eval.yml
name: Agent Quality Gate
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install deepeval
      - run: pytest tests/test_agent_quality.py -v
        # any test case below threshold → CI fails
Enter fullscreen mode Exit fullscreen mode

Threshold guidance:

  • Early development: 0.5 (loose gate, avoid blocking iteration)
  • Stable phase: 0.7 (standard requirement)
  • Critical paths: 0.85 (medical, financial, high-stakes)

RAGAS vs DeepEval: Complete Comparison 

Dimension          RAGAS                      DeepEval
─────────────────────────────────────────────────────────────────
Paradigm           Metric-first (batch)       Test-case-first (pytest)
Input format       Dataset (DataFrame)        LLMTestCase objects
Output             Score per metric           Pass/Fail + score + reason
CI integration     Needs wrapper              Native pytest, assert_test()
Tool eval          No built-in                ToolCorrectnessMetric
Custom metrics     Via custom scorers         Via BaseMetric subclass
Best for           Trend analysis, comparison CI gates, regression tests
─────────────────────────────────────────────────────────────────
Use RAGAS when:    Analyzing quality trends over time; comparing v1 vs v2
Use DeepEval when: Need clear pass/fail before PR merge; CI gate enforcement
Enter fullscreen mode Exit fullscreen mode

Use both, not one. They're complementary:

  • Per commit: DeepEval on 5-10 core cases (fast gate)
  • Weekly: RAGAS on 100+ samples (trend analysis)
  • Before release: both — RAGAS gives the trend picture, DeepEval gives the binary verdict

Summary

  1. ToolCorrectness at 20%: 4 of 5 cases, the Agent answered without calling tools — ToolCorrectness failed; this matches Article 05's 73% tool name accuracy; two frameworks surface the same problem from different angles
  2. The Faithfulness = 0 trap: when the Agent skips tools and answers directly, retrieval_context is empty, Faithfulness scores zero even if the answer is factually correct; Faithfulness only makes sense when context actually exists
  3. RAGAS for trends, DeepEval for gates: different paradigms, best used together — not competing

References

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

Top comments (0)