DEV Community

Cover image for AI Evaluation Series (07): Custom Benchmarks — From Business Scenarios to Evaluation Sets
WonderLab
WonderLab

Posted on

AI Evaluation Series (07): Custom Benchmarks — From Business Scenarios to Evaluation Sets

#ai #evaluation #benchmark #engineering

Three Situations Where Public Benchmarks Fall Short

MMLU, HELM, BIG-Bench measure general capability. Three situations require custom benchmarks:

Case 1: Domain too specialized

An automotive dealer's service knowledge base Q&A system gets questions like "What's the warranty period for the DSG transmission in the 2023 VW Magotan B8?" No public benchmark covers this — the domain expertise, customer phrasing, and answer format are all enterprise-specific.

Case 2: Data can't leave the organization

Medical, financial, and government scenarios contain sensitive information that can't be uploaded to third-party evaluation platforms. Benchmarks must be built and run internally.

Case 3: Continuous quality monitoring

Public benchmarks are static: run once, done. A custom benchmark adds new questions, updates ground truth when policies change, and tracks quality delta after every model or knowledge base update.

Construction Workflow

Step 1: Define Evaluation Scenarios

Before writing a single question, decide which user intents this benchmark covers. Don't try to cover everything.

Example scenario taxonomy for enterprise document Q&A:

# eval_scenarios.yaml
scenarios:
  - id: S01
    name: Fact Retrieval
    description: "User asks for a fact that exists explicitly in the documents"
    examples:
      - "What's the refund policy?"
      - "How long is the product warranty?"
    target_metric: Faithfulness (answer must stay within document content)

  - id: S02
    name: Process Guidance
    description: "User asks how to complete a specific action"
    examples:
      - "How do I request an invoice?"
      - "How do I change my delivery address?"
    target_metric: Answer Relevancy + Completeness

  - id: S03
    name: Comparison
    description: "User needs to compare multiple options"
    examples:
      - "What's the difference between standard and express shipping?"
      - "Gold vs Platinum membership?"
    target_metric: Context Recall (both options must be retrieved)

  - id: S04
    name: Boundary Testing
    description: "Questions with no answer in the knowledge base"
    examples:
      - "Do you accept cryptocurrency?" (not supported)
    target_metric: Rejection Rate (refuse, don't hallucinate)
Enter fullscreen mode Exit fullscreen mode

A practical benchmark covers 4-8 scenarios, 10-20 questions each, 50-150 total. Too few lacks statistical power; too many means high construction cost.

Step 2: Generate Questions

Method A: Manually written (most accurate, highest cost)

Pros: real user phrasing, no data contamination risk
Cons: slow, requires domain expert review
For: high-security scenarios, <50 questions
Enter fullscreen mode Exit fullscreen mode

Method B: LLM-generated + human review (recommended)

QUESTION_GEN_PROMPT = """Based on the following document, generate {n} test questions.

Requirements:
- Use natural conversational phrasing, simulating real users
- Cover these scenario types: {scenarios}
- Only ask questions the document can answer (except boundary tests)
- Don't repeat the same information point

Document:
{document}

Output one question per line."""
Enter fullscreen mode Exit fullscreen mode

After generation, humans do two things:

  1. Filter obvious errors (ambiguous, grammatical, duplicates)
  2. Spot-check 20%: verify the knowledge base actually contains an answer

Method C: Mine from production logs (most authentic)

If the system is live, real user queries are the most valuable source — these are questions users actually ask.

def sample_from_production_logs(logs, n=100):
    unique_queries = deduplicate(logs)
    # Stratified sample by scenario type
    return stratified_sample(unique_queries, n)
Enter fullscreen mode Exit fullscreen mode

Step 3: Prepare Ground Truth

Ground truth quality directly determines evaluation reliability.

Weak ground truth (too vague):

Question: What's the refund policy?
ground_truth: refunds are possible
Enter fullscreen mode Exit fullscreen mode

Strong ground truth (specific, aligned with document content):

Question: What's the refund policy?
ground_truth: Full refund within 7 days of purchase. 50% refund for 7-30 days.
              No refund after 30 days. Submit via order details page;
              processing takes 3-5 business days.
Enter fullscreen mode Exit fullscreen mode

Ground truth rules:

  • Quote directly from the document — don't paraphrase or summarize
  • Include specific numbers, proper nouns (these are the anchors Faithfulness evaluation checks)
  • If multiple correct answers exist, include all
  • For boundary tests, write "this information is not in the knowledge base"

Step 4: Difficulty Stratification

Stratification shows you where the system breaks down, giving more diagnostic value than a single pass rate.

Three difficulty levels:

Easy (should score near perfect):
  Answer exists in a single paragraph of a single document
  Example: question and answer are in the same chunk; direct retrieval hits

Medium (evaluates real capability):
  Requires integrating multiple paragraphs, or phrasing differs significantly
  Example: question uses "invoice request," document uses "billing certificate"

Hard (exposes bottlenecks):
  Cross-document reasoning, implicit inference, or edge cases
  Example: "I bought products A and B — is separate or combined refund better?"
Enter fullscreen mode Exit fullscreen mode

How to calibrate difficulty:

Run all questions through the system first, then assign difficulty based on actual pass rates — don't assign difficulty by intuition.

def calibrate_difficulty(eval_results: list[dict]) -> list[dict]:
    for result in eval_results:
        score = result["avg_score"]
        if score >= 0.85:
            result["difficulty"] = "Easy"
        elif score >= 0.60:
            result["difficulty"] = "Medium"
        else:
            result["difficulty"] = "Hard"
    return eval_results
Enter fullscreen mode Exit fullscreen mode

Step 5: Version Control

Evaluation sets need version control because:

  1. Prevents standard drift: if someone quietly fixes a ground_truth ("this was wrong, let me update it"), historical data becomes incomparable
  2. Tracks dataset evolution: which questions were added, removed, and why
  3. Enables rollback: run old evaluation set against old system to validate historical performance 
# eval_dataset_v1.2.yaml
metadata:
  version: "1.2.0"
  created_at: "2026-06-01"
  last_updated: "2026-07-01"
  total_questions: 52
  breakdown:
    easy: 20
    medium: 22
    hard: 10
  changelog:
    v1.2.0:
      - added 5 questions covering Apple Pay (new payment method)
      - updated ground_truth for Q023 (refund policy changed)
    v1.1.0:
      - calibrated difficulty from 1000-run baseline
      - removed 3 duplicate questions

questions:
  - id: Q001
    scenario: S01
    difficulty: Easy
    question: "What's your refund policy?"
    ground_truth: "Full refund within 7 days; 50% for 7-30 days; none after 30."
    added_in: "1.0.0"
    last_updated: "1.0.0"
Enter fullscreen mode Exit fullscreen mode

Version number rules:

MAJOR: scenario taxonomy changes (removing a category)
MINOR: new questions added, or ground_truth updated
PATCH: metadata, comments, formatting — no effect on evaluation results
Enter fullscreen mode Exit fullscreen mode

Evaluation Set Quality Checklist 

Question quality:
  □ Every question has a single unambiguous correct answer?
  □ Question phrasing is varied, not all "what is X?"
  □ No leading questions (answer isn't embedded in the question)?

Ground truth quality:
  □ Quoted directly from documents, not paraphrased?
  □ Contains specific numbers and proper nouns?
  □ Boundary tests explicitly say "not in knowledge base"?

Difficulty distribution:
  □ Easy questions are under 50%? (prevents inflated overall scores)
  □ Hard questions are at least 15%? (surface real problems)
  □ Difficulty assigned from actual pass rates, not intuition?

Coverage:
  □ Every scenario (S01-S04) has at least 5 questions?
  □ Boundary tests included?
  □ High-frequency production queries represented?
Enter fullscreen mode Exit fullscreen mode

Summary

  1. Public benchmarks test general capability; custom benchmarks test business scenarios: running MMLU on an enterprise Q&A system tells you nothing useful; 50-150 domain-specific questions reflect actual quality
  2. Difficulty stratification gives more diagnostic value than a single pass rate: 85% overall can be inflated by Easy questions; Hard question pass rate reveals where the system actually breaks
  3. Evaluation sets need version control: changing ground_truth without tracking the version makes historical comparisons meaningless

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

Top comments (0)