Why AI Systems Need Evaluation CI
Code CI prevents functional regression: someone changes A, B breaks, CI catches it immediately.
AI systems need the same protection. Three types of changes can quietly degrade quality without triggering any visible errors:
Prompt changes:
Modify system prompt to reduce token cost
→ Answer Relevancy may drop 15%
→ Without CI: users complain before anyone notices
Knowledge base updates:
Add new documents, update existing ones
→ Inconsistent chunking in new docs drops Context Recall
→ Without CI: weekly reports surface the problem days later
Model version updates:
LLM provider silently upgrades model weights
→ Output style shifts, Faithfulness drops
→ Without CI: only discoverable after enough bad samples accumulate
Two-Tier Strategy
Tier 1: Fast Evaluation (every commit)
Goal: fast (< 3 minutes), catch obvious quality regressions, not full coverage.
Design principles:
- 10-20 "golden cases" — the most important, most representative questions
- At least 1-2 cases per scenario category
- Prioritize edge cases that have historically failed
# fast_eval_cases.yaml
FAST_EVAL_CASES = [
{"id": "FE01", "scenario": "factual", "question": "What's the refund policy?", ...},
{"id": "FE02", "scenario": "factual", "question": "How long is the warranty?", ...},
{"id": "FE03", "scenario": "procedure", "question": "How do I request an invoice?", ...},
{"id": "FE04", "scenario": "procedure", "question": "How do I change my address?", ...},
{"id": "FE05", "scenario": "comparison", "question": "Gold vs Platinum membership?", ...},
{"id": "FE06", "scenario": "multi_hop", "question": "How much refund for ORD-004?", ...},
{"id": "FE07", "scenario": "edge", "question": "Do you accept Bitcoin?", ...},
# Historically difficult cases
{"id": "FE08", "scenario": "ambiguous", "question": "...", ...},
{"id": "FE09", "scenario": "multi_doc", "question": "...", ...},
{"id": "FE10", "scenario": "adversarial","question": "...", ...},
]
CI configuration:
# .github/workflows/fast_eval.yml
name: Fast Quality Gate
on:
pull_request:
paths:
- "prompts/**"
- "rag/**"
- "knowledge_base/**"
jobs:
fast-eval:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- run: pip install ragas deepeval
- name: Run fast evaluation
run: python eval/run_fast_eval.py --output fast_eval_result.json
- name: Check quality gates
run: python eval/check_gates.py fast_eval_result.json
# exits 1 on failure → blocks merge
- name: Comment on PR
if: always()
run: python eval/post_pr_comment.py fast_eval_result.json
Tier 2: Full Evaluation (scheduled or large change)
Goal: full coverage (100+ cases), trend tracking, release decision support.
# .github/workflows/full_eval.yml
name: Full Quality Evaluation
on:
schedule:
- cron: "0 2 * * 1" # every Monday at 2am
workflow_dispatch: # manual trigger supported
push:
branches: [main]
paths:
- "knowledge_base/**"
jobs:
full-eval:
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- uses: actions/checkout@v4
- run: pip install ragas deepeval
- run: python eval/run_full_eval.py --output full_eval_result.json
- run: python eval/compare_baseline.py full_eval_result.json baselines/latest.json
- name: Update baseline if on main
if: github.ref == 'refs/heads/main'
run: python eval/update_baseline.py full_eval_result.json
Quality Gate Design
What Blocks vs What Only Alerts
Not every metric drop should block a PR. If CI fails too often, teams start bypassing the gate.
# eval_gates.yaml
gates:
# Block PR (critical gates)
- metric: faithfulness
threshold: 0.70
mode: block
reason: "Hallucination risk — directly impacts user trust"
- metric: answer_relevancy
threshold: 0.65
mode: block
reason: "Off-topic answers — core function broken"
- metric: context_recall
threshold: 0.60
mode: block
reason: "Missing critical information — incomplete answers"
# Warn but don't block
- metric: context_precision
threshold: 0.70
mode: warn
reason: "Retrieval noise — quality impact but not critical"
- metric: tool_correctness
threshold: 0.60
mode: warn
reason: "Agent tool issues — needs attention but allows margin"
# Delta gates (based on change, not absolute value)
- metric: faithfulness
max_delta: -0.05
mode: block
reason: "This change significantly increased hallucination rate"
Delta gates are more sensitive than absolute gates:
def check_gates(current: dict, baseline: dict, gates: list) -> list[str]:
failures = []
for gate in gates:
metric = gate["metric"]
current_score = current[metric]
baseline_score = baseline.get(metric)
# Absolute threshold gate
if "threshold" in gate and current_score < gate["threshold"]:
failures.append(
f"[{gate['mode'].upper()}] {metric}: {current_score:.3f} < {gate['threshold']}"
)
# Delta gate
if "max_delta" in gate and baseline_score is not None:
delta = current_score - baseline_score
if delta < gate["max_delta"]:
failures.append(
f"[{gate['mode'].upper()}] {metric} dropped {delta:+.3f} "
f"(threshold: {gate['max_delta']:+.3f})"
)
return failures
Delta Reports: Show Reviewers Change, Not Just Score
PR evaluation comments shouldn't just say "Faithfulness: 0.82." A reviewer seeing that number has no context — was it higher last time? Lower? Did this change make things better or worse?
Good PR evaluation comment format:
## AI Quality Evaluation Report
Time: 2026-07-16 14:23 | Cases: 10 | Duration: 2.1 min
| Metric | Current | Baseline | Change | Status |
|--------|---------|----------|--------|--------|
| Faithfulness | 0.87 | 0.82 | **+0.05** | ✅ Improved |
| Answer Relevancy | 0.71 | 0.79 | **-0.08** | ⚠️ Warning |
| Context Recall | 0.88 | 0.84 | **+0.04** | ✅ Improved |
| Context Precision | 0.79 | 0.81 | -0.02 | ✅ Normal range |
| Tool Correctness | 0.60 | 0.73 | **-0.13** | 🚫 Blocked |
**Verdict: Tool Correctness dropped 0.13, exceeding threshold (-0.05). PR blocked.**
Failing case:
- FE06 (order refund query): expected get_order_status + calculate_refund,
actual: only calculate_refund called. Agent used the amount from the question
directly instead of fetching it from the order record.
Suggested fix: check whether this prompt change affected multi-step tool triggering logic.
Baseline Management
The baseline is the "last known good state." Poor baseline management makes gates useless.
Baseline update rules:
class BaselineManager:
def should_update_baseline(self, current: dict, previous: dict) -> bool:
# Rule 1: current results meaningfully better (avg +0.02 across metrics)
avg_delta = sum(current[k] - previous[k] for k in current) / len(current)
if avg_delta > 0.02:
return True
# Rule 2: baseline is stale (>30 days)
if baseline_age_days > 30:
return True
# Rule 3: explicit major version release (manual trigger only)
return False
def update_baseline(self, current: dict) -> None:
archive_current_baseline() # preserve history
save_as_new_baseline(current)
Don't do this:
- Update the baseline to make a failing CI pass ("the threshold is too strict, let me lower it")
- Store baseline without linking it to a code version tag (makes rollback impossible)
Full Evaluation Engineering Architecture
Code / Prompt / Knowledge Base Change
↓
Fast Eval (10 cases, < 3 min)
Pass → PR can merge
Fail → Block + PR comment with root cause
Weekly schedule
↓
Full Eval (100+ cases, ~1 hour)
↓ Compare with baseline (Delta report)
↓ Store in quality trend database
↓ Quality drop → notify team lead
↓ Consistent improvement → update baseline
Monthly
↓
Benchmark review
↓ Cases that always pass? Consider adding harder variants
↓ Cases that always fail? Check if it's a test problem or system problem
↓ New business scenarios to add?
↓ Update eval_dataset.yaml, bump version
Summary
- Two tiers solve the speed vs coverage tradeoff: fast evaluation (10 cases, 3 min) runs on every commit; full evaluation (100+ cases) runs weekly — different triggers, different purposes
- Delta gates are more sensitive than absolute gates: current score 0.82 is uninformative; this change caused a 0.08 drop is actionable
- PR comments should show change, not just score: "Tool Correctness dropped 0.13" points to action; "Tool Correctness: 0.60" does not
Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.
Find more useful knowledge and interesting products on my Homepage
Top comments (0)