What Happens Without Metrics
A RAG Q&A system launches. The engineers say "all tests passed" — API response time under 2 seconds, correct format, no crashes.
Two weeks later, users report "the AI often doesn't answer the actual question." Investigation reveals: retrieval recall is only 40%. More than half the user questions can't find relevant documents.
The problem was there before launch. No one had designed a "retrieval quality" metric, so no one saw it.
The L1/L2/L3 Framework
Metrics organize into three layers, each addressing a different class of question:
L1 — Business Outcome
The end goal: is the AI system creating value?
Examples: task completion rate, user adoption rate, user satisfaction
L2 — Output Quality
The middle layer: how good is the AI's actual output?
Examples: accuracy, relevance, completeness
L3 — System Health
The foundation: is the system running stably?
Examples: response latency, token cost, failure rate
Layer dependencies:
L3 fails → L2 degrades (timeouts cause truncated output) → L1 drops (tasks fail)
L3 healthy, L2 poor → L1 still low (users don't adopt low-quality output)
All three healthy → L1 reflects real value
Start diagnosing from L3 upward; reverse from L1 downward is much slower.
Metric Selection by Scenario
Scenario 1: Document Q&A (RAG)
L3 System health:
Response latency P90 → < 5s
Token cost per query → < 3000
Retrieval failure rate → < 1%
L2 Output quality:
Faithfulness → no hallucinations beyond retrieved content (RAGAS)
Answer Relevancy → actually addresses the question (RAGAS)
Context Precision → fraction of retrieved content that's useful (RAGAS)
Context Recall → fraction of relevant content that was retrieved (RAGAS)
L1 Business outcome:
Task completion rate → fraction of users who mark "question resolved"
Adoption rate → fraction of users who copy/cite the AI response
The critical metric: Context Recall — the most commonly skipped and most important RAG metric. If retrieval doesn't find the relevant content, generation quality is irrelevant.
Scenario 2: Code Generation
L3:
Response latency P90 → < 10s (code generation is slower; allow more)
Excessive output rate → avoid generating far more code than needed
L2:
Syntax correctness → can the output be parsed? (automatable)
Test pass rate → does the output pass unit tests? (automatable)
Usability → how much editing is needed? (human evaluation)
Security scan pass → does the output contain known vulnerabilities? (automatable)
L1:
Adoption rate → fraction of suggestions the user accepts
Edit distance → how much the user changed the suggestion (less = better)
The critical metric: Test pass rate — fully automatable, measurable per commit, a natural fit for CI integration.
Scenario 3: Document Summarization
L2:
Faithfulness → summary adds nothing not in the source (most important)
Coverage → source's key points appear in the summary
Conciseness → summary is noticeably shorter than source (otherwise no value)
Readability → LLM-as-Judge evaluation
The critical metric: Faithfulness. Summarization's greatest risk is "adding" information that wasn't in the source (hallucination) — more harmful than omitting a detail.
Scenario 4: Agent Task Completion
L2:
Tool call accuracy → correct tool selected, correct arguments (automatable)
Step efficiency → how many steps to complete the task (fewer is better)
Trajectory quality → is the reasoning path sound? (LLM-as-Judge)
L1:
Task completion rate → did the task get done? (the core metric)
First-run resolution → fraction completed without follow-up questions
The critical metric: Task completion rate, but define "completed" carefully — "artifact exists" versus "artifact meets quality standard" are different requirements.
Three Traps
Trap 1: Measuring Only L3
✗ Wrong:
"API returned 200, latency 1.2s, this version can go to production."
Problem: L3 health doesn't mean business value. The RAG example above — L3
completely normal, Context Recall at 40%.
Every release needs at least one L2 sampling evaluation.
Trap 2: Using BLEU/ROUGE for Semantic Quality
# Looks reasonable
rouge_score = rouge.compute(predictions=[output], references=[reference])
# The problem:
reference = "The capital of France is Paris."
output_1 = "Paris is the capital city of France." # same meaning, low ROUGE
output_2 = "The capital of France is Paris, the capital." # repetition, high ROUGE
# ROUGE scores output_2 higher, but output_2 is clearly worse
BLEU/ROUGE measure word overlap, not semantic correctness. For generative outputs, use LLM-as-Judge instead. BLEU/ROUGE only work for tasks with standard reference answers, like machine translation.
Trap 3: Setting Thresholds by Instinct
✗ Wrong:
"Let's set Faithfulness > 0.8 as the quality gate."
(Why 0.8? It sounds reasonable.)
✓ Right:
Step 1: Run 100 samples, get the current baseline (e.g., mean = 0.73)
Step 2: Manually inspect samples with Faithfulness < 0.6 — are they acceptable?
Step 3: Set threshold based on that inspection (e.g., 0.65)
Step 4: Document why samples below that threshold are unacceptable
Thresholds come from data distributions and business acceptability, not from alignment with round numbers.
Complete Metric Spec: Enterprise Document Q&A System
# eval_metrics.yaml
system: document-qa
version: "1.0"
metrics:
l3_system_health:
- name: response_latency_p90
target: "< 5000ms"
collection: trace_log
alert_threshold: 8000ms
- name: token_cost_per_query
target: "< 2000 tokens"
collection: llm_callback
alert_threshold: 4000
- name: retrieval_error_rate
target: "< 1%"
collection: error_log
l2_output_quality:
- name: faithfulness
tool: ragas
target: "> 0.80"
collection: weekly_sampling (n=100)
alert_threshold: 0.70
- name: answer_relevancy
tool: ragas
target: "> 0.75"
collection: weekly_sampling
- name: context_recall
tool: ragas
target: "> 0.70"
collection: weekly_sampling
note: "Most important retrieval quality metric"
- name: format_compliance
tool: rule_check
target: "100%"
collection: every_request
l1_business_outcome:
- name: task_completion_rate
target: "> 70%"
collection: user_feedback_widget
note: "Fraction of users clicking 'this solved my question'"
- name: adoption_rate
target: "> 50%"
collection: behavior_tracking
note: "Fraction of users who copy/cite the AI response"
Roadmap to a Working Metric System
Step 1 (no tools needed, do now):
□ Write down the system's core business goal (what is L1?)
□ List 3-5 most important L2 output quality dimensions
□ Confirm L3 basics are monitored (latency, error rate)
Step 2 (within 1 week):
□ Manually evaluate 50 real use cases to get baseline numbers per L2 metric
□ Set thresholds based on that baseline — not on intuition
Step 3 (ongoing):
□ Run L2 sampling eval before each release; compare to baseline
□ Check L1 data monthly; verify it moves consistently with L2
Summary
- L1/L2/L3 each have a distinct role: L3 monitors system stability, L2 evaluates output quality, L1 validates business value — missing any one layer creates a blind spot
- Scenario determines critical metrics: RAG systems need Context Recall above all; code generation needs test pass rate; Agents need task completion rate — copying another system's metrics doesn't mean they fit yours
- Thresholds come from data: run a baseline first, then set the threshold — don't set a threshold and then justify it with data afterward
Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.
Find more useful knowledge and interesting products on my Homepage
Top comments (0)