What Traditional Software Testing Assumes
Traditional software testing rests on three assumptions:
- Deterministic output: same input gives same output; tests are reproducible
- Correctness is decidable: a function returns 42 — right or wrong, there's a clear standard
- Single dimension: code either passes the test or it doesn't; no "kind of passes"
AI applications break all three.
Three Fundamental Problems
Problem 1: Non-Deterministic Output
The same Prompt gets different responses from Claude today versus tomorrow. Setting temperature=0 reduces randomness but doesn't eliminate it. Model updates silently change outputs for the same input.
Engineering implication:
# Traditional test
assert calculator(2, 3) == 5 # runs 100 times, passes 100 times
# AI evaluation
score1 = judge(llm.invoke("Explain the Transformer")) # 4.2/5
score2 = judge(llm.invoke("Explain the Transformer")) # 3.8/5
# 0.4 point gap — normal variance or a problem?
Sample multiple times and average. Three to five samples is usually enough; cost is 3-5x. Don't judge quality from a single run.
Problem 2: Subjective Quality
"Explain what a Transformer is" has countless correct answers. An explanation for a 10-year-old and one for an ML engineer can both be "good answers" while being completely different.
Subjectivity shows up as:
Same question, different good answers for different audiences
→ Depth: beginner vs expert
→ Style: analogy vs equations
→ Length: 100 words vs 2000
Same answer, different scores for different goals
→ Goal is "quick understanding" → short, clear answer scores high
→ Goal is "deep learning" → comprehensive answer scores high
Fix: define the scenario and audience before evaluating. Convert subjective judgment into a concrete scoring Rubric so LLM-as-Judge has clear criteria to follow.
Problem 3: Multi-Dimensional Quality
"AI responses should be good" unpacks into multiple independent dimensions that a single score can't capture:
Accuracy: content matches facts, no hallucinations
Relevance: the response addresses the user's actual question
Completeness: covers the core aspects of the question
Clarity: well-organized, easy to follow
Helpfulness: the user can actually act on this response
A response can be accurate but incomplete, or complete but unclear. Compressing 5 dimensions into 1 number loses too much information.
Four Evaluation Methods
No evaluation method works for every situation. Each carries costs. The choice depends on the scenario and budget.
Method Accuracy Speed Cost Best for
──────────────────────────────────────────────────────────────
Human evaluation Highest Slow (days) High Baseline setup, periodic sampling
Rule/auto metrics Medium Fast (ms) Low Format checks, keyword recall
LLM-as-Judge High Medium (min) Med Main workhorse; needs bias mitigation
A/B testing High Slow (weeks) High Production with real user behavior
Human Evaluation
Strengths: Highest accuracy, captures subtle quality differences, serves as the baseline for all other methods.
Weaknesses: Slow, expensive, doesn't scale. An evaluator reviews 50-100 samples per day; 1000 samples takes two weeks.
How to use: Build a "gold test set" for each critical scenario (about 100 samples, human-labeled), and use it as the reference standard. Validate automatic evaluation methods by comparing their results against the gold set.
Rule / Auto Metrics
Strengths: Millisecond speed, near-zero cost, easy to integrate into CI.
Weaknesses: Limited coverage; can't assess semantic quality. BLEU/ROUGE measure surface-level word overlap, not whether the meaning is correct.
# Useful automatic metrics
def check_format(output: str, schema: dict) -> bool:
"""Does the output conform to the JSON Schema?"""
def check_length(output: str, min_words: int, max_words: int) -> bool:
"""Is the length in the expected range?"""
def check_no_hallucination_markers(output: str) -> bool:
"""Does the output contain known hallucination phrases?"""
# Not useful for semantic quality
rouge_score = rouge.compute(predictions=[output], references=[reference])
# ROUGE penalizes semantically equivalent rephrasings and rewards verbatim repetition
How to use: Layer 2 structural checks — format compliance, length validation, prohibited phrase filtering. Don't use for semantic quality judgment.
LLM-as-Judge
Strengths: Scales well, understands semantics, evaluates quality dimensions that require reasoning.
Weaknesses: Susceptible to biases (covered in detail in Article 03), costs about 1/10th of human evaluation but 100x more than rule methods.
JUDGE_PROMPT = """Evaluate the following AI response (1-5 per dimension):
Dimensions:
1. Accuracy: Is the content factually correct?
2. Relevance: Does it address the user's question?
3. Clarity: Is it easy to understand?
User question: {question}
AI response: {answer}
Return as JSON: {{"accuracy": int, "relevance": int, "clarity": int}}"""
How to use: The main method for semantic quality dimensions that automatic metrics can't cover.
A/B Testing
Strengths: Tests actual user behavior, closest to business value. User clicks, adoption, and conversion are more trustworthy than model self-assessment.
Weaknesses: Requires sufficient traffic (statistical significance typically needs hundreds of exposures), takes weeks, and requires online experiment infrastructure.
How to use: Validate large changes (new model version, Prompt rewrite) for business impact. Not suitable for rapid iteration during development.
Combining Methods by Layer
In practice, the three approaches compose by layer:
Development (rapid iteration):
Rule checks (automatic) + LLM-as-Judge (sampling)
→ CI runs format checks on every commit
→ Daily sampling quality evaluation
Release (quality gate):
LLM-as-Judge (full or large sample) + human spot-check of critical cases
→ Release only when quality score exceeds threshold
Production (continuous monitoring):
A/B testing + real-time rule checks + periodic human sampling
→ Monitor quality drift, surface systemic problems
Summary
- The determinism trap: traditional "assert" thinking doesn't work for AI evaluation — multiple samples with statistics is more reliable than a single assertion
- Subjectivity isn't a blocker: subjective quality becomes operational through clear scenario definitions, audience specifications, and scoring rubrics
- No single best method: rules are fast but narrow in coverage; human evaluation is accurate but expensive; LLM-as-Judge is the best cost-effectiveness ratio for the main workload; A/B testing provides the final commercial validation
The next article goes into practice: how to decompose the vague goal "AI responses should be good" into a measurable L1/L2/L3 three-layer metric system.
Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.
Find more useful knowledge and interesting products on my Homepage
Top comments (0)