LLM applications fail silently at the semantic level. Standard unit tests verify that functions return values, but they cannot detect if the output is factually wrong, off-tone, or missing steps. Evals fix this by running a prompt, inspecting the output, and programmatically or judgmentally deciding its quality.
Why LLM Testing is Different
Unit tests break on non-determinism. Temperature-driven randomness means identical inputs produce varying valid outputs, which defeats static equality checks. Semantic equivalence ("Paris is the capital" vs. "Capital is Paris") makes string matching useless. Small model or prompt updates cause gradual quality drift that only surfaces at scale. Tone and accuracy require judgment that basic assertions cannot encode.
The Three Eval Types
1. Heuristic Evals check measurable properties: JSON validity, word counts, PII presence. They are fast and objective — ideal for regression gates on every pull request.
2. LLM-as-Judge uses a second LLM to grade output against a rubric. Best for subjective content too complex for code but too vast for manual review.
3. Human Evals are the ground truth. Use them to build your golden dataset, calibrate the LLM judge, and finalize decisions before major releases.
Building Your Eval Infrastructure
Start minimal: a Python script, a JSON file of test cases, and a loop comparing LLM output against assertions or a rubric. Curate a golden dataset of 500–1000 production-representative requests including edge cases and adversarial prompts. Refresh it quarterly to match user behavior drift.
Automate the harness in GitHub Actions. Block deployments when pass rates drop below your threshold (90% is a common starting point). Run fast heuristic tests on every PR and full LLM-judge evals nightly.
Key Tools
- PromptFoo — Open-source, YAML-based test cases with a built-in CLI runner and diff reporting.
- Braintrust — Hosted platform for dataset management and experimental comparison.
- Inspect — Open-source framework by the UK AI Safety Institute, optimized for rigorous benchmarking.
Best Practices
- Optimize for your data, not public benchmarks. MMLU scores do not reflect your app's production requirements.
- Prioritize coverage over perfection. A broad diverse dataset at 85% pass rate beats a narrow one at 99%.
- Account for provider drift. Anthropic and OpenAI silently update underlying models — run recurring full evaluations even without code changes.
Minimal Starting Path
Pull 100 real requests from production. Manually review 50 to build a ground-truth baseline. Script a basic test runner. Implement one heuristic check. This gives you more safety than zero automated testing and a foundation to build on.
References
- LLM Evals in 2026: A Practical Guide to Testing Your AI App: https://devtoollab.com/blog/llm-evals-guide-2026
- PromptFoo — https://promptfoo.dev
- Braintrust — https://braintrust.dev
- Inspect (UK AISI) — https://inspect.ai.uk.gov
Top comments (1)
Strong starting path. One addition that catches a lot of “green” pipelines: don’t gate only on the aggregate pass rate. Slice results by task type, risk level, language, input length, and known failure mode, then require a floor for each critical slice. A 92% overall score can hide a severe regression in the 3% of requests that trigger tool use or contain PII.
For LLM-as-judge, I’d also version the judge model, rubric, prompt, and dataset together, and periodically measure agreement against a blinded human sample. Pairwise comparisons plus confidence intervals are often more stable than absolute 1–5 scores. That turns the eval suite into a change detector, not just a dashboard.