Every LLM project hits the same wall: you change a prompt, the output "seems better," you ship it — and you have no idea if you actually improved anything or just moved the failures somewhere you didn't look. "It looks good" is not a metric.
Evaluation is the least exciting and most decisive part of building with LLMs. The teams that ship reliable products are the ones that figured out how to measure quality instead of eyeballing it. Here's how.
Why evaluating LLMs is genuinely hard
Traditional software has deterministic tests: given input X, assert output Y. LLMs break this. The output is open-ended, non-deterministic, and often has many valid forms — there's no single correct string to assert against. "Summarize this article" has a thousand good answers and no exact key. That's why the instinct is to just read a few outputs and vibe-check them — and why that instinct doesn't scale past a demo.
Building real evaluation
You climb from anecdote to measurement in a few steps:
1. Build an eval set. Collect a fixed set of representative inputs — including the weird edge cases and past failures. This is your test suite. The moment you have it, "did this change help?" becomes answerable instead of a feeling.
2. Decide what "good" means, concretely. Different tasks need different measures. For tasks with a right answer (classification, extraction), you can score exact or near-exact matches. For open-ended tasks, you define criteria: is it grounded in the source? Is it relevant? Is it in the right format? Vague quality becomes specific, checkable properties.
3. Use the right scorer. Some things you can check with code — is it valid JSON, does it contain the required field, is the number correct. For subtler qualities, a growing practice is LLM-as-judge: using a strong model, with a careful rubric, to score outputs at scale. It's imperfect and needs validation against human judgment, but it's how you evaluate thousands of open-ended responses without reading them all by hand.
4. Watch for regressions. Run the eval set on every meaningful change. That "small prompt tweak" that fixed one case may have broken three others — and without an eval, you'd ship it blind.
The trap of the flattering metric
The most important discipline is honesty about your own numbers. It's easy to design an evaluation that makes you look good — test on easy cases, pick lenient criteria, quietly ignore the failures. A metric you've rigged to pass tells you nothing. The unflattering measure you trust is worth more than the beautiful one you can't. That principle carries across everything I build, from LLM systems to machine-learning models where I hold the same line on evaluation.
The mindset shift
Stop asking "does this output look good?" and start asking "what's my score on the eval set, and did it go up?" The first is a vibe; the second is engineering. Every serious LLM product is built on that second question.
Measurement is what turns "I think it's better" into "it is better, here's by how much." More on how I approach it at www.divyakush.com.
Related reading
- Why LLMs hallucinate — and the patterns that actually stop it — because groundedness is one of the things you must evaluate for.
- Fine-tuning vs RAG vs prompting: pick the right lever — when to fine-tune, retrieve, or just prompt.
- Chain-of-thought: making a model think before it answers — why thinking step by step improves answers.
Divyakush Punjabi · Full-Stack & AI Engineer
Portfolio · GitHub · LinkedIn
Top comments (0)