How Do You Actually Know if Your AI Is Good? (Evaluating LLM Outputs)

Written by Syed Muhammad Ali Raza

Six articles into this series, and I've conveniently skipped over a question that should've come up way earlier, how do you actually know if any of this stuff you built is good. Not "it felt fine when I tried it three times," actually know, in a way you could show a teammate or defend in a code review.

I got away with skipping this for a while because in a personal project, "eh, looks right to me" is an acceptable bar. It stopped being acceptable the moment I changed one line of a prompt in a real project and had no idea whether I'd just made things better or quietly worse for cases I wasn't looking at. That's the exact moment evaluation, usually just called evals, stops being an academic nice to have and starts being the thing that separates a toy from something you can actually trust.

A real life example before the technical stuff

Think about how a driving school actually knows if a student is ready for their test. They don't just ask the student "do you feel confident driving now?" and take their word for it. They don't watch the student parallel park once, nail it, and declare them a great driver forever.

They run the student through a consistent set of specific scenarios, every time, for every student, parking, highway merging, stopping at a light, handling a pedestrian crossing, and they score each one against a clear standard. If a student aces parking but panics at merging, that's useful, specific information, not just a vague "pretty good overall" feeling. And critically, they run this same test after every round of practice, so they can actually tell if a particular lesson helped or made things worse for some specific skill.

That's the entire idea behind evaluating LLM outputs. Instead of vibing your way through "does this feel right," you build a consistent set of test cases, you score outputs against a real standard, and you rerun that same test every time something changes, so you know precisely what got better and what quietly broke.

Why "it looks right to me" genuinely doesn't work here

I want to be specific about why this instinct fails with LLMs particularly, because it's not obvious at first.

These models are extremely good at producing text that sounds confident and well formed, regardless of whether it's actually correct. A wrong answer from an LLM often reads exactly as smoothly as a right one, there's no visual cue, no hedging tone, nothing that tips you off the way a human who's unsure of themselves usually does. Skimming a handful of outputs and going "yeah that seems fine" is genuinely unreliable because your eye is trained to catch awkward phrasing, not factual or logical correctness.

There's also the randomness problem I mentioned back in the second article in this series, the same prompt can produce different outputs on different runs. Testing something once and calling it done tells you almost nothing about how it behaves across the range of things it'll actually be asked in the real world.

And changes compound in ways that are easy to miss. You tweak a prompt to fix one specific complaint, and it can quietly make five other cases worse, cases you weren't looking at when you made the change. Without a standing set of test cases you check every time, you have no way of catching that regression until an actual user does.

The three broad types of evals, and when each one fits

I organize evals into three buckets in my head, because the tooling and effort required for each is genuinely different.

Exact match or rule based checks work when there's a clearly correct, checkable answer. Did the output contain valid JSON matching a schema. Did it correctly extract a specific number from a document. Did a classification task pick the right category out of a fixed list. These are cheap, fast, deterministic, and you should use them anywhere they genuinely apply, because they're the most trustworthy category by far.

Human review is when a person actually reads the output and judges it against a rubric, is this response helpful, is the tone right, does it follow the instructions. This is the most reliable check for anything genuinely subjective, but it's slow and doesn't scale to checking thousands of outputs every time you tweak a prompt.

LLM as judge is a newer and increasingly common middle ground, using a separate model call to grade the output against a rubric, automatically, at scale. It's faster than human review and can handle subjective quality in a way rule based checks can't, but it comes with its own accuracy limits that I'll get into, it is not a free pass to stop thinking about evaluation quality.

Most real projects end up using a mix of all three, rule based checks wherever a real correct answer exists, LLM as judge for scaling up subjective quality checks, and human review as a periodic sanity check on whether the automated judge itself is actually behaving sensibly.

Let's build an actual eval suite, starting with rule based checks

I'll use a genuinely practical example, an LLM that's supposed to extract structured data from customer support messages, name, issue category, and urgency level, as JSON. This is a great first eval case because there's a real, checkable correct answer for each test case.

Step 1, define your test cases with known correct answers

test_cases = [
    {
        "input": "Hi, this is John, my app crashes every time I open it, please help urgently",
        "expected": {"name": "John", "category": "bug", "urgency": "high"}
    },
    {
        "input": "Hey it's Sarah, just wondering how to change my email on file, no rush",
        "expected": {"name": "Sarah", "category": "account", "urgency": "low"}
    },
    {
        "input": "This is Mike, I was charged twice for my subscription this month",
        "expected": {"name": "Mike", "category": "billing", "urgency": "medium"}
    }
    # a real eval suite would have dozens to hundreds of these,
    # covering edge cases, not just the easy obvious ones
]

That last comment matters more than it looks. Three test cases only tells you the model handles three easy examples. A real eval suite needs edge cases on purpose, ambiguous urgency, missing names, messages that don't clearly fit any category, because those are exactly the cases most likely to break.

Step 2, run the actual extraction and compare against the expected answer

import anthropic
import json

client = anthropic.Anthropic(api_key="your-api-key-here")

def extract_support_info(message):
    system_prompt = """
Extract the customer's name, issue category (bug, account, billing,
or other), and urgency (low, medium, high) from the message. Respond
with ONLY valid JSON in this exact format:
{"name": "...", "category": "...", "urgency": "..."}
"""
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=150,
        system=system_prompt,
        messages=[{"role": "user", "content": message}]
    )
    raw_text = response.content[0].text.strip()
    try:
        return json.loads(raw_text)
    except json.JSONDecodeError:
        return {"error": "invalid json", "raw": raw_text}


def run_extraction_eval(test_cases):
    results = []
    correct_count = 0

    for case in test_cases:
        actual = extract_support_info(case["input"])
        expected = case["expected"]

        is_correct = actual == expected
        if is_correct:
            correct_count += 1

        results.append({
            "input": case["input"],
            "expected": expected,
            "actual": actual,
            "correct": is_correct
        })

    accuracy = correct_count / len(test_cases)
    return accuracy, results


accuracy, results = run_extraction_eval(test_cases)
print(f"Accuracy: {accuracy * 100:.1f}%\n")

for r in results:
    status = "PASS" if r["correct"] else "FAIL"
    print(f"[{status}] {r['input'][:50]}")
    if not r["correct"]:
        print(f"    expected: {r['expected']}")
        print(f"    actual:   {r['actual']}")

This gives you a real number, not a feeling. Run this exact same suite before and after any prompt change, and you know immediately whether accuracy went up, down, or stayed the same, and exactly which specific cases broke if it dropped.

Step 3, a partial credit version, since exact match is sometimes too strict

Exact dictionary equality is harsh, if the model gets name and category right but urgency slightly wrong, exact match scores that as a total failure, which can hide useful signal about what's actually working.

def score_partial_credit(actual, expected):
    if "error" in actual:
        return 0.0

    fields = ["name", "category", "urgency"]
    correct_fields = sum(1 for f in fields if actual.get(f) == expected.get(f))
    return correct_fields / len(fields)


def run_partial_credit_eval(test_cases):
    scores = []
    for case in test_cases:
        actual = extract_support_info(case["input"])
        score = score_partial_credit(actual, case["expected"])
        scores.append(score)
        print(f"{case['input'][:50]}... -> {score * 100:.0f}%")

    average_score = sum(scores) / len(scores)
    print(f"\nAverage field accuracy: {average_score * 100:.1f}%")
    return average_score

This version tells you a lot more, if urgency is consistently the field that's wrong across many test cases, that's a specific, actionable signal, maybe your prompt needs clearer examples of how to judge urgency, rather than a vague "something's a bit off" feeling from eyeballing outputs.

Now the harder case, evaluating open ended text with an LLM judge

Structured extraction has a clean right answer. A lot of real tasks don't, summarization quality, whether a customer support reply sounds appropriately empathetic, whether a generated article actually stays on topic. For these, LLM as judge is the practical middle ground.

def llm_judge(generated_text, criteria):
    judge_prompt = f"""
You are evaluating a piece of AI generated text against a specific
criterion. Be strict and critical, do not default to a positive
score just to be agreeable.

Criterion: {criteria}

Text to evaluate:
{generated_text}

Respond with ONLY valid JSON in this format:
{{"score": <1 to 5>, "reasoning": "<brief explanation>"}}
"""
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=200,
        messages=[{"role": "user", "content": judge_prompt}]
    )
    try:
        return json.loads(response.content[0].text.strip())
    except json.JSONDecodeError:
        return {"score": None, "reasoning": "judge failed to return valid json"}


# example usage
generated_summary = "The article discusses how solar panels convert sunlight into electricity using photovoltaic cells, and covers basic installation costs."

result = llm_judge(
    generated_summary,
    "The summary should accurately reflect the main points without adding information not present in the original article."
)
print(result)

A few things I learned the hard way about doing this properly. Explicitly telling the judge to be strict and not default to agreeable high scores matters a lot, models genuinely do have a tendency toward being overly generous graders unless you push against it directly in the prompt. Giving a narrow, specific criterion per call, rather than one vague "is this good" question, gets you much more useful and consistent scores, "is this factually grounded in the source" is a much better question than "is this good."

And here's the part people skip, you have to periodically check whether your judge is actually judging correctly. Take a sample of the judge's scores, review them yourself, and see if you agree. An unreliable judge silently grading everything wrong is arguably worse than no automated evaluation at all, because it gives you false confidence.

def spot_check_judge(judge_results, sample_size=10):
    import random
    sample = random.sample(judge_results, min(sample_size, len(judge_results)))

    print("Manually review these judge decisions, do you agree?\n")
    for item in sample:
        print(f"Text: {item['text'][:80]}...")
        print(f"Judge score: {item['score']}, reasoning: {item['reasoning']}")
        print("-" * 40)

Putting it together, a small regression test you actually run every time

Here's the practical habit that made the biggest difference for me, treating evals exactly like unit tests, something that runs automatically whenever you change a prompt, not something you remember to do occasionally when you feel like it.

def run_full_eval_suite():
    print("=" * 50)
    print("RUNNING FULL EVAL SUITE")
    print("=" * 50)

    accuracy, extraction_results = run_extraction_eval(test_cases)
    print(f"\nExtraction exact match accuracy: {accuracy * 100:.1f}%")

    if accuracy < 0.8:
        print("WARNING, extraction accuracy below 80% threshold")

    partial_score = run_partial_credit_eval(test_cases)

    return {
        "extraction_exact_match": accuracy,
        "extraction_partial_credit": partial_score
    }


baseline_scores = run_full_eval_suite()
# save this somewhere, a json file, a spreadsheet, whatever
# then after any prompt change, rerun and compare against it directly

That warning threshold matters more than it looks like sitting there in one line. Deciding in advance what "good enough" actually means, and having the suite loudly flag it when you drop below that, turns evaluation from a thing you glance at into something that actually catches problems before they reach real users.

The honest limits of all this

I want to be straight about where this approach still falls short, because oversdelling evals is its own kind of mistake.

Your test cases are only as good as your imagination for what could go wrong, real users will inevitably hit inputs you didn't think to test for. The fix here isn't perfection up front, it's treating your eval suite as a living thing, every real bug or weird output a user reports should turn into a new permanent test case, so your suite genuinely gets stronger over time instead of staying frozen at whatever you thought of on day one.

LLM judges have their own blind spots and biases, and a judge model can share systematic weaknesses with the model being judged, especially if you're using the same model family for both. Periodic human review of the judge's own scoring isn't optional, it's the thing that keeps the whole automated system honest.

And a good score on your eval suite is evidence, not proof. It raises your confidence, it does not guarantee correctness on every possible input your system will ever see in the real world. Treat it as a strong signal you actively monitor and expand, not a checkbox you tick once and forget about.

Bringing this back to the whole series

Every technique in this series, RAG, fine-tuning, agents, multi-agent pipelines, all produce outputs you have to actually trust before shipping to real users. Evaluation is the piece that turns "I built something and it felt okay when I tried it a few times" into "I can show you exactly how well this performs, on which specific cases it struggles, and prove whether my last change actually helped." That difference, between vibes and actual evidence, is honestly what separates a fun weekend project from something you'd trust with a real product and real users.

---

If you've built an eval suite for something of your own, I'd genuinely like to hear what your test cases caught that you didn't expect, that's usually where the real lessons are.