Automating LLM Answer Evaluation with a Small Python Scoring Script

If you have ever evaluated a language model's output by hand, you know how quickly it stops scaling. Ten answers are fine. A thousand answers, re-run every time someone tweaks a prompt, is not. After spending a lot of time auditing LLM output for correctness, I have found that a small, boring Python script removes most of the pain — and, just as importantly, makes your evaluation consistent instead of depending on how tired the reviewer is.

This tutorial builds a minimal but genuinely useful scoring harness: give it a set of questions with known correct answers and the model's answers, and it returns a score plus a list of exactly which items failed and why. No frameworks, standard library only.

The idea

Most "did the model get it right?" checks fall into three buckets:

1. Exact match after light normalization (case, whitespace, punctuation).
2. Numeric match within a tolerance (the model says 3.14, the gold answer is 3.14159).
3. Contains — the gold answer appears somewhere in a longer response.

A good harness applies the right check per question rather than forcing everything through exact match, which is where naive scripts quietly report wrong numbers.

The gold set

Keep your evaluation data as plain JSON so anyone can edit it without touching code:

[
  {"id": "q1", "question": "What year did the first moon landing happen?", "answer": "1969", "match": "contains"},
  {"id": "q2", "question": "What is pi to two decimals?", "answer": "3.14", "match": "numeric", "tol": 0.01},
  {"id": "q3", "question": "Name the capital of France.", "answer": "Paris", "match": "contains"}
]

The match field is the important part: it tells the scorer how to compare, so each question is judged the way it should be.

The scorer

import json
import re


def normalize(text):
    """Lowercase, strip, collapse whitespace, drop trailing punctuation."""
    text = text.strip().lower()
    text = re.sub(r"\s+", " ", text)
    return text.rstrip(".!?,;:")


def first_number(text):
    """Pull the first number out of a string, or None if there isn't one."""
    m = re.search(r"-?\d+(?:\.\d+)?", text.replace(",", ""))
    return float(m.group()) if m else None


def is_correct(item, model_answer):
    kind = item.get("match", "exact")
    gold = item["answer"]

    if kind == "exact":
        return normalize(model_answer) == normalize(gold)

    if kind == "contains":
        return normalize(gold) in normalize(model_answer)

    if kind == "numeric":
        got = first_number(model_answer)
        want = first_number(gold)
        if got is None or want is None:
            return False
        return abs(got - want) <= item.get("tol", 0.0)

    raise ValueError(f"unknown match type: {kind!r}")


def score(gold_path, answers):
    """answers: dict of {id: model_answer_string}. Returns (score, failures)."""
    gold = json.load(open(gold_path, encoding="utf-8"))
    passed, failures = 0, []
    for item in gold:
        model_answer = answers.get(item["id"], "")
        if is_correct(item, model_answer):
            passed += 1
        else:
            failures.append({
                "id": item["id"],
                "question": item["question"],
                "expected": item["answer"],
                "got": model_answer or "(no answer)",
            })
    return passed / len(gold), failures

Running it

answers = {
    "q1": "The first moon landing was in 1969.",
    "q2": "Pi is about 3.14159",
    "q3": "The capital of France is Paris.",
}

acc, failures = score("gold.json", answers)
print(f"accuracy: {acc:.0%}")
for f in failures:
    print(f"FAIL {f['id']}: expected {f['expected']!r}, got {f['got']!r}")

Two things make this more useful than a one-line accuracy number. First, q1 passes even though the model wrapped 1969 in a full sentence, because contains/normalization handles the wrapping — an exact-match-only script would have marked it wrong and sent you chasing a non-bug. Second, every failure comes back with the question, the expected answer, and what the model actually said, so you can see why it failed instead of just that it failed.

Where to take it next

Once this is in place, small additions pay off quickly:

• Regression tracking: dump the failures to a file per run and diff two runs, so you can see exactly which items a prompt change broke or fixed.
• Weighting: add a weight field for questions that matter more.
• Guarding the input: reject a gold file where two items share an id, or where a numeric item's answer has no number in it — bad eval data causes more wrong conclusions than bad models do.

None of this is fancy, and that is the point. A hundred lines of readable Python turns "I looked at some outputs and they seemed okay" into a repeatable number with a paper trail — which is the difference between an opinion about a model and an actual evaluation of it.