DEV Community

Cover image for Test AI outputs by properties, not by one golden string
Conor Bronsdon
Conor Bronsdon

Posted on Originally published at chainofthought.show

Test AI outputs by properties, not by one golden string

An exact-string assertion can reject a valid model answer when several different outputs are acceptable. Check properties on a dataset instead. For each case, ask whether the output is grounded in the supplied source, follows the instruction, stays in scope, and avoids the unsafe response. Score with code where you can write the check, and with a rubric or an LLM judge where you cannot. Run that suite on every prompt change, model swap, or dependency bump.

This explainer draws from Chain of Thought episode 37 (August 2025) with Hamel Husain, founder of Parlance Labs, and episode 18 (April 2025) with Charity Majors, co-founder and CTO of Honeycomb.

Why does a golden string fail a correct model answer?

The same prompt can produce several answers that are all correct, so a test locked to one stored string fails any correct answer with different wording. Property checks ask what the answer must do.

Yash Sheth described the first job of evaluation as solving the measurement problem: quantifying an application's behavior as metrics.

Some checks are mechanical. The reply should name a source id the case requires, exclude a forbidden phrase, and stay under a word limit. Naming a source id does not prove the source supports the claim. That part belongs on a rubric.

Dashboard scores such as a hallucination score, a toxicity score, or a conciseness score do not replace those checks. Hamel Husain said they are way too generic and do not necessarily correlate with actual failures in the product.

"These generic metrics are most of the time not helpful at all. They're way too generic. They don't necessarily correlate with actual failures in your AI product."

Hamel Husain, founder of Parlance Labs, on Chain of Thought ep 37

Mindset Over Metrics: How to A… - Chain of Thought | AI Agents, Infrastructure & Engineering - Apple Podcasts

Podcast Episode · Chain of Thought | AI Agents, Infrastructure & Engineering · August 20, 2025 · 42m

favicon podcasts.apple.com

How do you build the dataset that makes the suite repeatable?

Build the first cases from failures, not from a coverage quota. Hamel Husain calls the method error analysis. Open a trace viewer, read traces, write notes on what went wrong, put the notes into categories, and count them. Prioritize the categories you see most often.

How many traces is the usual question. Theoretical saturation, a social-science term he uses, means stop when you learn nothing new. People stall on that phrase, so he offers a starter you can schedule: at least 100 traces. After you start, the quota matters less. The notes tell you which eval to write.

Outsourcing evaluations to developers is one of the biggest failure modes he sees, unless you are building a developer tool and the developer is the domain expert. Otherwise you are guessing. If the product is for lawyers, involve the lawyer.

"To trust LLM as a judge, you have to compare it to some human labels."

Hamel Husain, founder of Parlance Labs, on Chain of Thought ep 37

Before a judge can block a release, compare it with those human labels. The judge's verdict is another model output. If it disagrees with itself on the same case across runs, read the disputed cases before you move the threshold. A gate that flips on noise teaches the team to ignore it.

How do you gate a release on properties instead of a single score?

Keep a reviewed baseline pass rate in the repo with the cases. Fail the build when the new pass rate falls below it. Print the ids that failed. The rate alone does not say what to fix.

Run cheap checks before the judge. A string check is enough for a forbidden phrase. The judge should run only after those checks pass.

A minimal sketch to adapt.

import json
from pathlib import Path

def test_eval_pass_rate():
    cases = json.loads(Path("eval_cases.json").read_text())
    baseline = json.loads(Path("eval_baseline.json").read_text())["pass_rate"]
    assert cases, "Evaluation dataset must not be empty"
    failed = []
    for case in cases:
        reply = call_model(case["prompt"], case["source"])
        text = reply["text"]
        cheap_ok = (
            case["required_source_id"] in reply["source_ids"]
            and case["forbidden_phrase"].lower() not in text.lower()
            and len(text.split()) <= case["max_words"]
        )
        if not (cheap_ok and judge_against_rubric(case["source"], text, case["rubric"])):
            failed.append(case["id"])
    pass_rate = (len(cases) - len(failed)) / len(cases)
    assert pass_rate >= baseline, f"Pass rate dropped: {pass_rate}, failed: {failed}"
Enter fullscreen mode Exit fullscreen mode

Counting words with split() is coarse. Use a stricter method if the limit has hard product meaning.

What should you add after a production failure?

Charity Majors' point is that a lot of engineers are comfortable in the editor running tests, and they treat production as someone else's problem. She sees the influx of generated code straining the usual debugging path: find who changed it, and ask the expert what happened. She poses the follow-up: what if there is no expert, and what if you have to understand production on its own terms? When a failure reaches users, record what the system received and what it returned, then add the case that would have caught it.

"the source of truth is what is happening right now."

Charity Majors, co-founder and CTO of Honeycomb, on Chain of Thought ep 18

She said that in many sentences you can replace AI with automation and the sentence still holds, nine times out of ten by her count. Conor named nondeterminism as the exception, and Charity agreed. Charity was drawing that contrast when she said the whole premise of APIs is that you put in the same thing and get the same thing out.

FAQ

Why does assert-equals fail a correct answer?

The same prompt can produce more than one correct answer. Equality with a single golden string fails the correct answers that use different words.

How many traces should I review to start?

Hamel Husain recommends at least 100 traces as a concrete starter, then continuing until you are not learning anything new. That stopping rule is what he calls theoretical saturation.

Does an AI engineer need to train models?

No. You do not need to train models, and you do need data literacy. Hamel Husain said that validating a judge against human labels, and explaining why sampling can show noise in the judge, moves you quickly toward the skill set of a machine learning engineer or a data scientist.

Takeaway

  • Treat each test case as properties: a required source, a forbidden phrase, a word limit, and a rubric for whether the source supports the answer.
  • Run cheap code checks before paying for a judge.
  • Gate the build on a reviewed baseline, and print failed case ids beside the pass rate.
  • Compare the LLM judge with human labels before the judge can fail a build.
  • Add a case for every production break, fed by error analysis on traces.

The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.

Top comments (0)