The Pain: Your agent was fine yesterday. Today you changed one line in a prompt and everything breaks — and you have no idea which change did it. Traditional unit tests don't apply because agent output is nondeterministic.
What You'll Learn: How to build a three-element golden dataset (input, expected trajectory, expected output) and wire it into CI as a regression gate that blocks merges below a 90% pass rate — with a copy-paste, deterministic demo script that needs no API key.
Opening: One Line Changed, Everything Broke
I once changed one line in a prompt: the instruction for "check quote" went from "verify the recipient again" to "send directly". The reasoning was sound — one less validation step, 200 milliseconds faster. That evening, customer 003 received customer 007's quote sheet.
Final answer all green, process all wrong. In the previous article, LLM-as-Judge: How to Calibrate Your Referee, we learned that the judge itself must be calibrated, and calibration relies on a 20-case golden set. This article answers the question that one left open: a calibrated judge has to show up to work every day, and appear in court on a budget — 20 samples are enough to calibrate, but not enough to gate. An agent can collapse from a one-line prompt change; an agent without regression testing is walking a tightrope. The solution, in four characters' worth of meaning: gate it.
One: Why Agents Especially Need Regression Testing
Traditional software has compile-time checks, unit tests, and integration tests as layered safety nets. Agents get none of that — the prompt is the agent's behavioral code. Change one line, and the impact isn't one line of logic, it's the entire trajectory:
| What you changed | Where it can break |
|---|---|
| Added "keep it concise" | Tool-call arguments get dropped, everything downstream breaks |
| Switched model versions | Output style shifts, judges flip their verdicts en masse |
| Tweaked a system constraint | Whitelist fails, privilege escalation |
| Edited a tool description | Agent starts using the wrong tool |
Worse: agent output is nondeterministic. The same input can produce different trajectories on two runs, so the classic "assert output equals expectation" unit test simply doesn't apply. That's why we need a golden dataset — not assertions, but a behavior baseline: turn "mistakes made in the past" into "the defensive line of the future".
Two: The Three Elements of a Golden Dataset
In F1, Stop Testing Final Answers: Trajectory Evals Are the Truth About Agent Quality, we showed that answer-only testing is blind for agents — in that misfire, the answer was literally "already sent". So every sample in a golden set must contain three elements:
- Input: a real trigger scenario, e.g. "Look up customer 007's quote history and send it"
-
Expected trajectory: which tools should be called and in what order —
search_contact → get_quote_history → send_email. One extra step, one missing step, or a skipped step all count as errors - Expected output: the key facts the answer must cover, e.g. "recipient=007", "content=quote for 007"
All three are indispensable: test output only, and a trajectory error (sending when you shouldn't) goes unnoticed; test trajectory only, and a fact error (recipient mix-up) goes unnoticed. Together, they form the complete defense.
Three: Step by Step — Turning a Golden Set into a CI Gate
Here is the regression gate actually running in my system (simplified — save it as golden_regression.py, fully deterministic, no API key needed):
# golden_regression.py — Golden Dataset regression gate demo
# three elements: input / expected_trajectory / expected_facts
# gate: pass rate < 90% blocks the merge/release
GOLDEN = [
# input, expected trajectory (tool sequence), expected facts (must be covered in output)
{"input": "Look up customer 007's quote history and send it",
"trajectory": ["search_contact", "get_quote_history", "send_email"],
"facts": ["recipient=007", "content=007's quote"]},
{"input": "Look up customer 003's quote history and send it",
"trajectory": ["search_contact", "get_quote_history", "send_email"],
"facts": ["recipient=003", "content=003's quote"]},
{"input": "Only look up customer 012's quote history, do not send",
"trajectory": ["search_contact", "get_quote_history"],
"facts": ["no send action"]},
{"input": "Send customer 007 a greeting email",
"trajectory": ["search_contact", "send_email"],
"facts": ["recipient=007", "content=greeting"]},
{"input": "Look up customer 019's contact info",
"trajectory": ["search_contact"],
"facts": ["contact info returned"]},
{"input": "Look up customer 007's quote history and send with template",
"trajectory": ["search_contact", "get_quote_history", "send_email"],
"facts": ["recipient=007", "content=007's quote", "template=standard quote"]},
]
def run_agent_v1(input_text):
"""Legacy agent: correct behavior"""
cid = input_text.split("customer ")[1][:3]
traj = ["search_contact"]
if "quote history" in input_text:
traj.append("get_quote_history")
if ("send" in input_text or "Send" in input_text) and "do not send" not in input_text:
traj.append("send_email")
if traj[-1] == "send_email":
if "greeting" in input_text:
out = f"recipient={cid}, content=greeting, sent"
elif "template" in input_text:
out = f"recipient={cid}, content={cid}'s quote, template=standard quote, sent"
else:
out = f"recipient={cid}, content={cid}'s quote, sent"
else:
out = "no send action, contact info returned"
return traj, out
def run_agent_v2(input_text):
"""New agent: one line of logic changed — the send branch no longer respects 'do not send', and the recipient is hard-coded to 007"""
cid = input_text.split("customer ")[1][:3]
traj = ["search_contact"]
if "quote history" in input_text:
traj.append("get_quote_history")
# change point 1: unconditionally append the send action (ignoring "do not send")
traj.append("send_email")
# change point 2: recipient hard-coded to 007 (mix-up bug)
if "greeting" in input_text:
out = f"recipient=007, content=greeting, sent"
elif "template" in input_text:
out = f"recipient=007, content={cid}'s quote, template=standard quote, sent"
else:
out = f"recipient=007, content={cid}'s quote, sent"
return traj, out
def evaluate(item, traj, out):
"""Score against the three elements: trajectory must match exactly + all expected facts must be covered"""
if traj != item["trajectory"]:
return False, f"trajectory mismatch: expected {item['trajectory']} got {traj}"
missing = [f for f in item["facts"] if f not in out]
if missing:
return False, f"output missing facts: {missing}"
return True, "pass"
def regression(agent, threshold=0.90):
passed = 0
fails = []
for i, item in enumerate(GOLDEN, 1):
traj, out = agent(item["input"])
ok, msg = evaluate(item, traj, out)
if ok:
passed += 1
else:
fails.append((i, msg))
rate = passed / len(GOLDEN)
return passed, rate, fails
def gate(name, passed, rate, fails, threshold=0.90):
print(f"\n=== {name} ===")
print(f"Cases: {len(GOLDEN)} | Passed: {passed} | Pass rate: {rate:.0%}")
if rate >= threshold:
print(f"PASS: pass rate {rate:.0%} >= {threshold:.0%}, merge/release allowed")
else:
print(f"BLOCK: pass rate {rate:.0%} < {threshold:.0%}, merge/release blocked")
for i, msg in fails:
print(f" case#{i}: {msg}")
if __name__ == "__main__":
p1, r1, f1 = regression(run_agent_v1)
gate("v1 (legacy agent)", p1, r1, f1)
p2, r2, f2 = regression(run_agent_v2)
gate("v2 (after a one-line logic change)", p2, r2, f2)
Run it:
python3 golden_regression.py
Here is my real run output:
=== v1 (legacy agent) ===
Cases: 6 | Passed: 6 | Pass rate: 100%
PASS: pass rate 100% >= 90%, merge/release allowed
=== v2 (after a one-line logic change) ===
Cases: 6 | Passed: 3 | Pass rate: 50%
BLOCK: pass rate 50% < 90%, merge/release blocked
case#2: output missing facts: ['recipient=003']
case#3: trajectory mismatch: expected ['search_contact', 'get_quote_history'] got ['search_contact', 'get_quote_history', 'send_email']
case#5: trajectory mismatch: expected ['search_contact'] got ['search_contact', 'send_email']
Those three failed cases expose exactly two error classes, one-to-one with the three elements:
- case#2: output error — the recipient got mixed up to 007, so the expected fact "recipient=003" is missing. Output-level assertion can't catch it (it believes the send succeeded); fact-point comparison catches it.
- case#3 and case#5: trajectory errors — the input explicitly said "do not send" / "only look up contact info", yet the agent appended a send action. Trajectory-sequence comparison catches it.
That's the power of the gate: that one-line v2 change went unnoticed before the gate; after the gate it's thrown back at 50%. The error is caught before merge, and the cost is one fix on the dev side; caught at the user, the cost is zero trust.
Four: My Production Setup — The Four publish_gate Gates
A script running alone is meaningless; it only matters wired into the release pipeline. My content pipeline has four gates before release, and regression testing hangs on the last one:
# ci_gate.sh — hook golden regression into CI (simplified)
# any step exiting non-zero blocks the merge/release
python3 validate_article.py check article.md || exit 1
python3 check_series_continuity.py check || exit 1
python3 article_checker.py article.md || exit 1
python3 golden_regression.py --threshold 0.90 || exit 1
echo "all four gates passed, release allowed"
These four gates are real and running: validate_article checks format completeness (length, images, no fabrication), check_series_continuity checks the next-article hook char by char, article_checker runs the C1-C8 deep audit, and publish_gate gives the final go/no-go. The gate has genuinely blocked things — image references with stray spaces breaking uploads, series-hook titles that didn't match exactly, articles containing fabricated content. All of them were caught at the gate layer. The moment it blocks you, you thank it: it turns "discovered after release" into "discovered before release".
Five: Appearing in Court on a Budget — Regression Cost Control
The F2 hook said the calibrated judge has to "show up to work every day, and appear in court on a budget" — the loudest objection to regression testing is always cost. Here's my layered approach:
| Layer | Size | When | Cost |
|---|---|---|---|
| Smoke set | 20 cases | every commit | seconds, nearly free |
| Full set | 500 cases | before release / daily | minutes, budget-controlled |
| Production reflux | grows continuously | failing cases auto-feed back | more accurate over time, costlier but worth it |
Three cost-saving principles: ① dedupe samples — keep one representative case per error class, don't reinvent the wheel; ② stratified sampling — the smoke set covers high-frequency paths, the full set covers edge cases; ③ failure reflux — a case that newly fails in production gets added to the golden set within 24 hours, becoming the sample for the next round. A golden set is not a one-time asset; it's a compounding asset that gets thicker with use — this is the engineering version of F2's "a golden set is an asset, not a burden".
Deeper Thinking: Regression Is About Turning Past Mistakes into Future Defenses
After the gate was running, I re-understood what "regression testing" really is.
Regression testing isn't "testing" — it's memory. Traditional tests verify whether current code is correct; regression tests verify whether past mistakes have recurred. Every case added to your golden set is one more piece of "never make this mistake twice" memory in the system. It's the exact same structure as Series 3's From SOP to Immunity: mistake → ledger → fix → feed the rule back. On the personal level it's an error-ledger; on the organization level it's an SOP; on the agent level it's a golden dataset. All three run on the same Loop Engineering core.
This also explains why "a gate isn't a limitation — it's freedom": when you know changes won't break things, you dare to change often. Teams without regression testing are afraid to touch the prompt, because every change is a gamble; teams with a golden gate iterate daily, because risk is quantified and intercepted before merge. True engineering maturity isn't about never making mistakes — it's about letting mistakes happen at the lowest possible cost: in CI, not in production.
Closing
Today you learned: an agent can collapse from a one-line prompt change, and golden dataset's three elements (input, expected trajectory, expected output) + a 90% threshold = a CI gate. Regression testing isn't testing — it turns past mistakes into future defenses: smoke set runs every commit, full set runs before release, and failing samples reflux back for compounding growth.
The call to action is simple: build a 6-case golden set for your agent tonight — pick 6 scenarios you use most, write down the input, the expected trajectory, and the expected facts, and run golden_regression.py. Then hook it into a gate, and block merges when the pass rate drops below 90%. For the first time, you'll see: the change never even shipped, and the error was already caught.
Next, we pull the evaluation system from "gate" to "production retrospective" — The Production Retrospective of the Observability Trio: How Gate, Audit, and Correction Turn Together: trajectory tracing, audit ledger, correction loop — how the trio actually cooperates in production so the agent runs steadier over time.
About the author: Wu Ji (无记) — AI & digitalization practitioner focused on Agent engineering, Loop Engineering, and digital transformation. Practical, hands-on tutorials — follow along and it just works.
Top comments (0)