The Prompt Changed. Nothing Broke. That's the Problem.

The most expensive failure in prompt engineering is not a crash or a validation error. It is the silent regression where output quality degrades just enough that nobody notices until users do. A single example that looks fine in a notebook proves nothing about a prompt change. LLM output is stochastic by design, and a sample size of three is a coin flip. The fix is a small eval harness with golden cases, automated graders, and a history file that surfaces regressions the moment they appear.

A recent DEV discussion asked who tests the AI reviewer now that every developer has been promoted to one. The honest answer is that most prompt changes still ship without a test suite. We test our functions, our components, and our API contracts. Yet the one piece of code that accepts free-form text and returns free-form text gets changed by feel. That asymmetry is strange, because prompts are pure functions from text to text, and pure functions are what test suites were invented for.

An eval harness has three parts. Golden cases are a fixed corpus of inputs paired with expected behavior, collected from real traffic rather than from the examples that inspired the prompt. Graders decide whether an output passes, and they must be stricter than a human glance, which is why exact and structural checks beat vibes. The runner executes every case against the current prompt, records the results, and compares them with the previous run to expose flips.

Here is the golden-case file, one JSON object per line:

{"id": "extract_001", "mode": "json", "prompt": "Extract sender, amount, and currency from: 'Invoice 4421 from Acme GmbH, EUR 12,400, due 2026-09-30.'", "expected_fields": {"sender": "Acme GmbH", "amount": "12400", "currency": "EUR"}}
{"id": "summarize_001", "mode": "substring", "prompt": "Summarize this release note in two sentences: 'The scheduler now retries failed jobs with exponential backoff, and the dashboard exposes queue depth per worker.'", "required_tokens": ["retries", "queue depth"]}
{"id": "classify_001", "mode": "exact", "prompt": "Classify this ticket as bug, feature, or question: 'The export fails when the file name contains a space.'", "expected": "bug"}

Collecting golden cases is the part most people skip, and it is the part that determines whether the harness catches anything real. Start with production logs and pick the inputs that caused the most confusion: the ambiguous ticket, the malformed invoice, the edge-case classification that support had to correct by hand. For each one, write down the output you would accept, not the output the model happened to produce. The whole point is to lock in intent rather than current behavior.

The harness itself is deliberately small, because a harness you are afraid to edit is a harness that rots:

# eval_harness.py
import json
from pathlib import Path

def load_cases(path):
    return [json.loads(line) for line in Path(path).read_text().splitlines() if line.strip()]

def grade(case, output):
    mode = case.get("mode", "exact")
    if mode == "exact":
        return output.strip() == case["expected"].strip()
    if mode == "substring":
        return all(t in output for t in case["required_tokens"])
    if mode == "json":
        try:
            parsed = json.loads(output)
            return all(parsed.get(k) == v for k, v in case["expected_fields"].items())
        except json.JSONDecodeError:
            return False
    return False

def run(cases, model_fn):
    return {c["id"]: grade(c, model_fn(c["prompt"])) for c in cases}

def report(results, label):
    rate = sum(results.values()) / len(results)
    print(f"{label}: {rate:.0%} pass ({sum(results.values())}/{len(results)})")
    return rate

The model function is the only integration point, and it can point anywhere. A typical adapter calls a local endpoint that fronts the free model access, and the exact request shape depends on your deployment:

# adapter.py — pattern only; the exact endpoint shape depends on your deployment
import json, urllib.request

def model_fn(prompt):
    payload = json.dumps({"prompt": prompt, "max_tokens": 256}).encode()
    req = urllib.request.Request("http://localhost:8000/v1/completions", payload)
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.load(resp)["choices"][0]["text"].strip()

The history file is where the harness earns its keep. Each run appends a timestamped row with per-case pass or fail, and the comparison step flags any case that passed before and fails now. A five-point drop in the aggregate rate is often noise. A single golden case flipping from pass to fail is a signal, because golden cases are supposed to be stable. That flip is the silent regression, and catching it before the pull request merges is the entire point of the exercise.

# regression.py
import json, datetime
from pathlib import Path

HISTORY = Path("eval_history.jsonl")

def record(results):
    row = {"ts": datetime.datetime.now(datetime.timezone.utc).isoformat(), "results": results}
    with HISTORY.open("a") as f:
        f.write(json.dumps(row) + "\n")

def regressions(previous, current):
    return [cid for cid, ok in previous.items() if ok and not current.get(cid, False)]

Running this suite costs tokens, and running it on every prompt change costs more tokens, which is where the infrastructure choice matters. MonkeyCode is an open source project, and its current free tier includes 10 million tokens and a free server option, enough to run a fifty-case suite hundreds of times over without touching a paid account. The free server gives the harness a place to run on a schedule instead of on your laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The limits of this approach are real, and they are worth naming before you adopt it. Golden cases rot, because the corpus you collect in August will not represent November traffic; refresh it from production logs on a regular cadence. An LLM-as-judge grader can inherit the very regression you are trying to catch, which is why I prefer exact and structural graders for anything machine-readable. Reserve judge models for cases where no objective check exists, and remember that free quotas are for evaluation, not production load, and that terms change over time.

This harness is also the wrong tool for some jobs. If your task is open-ended creative writing, a pass or fail grade measures the wrong thing and gives you false confidence. If you need compliance-grade evidence, a pass rate on fifty golden cases is not an audit trail. The sweet spot is tasks with checkable properties: extraction, classification, structured output, and summarization with required coverage. That is most of what production prompts actually do.

The next time you are about to ship a prompt change based on three screenshots, run the suite instead and keep the history file. The first time it catches a regression you would have shipped, the harness will have paid for itself. The cost of running it is a few tokens and one afternoon of setup.