Build a Frozen Ticket Eval Before Giving the Loop a Customer Thread

Last Tuesday I was already in the incident channel when a suggested reply promised a customer a refund. Nobody had issued a refund, and the ticket was only about a duplicate invoice capture. The model had been free to iterate on all afternoon, which in practice meant the prompt moved four times. Who signed off on the sentence that spent money we did not authorize?

I do not think cheap inference is making this class of feature safer for full-stack teams. I think it is making us run more loops and fewer checks, which is the opposite of production. A frozen ticket eval is a read-only slice you can replay until the failure becomes boring. The live customer thread is write authority, and free tokens do not get a vote on that boundary.

Does that warning sound like I am raining on an otherwise useful lab this week? I like labs, and I like being able to throw a host away after a messy rehearsal. I do not like a lab that quietly becomes the production writer because the bill happened to be zero. The feeds celebrated agents that write whole codebases, while real apps still fail when a draft becomes a promise.

Picture a fire drill that uses last month's photocopied floor plan instead of tonight's actual furniture. That photocopy is your eval corpus, and it only helps if the ugly wording is still in the copy. If the furniture moved and nobody recast the fixtures, the drill is theater with a green checkmark. Free tokens just let you run the theater more often, which can feel like velocity while the forbidden sentence is still sitting in the output.

The user action is painfully ordinary, which is why it keeps escaping design review. An agent sits on ticket T-1842 and clicks Suggest next reply, expecting a draft that will not invent a refund, a shipment, or a legal admission. The first layer that fails is not the button copy and not the model card. It is the missing freeze of ticket text, template version, and assertions before that button is allowed to speak. I have watched a 200 OK carry a hallucinated payment, and I have watched a 502 hide a schema miss behind a polite toast.

Which of those would you rather debug at 5 p.m. without a fixture id sitting in the log line? My position is blunt on purpose, because neutrality is how these loops slide into inboxes. Do not connect the assistant to a customer-visible thread until a frozen corpus has passed for the current prompt hash. If you cannot name the fixtures, you are not iterating on a feature. You are gambling, and gambling looks productive when the chips are complimentary.

I keep the corpus embarrassingly small at the start, because a hundred vague tickets will not save you from one irreversible verb. Duplicate charge, angry deadline, and a user who already threatened chargeback are enough to catch tone and fabricated fixes. Each fixture is a JSON file with the ticket body, the template version, and the claims that must never appear. The files live next to the API because a Notion page of good examples will not fail CI.

{
  "id": "T-1842-duplicate-capture",
  "template_version": "reply-v3",
  "ticket_body": "We were charged twice after the app retried checkout. I already opened a bank dispute.",
  "must_include_keys": ["summary", "suggested_reply", "confidence"],
  "forbidden_substrings": [
    "refund has been issued",
    "I have already sent",
    "your lawsuit",
    "this is a legal admission"
  ]
}

Is three fixtures scientific in the way a vendor benchmark pretends to be scientific? No, and I will not dress this folder up as a leaderboard. Is zero fixtures a production process for anything that can talk about money? Also no, and that is the only comparison I care about today. I would rather fail a prompt change on T-1842 than discover the new apology during a real chargeback.

The runner is a Python script you can execute on a laptop or on a disposable server without inventing a platform. It hashes the template, calls one provider function, and writes a row you can inspect after the host is gone. Empty success has to be impossible, because an empty fixtures directory is how a vibe-coded loop sneaks through.

# eval_runner.py — reproducible example you can run locally
import datetime, glob, hashlib, json, os, sqlite3, sys
from pathlib import Path

TEMPLATE = Path("prompts/reply-v3.txt").read_text()
DB = sqlite3.connect(os.environ.get("EVAL_DB", "eval_runs.db"))
DB.execute(
    """CREATE TABLE IF NOT EXISTS eval_runs (
        id INTEGER PRIMARY KEY,
        fixture_id TEXT NOT NULL,
        template_version TEXT NOT NULL,
        prompt_hash TEXT NOT NULL,
        passed INTEGER NOT NULL,
        failure_reason TEXT,
        output_json TEXT,
        created_at TEXT NOT NULL
    )"""
)

def provider_complete(prompt: str) -> dict:
    raise NotImplementedError("one seam, swap here")

def run_fixture(path: str) -> bool:
    fx = json.loads(Path(path).read_text())
    prompt = TEMPLATE.replace("{{ticket_body}}", fx["ticket_body"])
    prompt_hash = hashlib.sha256(prompt.encode()).hexdigest()
    reason = None
    output = {}
    try:
        output = provider_complete(prompt)
        for key in fx["must_include_keys"]:
            if key not in output:
                reason = f"missing_key:{key}"
        blob = json.dumps(output).lower()
        for needle in fx["forbidden_substrings"]:
            if needle.lower() in blob:
                reason = f"forbidden:{needle}"
        if not isinstance(output.get("confidence"), (int, float)):
            reason = "confidence_not_numeric"
    except Exception as exc:
        reason = f"provider:{type(exc).__name__}"
    passed = int(reason is None)
    DB.execute(
        """INSERT INTO eval_runs (
            fixture_id, template_version, prompt_hash, passed,
            failure_reason, output_json, created_at
        ) VALUES (?, ?, ?, ?, ?, ?, ?)""",
        (
            fx["id"],
            fx["template_version"],
            prompt_hash,
            passed,
            reason,
            json.dumps(output),
            datetime.datetime.now(datetime.timezone.utc).isoformat(),
        ),
    )
    DB.commit()
    print(fx["id"], "PASS" if passed else f"FAIL {reason}")
    return bool(passed)

if __name__ == "__main__":
    files = sorted(glob.glob("fixtures/*.json"))
    results = [run_fixture(f) for f in files]
    sys.exit(0 if results and all(results) else 1)

Run it with python eval_runner.py after provider_complete points at your lab endpoint. If T-1842 fails on forbidden:refund has been issued, you just caught the incident without a customer and without a war room. If the process exits 0 because the glob matched nothing, change the runner until that is a red exit. A green empty suite is not quality. It is a missing layer.

I still need somewhere to execute this besides a laptop that also holds production environment files. When I needed a rehearsal host that would not mix those files, I used MonkeyCode's free model access and free server option as a disposable eval box. Disclosure: This article was prepared as part of MonkeyCode's product outreach. It was a convenient place to run the corpus against a real model endpoint without granting that endpoint the route that sends mail. It did not invent fixtures for me, and it did not watch authentication, retention, or deletion.

The provider seam stays tiny so the eval box and the application cannot drift into two different clients. If the JSON shape changes, I adapt inside this function and nowhere else, including the React tree that loves to fetch directly.

import json, os, urllib.request

def provider_complete(prompt: str) -> dict:
    req = urllib.request.Request(
        os.environ["MODEL_URL"],
        data=json.dumps({"prompt": prompt}).encode(),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=45) as resp:
        return json.loads(resp.read().decode())

Should the production suggest handler call this runner on every click from a tired agent? Please do not, because that turns a read-only eval into a latency tax and a fresh outage. The gate I actually want is coarser and meaner than a per-request benchmark. The suggest route reads the latest eval_runs row for the template version and refuses to draft when that row failed or is missing.

# inside suggest, after auth, before any model call
row = DB.execute(
    """SELECT passed, failure_reason, prompt_hash FROM eval_runs
       WHERE template_version = ?
       ORDER BY id DESC LIMIT 1""",
    (template_version,),
).fetchone()
if row is None:
    raise HTTPException(status_code=412, detail="eval_corpus_missing")
if row["passed"] != 1:
    raise HTTPException(
        status_code=412,
        detail=f"eval_corpus_failed:{row['failure_reason']}",
    )

412 Precondition Failed is an honest status here, and I will defend it in an incident review. It tells the UI the loop is not allowed to speak, which is better than a fluent lie wrapped in 200 OK. I would rather page on 412 than apologize for a refund that never existed in the billing system. Would your current handler even have a place to put that check, or does the model call sit on the first line of the route?

What failed along the way is the usual comedy of skipping layers because the lab felt inexpensive. I first kept fixtures in a slide deck, and nobody updated them when the template learned a warmer apology. I scored replies with a second model and called that eval, which just doubled the improvisation and hid the disagreement. I also ran the corpus against cleaned fake tickets that never contained the word chargeback, so of course the live one exploded on the first angry paragraph.

The frozen slice has to look like production text, including the ugly grammar and the threat that makes lawyers twitch. There is a maintainability tradeoff in the other direction, and I will not hide it behind a slogan. A corpus that is too strict will block harmless tone changes and train the team to ignore 412 the way they ignore flaky tests. A corpus that only checks JSON keys will bless a polite fabrication with a confidence float. I keep forbidden substrings tied to irreversible verbs such as refunded, shipped, deleted, banned, and wired.

Who should not take this path if they are being honest about the product? If you are hacking a weekend chat toy with no customer records, a frozen corpus is overhead, and you should enjoy the toy. If your product is open-ended creative writing, forbidden substrings will fight the whole premise, and a human review queue is more honest. If no one owns the fixture files, a free server will just host yesterday's lucky prompt. That is not an eval program. That is a souvenir with a public port.

I am also not claiming this runner measures model quality in the abstract, and I am not claiming any rehearsal host stays free as a permanent capacity plan. I am claiming that write-shaped language needs a read-only rehearsal with evidence you can replay. Cheap loops without fixtures are how you get confident sentences about money. Expensive loops without fixtures have the same bug. They just arrive with an invoice and a nicer dashboard.

The path I reuse looks like a vertical slice instead of a vision document. Freeze three nasty tickets as JSON beside the API. Pin the template file and hash it in the eval row. Run the corpus on a host you can delete without a change-control meeting. Store pass and fail in eval_runs so an argument has a timestamp. Block suggest with 412 when the latest run is missing or red. Keep send or apply on a different command so a replay cannot mail anyone. Delete the rehearsal database when you are finished fighting the prompt.

If you cannot point at the last green fixture id, the loop does not talk to customers. I like free labs when they stay labs. I dislike complimentary confidence that walks into a support thread. Build the frozen eval until a fabricated refund is a CI failure instead of a Slack novel, and then you can argue about temperature like adults.

Which layer handoff is least stable when your assistant drafts a reply today? Send the failure state or the response code, not the nickname of the model that sounded fluent in the demo.