Your Eval Ran Once. That's a Demo, Not a Measurement.

One green run proves almost nothing about a model. It proves one request succeeded, once, on your laptop, today.

I keep meeting the same five mistakes behind "we already tested it." Each one turns a demo into a number you cannot defend in a review.

Here's the catalog. Symptom, root cause, and the pattern that replaces it. Then a harness you can run for free.

1. The Single-Prompt Verdict

Symptom: someone says "it can't produce valid JSON." They tried it once.

Root cause: sampling variance gets confused with capability. One sample cannot separate the two.

Replacement: fix n before you form an opinion. Five runs is the floor. Ten is better if the output is short.

Ask yourself: would you ship a database after one query?

2. The Untimed Cold Start

Symptom: your p95 looks terrible. The dashboard blames the model.

Root cause: the first request pays connection and load costs. You averaged that into every percentile.

Replacement: send one warm-up call and throw it away. Then measure. Log both so the difference stays visible.

3. The Invisible Prompt Edit

Symptom: scores drift week to week. Nobody touched the code.

Root cause: the prompt lives in a chat window or a notebook cell. There is no hash to compare.

Replacement: prompts as files in the repo, hashed into every result row. Different hash means the numbers are not comparable. How do you diff a prompt you cannot see?

4. The Untracked Failure Body

Symptom: "it just failed." The ticket sits for three days.

Root cause: you logged the status code and dropped the response body. The actual reason was inside that body.

Replacement: store status, key headers, and a truncated body. Redact secrets before it hits git.

5. The Happy-Path Corpus

Symptom: the demo is beautiful. Real users break it within the hour.

Root cause: the corpus contains only clean inputs. Empty strings, huge pastes, and mixed languages never show up.

Replacement: a fixed corpus with labeled categories, committed next to the harness.

The Harness

This is stdlib-only Python. It has no dependencies and roughly sixty lines. I have not run it against your endpoint; treat any numbers you get as yours, not mine.

#!/usr/bin/env python3
"""eval_harness.py - fixed-corpus eval. Not production-hardened."""
import argparse, hashlib, json, os, time, urllib.error, urllib.request


def digest(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:12]


def call(prompt: str, user_input: str):
    body = json.dumps({
        "prompt": prompt,
        "input": user_input,
        "max_tokens": 256,
    }).encode("utf-8")
    req = urllib.request.Request(
        os.environ["MODEL_BASE_URL"].rstrip("/") + "/generate",
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + os.environ["MODEL_API_KEY"],
        },
        method="POST",
    )
    start = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=30) as resp:
            status, raw = resp.status, resp.read().decode("utf-8", "replace")
    except urllib.error.HTTPError as err:
        status, raw = err.code, err.read().decode("utf-8", "replace")
    except urllib.error.URLError as err:
        status, raw = 0, "transport: %s" % err.reason
    return status, round((time.perf_counter() - start) * 1000, 1), raw[:400]


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--prompt", default="prompts/classify.txt")
    ap.add_argument("--corpus", default="corpus/v1.jsonl")
    ap.add_argument("--runs", type=int, default=5)
    ap.add_argument("--out", required=True)
    args = ap.parse_args()

    prompt = open(args.prompt, encoding="utf-8").read()
    prompt_sha = digest(prompt)
    rows = [json.loads(l) for l in open(args.corpus, encoding="utf-8") if l.strip()]

    call(prompt, "warmup")  # discarded on purpose, see anti-pattern 2

    with open(args.out, "a", encoding="utf-8") as out:
        for row in rows:
            for run in range(args.runs):
                status, ms, body = call(prompt, row["input"])
                out.write(json.dumps({
                    "corpus_id": row["id"],
                    "category": row.get("category"),
                    "prompt_sha": prompt_sha,
                    "run": run,
                    "status": status,
                    "ms": ms,
                    "body_sha": digest(body),
                    "body": body,
                }) + "\n")


if __name__ == "__main__":
    main()

The corpus stays boring on purpose

Four categories are enough to start. Add more only when a real bug gives you a reason.

{"id": "c001", "category": "clean", "input": "Refund my order #4411."}
{"id": "c002", "category": "empty", "input": ""}
{"id": "c003", "category": "long", "input": "<paste about 8k tokens of logs here>"}
{"id": "c004", "category": "mixed_lang", "input": "我的订单 #4411 需要退款 refund please"}

Run it

mkdir -p prompts corpus results
printf 'Classify the request. Reply as JSON: {"intent": ..., "order_id": ...}' > prompts/classify.txt
export MODEL_BASE_URL="https://your-endpoint-here"
export MODEL_API_KEY="your-key-here"
python3 eval_harness.py --runs 5 --out results/2026-11-05.jsonl

Read it back with jq

jq -s 'group_by(.corpus_id)
  | map({id: .[0].corpus_id, n: length, ok: (map(select(.status==200))|length)})' \
  results/2026-11-05.jsonl

What to Gate On

Do not gate on vibes. Gate on rows you can re-read in a month.

Gate	Starting threshold	Why it matters
Pass rate, clean	>= 0.9 over 5 runs	Catches variance you would call a bug
Empty input	4xx, never 5xx	A crash is a bug; a rejection is a decision
prompt_sha	identical in every row	Otherwise you compared two different prompts
Warm-up row	excluded from latency	Keeps cold start out of p95
Failure body	present and redacted	The reason is in the body, not the code

Any row that fails a gate is a test case tomorrow. That is the whole return on this work.

Iterating Without Spending Anything

The expensive part is not the final run. It is the tenth rewrite of the harness.

That is where free model access earns its place. MonkeyCode offers free model access and a free server option, per the project operator. I am describing it as an option for smoke runs, not as a benchmark result.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Check the project's own current docs for what the free tier includes today. Token allowances, model lineups, and server specs change. I am not going to quote numbers that will be stale next quarter.

A sane loop looks like this:

1. Write the harness locally against the free model access.
2. Move it to the free server so your home network is out of the latency picture.
3. Commit results/*.jsonl to git. The free server's disk may not survive a restart.
4. Re-run the same corpus against whatever you ship to production.

Step 4 is the one people skip. Same corpus, same prompt hash, different endpoint. Now the comparison means something.

Limitations and Who Should Skip This

• Five runs is not a statistical test. It catches obvious variance. If you need confidence intervals, run more and say so.
• Free tiers are not contracts. Do not wire a CI gate to unversioned availability. Cache a known-good result set instead.
• Ephemeral disk will eat your evidence. Results belong in git, not on the box.
• Do not send regulated or customer data here. Use synthetic corpus rows until your data path is reviewed.
• Skip this if you need a hard latency SLA. A shared free server cannot promise you one.
• Skip the whole harness if you ship one prompt once. The overhead is not worth it below a few hundred requests.

The Part That Actually Matters

A single run is an anecdote with a timestamp. A corpus, a prompt hash, and a discarded warm-up make it a measurement.

None of that is clever. It is just the discipline of writing down what you did before you believe the result.

If you try the harness, keep the results file. That file is the only part worth sharing.