Earlier this week I did something deliberately boring: I pointed a regression harness at my own prompt and let it run for two straight days. The goal was simple — catch a bad prompt change before it reached production, not after a user filed a complaint. What I actually got was a 48-hour lesson in how often an evaluation harness lies to you, and how rarely the model is the one doing the lying.
The whole experiment ran on MonkeyCode's free server option with free model access, which meant the only thing I spent was attention. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Zero cost changed my workflow: I stopped rationing test runs and treated the harness like a flaky colleague instead of a source of truth. By the end, that distrust was the most valuable tool I had.
What I was actually testing
Every night, the harness compared a candidate prompt against a pinned baseline using a golden set of 40 support-ticket intents. Each case carried a label, a rubric, and a note about why it was tricky, and the harness scored both prompts with the same model call. Nothing about that design was exotic, and that was the point: I wanted a boring harness so any signal it produced would come from the model, not from my scaffolding.
# harness_core.py — simplified for this article
def evaluate(prompt, cases):
results = []
for case in cases:
answer = call_model(prompt, case["input"])
results.append(grade(answer, case["rubric"]))
return results
Fourteen failures came back over 48 hours. How many were the model's fault? Two. The other twelve were my harness, my fixture, or the server's cold shoulder.
The field log: what broke
The cold start looked like a model timeout. The server had been idle for hours, so the first call of the night took 40 seconds instead of 4. My harness treated any call over 15 seconds as a failure, and the whole run collapsed before the model ever saw the prompt. I now separate infrastructure timeouts from model timeouts before I judge either one.
Exact-match assertions graded wording, not meaning. The model answered "refund policy" where the baseline said "refunds", and my string comparison called it a regression. The model was right and the harness was wrong, which is the worst kind of failure because it trains you to ignore real signals.
One failing test poisoned the next. My fixture reused a shared client object, and a failed case left it in a broken state. The next three cases failed for reasons that had nothing to do with the prompt, and the candidate prompt took the blame. Shared state in an LLM harness is a landmine.
Rate limits masqueraded as bad answers. I fanned out ten parallel calls to speed things up, and the server pushed back with 429s. My harness recorded those as wrong outputs, so the candidate prompt looked like a disaster when it was actually a polite queue.
The triage script I'd repeat
After the second night, I stopped trusting the pass/fail column and started classifying every failure by its log fingerprint. The script below is the artifact I wish I had written on day one, because it turns a wall of red into a simple category count.
# triage.py — classify a harness failure by log fingerprint
import sys
def classify(line):
if "429" in line or "rate_limit" in line:
return "INFRA_RATE_LIMIT"
if "timeout" in line and "connect" in line:
return "INFRA_COLD_START"
if "timeout" in line:
return "MODEL_SLOW"
if "assert" in line and "exact" in line:
return "HARNESS_ASSERTION"
return "UNKNOWN"
for line in sys.stdin:
print(classify(line))
Run that against structured logs, count the categories, and you will quickly see which component is actually flaky. My 48-hour split was two model regressions, three infrastructure blips, and nine harness failures. The model was the most reliable part of the system, which is not a sentence I expected to write.
The decision table
| Observation | Repeat | Change |
|---|---|---|
| Cold starts caused false timeouts | Warm up with a tiny request before the real run | Use separate timeouts for infra and model |
| Exact-match caught wording changes | Grade with a rubric, not a string | Keep a human review for ambiguous cases |
| Shared fixture poisoned later cases | Rebuild client state per case | Run every case in isolation |
| 429s looked like bad answers | Log status codes separately from outputs | Add backoff and retry before judging |
Three things I would repeat: the warmup call, per-case isolation, and the triage script. Three things I would change: exact-match grading, parallel fan-out without backoff, and any assertion that cannot tell a 429 from a wrong answer.
Limitations: who should not copy this
This approach assumes your logs are structured enough to fingerprint, your golden set is small enough to review by hand, and your tolerance for false alarms is low. If you need a guaranteed nightly SLA, a free server is not the right foundation, and if your evaluation needs statistical significance, forty cases will not get you there. Free tiers also change without notice, so pin the date and the provider in your notes instead of in your assumptions.
Forty-eight hours of field notes taught me one thing: when an LLM pipeline misbehaves, the model is now the last suspect I check. The harness, the server, and my own assertions all lied to me first. If you have a flaky-harness story of your own, I would honestly love to read it in the comments.
Top comments (0)