Most developers pick a model the same way: they paste one prompt into a chat UI, like the answer, and start building on it. A week later the integration falls apart on edge cases nobody tested, and the 'evaluation' turns out to have been a vibe.
I got tired of doing this, so I built a tiny harness I now run against any model I have access to — including free tiers — before I write a single line of production code against it. It takes about 20 minutes to set up, and it has saved me from at least two bad architectural decisions.
This post walks through the harness, the decision table I use to interpret results, and where free model access genuinely fits (and where it doesn't).
Why one-prompt testing fails
A single prompt tests exactly one thing: whether the model got lucky on your prompt. It tells you nothing about:
- Consistency — does it give the same quality of answer on run 10 as on run 1?
- Structured output reliability — can it emit valid JSON every time, or only when it feels like it?
- Latency variance — is the p95 response time 2s or 40s? For anything user-facing, this matters more than the median.
- Failure shape — when it's wrong, is it obviously wrong (easy to catch) or plausibly wrong (dangerous)?
You need a fixed prompt suite, repeated runs, and mechanical checks. None of this requires paid infrastructure.
The setup: free models, free server
Two things made this cheap enough to run casually:
- Free model access. I use MonkeyCode, which offers free access to hosted models, so I can point the harness at a real endpoint without burning API budget during the exploratory phase.
- A free server option. MonkeyCode also provides a free server option, which is enough to host the harness script and store results — no VPS bill for what is essentially a cron job and a JSON file.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm describing how I actually use the free tier for evaluation work; I'm not claiming anything about quotas, specific models, or how long the free access lasts, because those can change and you should check the current terms yourself.
The harness itself is provider-agnostic — it works against any OpenAI-compatible endpoint, so you can rerun it against a paid provider later and compare like-for-like.
The harness
The core is a Python script with three parts: a prompt suite, a runner, and mechanical graders.
1. A fixed prompt suite
Mine lives in suite.json. The key rule: write the prompts before you look at any model's output, or you'll unconsciously tune them to flatter the model you already like.
[
{
"id": "json_extract_01",
"prompt": "Extract the dates from this text and return ONLY a JSON array of ISO-8601 strings: 'The meeting moved from March 3rd to 3/15, and the deadline is 2024-04-01.'",
"grader": "valid_json_array"
},
{
"id": "reasoning_01",
"prompt": "A bat and a ball cost $1.10 total. The bat costs $1.00 more than the ball. Then the ball's price doubles. What does the ball cost now? Answer with only the number.",
"grader": "exact:0.20"
},
{
"id": "code_01",
"prompt": "Write a Python function `dedupe_stable(items)` that removes duplicates while preserving first-occurrence order. Return only code.",
"grader": "runs_and_passes"
}
]
Keep the suite small (10–20 prompts) but make each prompt represent something your actual project needs. My suite is ~40% structured output, because that's what my projects break on.
2. The runner
import json, time, statistics, subprocess, tempfile, os
from openai import OpenAI
client = OpenAI(
base_url=os.environ["MODEL_BASE_URL"], # any OpenAI-compatible endpoint
api_key=os.environ["MODEL_API_KEY"],
)
MODEL = os.environ["MODEL_NAME"]
RUNS_PER_PROMPT = 5
def run_once(prompt):
start = time.monotonic()
resp = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": prompt}],
temperature=0.2,
)
latency = time.monotonic() - start
return resp.choices[0].message.content, latency
def grade(grader, output):
if grader == "valid_json_array":
try:
return isinstance(json.loads(output), list)
except Exception:
return False
if grader.startswith("exact:"):
return output.strip().strip("$.") == grader.split(":", 1)[1]
if grader == "runs_and_passes":
code = output.replace("```
python", "").replace("
```", "")
test = code + "\nassert dedupe_stable([1,2,1,3,2]) == [1,2,3]\nassert dedupe_stable([]) == []\n"
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:
f.write(test)
path = f.name
try:
subprocess.run(["python", path], check=True, timeout=10,
capture_output=True)
return True
except Exception:
return False
raise ValueError(f"unknown grader {grader}")
results = {}
for item in json.load(open("suite.json")):
passes, latencies, failures = 0, [], []
for _ in range(RUNS_PER_PROMPT):
out, lat = run_once(item["prompt"])
latencies.append(lat)
if grade(item["grader"], out):
passes += 1
else:
failures.append(out[:200])
results[item["id"]] = {
"pass_rate": passes / RUNS_PER_PROMPT,
"p50_latency": statistics.median(latencies),
"p95_latency": sorted(latencies)[int(len(latencies) * 0.95) - 1],
"sample_failures": failures[:2],
}
json.dump(results, open("results.json", "w"), indent=2)
print(json.dumps(results, indent=2))
Set MODEL_BASE_URL / MODEL_API_KEY / MODEL_NAME for whatever you're evaluating. On the free server I run this as a scheduled job and commit results.json to a git repo, which gives me a history of model behavior over time for free.
Five runs per prompt is a minimum, not a target. Bump it to 20+ if a result is close to a decision boundary.
3. Reading the results: my decision table
| Signal | Threshold I use | What I do |
|---|---|---|
| Structured-output pass rate | < 90% | Never use without a validation-and-retry loop |
| Reasoning pass rate | < 80% | Keep it out of any logic that feeds user-facing answers |
| p95 latency | > 3× p50 | Assume queueing under load; add timeouts + fallbacks |
| Failure shape | Plausibly wrong | Add an automated checker downstream, or don't use it |
| Same prompt, next week | Pass rate shifts | Treat the endpoint as mutable; pin behavior in CI |
The last row is the one people skip. Free tiers especially can change what's behind the endpoint. Re-running the same suite weekly against the same git history is how you notice a silent downgrade before your users do.
What I actually found
Running this against free access before committing has twice changed my plans:
- A model that looked great in chat scored 60% on my JSON extraction prompts — the failures were valid JSON with subtly wrong content, the worst kind. That project got a schema-validation layer it otherwise wouldn't have had.
- Another model passed everything but had p95 latency roughly 4× its median. Fine for my batch job, disqualifying for the interactive feature I'd originally planned. Knowing that before building the feature was the entire point.
Limitations and who shouldn't do this
- A 15-prompt suite is not a benchmark. It measures fitness for your tasks, nothing more. Don't publish the numbers as model comparisons.
- Free access is for evaluation and light workloads. Don't architect a production system around a free tier; quotas, availability, and the models behind the endpoint can change. Evaluate for free, then re-run the identical suite against the paid/production endpoint you actually intend to ship on.
-
Mechanical graders miss quality. A response can be valid JSON and still be a bad answer. I spot-check failures manually —
sample_failuresin the output exists for exactly that reason. - Skip this entirely if your task is a one-off, if you have no automatable notion of "correct," or if your prompts contain data you can't send to a third-party endpoint. The harness assumes API access is acceptable for your data.
Try it
If you're evaluating models right now: write ten prompts that represent your real workload before touching any chat UI, then run them mechanically. If you want somewhere to point the harness without spending money, MonkeyCode's free model access and free server are what I used for the setup above — but the script works against any compatible endpoint, and honestly the suite matters more than where it runs.
What's in your prompt suite? I'm curious which failure modes other people test for first.
Top comments (0)