Every week there's a new "best free model" thread, and every week I watch developers pick one based on vibes, a screenshot, or someone else's benchmark that has nothing to do with their workload. Then three days later they're rewriting prompts because the model falls apart on their actual inputs.
The fix isn't finding a better leaderboard. It's running a tiny, repeatable evaluation against your own tasks before you commit. This post walks through the harness I use. It's about 60 lines of Python, it works against any OpenAI-compatible endpoint, and it turns "which model should I use" from a debate into a diff.
The problem with picking models by feel
Generic benchmarks measure generic tasks. Your workload is not generic. A model that aces MMLU might mangle your domain-specific extraction prompts, and a model that's mediocre at chat might be perfectly fine at the one structured-output task you actually need.
So the unit of comparison shouldn't be the model. It should be your task, run identically against N models, scored the same way.
What you need
- A set of 10–30 real inputs from your project (sanitized — strip secrets and PII)
- For each input, either an expected output or a scoring rule
- One or more model endpoints to compare
On the endpoint side, free tiers are fine for this. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, which makes it a reasonable candidate to slot into a comparison like this — you can point the harness at it alongside whatever else you're evaluating, without paying for the experiment. The harness below doesn't care which endpoint it talks to, so treat that as one row in your results table, not the conclusion.
The harness
The design is boring on purpose: a JSONL file of cases, a runner that hits each endpoint with identical prompts, and a scorer. Determinism matters — pin the temperature to 0 and log raw responses so you can re-score later without re-calling the API.
# eval_harness.py — minimal model comparison harness
import json, time, hashlib, sys
from openai import OpenAI
ENDPOINTS = {
# name -> (base_url, model_id). Fill in your own.
"endpoint_a": ("https://api.example-a.com/v1", "model-a"),
"endpoint_b": ("https://api.example-b.com/v1", "model-b"),
}
def load_cases(path="cases.jsonl"):
with open(path) as f:
return [json.loads(line) for line in f if line.strip()]
def run_case(client, model, case):
start = time.time()
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": case["prompt"]}],
temperature=0,
)
latency = time.time() - start
text = resp.choices[0].message.content
return {"output": text, "latency_s": round(latency, 2)}
def score(case, output):
# Exact-match is the simplest honest scorer. Replace with your own rule:
# JSON-schema validation, regex, substring checks, a rubric, whatever fits.
expected = case.get("expected")
if expected is None:
return None # unscored case, review manually
return output.strip() == expected.strip()
def main(endpoint_name):
base_url, model = ENDPOINTS[endpoint_name]
client = OpenAI(base_url=base_url, api_key="YOUR_KEY")
results = []
for case in load_cases():
r = run_case(client, model, case)
r["case_id"] = hashlib.md5(case["prompt"].encode()).hexdigest()[:8]
r["score"] = score(case, r["output"])
results.append(r)
out = f"results_{endpoint_name}.jsonl"
with open(out, "w") as f:
for r in results:
f.write(json.dumps(r) + "\n")
scored = [r for r in results if r["score"] is not None]
if scored:
acc = sum(r["score"] for r in scored) / len(scored)
print(f"{endpoint_name}: {acc:.1%} exact-match on {len(scored)} cases")
if __name__ == "__main__":
main(sys.argv[1])
And the case file — one JSON object per line:
{"prompt": "Extract the version number from: 'Fixed crash in v2.14.1 when...' Reply with only the version.", "expected": "2.14.1"}
{"prompt": "Classify this ticket as billing, bug, or feature: 'I was charged twice for...' Reply with one word.", "expected": "billing"}
Run it once per endpoint, then diff the results_*.jsonl files. The per-case IDs make it obvious where a model fails, not just how often — that failure pattern is usually the real decision input.
What this actually tells you
Three things a leaderboard can't:
- Failure shape. A model at 85% accuracy that fails randomly is very different from one that fails only on multi-line inputs. The second is fixable with prompt changes; the first isn't.
- Latency on your prompts. Interactive tools and batch jobs have completely different tolerances. You now have numbers for your actual prompt lengths.
- Consistency. Rerun the same cases a day apart. Some endpoints drift more than others, and temperature 0 doesn't guarantee identical outputs on every provider.
Limitations, honestly
- Ten cases is a smoke test, not statistics. Small samples lie confidently. If the decision matters, grow the case set.
- Exact-match scoring penalizes models that answer correctly but verbosely. Write scorers that match your real acceptance criteria, not the easiest metric to code.
- Free tiers and free servers come with whatever rate limits, availability, and quota constraints the provider sets — check the current terms before you build on top of them, and don't design a production system around a free offering's generosity.
- This harness compares endpoints, not hosting, privacy, or data-retention policies. Those are separate questions and sometimes the deciding ones.
Who should skip this
If your task is genuinely one-shot and low-stakes, building an eval harness is overkill — just try the thing. And if your workload is dominated by a single well-supported commercial model with strict compliance requirements, the comparison question is probably already answered for you by policy.
For everyone else: the next time a "best free model" thread tempts you, spend an hour building your case file instead. If you want a zero-cost row in your results table, MonkeyCode's free model access and free server are one option worth pointing the harness at — then let your own cases decide.
What does your case file look like? I'd be curious what task types people find most discriminating between models.
Top comments (0)