Learn Why Open-Model Benchmarks Mislead by Building a Tiny Eval Harness

Every time an open-weight model trends — MiniMax's latest release is the current example lighting up my feed — the same thing happens: someone posts a benchmark screenshot, someone else posts a contradicting one, and the comments argue about which number is real.

As a student, I kept wondering: why do two honest evaluations of the same model disagree? So I built a tiny, reproducible eval harness to find out. The answer surprised me: most beginner-level benchmarks fail not because of the model, but because of the harness.

This post shows you the harness, the exact ways it lies, and how to fix it — about 80 lines of Python, standard library only.

What you will learn

A single number ("the model scores 78%") hides three silent failure modes: prompt formatting drift, sampling nondeterminism, and answer-parsing false matches. You will reproduce all three yourself, then fix them.

Prerequisites

• Python 3.11+ (tested on 3.12)
• Access to any LLM API endpoint (I'll note a free option below)
• ~30 minutes

The experiment setup

We'll evaluate a model on a deliberately trivial task: arithmetic word problems with exact numeric answers. Trivial on purpose — when the task is easy, any harness failure becomes visible instead of being blamed on "the model being bad."

For model access, I used MonkeyCode, which currently offers free model access and a free server option — handy when you're a student running repeated eval loops and don't want a credit-card meter ticking. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I bring it up here because it genuinely participates in the method: eval iteration only works if you can afford to re-run, and free access is what let me run the failing fixture below 20+ times to confirm the flakiness. That said, the harness below is plain HTTP — swap in any endpoint you already have. Check the provider's current terms yourself; free tiers change and I'm not claiming any quota or permanence.

One thing I appreciate as a learner: the open-source posture around this ecosystem — open-weight models like the MiniMax release, open tooling, reproducible scripts — is what makes this kind of "don't trust the number, test it yourself" exercise possible at all. Closed demos can't be audited. Open artifacts can.

The harness (v1 — intentionally naive)

# eval_v1.py — the version that lies to you
import json, urllib.request

API_URL = "https://your-endpoint.example/v1/chat/completions"  # swap yours
API_KEY = "YOUR_KEY"

CASES = [
    {"q": "Tom has 3 apples, buys 4 more. How many?", "answer": "7"},
    {"q": "A train goes 60 km in 2 hours. Speed in km/h?", "answer": "30"},
    {"q": "12 candies shared by 4 kids. Each gets?", "answer": "3"},
]

def ask(prompt: str) -> str:
    body = json.dumps({
        "model": "your-model-name",
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(
        API_URL, data=body,
        headers={"Authorization": f"Bearer {API_KEY}",
                 "Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.load(r)["choices"][0]["message"]["content"]

score = 0
for c in CASES:
    out = ask(c["q"])
    if c["answer"] in out:          # <-- bug #1
        score += 1
    print(repr(out))
print(f"Score: {score}/{len(CASES)}")

Run it. Then run it again. You will likely see different scores across runs — on a task a five-year-old could pass.

Failure fixture 1: parsing false matches

Look at the check c["answer"] in out. Now feed this expected output:

Tom starts with 3, buys 4, so 3 + 4 = 7. He has 7 apples.
Score: 3/3

Looks right. But change the expected answer of case 1 to "3" (the wrong answer) — the substring check still passes, because "3" appears in "starts with 3". Your harness just scored a wrong answer as correct. This is why published "scores" from quick scripts are unreliable.

Failure fixture 2: sampling nondeterminism

Most endpoints default to a nonzero temperature. Same prompt, different sampling, different prose — and sometimes different arithmetic slip-ups on harder cases. Run your real fixture 5 times and log every raw response:

# consistency check — the part most benchmarks skip
from collections import Counter
runs = [ask(CASES[1]["q"]) for _ in range(5)]
print(Counter(runs))

Expected output (yours will vary — that's the point):

Counter({"The speed is 30 km/h.": 3, "60/2 = 30 km/h": 1, "Speed = 30 km/h.": 1})

If one of those runs had said "35 km/h", a single-run benchmark would have either caught it or missed it by luck. Fix: pin temperature: 0 for evals, and report pass rate over N runs, not one.

Failure fixture 3: prompt formatting drift

Change "Tom has 3 apples, buys 4 more. How many?" to "Q: Tom has 3 apples, buys 4 more.\nA:" and re-run. On smaller models you can watch scores move from formatting alone. A benchmark that doesn't freeze its prompt template is measuring the template as much as the model.

The fixed harness (v2)

# eval_v2.py — honest enough to trust
import json, re, urllib.request
from collections import Counter

def grade(out: str, answer: str) -> bool:
    # extract the LAST number-like token; compare as strings
    nums = re.findall(r"-?\d+(?:\.\d+)?", out.replace(",", ""))
    return bool(nums) and nums[-1] == answer

def evaluate(ask_fn, cases, n_runs=3):
    for c in cases:
        prompt = f"Answer with one number only.\n{c['q']}"
        results = [grade(ask_fn(prompt), c["answer"]) for _ in range(n_runs)]
        print(f"{c['q'][:40]:40s} pass {sum(results)}/{n_runs}")

Key changes:

1. Structured output demand — "Answer with one number only" makes parsing nearly deterministic.
2. Real grading — extract the final numeric token instead of substring matching.
3. N-run reporting — pass 2/3 exposes flakiness that pass hides.

Expected output on a competent open model:

Tom has 3 apples, buys 4 more. How  pass 3/3
A train goes 60 km in 2 hours. Spee  pass 3/3
12 candies shared by 4 kids. Each g  pass 3/3

If you don't get 3/3 on trivial cases at temperature 0, the problem is your harness or prompt — not proof the model is bad.

What you should understand now

• A benchmark number is a claim about a system: model + prompt template + sampling params + grader. Change any one, change the number.
• Substring matching is the most common beginner grading bug.
• Flakiness is signal. pass 2/3 tells you something pass never will.
• This is why open models matter to learners: you can rerun, audit, and pin every layer. You can't audit a marketing screenshot — and it's why I tend to favor tools with an open-source ethos, where the moving parts stay inspectable.

Common mistakes

• Comparing two models using different prompt templates "because each model likes its own." Then you're measuring templates.
• Grading long-form answers with substring checks (the false-match bug scales up badly).
• Reporting a single run as a score. Minimum honest unit: pass rate over N runs with logged raw outputs.

Limitations, and who shouldn't use this

This harness is for learning how evals lie, not for publishing leaderboards. Three cases prove nothing about real capability; real eval suites (MMLU, GSM8K, etc.) exist for a reason and have their own documented pitfalls. If you need a decision-grade model comparison for production, use an established framework with contamination checks — don't extrapolate from a toy. Also: free tiers are a convenience for learning loops, not an SLA; design your scripts to survive endpoint changes.

Extension exercise

Add a deliberately adversarial case, e.g. "answer": "0.5" with the prompt "Half of one pizza is what fraction of a pizza?" — watch how many models answer "1/2" and fail your string grader. Then ask: is the model wrong, or is your grader wrong? That ambiguity is the entire lesson.

Before you run it: predict which of the three failure fixtures will bite your model first, and post your guess (and your Counter output) in the comments. The most interesting failures are the ones I haven't seen yet.