Pick Your LLM With a Scoreboard, Not a Hunch: A Two-File Eval That Costs Nothing to Re-Run

The decision nobody writes down

Ask a team why they run model X in production and the honest answer is usually "someone tried it in a chat window and it looked good." That works right up until the inputs stop looking like the demo: an edge case arrives, the model bluffs instead of abstaining, and the bad output flows straight into whatever consumes it downstream.

Code doesn't get this luxury. When behavior matters, we encode expectations as tests and re-run them every time something changes. Model selection deserves the same treatment, because "something changed" happens constantly in LLM land: you switch candidates, you edit the prompt, or the vendor ships a silent revision under an unchanged model name.

What follows is a minimal harness for that: a versioned set of labeled inputs, a deterministic grader, and a scoreboard you regenerate whenever you want. It runs against any free model endpoint and lives on any small always-on machine — the whole comparison phase can cost zero while you're still deciding.

Why the evaluation phase is the free phase

The questions you need answered before committing to a model are cheap to ask:

• Can anything on the market handle this task shape, even imperfectly?
• Which failure shows up first — broken JSON, invented fields, refusal when it should answer?
• Is the model weak, or is my prompt just ambiguous?

A free endpoint answers all three. And there's an underrated outcome: if every candidate fails identically, that tells you the bottleneck is task design, not model quality — which saves you from upgrading your way out of a prompt problem.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

In my setup, candidates are served through MonkeyCode's free model access, and the harness runs on its free server option, which also acts as the archive for past scoreboards. Neither choice is load-bearing: the script below pulls the base URL, credential, and model name from environment variables, so any OpenAI-compatible endpoint and any tiny VM work identically. I'm intentionally not quoting model catalogs, rate allowances, or how long the free tier lasts — those terms shift, so verify the current offering yourself before building on it.

The artifact: receipts, gold labels, and a grader that distrusts confidence

The walkthrough task: pull a total amount and currency out of messy receipt text. It's a good eval subject because the output is tiny, structured, and checkable by pure functions — no vibes required.

Three design commitments keep the harness meaningful:

1. Cases are files, not conversation logs. They live in git, diff like code, and grow every time production surprises you.
2. Deterministic checks first. Numeric comparison with tolerance, enum membership, JSON parseability. Human judgment is reserved for what a function genuinely cannot decide.
3. "I can't tell" is a valid answer with a score. A model that fabricates a total from ambiguous text is more dangerous than one that declines — the grader must reflect that asymmetry.

receipts.jsonl:

{"id": "r-001", "text": "COFFEE NORTHSIDE\\n2x latte 9.00\\nTOTAL $9.00\\nthank you", "gold": {"total": 9.00, "currency": "USD"}}
{"id": "r-002", "text": "Boulangerie Marie\\nCroissant 2,40 EUR\\nTotal: 2,40 EUR", "gold": {"total": 2.40, "currency": "EUR"}}
{"id": "r-003", "text": "TAXI RECEIPT\\nfare ...... 18.50\\ntip ....... 3.00\\nCARD ****4412", "gold": null}

That third row is the landmine: the receipt shows numbers but no explicit total. The honest answer is an abstention. A model that sums the lines and reports 21.50 has produced plausible-looking garbage — the exact thing that corrupts a downstream ledger.

receipt_eval.py (a starting point, not gospel):

import json, os, time, urllib.request, urllib.error

BASE  = os.environ["EVAL_BASE_URL"]      # any OpenAI-compatible endpoint
TOKEN = os.environ["EVAL_API_KEY"]
MODEL = os.environ["EVAL_MODEL"]

SYSTEM = (
    "Extract the grand total from the receipt. Respond with a single JSON object "
    '{"total": <number>, "currency": "USD|EUR|GBP"}. '
    "If no explicit grand total is printed, respond with: unclear"
)

def ask(receipt_text):
    payload = json.dumps({
        "model": MODEL,
        "temperature": 0,
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": receipt_text},
        ],
    }).encode()
    req = urllib.request.Request(
        BASE.rstrip("/") + "/chat/completions",
        data=payload,
        headers={"Authorization": "Bearer " + TOKEN,
                 "Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        return json.load(resp)["choices"][0]["message"]["content"].strip()

def judge(case, raw):
    """Return (points, tag). Fabrication is penalized below zero."""
    if raw.strip().lower() == "unclear":
        pred = None
    else:
        try:
            pred = json.loads(raw)
        except json.JSONDecodeError:
            return (-1, "not_json")
    gold = case["gold"]
    if gold is None:
        return (1, "abstained_ok") if pred is None else (-2, "fabricated_total")
    if pred is None:
        return (0, "gave_up")
    total_ok = isinstance(pred.get("total"), (int, float)) and \
               abs(pred["total"] - gold["total"]) < 0.01
    curr_ok = pred.get("currency") == gold["currency"]
    if total_ok and curr_ok:
        return (2, "full_match")
    return (0, "wrong_value" if not total_ok else "wrong_currency")

run = {"model": MODEL, "ts": int(time.time()), "cases": []}
earned = possible = 0
with open("receipts.jsonl") as fh:
    for line in fh:
        case = json.loads(line)
        points, tag = judge(case, ask(case["text"]))
        earned += points
        possible += 2 if case["gold"] else 1
        run["cases"].append({"id": case["id"], "tag": tag, "points": points})
        print(f"{case['id']:>7}  {tag:<16} {points:+d}")

run["score"] = round(100 * max(earned, 0) / possible, 1)
print(f"\nscore: {run['score']}  ({MODEL})")
with open("scoreboard.jsonl", "a") as out:
    out.write(json.dumps(run) + "\n")

No SDK, no framework — one standard HTTP shape, so pointing it at a different provider is an environment-variable change:

EVAL_BASE_URL="https://your-endpoint.example/v1" \
EVAL_API_KEY="..." \
EVAL_MODEL="candidate-a" \
python receipt_eval.py

Two details are doing quiet work here. The weighted tags encode your actual risk model: fabricated_total scores below giving up, because in a bookkeeping pipeline a confident wrong number costs more than a manual review queue. And results append to scoreboard.jsonl with a timestamp, so every run becomes one line of history you can diff later.

Reading a scoreboard without lying to yourself

Aggregates are the beginning of the analysis, not the end. A few habits that keep the numbers honest:

Habit	Why it matters
Diff per case, not per model	Two candidates at 85% can fail on disjoint cases — one might be shippable with a fallback path, the other not
Weight fabrication heavily	A wrong field on a real input is an annoyance; an invented answer on noise is a data-integrity incident
Freeze temperature=0 and the prompt string	Otherwise you're comparing runs, not models
Require a minimum case count	Under ~20 cases, noise picks winners; grow the set before trusting any ranking

Because every run lands in the same append-only file, "did candidate B actually beat candidate A, and on which inputs?" becomes a jq query instead of a memory exercise.

The boring machine is the point

Nothing in the harness needs orchestration. The useful trick is simply running it again later, which is what the always-on free server is for:

42 5 * * 1  cd /srv/receipt-eval && . ./env.sh && python receipt_eval.py >> weekly.log 2>&1

A weekly re-run against the same model name catches the failure mode no demo ever shows: the vendor revises the model silently, extraction behavior drifts, and your pipeline degrades without a single error being raised. With dated scoreboard lines, drift becomes a visible event with a before-and-after diff, not a mystery you debug for a weekend. One small box executing one small script is precisely the right amount of infrastructure here — the harness stays alive without turning into a second job.

Where this design runs out of road

• Free capacity is for exploration, not scale. Rate ceilings, context limits, and evolving terms are normal on no-cost tiers. A few dozen cases fit that world comfortably; a thousand-case nightly gate does not — plan for paid capacity when you get there, and treat that as a success signal.
• Function-graded scoring only covers verifiable outputs. Extraction, classification, format compliance: great fit. Summaries, code review comments, creative rewrites: not checkable by equality, and forcing them into this mold produces false confidence. Those need sampled human review or a judge model, with the subjectivity that implies.
• Your case file is a theory of production. Every green run means "green on these inputs." Feed sanitized real-world surprises back into the JSONL continuously, or the suite drifts into irrelevance even while it stays green.
• Abstention grading assumes your domain allows it. If your product must always answer, re-weight the rubric — but make that trade-off explicit rather than accidental.

Skip this if...

...you call a model interactively a few times a week with nothing automated downstream — the harness is ceremony at that scale. Also skip building cases from scratch if a maintained public benchmark genuinely mirrors your task; run that first. And if your output can't be mechanically checked at all, you want a sampling-and-review workflow, not this one.

But if you're about to wire a model into a pipeline based on playground impressions, one JSONL file and one grading script are all that stand between you and an evidence-based choice. If you want a zero-budget way to stand the loop up, MonkeyCode's free model access together with its free server is one concrete option — the endpoint stays swappable by design, which is exactly how an eval harness should treat its infrastructure. Version the cases, keep the scoreboard history, and let the next silent model update be something you measure instead of something you chase.