Stop Picking LLMs by Vibes: A Reproducible Evaluation Harness You Can Run for Free

Dakota Liu — Mon, 10 Aug 2026 08:25:00 +0000

Every week there's a new model, a new benchmark chart, and a new thread arguing about which one is "smartest." Meanwhile, the question that actually matters for your project — does this model handle my prompts, my edge cases, my output format? — goes untested.

I got tired of choosing models by scrolling opinions, so I built a tiny evaluation harness: a fixed set of prompts that represent my real workload, a scoring rubric, and a script that runs everything and produces a comparison table. The whole thing runs against free model access, so the evaluation itself costs nothing. This post walks through the harness, the rubric, and the failure modes I found.

The problem with vibe-based model selection

Public benchmarks measure general capability. Your workload is not general. In my case it's a mix of:

Summarizing messy, real-world text (logs, customer messages, meeting notes)
Generating structured output that must parse (JSON with a strict schema)
Refactoring small code snippets without breaking behavior

A model that tops a leaderboard can still mangle your JSON schema 30% of the time. The only way to know is to test your prompts against candidate models, repeatedly, with the same rubric.

Cost: the reason most people skip this

Running a real eval against paid APIs adds up fast, especially when you iterate on prompts and want to re-run the whole suite. That's the barrier I wanted to remove.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option, which is what I used to run the harness below without worrying about a bill while iterating. The harness itself is provider-agnostic — it talks to any OpenAI-compatible endpoint — so you can point it wherever you want.

The harness

The design has three parts: a prompt suite (YAML), a runner (Python), and a scoring pass. Everything is deterministic where possible: temperature 0, fixed seeds where supported, and results saved with timestamps so runs are comparable.

1. The prompt suite (`suite.yaml`)

- id: summarize_logs
  type: quality
  prompt: |
    Summarize the root cause from these log lines in two sentences:
    [2026-07-30 14:02:11] WARN retry 3/5 for upstream auth
    [2026-07-30 14:02:14] ERROR upstream auth: token expired, refresh failed: invalid_grant
    [2026-07-30 14:02:14] ERROR request aborted after retries
  rubric: |
    Must identify token expiry + failed refresh as root cause.
    Must not blame the network or the retry logic. Max 2 sentences.

- id: strict_json
  type: schema
  prompt: |
    Extract from the text below and return ONLY valid JSON matching:
    {"name": string, "price_usd": number, "in_stock": boolean}
    Text: "The Acme Anvil (ref A-9) costs $129.50 and is currently backordered."
  schema:
    type: object
    required: [name, price_usd, in_stock]

- id: refactor_keep_behavior
  type: quality
  prompt: |
    Refactor this Python function for readability. Do not change behavior.
    ```
{% endraw %}
python
    def f(xs):
        r = []
        for x in xs:
            if x % 2 == 0:
                r.append(x * x)
        return r
{% raw %}

    ```
  rubric: |
    Output must still square even numbers only, preserve order, return a list.
    Bonus: list comprehension. Penalty: any behavior change is an automatic fail.

Note the two scoring types. schema checks are objective — the output parses and validates or it doesn't. quality checks need a rubric; I grade those myself on a 0/1/2 scale rather than trusting an LLM judge for a suite this small.

2. The runner (`run_eval.py`)

import json, time, yaml, jsonschema
from openai import OpenAI

client = OpenAI(
    base_url="https://YOUR-ENDPOINT/v1",  # any OpenAI-compatible API
    api_key="YOUR-KEY",
)

MODELS = ["model-a", "model-b", "model-c"]  # candidates you're comparing

def run_prompt(model, item):
    resp = client.chat.completions.create(
        model=model,
        temperature=0,
        messages=[{"role": "user", "content": item["prompt"]}],
    )
    return resp.choices[0].message.content

def check_schema(output, schema):
    try:
        data = json.loads(output)
        jsonschema.validate(data, schema)
        return True
    except Exception:
        return False

results = []
for item in yaml.safe_load(open("suite.yaml")):
    for model in MODELS:
        out = run_prompt(model, item)
        row = {"id": item["id"], "model": model, "output": out, "ts": time.time()}
        if item["type"] == "schema":
            row["pass"] = check_schema(out, item["schema"])
        results.append(row)

json.dump(results, open(f"results_{int(time.time())}.json", "w"), indent=2)
print(f"Done: {len(results)} runs saved.")

One execution detail that matters more than the code: run each prompt at least 5 times per model, even at temperature 0. Output formatting is not always stable, and a single lucky pass on strict JSON tells you nothing. My suite is 12 prompts × 3 models × 5 reps = 180 calls per full run — exactly the kind of volume that's annoying on a metered API and fine on free access.

3. Scoring

For schema items, the harness reports pass rate directly. For quality items, I read the outputs side by side and grade against the rubric. Then I summarize in a decision table:

Prompt type	Model A	Model B	Model C
strict_json (pass rate)	100%	60%	100%
summarize_logs (avg rubric score)	1.6/2	1.8/2	1.2/2
refactor_keep_behavior	2/2	1/2	2/2
Median latency	1.9s	1.1s	2.7s

(These are illustrative numbers from a template run — yours will differ, which is the entire point.)

What I actually learned

Three findings surprised me when I first ran this:

Strict JSON was the biggest differentiator. Two models that felt equally "smart" in chat diverged hard on schema adherence. One added markdown fences around JSON roughly 40% of the time at temperature 0. If your pipeline parses model output, test this first.
Refactoring prompts expose behavior drift. One model "helpfully" changed x * x to x ** 2 (fine) and also filtered odd numbers differently (not fine). A rubric line that says "any behavior change is an automatic fail" catches this instantly.
Latency variance mattered more than mean latency. For interactive use, a model with a 1.1s median but occasional 8s spikes felt worse than a consistent 2s model.

Limitations and who shouldn't do this

Small suites overfit. 12 prompts tell you about those 12 prompts. Grow the suite from real failures as you hit them; don't treat the first table as truth.
Free tiers change. Availability, rate limits, and model lineups on any free offering (MonkeyCode's included) can shift. Treat free access as ideal for evaluation and iteration, and re-validate against your production provider before shipping.
This doesn't replace load testing or safety review. If you're in a regulated domain, a homemade rubric is a starting point, not compliance.
Don't bother if your usage is one-off. If you call an LLM twice a week for throwaway tasks, vibe-based selection is honestly fine. This harness pays off when model choice is baked into a pipeline.

Try it yourself

Steal the harness, write five prompts that represent your actual workload, and run them against two or three candidates. If you want zero-cost iterations while you tune the suite, MonkeyCode's free models and free server are a convenient place to point the script — but the table you produce matters more than where you run it.

What's in your prompt suite? I'm curious which test categories other people find most discriminating.

DEV Community: Dakota Liu