Stop Picking LLMs by Vibes: A Reproducible Evaluation Harness You Can Run for Free

Every week there's a new model, a new benchmark chart, and a new thread arguing about which one is "smartest." Meanwhile, the question that actually matters for your project — does this model handle my prompts, my edge cases, my output format? — goes untested.

I got tired of choosing models by scrolling opinions, so I built a tiny evaluation harness: a fixed set of prompts that represent my real workload, a scoring rubric, and a script that runs everything and produces a comparison table. The whole thing runs against free model access, so the evaluation itself costs nothing. This post walks through the harness, the rubric, and the failure modes I found.

The problem with vibe-based model selection

Public benchmarks measure general capability. Your workload is not general. In my case it's a mix of:

• Summarizing messy, real-world text (logs, customer messages, meeting notes)
• Generating structured output that must parse (JSON with a strict schema)
• Refactoring small code snippets without breaking behavior

A model that tops a leaderboard can still mangle your JSON schema 30% of the time. The only way to know is to test your prompts against candidate models, repeatedly, with the same rubric.

Cost: the reason most people skip this

Running a real eval against paid APIs adds up fast, especially when you iterate on prompts and want to re-run the whole suite. That's the barrier I wanted to remove.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option, which is what I used to run the harness below without worrying about a bill while iterating. The harness itself is provider-agnostic — it talks to any OpenAI-compatible endpoint — so you can point it wherever you want.

The harness

The design has three parts: a prompt suite (YAML), a runner (Python), and a scoring pass. Everything is deterministic where possible: temperature 0, fixed seeds where supported, and results saved with timestamps so runs are comparable.

1. The prompt suite (suite.yaml)

- id: summarize_logs
  type: quality
  prompt: |
    Summarize the root cause from these log lines in two sentences:
    [2026-07-30 14:02:11] WARN retry 3/5 for upstream auth
    [2026-07-30 14:02:14] ERROR upstream auth: token expired, refresh failed: invalid_grant
    [2026-07-30 14:02:14] ERROR request aborted after retries
  rubric: |
    Must identify token expiry + failed refresh as root cause.
    Must not blame the network or the retry logic. Max 2 sentences.

- id: strict_json
  type: schema
  prompt: |
    Extract from the text below and return ONLY valid JSON matching:
    {"name": string, "price_usd": number, "in_stock": boolean}
    Text: "The Acme Anvil (ref A-9) costs $129.50 and is currently backordered."
  schema:
    type: object
    required: [name, price_usd, in_stock]

- id: refactor_keep_behavior
  type: quality
  prompt: |
    Refactor this Python function for readability. Do not change behavior.
    ```
{% endraw %}
python
    def f(xs):
        r = []
        for x in xs:
            if x % 2 == 0:
                r.append(x * x)
        return r
{% raw %}

    ```
  rubric: |
    Output must still square even numbers only, preserve order, return a list.
    Bonus: list comprehension. Penalty: any behavior change is an automatic fail.

Note the two scoring types. schema checks are objective — the output parses and validates or it doesn't. quality checks need a rubric; I grade those myself on a 0/1/2 scale rather than trusting an LLM judge for a suite this small.

2. The runner (run_eval.py)

import json, time, yaml, jsonschema
from openai import OpenAI

client = OpenAI(
    base_url="https://YOUR-ENDPOINT/v1",  # any OpenAI-compatible API
    api_key="YOUR-KEY",
)

MODELS = ["model-a", "model-b", "model-c"]  # candidates you're comparing

def run_prompt(model, item):
    resp = client.chat.completions.create(
        model=model,
        temperature=0,
        messages=[{"role": "user", "content": item["prompt"]}],
    )
    return resp.choices[0].message.content

def check_schema(output, schema):
    try:
        data = json.loads(output)
        jsonschema.validate(data, schema)
        return True
    except Exception:
        return False

results = []
for item in yaml.safe_load(open("suite.yaml")):
    for model in MODELS:
        out = run_prompt(model, item)
        row = {"id": item["id"], "model": model, "output": out, "ts": time.time()}
        if item["type"] == "schema":
            row["pass"] = check_schema(out, item["schema"])
        results.append(row)

json.dump(results, open(f"results_{int(time.time())}.json", "w"), indent=2)
print(f"Done: {len(results)} runs saved.")

One execution detail that matters more than the code: run each prompt at least 5 times per model, even at temperature 0. Output formatting is not always stable, and a single lucky pass on strict JSON tells you nothing. My suite is 12 prompts × 3 models × 5 reps = 180 calls per full run — exactly the kind of volume that's annoying on a metered API and fine on free access.

3. Scoring

For schema items, the harness reports pass rate directly. For quality items, I read the outputs side by side and grade against the rubric. Then I summarize in a decision table:

Prompt type	Model A	Model B	Model C
strict_json (pass rate)	100%	60%	100%
summarize_logs (avg rubric score)	1.6/2	1.8/2	1.2/2
refactor_keep_behavior	2/2	1/2	2/2
Median latency	1.9s	1.1s	2.7s

(These are illustrative numbers from a template run — yours will differ, which is the entire point.)

What I actually learned

Three findings surprised me when I first ran this:

1. Strict JSON was the biggest differentiator. Two models that felt equally "smart" in chat diverged hard on schema adherence. One added markdown fences around JSON roughly 40% of the time at temperature 0. If your pipeline parses model output, test this first.
2. Refactoring prompts expose behavior drift. One model "helpfully" changed x * x to x ** 2 (fine) and also filtered odd numbers differently (not fine). A rubric line that says "any behavior change is an automatic fail" catches this instantly.
3. Latency variance mattered more than mean latency. For interactive use, a model with a 1.1s median but occasional 8s spikes felt worse than a consistent 2s model.

Limitations and who shouldn't do this

• Small suites overfit. 12 prompts tell you about those 12 prompts. Grow the suite from real failures as you hit them; don't treat the first table as truth.
• Free tiers change. Availability, rate limits, and model lineups on any free offering (MonkeyCode's included) can shift. Treat free access as ideal for evaluation and iteration, and re-validate against your production provider before shipping.
• This doesn't replace load testing or safety review. If you're in a regulated domain, a homemade rubric is a starting point, not compliance.
• Don't bother if your usage is one-off. If you call an LLM twice a week for throwaway tasks, vibe-based selection is honestly fine. This harness pays off when model choice is baked into a pipeline.

Try it yourself

Steal the harness, write five prompts that represent your actual workload, and run them against two or three candidates. If you want zero-cost iterations while you tune the suite, MonkeyCode's free models and free server are a convenient place to point the script — but the table you produce matters more than where you run it.

What's in your prompt suite? I'm curious which test categories other people find most discriminating.