DEV Community: Riley Wu

Hype Cycles Don't Ship My Code: How I Gate New LLMs with a Self-Written Eval Deck

Riley Wu — Mon, 10 Aug 2026 11:12:48 +0000

Every few weeks a new model dominates the timeline. The launch posts look the same every time: a cherry-picked demo, a benchmark chart, a wall of flame emojis — and no answer to the question that actually matters to me, which is "will this thing handle the boring, weird, half-documented work in my repos?"

I learned this the expensive way. I once adopted a trending model based on launch-day enthusiasm and spent a night cleaning up hallucinated CLI flags it had emitted with total confidence. Since then, every model that wants into my workflow has to pass a small deck of adversarial tasks I wrote myself. This post is the full method.

Launch metrics answer launch questions, not yours

Benchmarks are legitimate science — for the benchmark's questions. Your codebase asks different ones:

Does the model follow your formatter and your naming conventions without being told twice?
Can it deal with unglamorous glue: GitHub Actions YAML, a creaky ORM migration, a Helm values file?
Does it hold together when your prompt is long and messy instead of short and clean?
When it fails, does it fail loudly and catchably, or quietly and plausibly?

A viral screenshot tells you none of this. The fix isn't cynicism — it's a tiny verification loop that you own.

The artifact: a deck of task cards plus a dumb runner

I keep each eval as a card: a prompt file plus a machine-checkable contract, stored as plain files so adding a task never touches the runner.

cards/
  tasks/
    pr-summary.md          # summarize this diff for a reviewer
    migration-plan.md      # propose a zero-downtime schema change
    incident-timeline.md   # reconstruct events from these log lines
  contracts/
    pr-summary.yml
    migration-plan.yml
    incident-timeline.yml

A contract declares what a passing answer looks like:

required_terms: ["rollback", "index concurrently"]
forbidden_terms: ["LOCK TABLE", "truncate"]
max_words: 220
samples: 4
required_success_fraction: 1.0

The runner is one dependency-free file. It speaks to any OpenAI-compatible endpoint, so the model under test is just environment config:

// eval.mjs — Node 18+, stdlib only
import { readdir, readFile } from "node:fs/promises";
import { parse } from "node:util"; // placeholder-free: we hand-roll YAML below

const BASE = process.env.LLM_BASE_URL;  // e.g. any OpenAI-compatible endpoint
const KEY  = process.env.LLM_API_KEY;
const MODEL = process.env.LLM_MODEL;

// Tiny YAML-subset parser (only the keys our contracts use)
function parseContract(text) {
  const out = {};
  for (const line of text.split("\n")) {
    const m = line.match(/^(\w+):\s*(.+)$/);
    if (!m) continue;
    const [, k, v] = m;
    out[k] = v.startsWith("[")
      ? JSON.parse(v.replace(/'/g, '"'))
      : isNaN(+v) ? v.trim() : +v;
  }
  return out;
}

async function complete(prompt) {
  const r = await fetch(`${BASE}/v1/chat/completions`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: MODEL,
      messages: [{ role: "user", content: prompt }],
      temperature: 0.1,
    }),
  });
  return (await r.json()).choices[0].message.content;
}

function passes(answer, c) {
  if (answer.split(/\s+/).length > c.max_words) return false;
  const low = answer.toLowerCase();
  return c.required_terms.every(t => low.includes(t.toLowerCase()))
      && c.forbidden_terms.every(t => !low.includes(t.toLowerCase()));
}

for (const f of await readdir("cards/tasks")) {
  const id = f.replace(".md", "");
  const prompt = await readFile(`cards/tasks/${f}`, "utf8");
  const c = parseContract(await readFile(`cards/contracts/${id}.yml", "utf8"));

  let ok = 0;
  const lat = [];
  for (let i = 0; i < c.samples; i++) {
    const t0 = performance.now();
    const ans = await complete(prompt);
    lat.push(Math.round(performance.now() - t0));
    ok += passes(ans, c);
  }
  const frac = ok / c.samples;
  console.log(
    `${id.padEnd(22)} ${ok}/${c.samples}` +
    `  p50=${lat.sort((a,b)=>a-b)[Math.floor(lat.length/2)]}ms` +
    (frac >= c.required_success_fraction ? "  PASS" : "  REJECT")
  );
}

(The toy YAML parser is fine for flat contracts; swap in a real parser if your specs grow nesting.)

Three design choices carry most of the weight:

Sampled consistency, not single-shot luck. Four low-temperature runs per card, and the required fraction decides. A model that's right one time in four isn't skilled — it's a slot machine that paid out while you were watching.
Deny-lists for catastrophic modes. For anything infra-adjacent, certain words auto-fail the answer. I don't care how elegant the prose is if the migration plan contains truncate.
Tasks sourced from my own ticket history. Every card is a sanitized version of a real task from the last month. Curated benchmarks and real glue work correlate less than anyone wants to admit.

To run this you need an endpoint for the candidate model — ideally one that costs nothing while you're still in the poking-around phase. For my most recent round I pointed the runner at MonkeyCode, which offers free model access and a free server option; that meant an afternoon of iterating against the new release with no provisioning and no card on file. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

From raw results to an actual decision

A score line is not a verdict. I run each card's output through a short decision tree:

All cards pass, latency fine?
├─ yes → promote to one real, reversible task this week
└─ no  → is the failure concentrated in ONE card category?
    ├─ yes → adopt it everywhere EXCEPT that category
    ├─ no, failures are flaky/random → reject; re-test next checkpoint
    └─ no, failures are systemic → reject

The single-category-failure branch is where the real value lives. Models aren't uniformly good or bad — they have a shape. The goal isn't to crown a winner; it's to find a model whose weak spots never intersect your workload. A model that can't write commit messages but nails log analysis is a fantastic fit for my on-call triage and a terrible fit for my release notes.

Why open availability is the real story

This workflow only exists because more models are now reachable without an enterprise contract. That trend — whatever vendor's release is trending this week — quietly changes the epistemics: "trust the launch post" becomes "verify on your own machine tonight". It's the same deal open source always offered, applied to weights instead of source. Providers that make independent checking cheap (free model access, a free server tier — MonkeyCode's current arrangement is one example) are removing the last excuse for skipping due diligence. Providers that make it expensive are making a statement too.

Honest limitations

This catches dealbreakers, not gradients. A dozen cards can't finely rank models. Never sign an annual contract on this evidence alone.
Term matching is style-blind. An answer can pass every contract and still be ugly. I hand-read two full responses per model, no exceptions.
Free access is a snapshot, not a covenant. Pricing and availability change. That's precisely why the runner only assumes the OpenAI-compatible API shape — switching endpoints is an env var, not a refactor.
Proprietary code needs clearance first. Sending internal snippets to any external endpoint, free or paid, is a decision your security policy must authorize.
If your org already runs a mature eval platform, skip the harness — though I'd argue the forbidden-terms deny-list trick is still worth stealing.

Wrapping up

The next model launch is already scheduled somewhere. You don't have to choose between drinking the hype and ignoring the field — spend one evening turning your recent tickets into task cards, run the deck, and let your own results argue. If you need a zero-cost endpoint to run this against, MonkeyCode's free tier is a convenient place to start. One thing I'd genuinely like to hear: which catastrophic-failure terms land on other people's deny-lists — mine skews heavily toward destructive filesystem and SQL verbs, which is either prudence or scar tissue.

Pick Your LLM With a Scoreboard, Not a Hunch: A Two-File Eval That Costs Nothing to Re-Run

Riley Wu — Mon, 10 Aug 2026 08:55:05 +0000

The decision nobody writes down

Ask a team why they run model X in production and the honest answer is usually "someone tried it in a chat window and it looked good." That works right up until the inputs stop looking like the demo: an edge case arrives, the model bluffs instead of abstaining, and the bad output flows straight into whatever consumes it downstream.

Code doesn't get this luxury. When behavior matters, we encode expectations as tests and re-run them every time something changes. Model selection deserves the same treatment, because "something changed" happens constantly in LLM land: you switch candidates, you edit the prompt, or the vendor ships a silent revision under an unchanged model name.

What follows is a minimal harness for that: a versioned set of labeled inputs, a deterministic grader, and a scoreboard you regenerate whenever you want. It runs against any free model endpoint and lives on any small always-on machine — the whole comparison phase can cost zero while you're still deciding.

Why the evaluation phase is the free phase

The questions you need answered before committing to a model are cheap to ask:

Can anything on the market handle this task shape, even imperfectly?
Which failure shows up first — broken JSON, invented fields, refusal when it should answer?
Is the model weak, or is my prompt just ambiguous?

A free endpoint answers all three. And there's an underrated outcome: if every candidate fails identically, that tells you the bottleneck is task design, not model quality — which saves you from upgrading your way out of a prompt problem.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

In my setup, candidates are served through MonkeyCode's free model access, and the harness runs on its free server option, which also acts as the archive for past scoreboards. Neither choice is load-bearing: the script below pulls the base URL, credential, and model name from environment variables, so any OpenAI-compatible endpoint and any tiny VM work identically. I'm intentionally not quoting model catalogs, rate allowances, or how long the free tier lasts — those terms shift, so verify the current offering yourself before building on it.

The artifact: receipts, gold labels, and a grader that distrusts confidence

The walkthrough task: pull a total amount and currency out of messy receipt text. It's a good eval subject because the output is tiny, structured, and checkable by pure functions — no vibes required.

Three design commitments keep the harness meaningful:

Cases are files, not conversation logs. They live in git, diff like code, and grow every time production surprises you.
Deterministic checks first. Numeric comparison with tolerance, enum membership, JSON parseability. Human judgment is reserved for what a function genuinely cannot decide.
"I can't tell" is a valid answer with a score. A model that fabricates a total from ambiguous text is more dangerous than one that declines — the grader must reflect that asymmetry.

receipts.jsonl:

{"id": "r-001", "text": "COFFEE NORTHSIDE\\n2x latte 9.00\\nTOTAL $9.00\\nthank you", "gold": {"total": 9.00, "currency": "USD"}}
{"id": "r-002", "text": "Boulangerie Marie\\nCroissant 2,40 EUR\\nTotal: 2,40 EUR", "gold": {"total": 2.40, "currency": "EUR"}}
{"id": "r-003", "text": "TAXI RECEIPT\\nfare ...... 18.50\\ntip ....... 3.00\\nCARD ****4412", "gold": null}

That third row is the landmine: the receipt shows numbers but no explicit total. The honest answer is an abstention. A model that sums the lines and reports 21.50 has produced plausible-looking garbage — the exact thing that corrupts a downstream ledger.

receipt_eval.py (a starting point, not gospel):

import json, os, time, urllib.request, urllib.error

BASE  = os.environ["EVAL_BASE_URL"]      # any OpenAI-compatible endpoint
TOKEN = os.environ["EVAL_API_KEY"]
MODEL = os.environ["EVAL_MODEL"]

SYSTEM = (
    "Extract the grand total from the receipt. Respond with a single JSON object "
    '{"total": <number>, "currency": "USD|EUR|GBP"}. '
    "If no explicit grand total is printed, respond with: unclear"
)

def ask(receipt_text):
    payload = json.dumps({
        "model": MODEL,
        "temperature": 0,
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": receipt_text},
        ],
    }).encode()
    req = urllib.request.Request(
        BASE.rstrip("/") + "/chat/completions",
        data=payload,
        headers={"Authorization": "Bearer " + TOKEN,
                 "Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        return json.load(resp)["choices"][0]["message"]["content"].strip()

def judge(case, raw):
    """Return (points, tag). Fabrication is penalized below zero."""
    if raw.strip().lower() == "unclear":
        pred = None
    else:
        try:
            pred = json.loads(raw)
        except json.JSONDecodeError:
            return (-1, "not_json")
    gold = case["gold"]
    if gold is None:
        return (1, "abstained_ok") if pred is None else (-2, "fabricated_total")
    if pred is None:
        return (0, "gave_up")
    total_ok = isinstance(pred.get("total"), (int, float)) and \
               abs(pred["total"] - gold["total"]) < 0.01
    curr_ok = pred.get("currency") == gold["currency"]
    if total_ok and curr_ok:
        return (2, "full_match")
    return (0, "wrong_value" if not total_ok else "wrong_currency")

run = {"model": MODEL, "ts": int(time.time()), "cases": []}
earned = possible = 0
with open("receipts.jsonl") as fh:
    for line in fh:
        case = json.loads(line)
        points, tag = judge(case, ask(case["text"]))
        earned += points
        possible += 2 if case["gold"] else 1
        run["cases"].append({"id": case["id"], "tag": tag, "points": points})
        print(f"{case['id']:>7}  {tag:<16} {points:+d}")

run["score"] = round(100 * max(earned, 0) / possible, 1)
print(f"\nscore: {run['score']}  ({MODEL})")
with open("scoreboard.jsonl", "a") as out:
    out.write(json.dumps(run) + "\n")

No SDK, no framework — one standard HTTP shape, so pointing it at a different provider is an environment-variable change:

EVAL_BASE_URL="https://your-endpoint.example/v1" \
EVAL_API_KEY="..." \
EVAL_MODEL="candidate-a" \
python receipt_eval.py

Two details are doing quiet work here. The weighted tags encode your actual risk model: fabricated_total scores below giving up, because in a bookkeeping pipeline a confident wrong number costs more than a manual review queue. And results append to scoreboard.jsonl with a timestamp, so every run becomes one line of history you can diff later.

Reading a scoreboard without lying to yourself

Aggregates are the beginning of the analysis, not the end. A few habits that keep the numbers honest:

Habit	Why it matters
Diff per case, not per model	Two candidates at 85% can fail on disjoint cases — one might be shippable with a fallback path, the other not
Weight fabrication heavily	A wrong field on a real input is an annoyance; an invented answer on noise is a data-integrity incident
Freeze `temperature=0` and the prompt string	Otherwise you're comparing runs, not models
Require a minimum case count	Under ~20 cases, noise picks winners; grow the set before trusting any ranking

Because every run lands in the same append-only file, "did candidate B actually beat candidate A, and on which inputs?" becomes a jq query instead of a memory exercise.

The boring machine is the point

Nothing in the harness needs orchestration. The useful trick is simply running it again later, which is what the always-on free server is for:

42 5 * * 1  cd /srv/receipt-eval && . ./env.sh && python receipt_eval.py >> weekly.log 2>&1

A weekly re-run against the same model name catches the failure mode no demo ever shows: the vendor revises the model silently, extraction behavior drifts, and your pipeline degrades without a single error being raised. With dated scoreboard lines, drift becomes a visible event with a before-and-after diff, not a mystery you debug for a weekend. One small box executing one small script is precisely the right amount of infrastructure here — the harness stays alive without turning into a second job.

Where this design runs out of road

Free capacity is for exploration, not scale. Rate ceilings, context limits, and evolving terms are normal on no-cost tiers. A few dozen cases fit that world comfortably; a thousand-case nightly gate does not — plan for paid capacity when you get there, and treat that as a success signal.
Function-graded scoring only covers verifiable outputs. Extraction, classification, format compliance: great fit. Summaries, code review comments, creative rewrites: not checkable by equality, and forcing them into this mold produces false confidence. Those need sampled human review or a judge model, with the subjectivity that implies.
Your case file is a theory of production. Every green run means "green on these inputs." Feed sanitized real-world surprises back into the JSONL continuously, or the suite drifts into irrelevance even while it stays green.
Abstention grading assumes your domain allows it. If your product must always answer, re-weight the rubric — but make that trade-off explicit rather than accidental.

Skip this if...

...you call a model interactively a few times a week with nothing automated downstream — the harness is ceremony at that scale. Also skip building cases from scratch if a maintained public benchmark genuinely mirrors your task; run that first. And if your output can't be mechanically checked at all, you want a sampling-and-review workflow, not this one.

But if you're about to wire a model into a pipeline based on playground impressions, one JSONL file and one grading script are all that stand between you and an evidence-based choice. If you want a zero-budget way to stand the loop up, MonkeyCode's free model access together with its free server is one concrete option — the endpoint stays swappable by design, which is exactly how an eval harness should treat its infrastructure. Version the cases, keep the scoreboard history, and let the next silent model update be something you measure instead of something you chase.