Hype Cycles Don't Ship My Code: How I Gate New LLMs with a Self-Written Eval Deck

Every few weeks a new model dominates the timeline. The launch posts look the same every time: a cherry-picked demo, a benchmark chart, a wall of flame emojis — and no answer to the question that actually matters to me, which is "will this thing handle the boring, weird, half-documented work in my repos?"

I learned this the expensive way. I once adopted a trending model based on launch-day enthusiasm and spent a night cleaning up hallucinated CLI flags it had emitted with total confidence. Since then, every model that wants into my workflow has to pass a small deck of adversarial tasks I wrote myself. This post is the full method.

Launch metrics answer launch questions, not yours

Benchmarks are legitimate science — for the benchmark's questions. Your codebase asks different ones:

• Does the model follow your formatter and your naming conventions without being told twice?
• Can it deal with unglamorous glue: GitHub Actions YAML, a creaky ORM migration, a Helm values file?
• Does it hold together when your prompt is long and messy instead of short and clean?
• When it fails, does it fail loudly and catchably, or quietly and plausibly?

A viral screenshot tells you none of this. The fix isn't cynicism — it's a tiny verification loop that you own.

The artifact: a deck of task cards plus a dumb runner

I keep each eval as a card: a prompt file plus a machine-checkable contract, stored as plain files so adding a task never touches the runner.

cards/
  tasks/
    pr-summary.md          # summarize this diff for a reviewer
    migration-plan.md      # propose a zero-downtime schema change
    incident-timeline.md   # reconstruct events from these log lines
  contracts/
    pr-summary.yml
    migration-plan.yml
    incident-timeline.yml

A contract declares what a passing answer looks like:

required_terms: ["rollback", "index concurrently"]
forbidden_terms: ["LOCK TABLE", "truncate"]
max_words: 220
samples: 4
required_success_fraction: 1.0

The runner is one dependency-free file. It speaks to any OpenAI-compatible endpoint, so the model under test is just environment config:

// eval.mjs — Node 18+, stdlib only
import { readdir, readFile } from "node:fs/promises";
import { parse } from "node:util"; // placeholder-free: we hand-roll YAML below

const BASE = process.env.LLM_BASE_URL;  // e.g. any OpenAI-compatible endpoint
const KEY  = process.env.LLM_API_KEY;
const MODEL = process.env.LLM_MODEL;

// Tiny YAML-subset parser (only the keys our contracts use)
function parseContract(text) {
  const out = {};
  for (const line of text.split("\n")) {
    const m = line.match(/^(\w+):\s*(.+)$/);
    if (!m) continue;
    const [, k, v] = m;
    out[k] = v.startsWith("[")
      ? JSON.parse(v.replace(/'/g, '"'))
      : isNaN(+v) ? v.trim() : +v;
  }
  return out;
}

async function complete(prompt) {
  const r = await fetch(`${BASE}/v1/chat/completions`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: MODEL,
      messages: [{ role: "user", content: prompt }],
      temperature: 0.1,
    }),
  });
  return (await r.json()).choices[0].message.content;
}

function passes(answer, c) {
  if (answer.split(/\s+/).length > c.max_words) return false;
  const low = answer.toLowerCase();
  return c.required_terms.every(t => low.includes(t.toLowerCase()))
      && c.forbidden_terms.every(t => !low.includes(t.toLowerCase()));
}

for (const f of await readdir("cards/tasks")) {
  const id = f.replace(".md", "");
  const prompt = await readFile(`cards/tasks/${f}`, "utf8");
  const c = parseContract(await readFile(`cards/contracts/${id}.yml", "utf8"));

  let ok = 0;
  const lat = [];
  for (let i = 0; i < c.samples; i++) {
    const t0 = performance.now();
    const ans = await complete(prompt);
    lat.push(Math.round(performance.now() - t0));
    ok += passes(ans, c);
  }
  const frac = ok / c.samples;
  console.log(
    `${id.padEnd(22)} ${ok}/${c.samples}` +
    `  p50=${lat.sort((a,b)=>a-b)[Math.floor(lat.length/2)]}ms` +
    (frac >= c.required_success_fraction ? "  PASS" : "  REJECT")
  );
}

(The toy YAML parser is fine for flat contracts; swap in a real parser if your specs grow nesting.)

Three design choices carry most of the weight:

• Sampled consistency, not single-shot luck. Four low-temperature runs per card, and the required fraction decides. A model that's right one time in four isn't skilled — it's a slot machine that paid out while you were watching.
• Deny-lists for catastrophic modes. For anything infra-adjacent, certain words auto-fail the answer. I don't care how elegant the prose is if the migration plan contains truncate.
• Tasks sourced from my own ticket history. Every card is a sanitized version of a real task from the last month. Curated benchmarks and real glue work correlate less than anyone wants to admit.

To run this you need an endpoint for the candidate model — ideally one that costs nothing while you're still in the poking-around phase. For my most recent round I pointed the runner at MonkeyCode, which offers free model access and a free server option; that meant an afternoon of iterating against the new release with no provisioning and no card on file. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

From raw results to an actual decision

A score line is not a verdict. I run each card's output through a short decision tree:

All cards pass, latency fine?
├─ yes → promote to one real, reversible task this week
└─ no  → is the failure concentrated in ONE card category?
    ├─ yes → adopt it everywhere EXCEPT that category
    ├─ no, failures are flaky/random → reject; re-test next checkpoint
    └─ no, failures are systemic → reject

The single-category-failure branch is where the real value lives. Models aren't uniformly good or bad — they have a shape. The goal isn't to crown a winner; it's to find a model whose weak spots never intersect your workload. A model that can't write commit messages but nails log analysis is a fantastic fit for my on-call triage and a terrible fit for my release notes.

Why open availability is the real story

This workflow only exists because more models are now reachable without an enterprise contract. That trend — whatever vendor's release is trending this week — quietly changes the epistemics: "trust the launch post" becomes "verify on your own machine tonight". It's the same deal open source always offered, applied to weights instead of source. Providers that make independent checking cheap (free model access, a free server tier — MonkeyCode's current arrangement is one example) are removing the last excuse for skipping due diligence. Providers that make it expensive are making a statement too.

Honest limitations

• This catches dealbreakers, not gradients. A dozen cards can't finely rank models. Never sign an annual contract on this evidence alone.
• Term matching is style-blind. An answer can pass every contract and still be ugly. I hand-read two full responses per model, no exceptions.
• Free access is a snapshot, not a covenant. Pricing and availability change. That's precisely why the runner only assumes the OpenAI-compatible API shape — switching endpoints is an env var, not a refactor.
• Proprietary code needs clearance first. Sending internal snippets to any external endpoint, free or paid, is a decision your security policy must authorize.
• If your org already runs a mature eval platform, skip the harness — though I'd argue the forbidden-terms deny-list trick is still worth stealing.

Wrapping up

The next model launch is already scheduled somewhere. You don't have to choose between drinking the hype and ignoring the field — spend one evening turning your recent tickets into task cards, run the deck, and let your own results argue. If you need a zero-cost endpoint to run this against, MonkeyCode's free tier is a convenient place to start. One thing I'd genuinely like to hear: which catastrophic-failure terms land on other people's deny-lists — mine skews heavily toward destructive filesystem and SQL verbs, which is either prudence or scar tissue.