I Stopped Reading Model Release Threads and Built a Release-Day Eval Ritual Instead

#ai #testing #productivity #tooling

Release day used to cost me a whole evening

Every time a fresh open-weight model drops, the same ritual plays out on my timeline: cherry-picked outputs, leaderboard crops, and a hundred replies arguing about benchmarks that measure workloads nothing like mine. I used to burn evenings reading those threads. Then I realized the only question I actually care about is narrow and personal: will this thing survive contact with my Monday?

My Monday is not a leaderboard prompt. It's untangling a half-migrated CI pipeline, writing a migration script I fully expect to run once, and figuring out why a linter suddenly hates code it accepted last sprint. So I built a small, repeatable release-day ritual that answers my question in under an hour, and this post is the whole setup.

The ritual: six probes, a spreadsheet, and one rule

The rule is the important part, so I'll put it first: the probes never change. A new model doesn't get a new test designed around its marketing page. It gets the same six probes every other model got, and it wins or loses on the same terms.

My six, mapped to things I genuinely do:

The migration script — given a CSV with messy quoting and a target schema, produce a one-shot Python import script. Verified by running it and diffing the output table.
The broken pipeline — a GitHub Actions YAML with a subtle matrix-variable typo and a caching key that silently never hits. Verified by a checker script that knows both faults.
The type-narrowing puzzle — a TypeScript snippet where a union type needs narrowing through a third-party API response. Verified by tsc --noEmit on a fixed test file.
The regex autopsy — a pathological regular expression from an old log parser. The model must explain what it matches, then rewrite it readably. Verified by running both against a fixture log and diffing match sets.
The docstring lie — a function whose docstring describes behavior the code doesn't have. The model must notice the mismatch before fixing anything. Verified by grepping its response for an explicit contradiction statement.
The refusal test — a request to use a deprecated, known-vulnerable library with an explicit instruction in the prompt forbidding it. Verified by scanning the output for the forbidden import.

Probes 5 and 6 are the ones I trust most. Generation quality is table stakes now; what separates models in practice is whether they notice things and whether they obey negative instructions. Those two properties predict how a model behaves inside a long agentic loop far better than any coding benchmark I've read.

The runner

The harness is a plain Bash driver plus per-probe checker scripts. Nothing clever — cleverness is where reproducibility goes to die:

#!/usr/bin/env bash
# probe_run.sh <endpoint> <api_key> <model_name>
# Runs every probe in ./probes against one model, appends results to ledger.csv
set -uo pipefail

ENDPOINT="$1"; KEY="$2"; MODEL="$3"
LEDGER="ledger.csv"

call_model() {
  local prompt_file="$1"
  curl -sS --max-time 240 "$ENDPOINT/chat/completions" \
    -H "Authorization: Bearer $KEY" \
    -H "Content-Type: application/json" \
    -d "$(jq -n \
      --arg m "$MODEL" \
      --rawfile p "$prompt_file" \
      '{model: $m, temperature: 0,
        messages: [{role:"user", content: $p}]}')" \
  | jq -r '.choices[0].message.content'
}

for probe in probes/*/; do
  name=$(basename "$probe")
  workdir=$(mktemp -d)

  start=$(date +%s)
  call_model "$probe/prompt.txt" > "$workdir/response.md"
  elapsed=$(( $(date +%s) - start ))

  # Each probe ships its own checker: exit 0 = pass, anything else = fail.
  if bash "$probe/check.sh" "$workdir" > "$workdir/check.log" 2>&1; then
    verdict="pass"
  else
    verdict="fail"
  fi

  echo "$MODEL,$name,$verdict,${elapsed}s,$(date -I)" >> "$LEDGER"
  printf '%-22s %-5s %ss\n' "$name" "$verdict" "$elapsed"
done

Each probe directory contains exactly three things: prompt.txt (the full task text, including starter code inline), check.sh (a mechanical verifier), and fixtures/ if the checker needs reference data. The TypeScript probe's checker is literally cp fixture_test.ts "$1"/ && npx tsc --noEmit plus an exit code. The refusal probe's checker is two grep calls. If a checker ever needs me to eyeball something, that probe gets redesigned — subjective scoring is how benchmark arguments start, and I built this thing specifically to stop having those.

Because everything speaks the OpenAI-compatible chat format, adopting a new model means changing one CLI argument. That single design decision is what makes release week a routine instead of a research project.

What the probes run against

I don't own a GPU that makes local inference pleasant, so my setup is hosted. Two environments cover everything:

A free hosted model endpoint for the interactive pass, when I'm poking at a single new release and want answers in minutes.
A free server for the unattended case, when I want to queue several candidate models overnight and read the ledger over coffee.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. In my case, MonkeyCode's free model access is the endpoint I aim the script at for day-one comparisons, and its free server option is where the overnight batches run so my laptop can sleep. Nothing in the harness is tied to it — that's deliberate, because experiment infrastructure should outlive any single provider. If a free tier happens to fit your budget for this kind of testing, it's a sensible place to start; if you outgrow it, the same script points anywhere.

Reading the ledger: my rubric

The ledger accumulates one row per probe per model, and over a few release cycles it becomes genuinely useful. How I act on it:

Pattern in the ledger	My interpretation	My move
6/6 with sane latency	Candidate for real work	Two-week trial on one low-risk project
Fails probe 5 (docstring lie)	Doesn't notice contradictions	Never use for review or debugging tasks
Fails probe 6 (refusal)	Ignores negative instructions	Excluded from any agentic/automated loop
Passes everything but 3x slower than my current model	Quality is fine, budget is wrong	Batch/offline use only
Fails 3+ probes	Not for my workload	Ignore the discourse, move on

The refusal row deserves emphasis. A model that uses a forbidden library inside a 30-line probe will absolutely violate constraints inside a 3,000-line autonomous run, where nobody is watching. Probe failures there are disqualifying regardless of how good everything else looks.

Honest limits

Six probes is a personal filter, not science. It tells me whether a model fits my week. It says nothing about the model's general quality, and I don't publish the numbers as if it did.
Temperature zero is not determinism. Providers batch, kernels vary, and I've seen identical prompts flip a probe across runs. Anything flaky gets three reruns, and flakiness itself counts as a fail.
The probes rot. My job changes, so twice a year I audit the suite: probes that no longer mirror real work get replaced, and every model I still care about gets re-run against the new suite.
Free access is a gift, not a foundation. Quotas, model catalogs, and pricing change. I never wire a free tier into anything that must keep working; the ledger format makes switching endpoints trivial, on purpose.
Don't copy my six. If your week is embedded firmware or data engineering, my probes will tell you nothing. Build two probes from your last two real tickets and you'll learn more than this entire post taught you.

The point

Models will keep shipping faster than trustworthy reviews can be written. The calm response isn't finding better threads to read — it's owning a tiny, boring, personal eval that turns every release into a 45-minute experiment with a CSV at the end. Mine took one weekend to build and has paid for itself every release week since.

If you build your own suite, the interesting part won't be your verdicts — it'll be which probes you chose. That list says more about how you actually work than any leaderboard says about the models.