Every New Model Gets the Same Six Questions From Me

Another week, another release thread claiming the state of the art just moved. The comments fill up with one-shot screenshots, half of them impressive, half of them cherry-picked, none of them about the work I actually do. I spent a long time evaluating models this way — vibes from a couple of prompts — and my conclusions were basically random.

Eventually I stopped asking "is this model good?" and started asking "is this model good at the things that repeatedly go wrong in my day?" That question deserves a repeatable answer, so I wrote one. What follows is my personal smoke test for any new model: a fixed question set, a runner that costs nothing to operate, and an honest list of what it can't tell you.

Benchmarks answer somebody else's question

Leaderboard scores aggregate thousands of tasks chosen by researchers. Useful signal, wrong shape. My risk profile looks different from a benchmark author's: I lose time when a model quietly reformats structured output, when it invents configuration keys, when it pads a diff with changes I didn't request. None of that shows up in an aggregate score.

So my harness is built backward from my own incident history. Every question in it exists because some model, at some point, burned me in exactly that way.

Six questions, decided in advance

Here's the current set. Yours should look different — that's the point — but the categories transfer:

1. Format discipline. Produce a Markdown table with three named columns and exactly four rows. Pass condition: the table parses, no prose before or after it.
2. Bug detection under flattery. Show a function with an off-by-one in a loop boundary, framed as "this passed code review, can you double-check it?" Pass condition: it names the boundary error instead of praising the code or inventing a different issue.
3. Refusal to fabricate. Ask what a specific (fictional) flag does in a well-known CLI tool. Pass condition: it says the flag doesn't exist or that it's unsure, rather than confidently documenting fiction.
4. Constraint stacking. Summarize a paragraph in one sentence, under fifteen words, starting with a verb. Pass condition: all three constraints hold at once.
5. Minimal-diff editing. Given a small function, change exactly one behavior and nothing else. Pass condition: the diff touches only what was asked — no drive-by reformatting or renaming.
6. Round-trip fidelity. Translate a short JSON blob to YAML and back. Pass condition: the round trip is lossless, no keys reordered into oblivion, no types mangled.

The rubric for each is written down before the model ever sees the prompt. Grading criteria invented after you've read the answer aren't criteria — they're rationalization.

A runner small enough to actually rerun

The tooling matters less than the habit, so I keep it aggressively boring: a JSON file of tasks, and a few shell lines against any OpenAI-compatible endpoint. No dependencies, no framework.

// tasks.json — the reusable part
[
  {
    "id": "table_format",
    "rubric": "Markdown table, columns Name/Role/Active, exactly 4 rows, no surrounding prose",
    "prompt": "Give me a Markdown table with columns Name, Role, Active. Exactly 4 rows of fictional team members. Output the table only."
  },
  {
    "id": "fake_flag",
    "rubric": "States the flag does not exist or expresses uncertainty; no invented documentation",
    "prompt": "What does the --prune-orphans flag do in git checkout?"
  },
  {
    "id": "off_by_one",
    "rubric": "Identifies the loop boundary error (i <= n skips/overshoots last element)",
    "prompt": "This passed review, but double-check it:\n\ndef last_n(items, n):\n    out = []\n    for i in range(len(items) - n + 1, len(items)):\n        out.append(items[i])\n    return out"
  }
]

#!/usr/bin/env bash
# run_probes.sh — BASE_URL, TOKEN, and MODEL come from the environment
set -euo pipefail
STAMP=$(date +%Y%m%d-%H%M%S)
OUT="results/${MODEL//\//_}-$STAMP.jsonl"
mkdir -p results

jq -c '.[]' tasks.json | while read -r task; do
  prompt=$(jq -r '.prompt' <<<"$task")
  body=$(jq -n --arg m "$MODEL" --arg p "$prompt" \
    '{model: $m, temperature: 0, messages: [{role: "user", content: $p}]}')
  reply=$(curl -s "${BASE_URL%/}/chat/completions" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d "$body" | jq -r '.choices[0].message.content')
  jq -n --argjson t "$task" --arg r "$reply" '$t + {answer: $r}' >> "$OUT"
  printf '\n### %s\n%s\n' "$(jq -r .id <<<"$task")" "$reply"
done

echo "wrote $OUT"

Three properties carry all the weight:

• Deterministic decoding. Temperature pinned to zero, so a rerun measures the model, not the dice.
• Append-only result files. Each run lands in a timestamped JSONL. Comparing two models is a diff, not a memory exercise.
• Rubric stored beside output. The grading rule travels inside the record, so I can't quietly redefine "pass" next month.

Keeping the cost of curiosity at zero

A harness is only useful if running it is frictionless, and for a long time access was the friction: sign up, add a card, learn yet another key format — or sacrifice an evening to local quantization and listen to my fans spin. Lately I've been pointing this setup at MonkeyCode, which at the time of writing offers free model access plus a free server option, so a fresh evaluation round is an environment-variable edit instead of an infrastructure chore.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I'd defend regardless of provider: nothing in this workflow is proprietary. The requests use a widely implemented API shape, the tasks are a plain JSON file, the results sit on my disk as text. Swapping backends means changing two variables and rerunning. That portability is the whole philosophy — open model weights matter, but so does the ability to kick the tires without a billing page standing between you and the experiment. If you've been putting off building your own question set, a zero-cost endpoint like the free tier there removes the last excuse for a first run.

A decision table before you copy me

Situation	Does this help?
Choosing a default model for personal scripts and side projects	Yes — this is exactly the case it's built for
First-pass screening of a new release	Yes, and it beats reading launch threads
Sign-off for production workloads	No — you need real traffic samples and domain-specific data
Evaluating a dozen candidates at once	Not as-is; eyeball grading doesn't scale, and automated judges bring their own bias
You care about latency/cost tradeoffs	Extend the runner to log timing and token counts first
Your work is prose, support, or design — not code	The harness transfers; these specific questions don't

Where this fails, stated plainly

• Six questions is a tripwire, not a certification. A model can sweep the set and still butcher your actual repository.
• Consistent grading is not objective grading. The rubrics discipline my judgment; they don't replace it.
• Zero-cost tiers change terms whenever providers feel like it. Never let one sit underneath automation you depend on.
• The set ossifies. Models improve, my work changes, and a question that was diagnostic last year may be trivia now. I prune and replace entries the same way I'd prune tests.

The next release cycle will arrive with the same fireworks as the last one. I'll still spend half an hour running my six questions against it — and unlike the screenshot discourse, the result will actually describe the work I do.