DEV Community: Finley Zhou

I Stopped Trusting My Gut on New Open Models. A 30-Minute Scoring Loop Replaced It.

Finley Zhou — Mon, 10 Aug 2026 11:17:20 +0000

Open model releases have turned into a weekly event. Another checkpoint, another chart, another round of confident replies under it — and somewhere in that noise I'm supposed to decide whether this thing deserves a slot in my daily workflow. For a long time my process was: open a chat, throw it a prompt I half-remember, and let my mood write the review.

That process failed me more than once. A model I dismissed after one bad answer turned out fine. A model I adopted on a good first impression hallucinated its way through a real refactor. The common thread: I was reacting, not measuring.

So I built a scoring loop that fits in a coffee break and produces a written verdict instead of a feeling. This post walks through the whole thing — the task file, the runner, the scoring discipline — with code you can lift directly.

Why first impressions lie

Three biases make casual testing useless for model selection:

Prompt lottery. A single prompt tells you how the model handled that prompt, at that phrasing. Rephrase it and you might get a different model entirely.
Recency of the last demo. Whatever the last screenshot in your feed showed becomes your anchor, regardless of what you actually need.
Readability as a proxy for correctness. Fluent, well-formatted wrong answers feel right. Code that looks clumsy but passes feels wrong. My gut consistently scores style over substance.

The antidote is unglamorous: fix the inputs, automate the run, and score against criteria you wrote before you saw any output.

Design: tasks live in a file, verdicts live in a log

Instead of a hardcoded list, I keep my tasks in a plain JSONL file. One line per task. It diffs cleanly in git, it's easy to extend, and the runner doesn't care what model it's pointed at.

{"id": "fix-off-by-one", "kind": "code", "prompt": "Return ONLY Python code. Fix the bug: this should sum values at even indices.\n\ndef even_sum(nums):\n    total = 0\n    for i in range(1, len(nums), 2):\n        total += nums[i]\n    return total\n", "check": "assert even_sum([1,2,3,4,5]) == 9\nassert even_sum([]) == 0\nassert even_sum([7]) == 7\nassert even_sum([2,9,2,9]) == 4\n"}
{"id": "recall-buried-fact", "kind": "manual", "prompt": "Read this document and answer: which port does the staging worker bind to? [paste a long doc with the answer '8471' on line ~200]", "rubric": "states 8471 = 1pt; hedges instead of inventing when unsure = 1pt"}
{"id": "schema-strict", "kind": "manual", "prompt": "Reply with JSON only, keys: name, count, tags. Item: a pack of 12 black gel pens.", "rubric": "valid JSON = 1pt; exactly 3 keys = 1pt; tags is an array = 1pt"}

Two categories, deliberately:

code tasks are self-scoring. The output gets executed against assertions. No interpretation, no generosity.
manual tasks carry a pre-written rubric. The rubric exists before the answer does. That's the whole trick — it turns "did I like this?" into "did it hit these specific points?"

I keep roughly ten tasks total, drawn from what I actually did last month: a debugging case from a real ticket, a summarization over a genuinely long document, one strict-formatting task, one "please don't do this" safety-adjacent task. If a task doesn't mirror real work, it gets cut.

The runner

Small, standard-library-only, works against any OpenAI-compatible endpoint:

#!/usr/bin/env python3
"""score_loop.py — run tasks.jsonl against a model, log a verdict.

  BASE_URL=http://localhost:8000/v1 MODEL=new-hotness \
  python score_loop.py tasks.jsonl
"""
import json, os, subprocess, sys, tempfile, time, urllib.request

BASE, MODEL = os.environ["BASE_URL"].rstrip("/"), os.environ["MODEL"]
KEY = os.environ.get("API_KEY", "x")

def ask(prompt):
    body = json.dumps({
        "model": MODEL, "temperature": 0,
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"})
    with urllib.request.urlopen(req, timeout=240) as r:
        return json.load(r)["choices"][0]["message"]["content"]

def grade_code(answer, check):
    # strip a fenced block if the model added one
    if "```

" in answer:
        inner = answer.split("

```")[1]
        answer = inner.split("\n", 1)[1] if "\n" in inner else inner
    with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:
        f.write(answer + "\n" + check)
        path = f.name
    try:
        subprocess.run([sys.executable, path], timeout=10,
                       capture_output=True, check=True)
        return 1
    except Exception:
        return 0

log = {"model": MODEL, "at": time.strftime("%Y-%m-%d %H:%M"), "rows": []}
for line in open(sys.argv[1]):
    t = json.loads(line)
    out = ask(t["prompt"])
    row = {"id": t["id"], "kind": t["kind"]}
    if t["kind"] == "code":
        row["score"] = grade_code(out, t["check"])
    else:
        row["rubric"] = t["rubric"]          # score these by hand, rubric first
        row["answer"] = out[:2000]
    log["rows"].append(row)
    print(t["id"], "->", row.get("score", "needs manual grade"))

name = f"verdict_{MODEL.replace('/', '_')}_{int(time.time())}.json"
json.dump(log, open(name, "w"), indent=2)
print("wrote", name)

Three details I'd keep in any rewrite:

Deterministic settings. Temperature 0 won't make a model deterministic, but it removes the most obvious source of run-to-run noise.
Execution beats inspection for code. The model doesn't get partial credit for a persuasive explanation of a broken fix.
Every run leaves an artifact. The verdict files accumulate. Six months from now, "why are we on this model?" has a paper trail.

Manual rubric scoring takes me maybe five minutes for the whole file, and because the rubric was written up front, it's grading rather than rationalizing.

The verdict rule

After scoring, one sentence goes into the log: adopt, trial, or skip.

Adopt — beats or matches my current model on code tasks and produces zero fabricated answers on the recall/safety rows.
Trial — mixed results but wins somewhere specific; it gets one week of real side-task usage before a final call.
Skip — anything else. No "it seemed promising." Promising is how mediocre tools colonize your workflow.

One nuance I now score explicitly: failure shape. On the recall task, a wrong confident answer is worse than a wrong hedged one, and both are worse than a correct "the document doesn't say." I'd rather run a model that loses loudly than one that loses politely.

Running it for free

The loop needs an endpoint and nothing else. I've been running mine through MonkeyCode, which currently offers free model access together with a free server option — enough to point BASE_URL at it and let the runner work through the file. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I like about this pattern existing at all is that it echoes how the open model ecosystem works in the first place. Weights get published, compatible APIs get shared, and suddenly the interesting question shifts from "can you afford to try this" to "what did you find when you did." Free inference and a free machine to run it on are the same instinct showing up in commercial form — the eval stops being a procurement decision and becomes a half-hour experiment. Anyone who wants to check a benchmark claim instead of quote-tweeting it can. The fully local route works too; the runner above doesn't care whether the endpoint is llama.cpp on your own box or a hosted free tier.

Where this breaks down

Ten tasks is a filter, not a benchmark. It answers "worth a real trial?" and nothing more. Don't cite your personal pass rates as model rankings.
The suite ages. Tasks you wrote last quarter may no longer represent your work. I prune mine when I notice I'm scoring models on problems I no longer have.
Greedy decoding flatters consistency. A model that looks stable at temperature 0 may wander at the settings you use for drafting or brainstorming. Spot-check at real settings before adopting for those tasks.
Free options are moving targets. Lineups, rate limits, and availability change; read the current terms, and never wire a free tier into anything production-shaped.
Formal contexts need formal methods. Procurement, compliance, or research comparisons call for established suites and real methodology — this loop is for personal tooling choices.
Don't run it weekly. Most releases aren't relevant to your workload. Trigger the loop on a concrete reason — a license you'd actually use, a size you can self-host, a capability gap in your current setup — not on hype volume.

Closing thought

The release cadence isn't slowing down, so the choice is between faster opinions and better filters. A task file, a runner that executes instead of admires, and a rubric written before the answers exist — that's the whole system. If you want a zero-cost endpoint to run it against, MonkeyCode's free model access and server are one place to start; a local server works identically. The point isn't where it runs — it's that next time a checkpoint trends, you reach for the runner instead of the reply button.

If you've built something similar, I'd love to hear which tasks made your personal cut — that's the part I'm always tempted to steal.

Don't Trust the Demo: A Repeatable Test Harness for Evaluating Free AI Coding Models

Finley Zhou — Mon, 10 Aug 2026 08:57:46 +0000

Every few weeks a new coding model shows up with a polished demo, and every few weeks I watch someone wire it straight into their editor because the demo looked good. Then the surprises arrive: it hallucinates an API that doesn't exist, it rewrites a file it wasn't asked to touch, or it burns an afternoon on a task a grep would have finished in thirty seconds.

The fix isn't cynicism — it's measurement. You don't need paid API access or a fancy eval suite to find out whether a model actually helps your work. You need a small, repeatable harness and an hour of honesty.

This article walks through the harness I use. Everything below is runnable, and the whole thing works with free tiers — so cost is not an excuse to skip it.

Why a harness instead of vibes

The recent discussions here about AI agents and tool boundaries circle around the same problem: we grant tools trust before we've measured what they do with it. Anecdotal evaluation has three known failure modes:

Primacy bias — the first impressive output anchors your judgment of the model.
Task drift — you start evaluating on whatever the model happens to be good at, instead of your actual work.
No baseline — you never compare against "just do it by hand," so a net loss still feels like a win.

A tiny harness fixes all three, because the task list is fixed before you see any output.

Step 1: Build a task list from your git history

Don't use someone else's benchmark. Mine your own repo for tasks you've actually done:

# Find your last 20 non-trivial commits as candidate tasks
git log --oneline --since="3 months ago" \
  -- . ':(exclude)*.lock' ':(exclude)*.min.*' | head -20

Pick 8–12 commits that represent your real distribution of work. For each one, write a task card:

## Task 04: retry logic for flaky HTTP client
- Input given to model: the commit message + the diff of the *parent* commit
- Expected output: a patch equivalent to the real fix
- Pass criteria: existing tests still pass; retry only on 5xx/timeout;
  no change to public function signatures
- Category: bugfix
- Human time it originally took: ~40 min

The "human time it originally took" field is your baseline. Without it, you're measuring speed against nothing.

Step 2: The scoring rubric

Keep it binary and boring. Fancy rubrics invite self-deception.

Dimension	Pass condition	Weight
Correctness	Patch applies and your test suite passes	3
Scope discipline	Touches only files/functions the task implies	2
Fabrication	References no nonexistent APIs, files, or flags	2
Time	Faster than your recorded human baseline	1

A model that scores high on correctness but fails scope discipline is a liability in a shared codebase — that's the tool-boundary problem showing up in miniature. Record results in a plain CSV so you can diff runs across model versions later.

Step 3: Run it without paying anything

This is where free access matters. I'm currently using MonkeyCode for this loop — Disclosure: This article was prepared as part of MonkeyCode's product outreach. — because it offers free model access plus a free server option, which covers the two things this harness needs: a model to evaluate and somewhere to run the harness repeatedly. I can't speak to quotas, specific model lineups, or how long the free tier lasts, so treat it as "free right now" and design accordingly.

The harness itself is deliberately portable — a shell script, not a platform feature:

#!/usr/bin/env bash
# eval.sh <task_dir> — proposed structure, adapt paths to your setup
set -euo pipefail
TASK_DIR="$1"

# 1. Snapshot the repo state
git stash -u 2>/dev/null || true

# 2. Feed task card to the model (via your client of choice),
#    capture the proposed patch to $TASK_DIR/output.patch

# 3. Score it
git apply --check "$TASK_DIR/output.patch" && echo "APPLIES: yes" || echo "APPLIES: no"
git apply "$TASK_DIR/output.patch"
npm test --silent > "$TASK_DIR/test.log" 2>&1 && echo "TESTS: pass" || echo "TESTS: fail"
git diff --name-only HEAD > "$TASK_DIR/files_touched.txt"

git checkout -- . && git clean -fd

The key property: the model never sees the scoring step. If your eval tool and your eval rubric live in the same prompt, you're grading homework with the answer key visible.

Step 4: What to do with the results

After one run you'll have a per-category picture. The useful output isn't "model X is good" — it's a routing table for your own work:

Bugfixes pass, refactors fail → use the model for triage, not for restructuring.
Fabrication failures cluster on one library → paste that library's real API docs into context, or stop using the model for that dependency.
Everything passes but slower than baseline → the model is a correctness aid, not a speed aid. Use it for review, not generation.

Re-run the harness whenever you switch models or the provider updates one. Because it's just a script plus a CSV, the rerun costs you an hour, not a migration.

Limitations — read this before adopting

8–12 tasks is a small sample. This harness tells you about your work, not about the model in general. Don't publish your CSV as a benchmark; it's a personal routing table.
Your git history is biased toward what you already attempted solo. Tasks you avoided entirely (unfamiliar languages, big greenfield design) are invisible to this method.
Free tiers change. Any setup that depends on a specific provider's free offering should be treated as temporary. The script above is provider-agnostic for exactly this reason — swap the client, keep the scoring.
Time-to-pass isn't value. A model that's 20% faster but needs your full attention while it works may be a net loss compared to 40 uninterrupted minutes of your own focus. The rubric can't see that; you have to.

Who should skip this

If you write code fewer than a few hours a week, the setup cost exceeds the payoff — just try a model on two real tasks and trust the result. And if your work involves code you can't send to any third-party endpoint, free cloud models are off the table entirely; run something local or don't run anything.

For everyone else: the demo will always be flattering. Your git history won't be. Measure against the second one.