DEV Community: Morgan Xu

A New Open-Weight Model Drops Every Week. Here's the 30-Minute Eval Harness I Run Before Believing Any Benchmark

Morgan Xu — Mon, 10 Aug 2026 10:59:39 +0000

If your feed looks anything like mine this week, it's wall-to-wall hot takes about the latest open-weight coding model release — right now it's MiniMax's H3 getting the treatment, last month it was something else, next month it'll be something newer. The pattern is always the same: a launch post, a set of vendor benchmark tables, a wave of "this changes everything" threads, and then, quietly, a trickle of "actually it's mediocre at X" follow-ups.

I'm not going to tell you whether H3 is good. I haven't run it through enough of my own workloads to say, and honestly, neither have most of the people posting about it. What I can give you is something more durable: a small, reproducible evaluation harness that takes about 30 minutes to set up and tells you whether any newly released model is worth switching to for your code, not for a benchmark suite.

Why vendor benchmarks don't answer your actual question

The question a launch benchmark answers is: "How does this model perform on a standardized set of problems under controlled prompting?" The question you actually have is: "Will this model waste less of my time than the one I'm already using, on the kind of work I actually do?"

These diverge fast. Your work has a specific language mix, a specific repo style, specific tolerance for verbosity, specific failure modes you can't stand (mine: confidently inventing API methods that don't exist). A two-point swing on a public leaderboard says almost nothing about any of that.

So the harness below is built around your tasks, scored automatically where possible and by checklist where not.

The harness: 12 tasks, 3 scoring tiers

The whole thing is a directory of task prompts plus a runner script. Here's the structure:

model-eval/
├── tasks/
│   ├── 01_fix_failing_test.md
│   ├── 02_refactor_for_readability.md
│   ├── 03_explain_unfamiliar_code.md
│   ├── ...
│   └── 12_migration_snippet.md
├── runner.py
├── score.py
└── results/

Each task file is a prompt plus an expected-output specification. Three tiers of tasks, four each:

Tier 1 — mechanically verifiable. Tasks where the output can be checked by a script: "fix this failing pytest," "write a function passing these assertions," "produce valid JSON matching this schema." These get scored by actually executing the output.

Tier 2 — checklist verifiable. "Refactor this function" tasks scored against an explicit rubric: no behavior change (run the existing tests), no new dependencies, cyclomatic complexity not increased. Partially automatable.

Tier 3 — judgment tasks. "Explain what this regex does," "review this diff for security issues." Scored by you, but against a written answer key you make before looking at model output, so you're not grading on vibes.

Here's the runner — deliberately boring, provider-agnostic, ~40 lines:

import json, subprocess, sys, time
from pathlib import Path

def run_task(client, model: str, task_file: Path) -> dict:
    prompt = task_file.read_text()
    start = time.time()
    response = client.complete(model=model, prompt=prompt, max_tokens=2048)
    elapsed = time.time() - start
    return {
        "task": task_file.stem,
        "model": model,
        "output": response.text,
        "latency_s": round(elapsed, 2),
        "tokens_out": response.usage.completion_tokens,
    }

def score_tier1(result: dict, task_dir: Path) -> bool:
    """Write model output to a file, run the task's checker script."""
    candidate = task_dir / "_candidate.py"
    candidate.write_text(result["output"])
    checker = task_dir / "check.sh"
    proc = subprocess.run(["bash", str(checker)], capture_output=True)
    return proc.returncode == 0

if __name__ == "__main__":
    model = sys.argv[1]
    client = get_client()  # whatever provider you're testing
    results = []
    for task in sorted(Path("tasks").glob("*.md")):
        r = run_task(client, model, task)
        r["tier1_pass"] = score_tier1(r, Path("tasks")) if is_tier1(task) else None
        results.append(r)
    Path(f"results/{model.replace('/', '_')}.json").write_text(
        json.dumps(results, indent=2)
    )

Twelve tasks feels small. It is small — deliberately. You'll actually finish a 12-task eval for every new model that drops. You will not finish a 200-task eval more than once, and an eval you run once is a demo, not a tool.

Where to run it without torching your budget

The annoying part of doing this properly used to be access: to compare a new open-weight model against your incumbent, you need both models available through something you can script against, and paying per-token across several providers for eval runs adds up fast, especially when you're iterating on the harness itself.

This is where I've been using MonkeyCode. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The relevant bit for this workflow is that it offers free access to a set of models plus a free server option, which means the "spin up an endpoint, point the runner at it, tear it down" loop costs me nothing while I'm debugging the harness — and harness debugging is where most of the token spend actually goes, not the final eval run. I'd rather burn my mistakes on a free tier and reserve paid API calls for the comparisons that matter.

What I appreciate beyond the pricing is the posture: the project leans into open source — the tooling is out in the open, you can read what the client is actually sending, and you're not reverse-engineering a black box to figure out why your eval numbers look weird. For an eval harness specifically, that transparency matters more than for casual use, because "I don't know what the wrapper did to my prompt" invalidates your results.

If you want to try this workflow yourself, the free server option is enough to get the runner above working end-to-end; grab a task from your own recent git history and start there.

What I learned running this against recent releases

Without naming specific scores (your tasks will differ, and that's the whole point), the pattern across the last few open-weight releases I've put through this harness:

Tier 1 gaps are smaller than marketing implies. Most current-generation open coding models pass the mechanically-verifiable tasks at similar rates. The differentiation has moved to Tier 2 and 3.
Verbosity is a hidden cost. One model I tested passed tasks at the same rate as my incumbent but produced 40% more output tokens per task — more to review, more latency, more cost per accepted suggestion. My harness counts tokens_out for exactly this reason.
Failure style matters more than failure rate. I keep a one-line note per failed task: "wrong but obviously wrong" vs "wrong and confident-looking." Models in the second category get rejected even with decent pass rates, because review time is my real bottleneck.
Latency on your hardware/endpoint is not the benchmark's latency. A model that looks great in a datacenter demo can be unusable if the endpoint you're actually hitting is slow. Measure it yourself; it's two lines of code, as above.

Limitations, and who shouldn't bother

Twelve tasks can't measure everything. This harness is a smoke test, not a certification. It's good at answering "is this worth a week-long trial," bad at answering "is this safe for production code review."
Task selection bias is real. If your tasks are all Python bugfixes, you'll pick the model that's best at Python bugfixes. That's fine if that's your job; it's a trap if it isn't. Rebuild the task set when your work changes.
Judgment-tier scoring is still you. The pre-written answer key reduces but doesn't eliminate grader drift.
If you only use a model for autocomplete-style completions, this harness is overkill — you're better off measuring accept-rate in your editor for a week.
Free tiers change. Any free model access or free server option — MonkeyCode's included — is an availability claim, not a permanent guarantee. Design your harness so the client is swappable and you're never locked into one endpoint.

The actual takeaway

The next time a model drops and the feed explodes — H3 today, something else in three weeks — the correct response isn't hype or dismissal. It's: pull the release, run your twelve tasks, compare against your incumbent on your rubric, and make a boring evidence-based decision in an afternoon. The teams that get value from the current pace of model releases aren't the ones with the strongest opinions; they're the ones with the cheapest evaluation loop.

Build the harness once. Every future launch post becomes a 30-minute chore instead of a week of discourse.

Green Tests Lie: How I Gate AI-Generated Patches Before They Touch Main

Morgan Xu — Mon, 10 Aug 2026 08:36:35 +0000

Every team chat I'm in has landed on the same uncomfortable question: an agent can write a plausible patch in ninety seconds, but who vouches for it? Reading the diff by eye feels like due diligence. In practice, it mostly rewards fluency — and generated code is fluent by construction.

The stance I settled on: treat agent output like a pull request from a stranger with no commit history. You wouldn't merge that because it read nicely. You'd merge it because it survived your gauntlet. Below is the gauntlet I run, built entirely on compute that costs nothing, followed by an honest list of where it falls apart.

Why eyeballing diffs keeps letting bad patches through

When I audit agent patches that got a thumbs-up and later caused trouble, three defects recur:

Prompt-shaped correctness. The code nails the scenario in the request and crumbles on the adjacent ones — empty input, DST boundaries, a record with ten thousand rows.
Scope creep in the edit set. You asked for a fix in one module; the patch also nudges a lint config or a lockfile you never mentioned.
Gates quietly lowered. The suite is green because a strict assertion became a loose one, or a test file got an early return.

None of these announce themselves in a casual read. So the process below makes each one impossible to skip.

The gauntlet, stage by stage

Four stages, each producing an artifact you can revisit months later:

Quarantine — run the patch in an environment with no network and no secrets.
Scope audit — diff the patch against what was actually requested.
Adversarial probes — tests you write before looking at the implementation.
Decision record — one file capturing inputs, outputs, and verdict.

Stage 1: Quarantine

The test environment should cost less to destroy than to disinfect. Mine is a scratch clone executed inside a container with networking disabled:

#!/usr/bin/env bash
# quarantine.sh — run an untrusted patch with no way in or out
set -euo pipefail

REPO=$1
PATCHFILE=$2
WORKDIR=$(mktemp -d)
trap 'rm -rf "$WORKDIR"' EXIT

git clone -q "$REPO" "$WORKDIR/tree"

docker run --rm \
  --network none \
  --cap-drop ALL \
  --read-only \
  --tmpfs /tmp \
  -v "$WORKDIR/tree":/repo \
  -v "$PATCHFILE":/queued.patch:ro \
  -w /repo \
  local/gate-runner:py312 \
  sh -c 'git apply /queued.patch && python -m pytest -x -q'

Two properties carry all the weight: traffic cannot escape (--network none), and nothing sensitive can wander in (no credential mounts, no capabilities, read-only root). A useful mental model: you're grading the artifact, not the agent. It doesn't matter where or how the model executed — only whether the patch survives this box.

Stage 2: Scope audit

Before reading a single implementation line, get the full inventory of what the patch touches:

git apply --stat /queued.patch
git apply --check /queued.patch   # a patch that won't apply cleanly gets rejected on the spot

Then classify every path relative to the original request:

Path in patch	Asked for?	Verdict
`src/parse_date.py`	yes	normal review
`tests/test_parse_date.py`	no	manual line-by-line
`pyproject.toml`	no	reject pending justification

The one heuristic I'd tattoo on the process: unrequested modifications to tests, CI configs, or dependency pins start with a presumption of guilt. Softening a gate is the cheapest way to make a wrong patch look right.

Stage 3: Adversarial probes

A passing project suite only tells you the patch preserved behaviors someone previously thought to test. Everything else is on you. I write a small set of hostile probes aimed at the change's probable weak spots. Say the patch rewrote a date-string parser:

# probes/test_date_hostile.py — written by me, never shown to the model
import pytest
from src.parse_date import parse_date

def test_rejects_feb_29_on_non_leap_year():
    with pytest.raises(ValueError):
        parse_date("2025-02-29")

def test_no_silent_rollover_on_month_13():
    with pytest.raises(ValueError):
        parse_date("2025-13-01")

def test_whitespace_and_unicode_padding_not_accepted():
    with pytest.raises(ValueError):
        parse_date("\u20032025-01-01")

def test_epoch_boundary_does_not_wrap_negative():
    assert parse_date("1969-12-31").year == 1969

One discipline makes this stage honest: write the probes before you open the diff. Once you've seen the implementation, your supposedly independent tests drift toward the cases the model already covers. Attack first, read second.

Stage 4: Decision record

Every accepted patch leaves one plain-text bundle:

ask: "make parse_date strict about malformed input"
date: 2026-08-09
compute: free model access + free server via MonkeyCode
scope: 1 requested file, 0 surprise files
project suite: 312 passed
hostile probes: 4 passed (attempt 1 failed 2, regenerated, then passed)
verdict: merged; human pass over parse_date.py:8-41

This feels like paperwork until a September merge quietly regresses in November. Then the record turns a forensic afternoon into a ten-minute lookup.

Doing this on zero budget

The gauntlet consumes two things: repeated model calls (hostile probes regularly send a patch back — budget two or three regeneration rounds per change) and sacrificial compute to run them in.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

My current loop runs on MonkeyCode's free model access together with its free server option, which suits exactly this pattern: frequent, low-stakes regeneration where per-call pricing would otherwise push you to ration attempts. That said, nothing in the four stages depends on any specific vendor — any model endpoint plus any disposable machine (a spare laptop, a CI free tier) runs the identical gauntlet. If you're assembling this from zero, rehearsing the workflow on a free tier is a reasonable first step before committing budget anywhere.

Where the gauntlet breaks

It proves behavior, not intent. Hostile probes won't surface a deliberately concealed flaw. Human diff review remains required — this process feeds it evidence, it never replaces it.
A container is hygiene, not a vault. No-network, dropped-capability defaults are prudent, not a promise against a determined exploit. Never quarantine untrusted patches on a host with privileges you'd miss.
Weak probes, false confidence. Stage 3 carries the rigor. If you phone it in, the remaining stages merely document your complacency.
Skip the ceremony where it doesn't pay: one-token typo fixes don't justify four stages. Conversely, irreversible domains — payments, signing, authorization — need a domain expert reading every line regardless of what any suite says.
Free tiers shift. The model and server availability described here are operator-supplied claims at the time of writing; re-verify before baking them into a team process.

The actual takeaway

The meaningful change is one of phrasing: stop asking "does this generated patch look correct?" and start asking "what can I demonstrate about it?" An offline quarantine, a scoped file audit, probes authored before the reveal, and a saved verdict — that chain is a demonstration, and today it costs essentially nothing to run.