DEV Community: Sam Li

The Model Passed Your Benchmark. Now Stop Merging Its Code Blindly

Sam Li — Fri, 07 Aug 2026 03:15:45 +0000

A few weeks ago I wrote about building a reproducible test harness for comparing free AI coding models before you commit. That harness answers one question: which model should I use?

It does not answer the harder follow-up: once a model generates a patch for my real codebase, when is it safe to merge?

This week there was a great discussion on DEV about "understanding over origin" — the idea that it doesn't matter whether code came from a human or a model, only whether someone actually understands it. I agree with the principle, but principles don't survive contact with a busy afternoon. What survives is a checklist with teeth. So here is the pipeline I bolted onto my model harness: every AI-generated patch has to pass through a scripted review gate before I even read it, and the script produces a scorecard that tells me how carefully I need to read it.

The problem with eyeballing diffs

When a model produces a 40-line diff that looks idiomatic, my brain does a dangerous thing: it pattern-matches on style and skips semantics. The code reads like something I'd write, so I approve it like something I'd write. The failures I've actually shipped from AI-generated code were never syntax errors — the tests even passed. They were things like:

A retry loop that retried on the wrong exception type, so real errors got swallowed.
A query filter that was subtly wider than the one it replaced (tests passed because fixtures were too small to notice).
A dependency added for a one-liner the standard library already covers.

All three would have been caught by asking four boring questions before reading the code. So I scripted the questions.

The review gate: a reproducible artifact

The gate is a small shell script. It takes a patch file, applies it to a throwaway worktree, and runs four checks. It never touches my working branch, and it prints a one-line verdict at the end.

#!/usr/bin/env bash
# review-gate.sh <patch-file> <base-branch>
set -euo pipefail

PATCH="$1"
BASE="${2:-main}"
WT=$(mktemp -d /tmp/ai-review.XXXXXX)

echo "== 1. Isolate =="
git worktree add --detach "$WT" "$BASE" >/dev/null
if ! git -C "$WT" apply --check "$PATCH" 2>/dev/null; then
  echo "VERDICT: REJECT (patch does not apply cleanly to $BASE)"
  git worktree remove --force "$WT"; exit 1
fi
git -C "$WT" apply "$PATCH"

echo "== 2. Full test suite (not just the touched package) =="
if ! (cd "$WT" && npm test --silent 2>&1 | tail -5); then
  echo "VERDICT: REJECT (tests fail)"
  git worktree remove --force "$WT"; exit 1
fi

echo "== 3. Diff surface audit =="
git -C "$WT" diff "$BASE" --stat
NEW_DEPS=$(git -C "$WT" diff "$BASE" -- package.json \
  | grep -c '^+.*".*":' || true)
echo "new dependency entries: $NEW_DEPS"

echo "== 4. Behavior-change heuristics =="
# Flag the patterns that have burned me before
SUSPECT=$(git -C "$WT" diff "$BASE" \
  | grep -E '^\+.*(catch|except|retry|timeout|WHERE|filter\()' \
  | wc -l | tr -d ' ')
echo "error-handling/query lines added: $SUSPECT"

if [ "$NEW_DEPS" -gt 0 ] || [ "$SUSPECT" -gt 5 ]; then
  echo "VERDICT: REVIEW CLOSELY (expanded surface area)"
else
  echo "VERDICT: STANDARD REVIEW"
fi

git worktree remove --force "$WT"

Adapt steps 2–4 to your stack. The point isn't my specific greps — it's that the checks run before my opinion does, every time, identically. Step 1 exists because a model's patch often "works" only against the stale context it was shown; applying it to a clean checkout of current main is itself a finding.

Where the free compute fits

This loop has two compute-hungry halves: generating candidate patches, and running the gate (full test suites per candidate get slow). Both halves are exactly where I've been using MonkeyCode. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Concretely: I use MonkeyCode's free model access to generate two or three independent candidate patches for the same task, and its free server option to run the gate script against each candidate so my laptop isn't tied up running three test suites in parallel. Generating multiple candidates is the underrated move here — when two independent generations converge on the same approach, my review confidence goes up; when they diverge, the divergence points are exactly where I read line-by-line.

I'm deliberately not quoting quotas, model names, or performance numbers, because those change and you should check the current state yourself. What matters for this workflow is just: the generation step and the sandboxed-test step cost me nothing and don't monopolize my machine.

The decision table

After the gate runs, I use this to decide how much of my own attention to spend:

Gate output	My review depth	Typical action
Patch doesn't apply	None	Re-prompt with fresher context; don't hand-fix
Tests fail	Read the failure only	Regenerate or discard; never patch the patch blindly
Passes, no new deps, low suspect count	Read every changed line once	Merge after reading
Passes, new deps or high suspect count	Read lines + write one adversarial test	Merge only if my new test passes
Two candidates diverge on approach	Read both diffs at the divergence point	Pick one, write down why in the commit message

The last row is the one that pays rent. "Write down why" is my personal guardrail for the understanding-over-origin principle: if I can't explain in one sentence why this diff is correct, it doesn't merge, regardless of how green the tests are.

Limitations, and who shouldn't do this

The heuristics in step 4 are mine, tuned to my past failures. Yours will be different. Start with an empty list and add a pattern every time AI-generated code burns you — the script should grow scar tissue, not ship with mine.
Green tests are a floor, not a ceiling. If your test suite has weak coverage, the gate's value collapses to step 1 and the greps. Fix coverage first.
Don't run this for trivial diffs. A five-line config change doesn't need a worktree and a scorecard; ceremony has a cost too.
This is not a security review. Dependency additions get flagged, not audited. If the model adds a package, that package still needs a real look.
If your repo's tests can't run in a clean checkout (hidden local state, manual env setup), step 1 will fail constantly and teach you nothing. That's actually a useful signal about your repo, but fix it before adopting the gate.

Closing

Model comparison tells you which generator to trust on average. A review gate tells you whether to trust this specific patch. The first question is interesting; the second is the one that decides what your users run. If you're generating candidates with free model access anyway, the marginal cost of gating every patch through a clean-worktree script is about fifteen lines of bash — and one honest commit message at a time, it keeps the understanding on your side of the merge button.

If you've built your own version of this — especially the failure-pattern greps — I'd genuinely like to see what's in your list.

A Reproducible Test Harness for Comparing Free AI Coding Models Before You Commit

Sam Li — Wed, 05 Aug 2026 07:05:57 +0000

Scrolling DEV this week, a pattern is hard to miss: local AI workspaces, multi-agent orchestration, and arguments about whether agent metrics even mean anything. Underneath all of it is one practical question most of us actually face: with so many models available, how do you decide which one deserves a place in your daily coding loop — without paying for the privilege of finding out?

This article describes a small, reproducible harness you can run against any coding-model endpoint, including free tiers. The harness is the point; the specific provider is interchangeable.

Why "just try it" doesn't scale

Vibe-testing a model on one or two prompts tells you almost nothing. Models are inconsistent across task types: a model that writes a beautiful regex may fumble a multi-file refactor, and single-run results are noisy because outputs are non-deterministic. What you want is:

A fixed set of tasks that resemble your real work.
A repeatable scoring method you can re-run when models change.
Zero or near-zero cost while you're still evaluating.

That third requirement is where free model access and free server options matter. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option, which makes it a convenient target for this kind of evaluation loop — you can burn through dozens of test runs without watching a billing dashboard. That said, everything below works against any OpenAI-compatible endpoint, so treat MonkeyCode as one possible backend, not the subject of the article.

The artifact: a minimal evaluation harness

The harness below (Python 3.10+, no dependencies beyond requests) runs a task suite against a chat-completions endpoint and records results to JSONL. Tasks are scored by automated checks where possible — compilability, unit tests, presence of required symbols — because LLM-as-judge adds cost and noise during a free-tier evaluation.

#!/usr/bin/env python3
"""eval_harness.py — reproducible coding-model smoke test.
Label: reference implementation; adapt task checks to your stack."""
import json, subprocess, tempfile, time, sys
from pathlib import Path
import requests

ENDPOINT = "http://localhost:8000/v1/chat/completions"  # free server or any compatible endpoint
MODEL = "your-model-name"                              # set per run, record it in output

TASKS = [
    {
        "id": "retry-helper",
        "prompt": "Write a Python function `with_retry(fn, attempts=3, backoff=0.5)` "
                  "that retries on Exception with exponential backoff. Code only.",
        "check": ["def with_retry", "attempts", "backoff"],
    },
    {
        "id": "sql-injection-fix",
        "prompt": "This code is vulnerable: `cursor.execute('SELECT * FROM users WHERE name = \"' + name + '\"')`. "
                  "Rewrite it safely in Python and explain in one sentence.",
        "check": ["?", "%s"],  # any parameterized form counts
    },
    {
        "id": "ts-debounce",
        "prompt": "Write a typed TypeScript `debounce` function with generic parameter preservation. Code only.",
        "check": ["function debounce", "setTimeout", "...args"],
    },
]

def run_task(task):
    started = time.time()
    resp = requests.post(ENDPOINT, json={
        "model": MODEL,
        "messages": [{"role": "user", "content": task["prompt"]}],
        "temperature": 0.2,
    }, timeout=120)
    resp.raise_for_status()
    text = resp.json()["choices"][0]["message"]["content"]
    hits = sum(1 for needle in task["check"] if needle in text)
    return {
        "task": task["id"],
        "model": MODEL,
        "latency_s": round(time.time() - started, 2),
        "check_score": f"{hits}/{len(task['check'])}",
        "output_chars": len(text),
        "output": text[:2000],  # truncate for review
    }

if __name__ == "__main__":
    out = Path("results.jsonl")
    with out.open("a") as f:
        for rep in range(3):  # 3 reps to catch non-determinism
            for task in TASKS:
                row = run_task(task)
                row["rep"] = rep
                f.write(json.dumps(row) + "\n")
                print(f"{row['model']} | {row['task']} | rep {rep} | "
                      f"{row['check_score']} | {row['latency_s']}s")

Three deliberate design choices:

Three repetitions per task. One-shot scores lie. If a model passes retry-helper once in three tries at temperature 0.2, that's a different verdict than 3/3.
Substring checks, not vibes. Crude, but objective and free. For Python tasks, you can extend this to actually exec the extracted code inside a subprocess with a timeout — I've sketched that as subprocess/tempfile imports above, and it's worth doing for tasks where correctness is testable.
JSONL output. Append-only, diffable, and easy to chart later. When a provider rotates models, re-run the suite and diff.

Decision table: when this evaluation loop makes sense

Situation	Free-tier harness a good fit?	Why
Picking a daily-driver coding model	Yes	You need many runs; cost would otherwise gate the experiment
Comparing latency on your own hardware/network	Yes, with caveats	Free servers may be shared or queued — record latency, don't over-trust it
Validating a model for a security-critical codebase	Partially	Free tiers are fine for capability screening, not for final sign-off
Benchmarking for a published claim	No	Uncontrolled environments and undisclosed quotas make results non-publishable
Sustained production workload	No	Free access is for evaluation; availability and limits can change

Limitations, honestly

Substring checks are shallow. They verify the model produced something shaped like the answer, not that it's correct. Extend the harness to execute code where you can.
Free-tier performance is not production performance. Queuing, shared capacity, and rate limits can distort latency numbers. Treat latency from a free server as a floor for stability testing, not a ceiling for speed.
No permanence assumptions. Free model access and free servers are availability claims, not guarantees. Build your harness so swapping the endpoint is a one-line change — the script above does exactly that.
Small task suites overfit. Three tasks is a starting skeleton. Grow it with tasks extracted from your recent commits and bug reports; that's the only distribution that matters.

Who should skip this approach

If you need a defensible benchmark for procurement or publication, use an established benchmark with controlled conditions. If your evaluation requires proprietary code that can't leave your infrastructure, don't send it to any hosted endpoint, free or paid — run local models instead (the recent DEV posts on local AI workspaces cover that path well). And if you already have a paid model that measurably works, the switching cost may exceed whatever a free alternative saves you.

Closing thought

The useful habit here isn't any particular provider — it's treating model selection as an experiment with a harness you own. Free access tiers, whether MonkeyCode's or anyone else's, lower the cost of running that experiment to roughly your time. If you build a task suite from your own work and re-run it whenever models rotate, you'll always know what you're actually getting — which is more than most benchmark screenshots can tell you.

If you end up extending the harness with real code execution, I'd be curious what checks you found most predictive of real-world usefulness — drop a note in the comments.