Stop Trusting Vibes: A Reproducible Harness for Comparing AI Coding Models on Your Own Codebase

Alex Zhu — Wed, 05 Aug 2026 06:53:44 +0000

Most comparisons of AI coding models are useless to you. Not because the authors are dishonest, but because they test on their problems: greenfield LeetCode-style prompts, demo TODO apps, or a framework you don't use. Your codebase has different failure modes — a weird build system, a legacy module nobody wants to touch, tests that take 40 minutes.

This article is a small, reproducible harness you can run in an afternoon to compare coding models against your own repository, with scoring based on your own test suite instead of vibes. The artifact is ~120 lines of shell and Python, plus a scoring rubric you can adapt.

The core idea

Instead of asking "which model is best?", ask: on a fixed set of real tasks from my repo, which model produces patches that pass my tests, fastest, with the least hand-holding?

That gives you three measurable axes:

Correctness — does the resulting diff pass the relevant tests?
Edit locality — did the model touch only the files it should have?
Iteration cost — how many prompt rounds did it take to get there?

Step 1: Build a task set from your own git history

The cheapest source of realistic tasks is your own commit log. Find commits that fixed a bug or added a small feature, then check out the parent commit and ask the model to reproduce the fix (without showing it the actual fix).

#!/usr/bin/env bash
# extract_tasks.sh — mine candidate tasks from git history
# Usage: ./extract_tasks.sh <repo_path> <count>
set -euo pipefail
REPO="$1"; COUNT="${2:-8}"
cd "$REPO"

# Small, self-contained commits: <= 3 files, <= 80 changed lines, has a test file touched
git log --oneline --no-merges -n 300 | while read -r sha msg; do
  files=$(git diff-tree --no-commit-id --name-only -r "$sha" | wc -l)
  lines=$(git diff --shortstat "$sha^" "$sha" | grep -oE '[0-9]+ insertion|[0-9]+ deletion' | grep -oE '[0-9]+' | paste -sd+ | bc)
  if [ "$files" -le 3 ] && [ "${lines:-999}" -le 80 ]; then
    echo "$sha|$files|$lines|$msg"
  fi
done | head -n "$COUNT"

Filter the output by hand. You're looking for tasks where the commit message describes the goal clearly enough to serve as a prompt. For each chosen task, record:

{
  "id": "task-03",
  "parent_sha": "a1b2c3d",
  "prompt": "Fix the bug where empty query strings crash the /search handler. Behavior should return HTTP 400 with a JSON error body.",
  "test_command": "pytest tests/test_search.py -x",
  "forbidden_paths": ["docs/", ".github/"]
}

The forbidden_paths field is deliberate: a model that "fixes" the bug by rewriting your CI config should be penalized.

Step 2: A runner that scores patches, not prose

The runner checks out the parent commit, hands the prompt to the model, applies the returned diff, runs your test command, and scores the result. The scoring is intentionally boring — boring scoring is reproducible scoring.

# score_run.py — evaluate one model output on one task
import subprocess, sys, json, pathlib

def run(cmd, cwd, timeout=300):
    return subprocess.run(cmd, cwd=cwd, shell=True,
                          capture_output=True, text=True, timeout=timeout)

def score(task, patch_text, workdir):
    result = {"task": task["id"], "applied": False,
              "tests_pass": False, "files_touched": [],
              "forbidden_touched": False, "score": 0}

    patch = pathlib.Path(workdir) / "candidate.patch"
    patch.write_text(patch_text)

    if run(f"git apply --check candidate.patch", workdir).returncode != 0:
        return result  # score 0: patch doesn't even apply
    run("git apply candidate.patch", workdir)
    result["applied"] = True

    touched = run("git diff --name-only", workdir).stdout.split()
    result["files_touched"] = touched
    result["forbidden_touched"] = any(
        t.startswith(tuple(task["forbidden_paths"])) for t in touched)

    t = run(task["test_command"], workdir, timeout=600)
    result["tests_pass"] = (t.returncode == 0)

    # Scoring rubric (adjust weights to what your team cares about)
    s = 0
    if result["tests_pass"]: s += 60
    elif result["applied"]: s += 10
    if result["tests_pass"] and not result["forbidden_touched"]: s += 25
    if len(touched) <= 3: s += 15
    result["score"] = s
    return result

if __name__ == "__main__":
    task = json.loads(pathlib.Path(sys.argv[1]).read_text())
    patch = pathlib.Path(sys.argv[2]).read_text()
    print(json.dumps(score(task, patch, sys.argv[3]), indent=2))

Run each task in a clean worktree (git worktree add /tmp/eval-task-03 <parent_sha>) so failures can't contaminate each other. Always reset between models.

Step 3: Run it without burning an API budget

Here's the practical constraint: a serious comparison — say 8 tasks × 3 models × 2 repetitions — is 48 full coding sessions. On metered APIs that adds up fast enough that most people skip the repetition and end up with statistically meaningless one-shot results.

This is where free tiers become genuinely useful rather than a gimmick. I ran my harness through MonkeyCode, which offers free access to coding models and a free server option you can point the runner at, so the repetition cost of the experiment is time, not money. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest caveats about any free tier, MonkeyCode included: free model lineups and rate limits change, so treat the specific models available as a snapshot, not a permanent fixture — and design your harness so the model endpoint is a config value, not a hardcoded assumption. If the free option disappears next quarter, your task set and scorer still work against whatever you point them at next.

What my rubric surfaced that leaderboard reading didn't

A few patterns only became visible because the tasks came from a real repo:

Patch application failures are more common than test failures. Models that produce beautiful explanations sometimes emit diffs with wrong context lines for your exact file versions. git apply --check catches this before you waste a test run.
Edit locality is a proxy for review cost. A model that passes tests but touches 11 files creates a worse review experience than one that touches 2, even at equal correctness. That's why locality is in the rubric.
Repetition matters more than model choice. The gap between two runs of the same model on the same task was sometimes larger than the gap between two different models. One-shot comparisons are noise; run everything at least twice.

Limitations and who should skip this

Small task bias. Mining small commits means your harness tests small fixes. It says nothing about multi-hour refactoring or architectural work. If that's what you need from a model, this harness under-measures.
Your tests are the oracle. Weak test suites produce inflated scores. If a task's test_command only covers the happy path, treat its score with suspicion.
Free tiers aren't a benchmark platform. Rate limits can distort wall-clock measurements, and you typically can't pin a model version. Use free access to screen candidates; re-confirm the winner on the exact paid tier/version you'd actually deploy.
Don't run this on code you can't share. Unless the service's data terms explicitly cover your use case, use a sanitized or open-source repo. A free endpoint is not a compliance review.

If your team already has a solid internal eval suite, or you work on regulated codebases where no external model call is permitted, this whole approach is the wrong tool.

Takeaway

"Which model is best" is a question with no shelf life. "Which model passes my tests on my task distribution" is a question you can re-answer every quarter with the same harness. Build the task set once, keep the scorer boring, make the endpoint swappable — and the next time a new model drops, you'll have an answer by lunch instead of a hot take.

If you want to try this without a metered bill, the free model access and free server option at MonkeyCode are a reasonable starting point for the screening pass — just keep the caveats above in mind before you trust the numbers.

What does your team use as the oracle for AI-generated patches — tests, human review sampling, or something else? Curious what's worked.

DEV Community: Alex Zhu