Stop Benchmarking Coding Models on Strangers' Bugs: A Reproducible Harness for Your Own Repo

Open-weight coding models are having a moment. MiniMax's recent open releases have been all over my feed, and every announcement comes with the same problem: the benchmarks are always somebody else's. SWE-bench scores and leaderboard deltas tell you very little about whether a model can fix your flaky pagination bug in your weird legacy service.

So I stopped reading leaderboards and built a tiny harness that runs candidate models against bugs I actually care about. This post is that harness: a ~80-line Python runner, a task format, and a decision table for where to run it cheaply. (Check MiniMax's official repos and model cards for current versions and licenses before you build anything on them — release cadence is fast and my notes will go stale.)

The idea: a personal bug corpus

Every time you fix a non-trivial bug, capture it while the context is fresh:

tasks/
  001-pagination-off-by-one/
    task.md          # problem statement, exactly what you'd paste into a chat
    repo_snapshot/   # minimal code needed to reproduce (strip secrets!)
    test.sh          # exits 0 if the fix is correct
  002-race-in-cache-invalidation/
    ...

The test.sh is the important part. Not "the answer looks right" — an executable check. Mine are usually just a pytest invocation plus one assertion about the fix's behavior:

#!/usr/bin/env bash
# test.sh — run inside the snapshot directory
set -e
cd repo_snapshot
python -m pytest tests/test_pagination.py -q --tb=short

Ten to twenty real bugs from your own codebase beats any public benchmark for your decision-making, because the distribution actually matches your work.

The runner

The runner talks to any OpenAI-compatible endpoint, so it works against a locally hosted open-weight model or a hosted API with zero code changes:

#!/usr/bin/env python3
"""eval_runner.py — run your bug corpus against any OpenAI-compatible model."""
import json, subprocess, sys, time
from pathlib import Path
from openai import OpenAI

client = OpenAI()  # reads OPENAI_BASE_URL and OPENAI_API_KEY from env

SYSTEM = (
    "You are a senior engineer. The user gives you a bug report and a code "
    "snapshot. Respond with a unified diff that fixes the bug. Output ONLY "
    "the diff, no prose."
)

def run_task(task_dir: Path, model: str) -> dict:
    prompt = (task_dir / "task.md").read_text()
    start = time.time()
    resp = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": prompt},
        ],
        temperature=0.2,
    )
    latency = time.time() - start
    diff = resp.choices[0].message.content

    # Apply the proposed diff to a throwaway copy, then run the check.
    workdir = task_dir / "repo_snapshot"
    applied = subprocess.run(
        ["git", "apply", "-"], input=diff, text=True,
        capture_output=True, cwd=workdir,
    )
    if applied.returncode != 0:
        return {"task": task_dir.name, "model": model, "verdict": "patch_failed"}

    check = subprocess.run(
        ["bash", str(task_dir / "test.sh")], capture_output=True
    )
    # Always reset so the next model gets a clean snapshot.
    subprocess.run(["git", "checkout", "."], cwd=workdir)
    return {
        "task": task_dir.name, "model": model,
        "verdict": "pass" if check.returncode == 0 else "fail",
        "latency_s": round(latency, 1),
        "tokens": resp.usage.total_tokens if resp.usage else None,
    }

if __name__ == "__main__":
    models = sys.argv[1:]
    results = []
    for task in sorted(Path("tasks").iterdir()):
        if not task.is_dir():
            continue
        for m in models:
            results.append(run_task(task, m))
            print(json.dumps(results[-1]))
    Path("results.json").write_text(json.dumps(results, indent=2))

Usage:

export OPENAI_BASE_URL="https://your-endpoint/v1"
export OPENAI_API_KEY="..."
python eval_runner.py model-a model-b model-c

Each result is one JSON line, so you can jq it, diff runs, or throw it in a spreadsheet. Keep temperature fixed across candidates or the comparison is meaningless.

What my first run actually showed

A few honest observations from running this against my own corpus:

1. Leaderboard ranking did not predict my ranking. A model with weaker public scores fixed my gnarliest concurrency bug; the "top" model produced a plausible-looking diff that failed the test in a subtle way. This is exactly why executable checks matter.
2. Patch-apply failures were the silent killer. About a fifth of failures across models weren't wrong fixes — they were malformed diffs. Adding "output ONLY the diff" to the system prompt cut that roughly in half. Prompt discipline is part of the eval.
3. Latency variance matters for agent loops. If you're wiring a model into a multi-step agent, a model that's 30% smarter but 4x slower can lose on total wall-clock time to a fix.

Where to run this: a decision table

Situation	Good fit
You have a modern GPU and want full control + privacy	Self-host the open weights (vLLM / llama.cpp)
No GPU, want to try several open models quickly	A hosted service with free model access
One-off experiments, CI eval on a schedule	A free server tier rather than renting a GPU
Regulated data that can't leave your network	Self-host only, no exceptions

For the middle two rows, I've been using MonkeyCode: it offers free access to coding models and a free server option, which is enough to run a corpus like this without provisioning hardware. The OpenAI-compatible API pattern above means the harness doesn't care which backend is behind the URL. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you want to replicate this setup, their free tier is a low-friction starting point — but the harness works identically against a local vLLM instance, and I'd encourage you to try both.

On the open-source spirit

The reason this harness is possible at all is the open ecosystem around it: open-weight models you can actually inspect and self-host, an OpenAI-compatible API convention that decouples tools from vendors, and open tooling like vLLM and pytest doing the heavy lifting. The healthiest thing a tool in this space can do is stay compatible with that ecosystem instead of locking you into one model. I care more that MonkeyCode gives free access to try models and a free server to run experiments than about any single feature — that interoperability is what open-source culture is supposed to produce.

Limitations, and who shouldn't do this

• Sample size. Twenty of your bugs is still a tiny, biased sample. Treat results as directional, not scientific.
• Contamination. If your bug fix is on a public repo, models may have memorized it. Prefer unreleased code or private repos.
• Free tiers have limits. Quotas, rate limits, and model availability on any free offering can change without notice; don't build critical CI on one without checking current terms.
• Not for regulated code. If your code can't leave your network, skip hosted options entirely — self-host only.
• Prompt sensitivity. A model that "loses" at temperature 0.2 with my system prompt might win with better prompting. The harness measures a workflow, not pure capability.

Takeaway

The next time a new open-weight model trends — MiniMax today, someone else next month — don't ask "is it good?" Ask "does it fix my bugs?" A personal corpus plus 80 lines of runner gives you an answer in an afternoon, and it keeps working no matter which model the hype cycle serves up next.