Consider a small team that sees two model releases in the same week. One is DeepSeek-V4-Pro-0813, described as cheap and good at code. Another is Grok 4.6, shared around as a strong debugging model. The team adds one to an issue triage bot because the per-token price looks negligible. Two days later, a patch closes the issue but rewrites retry logic in three unrelated modules.
The cheap model was not the problem. The missing check was.
New release notes are inputs, not evidence. A model can be cheap, capable on a public benchmark, and still be wrong about a private repository. The useful question is narrower: does the model fix this codebase without breaking something else?
The problem with benchmark-first model selection
Most benchmark comparisons flatten a model into a single number. That number hides the details that matter for a specific repository:
- Which error patterns appear in the codebase
- Which modules are allowed to change
- How small a patch should be
- Whether the test suite passes after the fix
A model can score well on general coding tasks and still produce broad patches that touch unrelated concerns. For a team with limited review time, that is the cost that shows up later.
The artifact: replay historical bugfixes
The harness starts from a simple idea. A repository already contains a record of its own bugs: the fix commits. Each fix commit has a parent state with a bug, a commit message, and a human patch that closed the issue.
The workflow is:
- Pull the last ten commits that mention 'fix' or 'bug'.
- For each commit, checkout the parent state.
- Ask the model for a minimal patch using the commit message as the prompt.
- Apply the model patch and run the test command.
- Compare the model patch to the human patch.
This produces five practical signals: targeted-test pass rate, patch application failures, unrelated file changes, patch size ratio, and runtime.
Prepare the cases
A small script can turn git history into a replay set.
python prepare_cases.py /path/to/repo
The script writes one JSON line per case.
#!/usr/bin/env python3
import json
import subprocess
import sys
from pathlib import Path
repo = Path(sys.argv[1] if len(sys.argv) > 1 else '.')
out = Path('cases.jsonl')
def git(*args: str) -> str:
return subprocess.check_output(['git', '-C', str(repo), *args], text=True)
log = git('log', '--grep=fix', '--grep=bug', '--all-match', '--oneline', '-n', '10')
cases = []
for line in log.strip().splitlines():
sha = line.split()[0]
parent = git('rev-parse', f'{sha}^').strip()
message = git('show', '-s', '--format=%B', sha).strip()
human_diff = git('diff', f'{sha}^', sha)
cases.append({
'sha': sha,
'base': parent,
'message': message,
'human_diff': human_diff,
})
with out.open('w') as f:
for case in cases:
f.write(json.dumps(case) + '\n')
print(f'wrote {len(cases)} cases to {out}')
The ten-case limit is deliberate. It keeps the run cheap, fast to review, and small enough to repeat when a new model appears.
Run the replay
Disclosure: This article was prepared as part of MonkeyCode's product outreach. The evaluator can point at MonkeyCode's free model access and free server option through an OpenAI-compatible base URL, but the method is not tied to that product.
The evaluator below is an unexecuted sketch. Replace the model call and test command with the repository's actual test runner.
#!/usr/bin/env python3
import json
import os
import subprocess
import tempfile
from pathlib import Path
from openai import OpenAI
client = OpenAI(
api_key=os.environ['API_KEY'],
base_url=os.environ.get('BASE_URL', 'https://api.openai.com/v1'),
)
def ask_model(message: str) -> str:
response = client.chat.completions.create(
model=os.environ.get('MODEL', 'gpt-4o-mini'),
messages=[
{
'role': 'system',
'content': 'You are a careful code reviewer. Return only a unified diff that fixes the bug described in the commit message. Do not add comments or explanations.',
},
{'role': 'user', 'content': message},
],
temperature=0.2,
)
return response.choices[0].message.content or ''
def files_from_diff(diff: str) -> set[str]:
return {line.removeprefix('+++ b/') for line in diff.splitlines() if line.startswith('+++ b/')}
def run_case(repo: Path, case: dict) -> dict:
model_diff = ask_model(case['message'])
patch_path = Path(tempfile.mkdtemp()) / 'model.diff'
patch_path.write_text(model_diff)
result = {
'sha': case['sha'],
'applied': False,
'test_pass': None,
'unrelated_files': 0,
'size_ratio': 0.0,
'error': None,
}
check = subprocess.run(
['git', '-C', str(repo), 'apply', '--check', str(patch_path)],
capture_output=True,
text=True,
)
if check.returncode != 0:
result['error'] = check.stderr[:200]
return result
subprocess.run(
['git', '-C', str(repo), 'apply', str(patch_path)],
capture_output=True,
text=True,
)
result['applied'] = True
test_cmd = os.environ.get('TEST_CMD', 'pytest -q')
test = subprocess.run(
test_cmd.split(),
cwd=repo,
capture_output=True,
text=True,
)
result['test_pass'] = test.returncode == 0
human_files = files_from_diff(case['human_diff'])
model_files = files_from_diff(model_diff)
result['unrelated_files'] = len(model_files - human_files) + len(human_files - model_files)
human_size = max(len(case['human_diff'].encode()), 1)
result['size_ratio'] = round(len(model_diff.encode()) / human_size, 2)
subprocess.run(['git', '-C', str(repo), 'reset', '--hard'], capture_output=True)
return result
repo = Path(os.environ.get('REPO_DIR', '.'))
cases = [json.loads(line) for line in Path('cases.jsonl').read_text().splitlines() if line.strip()]
for case in cases:
result = run_case(repo, case)
print(json.dumps(result))
Run this in a disposable git worktree, not on the main checkout. The reset command is intentionally blunt.
git worktree add /tmp/model-harness HEAD
REPO_DIR=/tmp/model-harness API_KEY=... BASE_URL=... python evaluate.py
Read the output as a gate, not a leaderboard
Each JSON line answers a narrow question: did this model replay a known fix without expensive side effects?
| Signal | Acceptable | Investigate |
|---|---|---|
| Targeted-test pass rate | 8 or more of 10 cases | 7 or fewer |
| Patch application failures | 0 failures | 1 or more |
| Unrelated files per patch | 0 in most cases | More than 1 |
| Patch size ratio | Less than 1.5x the human fix | More than 3x |
A single case is not a verdict. Run the set more than once because model output can vary. A model that passes nine cases once and five cases on the second run is not stable enough for unattended use.
Where a free endpoint and free server fit
The main excuse for skipping a local eval is cost. That excuse weakens when the model call can go through a free endpoint and the harness can run on a free server.
For this workflow, the free tier is best treated as a qualification stage. It can run the ten-case replay nightly or whenever a release note starts circulating. It should not be the only review step before merging production code.
Set BASE_URL and API_KEY once, point the evaluator at the desired model identifier, and keep the same cases. That gives a repeatable comparison across models, including names like DeepSeek-V4-Pro-0813 or Grok 4.6 when they appear in the feed.
Limitations and who should not use this
This harness is intentionally small, and that is also its limit.
- Historical fixes may already be in the model's training data. A memorized patch can inflate the pass rate.
- A passing patch is not automatically correct. The diff still needs human review.
- The commit message may not contain enough context. Add the failing test or stack trace when possible.
- A weak test suite produces weak signal.
- The reset and apply steps are unsafe on a dirty checkout. Always run the harness in a worktree or container.
- Ten cases cannot validate model safety, security, or long-term maintainability.
Do not use this approach where the cost of a bad patch is high, where no test suite exists, or where manual review is already faster than maintaining the harness.
The cheapest model is not the one with the lowest price. It is the one that clears this codebase's own regression gate with the smallest footprint. Run the ten-case replay before the next release note becomes a merge request.
Top comments (0)