I Grade Every New Coding Model Against My Own Bug History — Here's the Scorecard

A model shipped this week. The screenshots arrived within hours: impossible refactors, one-shot fixes, declarations that everything before it is obsolete. I stopped arguing with launch-week sentiment a while ago. Instead I do something more boring and more useful: I grade the new release against a scorecard built from my own git history, and the scorecard decides whether the model gets anywhere near my working tree.

This is a different framing from the usual "run a benchmark" advice. I'm not benchmarking. I'm auditing a candidate against incidents I've already lived through — and the grading is weighted, because not all failures cost the same.

Why scorecards beat impressions

When I post-mortemed my own bad model adoptions, the mistakes clustered into three buckets:

• Selection effects in what I saw. Public demos are chosen because they succeed. My code has none of that filtering.
• Recall masquerading as reasoning. Tasks that look like public GitHub code can be answered from memory. My evaluation tasks must come from private history.
• Noisy first impressions. A model that shines on a blank file can still wreck a mature module with house conventions. The second case is what I actually pay for.

A scorecard handles all three: frozen tasks from private PRs, mechanical checks, and weights that reflect blast radius rather than vibes.

Step 1: Mine your own git history for tasks

The fastest way to build a task suite is to recover work you've already verified. I scan for small, self-contained fixes:

# Candidate commits: single-file changes, small diffs, likely bugfixes
git log --since="-6 months" --name-only --oneline \
  | awk '/^[0-9a-f]{7,}/ {sha=$1} /\.[a-z]+$/ {print sha, $0}' \
  | sort | uniq -c | awk '$1 == 1 {print}' > candidates.txt

From that list I hand-pick eight episodes: a null-handling regression, a race in a background worker, a migration that needed a rollback path, a refactor tangled in an internal style guide. Each becomes:

suite/worker-race/
  brief.md        # the task, written once, model-agnostic
  before.patch    # starting state of the code
  check.sh        # mechanical verification, exit 0 = pass

Two rules keep this honest: every task has a known-good resolution (because I merged one), and check.sh never gets edited to accommodate a model. A confusing brief is a defect in the task, not evidence about the model.

Step 2: Weighted scoring, not pass counting

Raw pass rate hides the failures that actually hurt. I assign each task a weight reflecting what a wrong answer would cost in production, then let a script do the arithmetic:

#!/usr/bin/env python3
"""grade.py — read results/*.json, emit a weighted verdict."""
import json, glob, sys

# weight = blast radius if this category fails silently
WEIGHTS = {"correct": 1.0, "scope": 3.0, "revertable": 2.0}

def grade(result):
    score, total = 0.0, 0.0
    for task in result["tasks"]:
        total += sum(WEIGHTS.values())
        if task["tests_pass"]:            score += WEIGHTS["correct"]
        if not task["touched_out_of_scope"]: score += WEIGHTS["scope"]
        if task["diff_reverts_cleanly"]:   score += WEIGHTS["revertable"]
    return score / total

for path in glob.glob("results/*.json"):
    r = json.load(open(path))
    print(f"{r['model']:40s} {grade(r):.2%}")

The deliberate asymmetry: scope discipline is weighted three times higher than correctness. In my incident history, the expensive model behavior was never "the fix was wrong" — tests catch that. It was "the fix was right and it also rewrote two files nobody asked about." A candidate at 91% with surgical diffs outranks one at 95% that wanders. The weighting table is the actual decision document; tune the weights to your own incident log, not to mine.

I also log latency and token counts per task, because "free" and "cheap" claims only mean something measured against your workload's shape.

Step 3: One attempt, frozen conditions

Every (model, task) pair gets exactly one shot at temperature zero. Retries and re-prompting measure my patience, not the model. The runner is a thin loop that applies before.patch, invokes whatever endpoint I'm testing through a generic wrapper, applies the returned diff, runs check.sh, and dumps a JSON record into results/. Swapping in this week's release is one environment variable — the model is the variable; everything else stays frozen so scores are comparable across weeks.

Running this without a line item in the budget

Scoring every new release against paid APIs adds up fast, and self-hosting a day-one model eats the hour you were trying to save. For these grading passes I currently use MonkeyCode, which offers free access to a selection of models along with a free server option for the runner side — a full eight-task grading run costs me the time, not an invoice. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Caveats I'd apply to any free tier, this one included: treat it as an evaluation environment, not a production commitment, and don't assume a hosted endpoint behaves identically to the vendor's reference deployment. If a candidate passes and later graduates to real use, I re-grade it against the production endpoint before cutover — I've seen endpoint differences flip a verdict.

Passing earns a probation period, not access

A strong score buys the model one week as a suggestion-only assistant: it proposes, I read every line, nothing merges itself. Week two it may open draft PRs on low-risk paths. Proximity to CI-gated auto-merge comes later and stays behind required human approval. The scorecard is a necessary gate, never a sufficient one.

Where this breaks down

• Eight weighted tasks is a smoke screen, not a ranking system. It eliminates disasters and wanderers; it won't finely separate two good candidates. Grow the suite before making expensive commitments.
• Single-shot, temperature-zero grading is unfair to agent-loop models. If your workflow is multi-turn, build a frozen multi-turn variant — but freeze it just as hard.
• Small or mostly-greenfield repos have less to lose. This scorecard pays for itself on large, old, convention-heavy codebases.
• Data handling comes first. If code can't leave your perimeter, don't send it to any third-party endpoint, free tiers least of all.

The feed will crown another model next week. The scorecard doesn't care — it takes an hour, speaks in weights, and grades against bugs I've actually paid for. If you're assembling your own version and want a zero-cost place to run it, MonkeyCode's free model access and server are a reasonable bench; the git history, the checks, and the weights have to come from you.