Judge New Models With the Bugs That Already Burned You

Every model launch arrives with confetti: a chart, a cherry-picked demo, a thread full of people declaring the old stack dead. I still read those threads, but I no longer let them choose my tools. A public score can tell me which system wins across a crowd of strangers. It cannot tell me whether a model will respect the weird constraints that keep my own projects upright: the retry rule that only fails during deploys, the ORM migration that is fine until it meets a legacy table, the timezone edge case that appears every fiscal rollover.

So I stopped asking, “Is this model good?” and started asking, “Does it step on the same landmines I already paid to learn?” The raw material for that answer is not on a leaderboard. It is in old incident notes, reverted commits, and issue threads with titles nobody should ever have to write twice.

A scoreboard cannot see your blast radius

General evals are useful for broad comparison, but they sample a public universe. My expensive failures are local. One came from a payment exporter that rounded correctly in tests and still produced a one-cent mismatch when a customer’s daylight-saving boundary landed inside a settlement window. Another was a websocket client that looked stable in staging, then amplified reconnects after a proxy timeout and turned a small network blip into a queue backup. A third was a CLI that worked everywhere except CI images with a different $HOME layout.

None of those are famous benchmark problems. All of them are exactly the kind of context-shaped failure that makes a “better” model feel worse in practice. If a coding assistant rewrites the retry loop and removes jitter because it looks untidy, aggregate accuracy will not save my evening.

That is why my evaluation unit is not a prompt. It is a scar: a small, runnable package built from something that once hurt, with the embarrassing details preserved enough to matter.

Build a scar suite, not a demo reel

The process starts after the fix, when the bruise is still visible.

1. Pick 5–8 closed incidents with teeth. Favor bugs where the wrong answer looked plausible: off-by-one retries, narrowing types under a strict config, path handling on another OS, cache invalidation around partial writes. Skip pure typos unless they reveal a real misunderstanding.
2. Cut each case down until it runs alone. Copy the smallest module that still shows the behavior. Replace private services with fakes, but keep the constraint that made the bug real. If the case needs your whole monorepo, it is archaeology, not an eval.
3. Write the oracle from the incident, not from taste. The check should encode the failure mode: “must include bounded backoff,” “must not emit naive UTC midnight,” “must return exit code 2 on unreadable config.” A prettier solution that breaks the constraint fails.
4. Date and rotate the suite. Models change, stacks change, and yesterday’s sharp edge becomes today’s routine. I keep a retired/ folder so old cases stop pretending they represent current risk.
5. Keep secrets out. Scrub tokens, customer names, hostnames, and internal URLs. A personal benchmark is still a liability if it leaks the map to your house.

A case directory is deliberately plain:

scars/
  2026-04_dst-cent-drift/
    prompt.md
    starter/
    verify.sh
    origin.md
  2026-06_ws-reconnect-storm/
    prompt.md
    starter/
    verify.sh
    origin.md

origin.md is short and human: what paged you, what the fix taught you, why the verifier is strict. That note is what turns a pass/fail count into engineering memory.

A small runner that only cares about evidence

I want the harness to be boring enough that I trust it when I am tired. This version uses only the Python standard library, talks to any OpenAI-compatible chat endpoint, and records JSONL so results are easy to diff later:

#!/usr/bin/env python3
# Replay a local scar suite against one candidate model.
# EVAL_BASE_URL=https://host/v1 EVAL_API_KEY=... python scar_run.py MODEL scars/
import json, os, shutil, subprocess, sys, tempfile, time, urllib.request
from pathlib import Path

BASE = os.environ.get('EVAL_BASE_URL', '').rstrip('/')
KEY = os.environ.get('EVAL_API_KEY', '')

if not BASE or not KEY:
    raise SystemExit('set EVAL_BASE_URL and EVAL_API_KEY')

def chat(model, prompt):
    body = json.dumps({
        'model': model,
        'temperature': 0,
        'messages': [{'role': 'user', 'content': prompt}],
    }).encode()
    req = urllib.request.Request(
        BASE + '/chat/completions',
        data=body,
        headers={'authorization': 'Bearer ' + KEY, 'content-type': 'application/json'},
        method='POST',
    )
    with urllib.request.urlopen(req, timeout=120) as r:
        data = json.loads(r.read().decode())
    return data['choices'][0]['message']['content']

def verify(case_dir, candidate_text):
    with tempfile.TemporaryDirectory() as tmp:
        work = Path(tmp) / 'work'
        shutil.copytree(case_dir / 'starter', work)
        (work / 'candidate.patch').write_text(candidate_text)
        probe = subprocess.run(
            ['bash', str(Path(case_dir / 'verify.sh').resolve())],
            cwd=work, capture_output=True, text=True, timeout=60,
        )
        return probe.returncode == 0, (probe.stdout + probe.stderr)[-500:]

def main(model, suite):
    out = Path('scar_results.jsonl').open('a')
    for case in sorted(Path(suite).iterdir()):
        if not (case / 'prompt.md').exists():
            continue
        prompt = (case / 'prompt.md').read_text() + '\nReturn only a unified diff for starter/.'
        began = time.time()
        try:
            text = chat(model, prompt)
            ok, log = verify(case, text)
        except Exception as e:
            ok, log = False, 'harness-error: ' + repr(e)
        rec = {'model': model, 'case': case.name, 'ok': ok, 'sec': round(time.time() - began, 2), 'log': log}
        out.write(json.dumps(rec) + '\n')
        print(('PASS' if ok else 'FAIL'), case.name, rec['sec'], 's')
    out.close()

if __name__ == '__main__':
    main(sys.argv[1], sys.argv[2] if len(sys.argv) > 2 else 'scars')

The important choices are not the HTTP calls. First, the model is asked for a diff, so accidental prose usually breaks application and gets counted as a failure instead of being politely ignored. Second, every record keeps a verifier log tail, because “missed an import” and “removed rate limiting under load” both score zero while demanding different reactions. Third, latency lives next to correctness; a model that is right after a long think may be perfect for nightly migration review and miserable in an editor.

Turn counts into decisions, not mythology

The output is only useful if it changes a small, concrete choice. I read a fresh run like this:

Signal in the JSONL	Likely meaning	Next move
One old case fails, everything else clean	Possible task quirk, possible real weakness	Open origin.md and hand-check before caring
Failures cluster around one incident family	The model shares my blind spot	Treat as workload-specific evidence
All green on old cases	Suite may be stale or contaminated	Mine newer bugs and retire solved shapes
Correct but consistently slow	Fine for batch review, risky for interactive help	Route by task, not by hype
Same case flips between runs	Unstable on my constraints	Do not trust a single score

This is deliberately unheroic. I am not crowning a champion. I am deciding whether a candidate deserves a longer trial in the places where I already know the floor is thin.

Cost is the difference between ritual and aspiration

The reason many personal eval plans die is not philosophy; it is friction. A serious pass means several models, several seeds, and enough repetition that a lucky run does not become religion. On a metered bill, that can turn a healthy habit into a quarterly exception.

Open-weight releases help because the same artifact can be reached through more than one host, which weakens the idea that evaluation belongs to whichever endpoint sent the most polished announcement. I also run these replays through MonkeyCode where its current free model access and free server option make the sweep cheap enough to do after a release instead of merely intending to. Disclosure: This article was prepared as part of MonkeyCode's product outreach. That matters here for one practical reason: the harness above is just files and environment variables, so if an offer changes, EVAL_BASE_URL changes and the methodology survives. I would not build a durable habit around any plan, free or paid, that required rewriting the tests when billing changed.

There is also a credibility benefit. When the model is portable and the eval cases are inspectable, another engineer can rerun the claim and argue with the logs instead of the vibe.

Where this can mislead you

• Small suites lie by omission. Eight scars can say, “this trips on my known traps,” not “this is universally better.” The moment you hear yourself making a market ranking from six folders, stop.
• Yesterday’s bug becomes today’s memorized answer. If a case is based on a public repo or a famous postmortem, assume leakage is possible. Prefer private incidents and keep mining.
• A verifier can be too strict. If the check demands one exact implementation, you may reject a safer design. Review failures before converting them into dogma.
• Free access is an availability statement, not a vow. Do not hard-code quotas, model lifetimes, or permanence into your team process. Keep endpoints swappable and archive results.
• This is not compliance. Security reviews, regulated releases, procurement, and customer-facing guarantees need broader coverage, threat modeling, and human sign-off. A scar suite can narrow the field; it cannot absolve the decision.

Skip it if your risk is already formal

If you are a solo builder or a small team drowning in release notes, this gives you a repeatable Monday move: take the announcement, run the suite against the bugs that taught you something, keep the log, and get back to work. If you are choosing a platform for hundreds of developers, use this only as an early filter before a real evaluation program with representative tasks, privacy review, and rollout criteria.

The deeper shift is that model choice stops being a spectator sport. Your git history is a private benchmark shaped by the exact ways your system fails. Freeze a few of those lessons while they are still sharp, replay them when the next model arrives, and let the most painful old bug cast the first vote. If you try it, I would rather hear which resurrected incident broke the new hotness than which badge it earned somewhere else.