New Model Dropped? Run Your Own Git History Through It First

#ai #llm #python #productivity

The release notes say it's faster. The launch thread says it beats everything. Three people I follow have already switched. And yet, every time I've switched on that basis alone, I've quietly switched back two weeks later after the model mangled a migration script or confidently explained a bug that didn't exist.

The gap is simple: public benchmarks measure what benchmarks measure. My daily work is a pile of half-remembered Django internals, shell one-liners I keep re-googling, and docstrings for functions nobody else will ever read. None of that shows up on a leaderboard. So instead of arguing about rankings, I built a small ritual: when a model worth caring about appears, I spend one evening replaying my own recent work through it. This post is that ritual — the corpus format, the runner, and the scorecard I use to decide.

Ask three separate questions, not one

"Is this model good?" is unanswerable. These are not:

Does it survive my prompt shapes? Mine are ugly: truncated stack traces, two files pasted back to back, instructions like "don't change the public API."
Can I afford to be careless with it? A model that's perfect but expensive gets saved for special occasions, which means it never actually helps.
How does it fail? Quiet, hedged uncertainty I can work with. Invented certainty I cannot.

No single number answers these. A dozen prompts from your own history does.

Mine your own history for test cases

I keep everything in one folder, one YAML file per case, pulled from things I actually asked in the past month. My current set has twelve cases across four categories:

Untangle — "This bash script has a quoting bug somewhere. Find it and explain why it only breaks on filenames with spaces." Pass = names the real quoting issue, not a cosmetic one.
Write-under-constraint — "Add type hints to this function without changing any runtime behavior, Python 3.9 compatible." Pass = the file still imports and mypy is happy.
Diagnose — "Here's a traceback plus the 30 lines around it. Where's the fault?" Pass = correct file and correct reasoning.
Translate — "Rewrite this jQuery snippet as vanilla JS, keep the debounce." Pass = behavior preserved, no library left in.

Each file looks like this:

id: bash-quoting-02
category: untangle
prompt: |
  This script breaks on filenames with spaces. Why?
  <script pasted here>
checks:
  mechanical: "bash -n answer.sh"   # syntax must be valid
  by_eye:
    - "identifies word splitting, not just 'add quotes somewhere'"
    - "doesn't rewrite the whole script unprompted"

Two kinds of checks, on purpose. Anything a machine can verify — parses, compiles, valid JSON, exit code zero — gets verified by a machine. Human attention is reserved for the cases where correctness is a judgment call, because that's where models get sneaky.

The runner (deliberately boring)

One file, no dependencies beyond the standard library, works against any OpenAI-compatible endpoint:

#!/usr/bin/env python3
"""replay.py — feed a YAML prompt corpus to any chat-completions endpoint."""
import json, os, subprocess, sys, time
from pathlib import Path
from urllib import request

def ask(base, key, model, prompt):
    payload = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,   # we're comparing models, not sampling moods
    }).encode()
    req = request.Request(
        f"{base.rstrip('/')}/chat/completions", data=payload,
        headers={"Authorization": f"Bearer {key}",
                 "Content-Type": "application/json"})
    start = time.monotonic()
    with request.urlopen(req, timeout=180) as resp:
        data = json.loads(resp.read())
    return data["choices'][0"]["message"]["content"], time.monotonic() - start

The rest of the script loops over the YAML files, writes each answer to answers/<model>/<id>.md, runs any mechanical check against the extracted code block, and prints a one-line summary. Run it once per model:

REPLAY_KEY=... python replay.py ./cases https://endpoint.example/v1 model-name

A few choices that matter more than they look:

Temperature zero. You want the model's default behavior, not a lucky sample. (Caveat below: zero isn't guaranteed deterministic, so re-run anything surprising.)
Answers saved as files, not printed. Diffing two models' answers side by side in an editor is worth more than any score.
Timeouts are generous. A slow-but-right answer is data; a killed request is noise.

Making evaluation cheap enough to actually do

Here's the honest reason most people never do this: running a dozen long prompts against every new release costs money, so we outsource the decision to whoever tweets loudest. The fix isn't discipline, it's making the experiment nearly free.

My current setup: I point the runner at MonkeyCode, which provides free model access and a free server option, and use it purely as an evaluation bench — new model shows up, I replay my twelve cases that evening, no invoice attached. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free tier at monkeycode.ai is enough for a corpus this size if you want to replicate the setup; the runner itself is endpoint-agnostic, so nothing here locks you to one provider.

The separation is the point: free tier for experiments, paid-and-trusted endpoint for anything that ships. Evaluation traffic tolerates rate limits and queues; production traffic doesn't.

The scorecard that decides

After the mechanical checks run and I've read the remaining answers (fifteen minutes, coffee), each model gets judged against:

Test	Bar it has to clear
Mechanical checks vs. my current model	Equal or better
Any silently-wrong code in Untangle/Translate?	One is disqualifying
When unsure, does it say so?	Must hedge; bluffing = out
Latency on my longest prompt	Interactive-usable, not batch-usable
Sustainable at my daily volume?	Firm yes

Two hard-won rules:

Bluffing is the only fatal flaw. In the Diagnose category, a model that answers "most likely X, but verify Y" is a colleague. A model that fabricates a precise-sounding root cause is worse than no model, because it fails at the exact moment you were about to trust it.

Score per category, never in aggregate. I currently use one model for untangle/diagnose work and a different one for write-under-constraint tasks, because the last three "best overall" models I tested each lost at least one category badly. Winners rotate; categories don't.

Where this breaks down

Twelve cases is a smoke alarm, not a laboratory. It reliably catches "obviously worse for my work." It cannot detect "marginally better on average." If you need that, you need a real eval suite.
The by-eye review doesn't scale. That's the cost of testing what you actually care about. Accept it.
Free endpoints come with real limits — throttling, queues, terms that can change. Fine for an evening experiment; never wire one into a user-facing path.
Temperature zero isn't a determinism guarantee on most inference stacks. Double-run anything surprising before drawing conclusions.
Skip the whole ritual if you prompt twice a month, or if your employer mandates a model — then your energy belongs in the security review, not a personal harness.

The point

When the next release floods your feed with charts, you don't need an opinion about the charts. You need one evening, twelve prompts from your own history, and a scorecard that punishes bluffing. Models that survive your real work earn a slot; the rest were launch marketing with an API key.

If you've gone further down this road — especially better ways to automate the "by eye" half — I'd love to compare notes in the comments.