A New 'Cheap and Good' Model Dropped. Here's How I Decide If It Earns a Place in My Stack

Every few weeks a new model release lands with the same pitch: cheaper, faster, smarter. The timeline fills with hot takes within hours, and by the next day half of them are quietly deleted. I stopped reading launch-day benchmarks a while ago, because none of them answer the only question that matters to me: does this model fail on the kinds of tasks I actually give it?

This post is the evaluation workflow I run whenever a shiny new budget model shows up. It costs almost nothing, takes an evening, and produces a decision I can defend to my future self. It's model-agnostic — swap in whatever just launched.

The core idea: your repo is the benchmark

Public benchmarks measure average performance on average problems. My problems are not average: a legacy Django app with a cursed settings file, a data pipeline with timezone bugs that only appear in February, a test suite that depends on execution order. A model that scores 90% on a leaderboard can still be useless to me if it confidently rewrites my migration files.

So the harness below runs candidate models against tasks extracted from my own git history, in an isolated environment, and scores the diffs mechanically before I ever read a single output.

Step 1: Mine your own bugs into a task file

I keep a running list of past bugs that cost me real time. For each one I record: the failing behavior, the file(s) involved, and the fix that actually worked. Then I turn each into a task prompt with the fix removed:

[
  {
    "id": "tz-off-by-one",
    "context_files": ["pipeline/schedule.py", "tests/test_schedule.py"],
    "prompt": "Jobs scheduled between 23:30 and 00:00 UTC occasionally run a day late. Find and fix the bug. Do not change the public API of schedule_next_run().",
    "known_good_patch": "patches/tz_off_by_one.diff",
    "acceptance": "pytest tests/test_schedule.py -q"
  }
]

Ten to fifteen of these is enough. If you don't have a list, start one today — I wrote previously about using past bugs to judge new models, and this is the same discipline with automation wrapped around it.

Step 2: Run candidates in a sandbox, never on your machine

Each candidate model gets the same task file, the same context files, and a throwaway environment. The isolation matters twice: once for safety (an agent with shell access and a bad idea can do real damage — I wrote a separate post about sandboxing coding agents before giving them your shell), and once for fairness (a polluted environment contaminates results).

My runner is deliberately boring:

import json, shutil, subprocess, tempfile
from pathlib import Path

def run_task(task, model_client, repo_root):
    workdir = Path(tempfile.mkdtemp(prefix="eval_"))
    shutil.copytree(repo_root, workdir / "repo", dirs_exist_ok=True)

    diff = model_client.solve(
        prompt=task["prompt"],
        context_files=[workdir / "repo" / f for f in task["context_files"]],
    )

    apply = subprocess.run(
        ["git", "apply", "-"], input=diff, text=True,
        cwd=workdir / "repo", capture_output=True,
    )
    if apply.returncode != 0:
        return {"id": task["id"], "result": "patch_rejected"}

    test = subprocess.run(
        task["acceptance"], shell=True,
        cwd=workdir / "repo", capture_output=True, timeout=300,
    )
    return {
        "id": task["id"],
        "result": "pass" if test.returncode == 0 else "fail",
        "stderr_tail": test.stderr[-500:].decode(errors="replace"),
    }

This is a starting point, not a product — adapt the model_client interface to whatever provider you're testing. The point is that every model faces identical conditions and the first scoring pass requires zero human judgment.

Step 3: Score in three buckets, not one number

I record results in a table like this (fill in your own numbers — mine vary per release):

Outcome	What it means	My tolerance
Pass, clean diff	Safe to delegate similar tasks	Unlimited use
Pass, messy diff	Works but creates review burden	Supervised use only
Fail confidently	Claims success, tests disagree	Disqualifying for autonomous use
Fail honestly	Says "I'm not sure"	Acceptable, sometimes preferable

The third row is the one benchmarks never show you. A model that fails loudly is a tool; a model that fails while reporting success is a liability. When a hot new release lands, this is the bucket I'm watching.

Where free access changes the math

The obvious objection to running this harness on every new release is cost — both API spend and the machine to run it on. This is where I've been using MonkeyCode: it offers free access to a rotating set of models and a free server option, which maps neatly onto this workflow. The candidate model runs through their free model access, and the sandbox runner runs on their free server instead of my laptop, so a full evaluation evening costs me nothing but time.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest caveats on that: free tiers change, so check current availability before building a habit around them; and the free model lineup won't always include the exact release you want to test — when it doesn't, I run that one candidate through its own provider's cheapest tier and keep everything else on the free setup.

Step 4: The promotion ladder

A model that survives the harness doesn't get my trust — it gets a probationary role:

1. Week 1–2: draft-only. It writes, I review every line, nothing merges unedited.
2. Week 3–4: low-risk autonomy. Test fixes, doc updates, rename refactors.
3. Ongoing: re-run the harness after every provider-side model update, because models silently change under a fixed name.

Most "cheap and good" releases stall at step 1 for me, and that's fine — a drafting assistant that saves twenty minutes a day is already worth having. The harness just tells me which rung a model has earned, instead of letting launch-day hype decide.

Limitations, and who shouldn't bother

• Sample size. Fifteen tasks can't characterize a model. It can only catch disqualifying behavior early, which is all I'm asking of it.
• Contamination. If your historical bugs resemble public training data, results skew optimistic. Prefer recent, private-codebase bugs.
• Prompt sensitivity. Identical prompts across models is fair for comparison but may undersell a model that needs different prompting style.
• Skip this entirely if you're evaluating a model for a one-off task, or if your work is mostly greenfield code with no regression surface to test against — the harness's value comes from having real past failures to replay.

The takeaway

The question is never "is the new model good?" It's "good at my failures, under my constraints, at this price?" A task file mined from your own git history, a sandbox, and a mechanical scoring pass will answer that in one evening — and the answer will still be true after the hype cycle moves on. If you want to try the harness without spending anything, MonkeyCode's free model access and free server are one way to stand it up; the code above doesn't care where it runs.

What's in your task file? I'd genuinely like to know which historical bugs other people use as their model litmus test.