Route by Task, Not by Hype: A Budget-Aware Harness for Trying New Coding Models

#ai #tooling #llm #python

Every few weeks a new checkpoint drops and the timeline fills up with claims that it's cheaper, smarter, and about to change everything. Some of those claims hold up. Many don't. And even when a model genuinely is better on public leaderboards, that tells you almost nothing about whether it's better on your codebase, your tasks, and your budget.

I wrote previously about building a reproducible harness before wiring any model into your workflow. This article is the sequel nobody asked for but everybody needs: once you have a harness, how do you evaluate a steady stream of new models without spending a steady stream of money?

The answer I keep coming back to is routing by task difficulty: don't run your whole eval suite against every candidate. Tier your tasks, send the cheap ones to cheap models, and reserve expensive runs for the cases that actually discriminate between models.

The problem with "run everything against everything"

If your eval suite has 60 tasks and a new model appears every two weeks, naive evaluation costs scale linearly forever. Worse, most of those runs are wasted signal:

Easy tasks (rename a variable, write a docstring, fix an obvious off-by-one) are solved by almost every current model. Running a frontier-priced model on them tells you nothing.
Medium tasks (implement a small feature against an existing test, refactor across two files) are where models actually diverge.
Hard tasks (multi-file reasoning, subtle concurrency bugs, unfamiliar framework internals) discriminate strongly but are few — and they're where failures are expensive to verify.

So the harness should spend its budget where the signal is.

A concrete artifact: a tiered router in ~80 lines of Python

Here's a minimal, runnable sketch. It assumes your eval tasks are JSON files with a tier field (easy, medium, hard) and a verify command you can execute (a test suite, a diff check, whatever your harness already uses).

# router.py — tiered evaluation router (working sketch, adapt to your harness)
import json, subprocess, sys, time
from pathlib import Path

# Model pool: map a logical name to whatever client you use.
# Keep this boring and swappable — the point is the routing, not the SDK.
MODELS = {
    "cheap":   {"run": lambda prompt: call_model("cheap-model-id", prompt)},
    "mid":     {"run": lambda prompt: call_model("mid-model-id", prompt)},
    "strong":  {"run": lambda prompt: call_model("strong-model-id", prompt)},
}

# Routing policy: which pool each tier starts in, and escalation rules.
POLICY = {
    "easy":   {"start": "cheap", "escalate_to": "mid"},
    "medium": {"start": "mid",    "escalate_to": "strong"},
    "hard":   {"start": "strong", "escalate_to": None},
}

def call_model(model_id: str, prompt: str) -> str:
    # Placeholder — wire in your provider SDK or local server here.
    raise NotImplementedError

def verify(task: dict, patch: str) -> bool:
    """Apply the model's patch and run the task's verification command."""
    workdir = apply_patch(task, patch)          # your existing harness logic
    result = subprocess.run(task["verify"], cwd=workdir, shell=True)
    return result.returncode == 0

def evaluate_task(task: dict) -> dict:
    tier = POLICY[task["tier"]]
    model_key = tier["start"]
    while True:
        patch = MODELS[model_key]["run"](task["prompt"])
        passed = verify(task, patch)
        if passed or tier["escalate_to"] is None:
            return {"task": task["id"], "tier": task["tier"],
                    "solved_by": model_key, "passed": passed}
        model_key = tier["escalate_to"]         # retry on the next tier up

def main(suite_dir: str):
    results = [evaluate_task(json.loads(p.read_text()))
               for p in Path(suite_dir).glob("*.json")]
    for r in results:
        print(json.dumps(r))

if __name__ == "__main__":
    main(sys.argv[1])

The interesting properties:

Escalation, not blanket coverage. Easy tasks only ever touch expensive models if the cheap model fails them — which is itself a signal worth recording.
The solved_by field is the real output. Over time it answers: "for my tasks, does the mid-tier model actually earn its price over the cheap one?"
The pool is swappable. When a new checkpoint appears, you add one entry to MODELS and re-run. The suite, not the hype cycle, decides where it lands.

A note on where the cheap tier comes from

The cheap tier only works if it costs roughly nothing to run at eval volume. Disclosure: This article was prepared as part of MonkeyCode's product outreach. One practical option I've used for this tier is MonkeyCode, which offers free model access and a free server option — that's a reasonable fit for the high-volume, low-stakes end of the pool where you'd otherwise be rationing API calls. If you want to try this routing pattern, that's the tier I'd point it at first; check their docs for what's currently available rather than trusting any article (including this one) about specifics.

Detecting drift: the regression gate

Routing saves money, but there's a subtler failure mode: you swap a model into a tier and silently get worse. Guard against it with a frozen "golden subset" — 10–15 tasks with known-good outcomes — that runs on every pool change:

Check	What it catches	Threshold example
Golden-subset pass rate	Silent capability regression on your real tasks	Must not drop vs. current pool
Escalation rate per tier	A tier model quietly getting weaker (more retries upward)	Alert if +20% week over week
Verify-command runtime	A model producing bloated patches that slow your test loop	Flag outliers for review
Empty/trivial patch rate	Lazy outputs that pass weak verifiers	Any nonzero rate → inspect

The golden subset should include at least two tasks where a previously-hyped model famously failed for you. Those are your canaries.

A decision table for new-model claims

When the next "cheaper and better" checkpoint drops, run this before touching your pool:

Question	If yes	If no
Is there a primary source (provider release notes, official evals)?	Read it; ignore aggregators	Wait — you have nothing to test against
Does the claim cover your task types (not just chat benchmarks)?	Add it as a pool candidate	Note it, don't route to it yet
Does it pass your golden subset at its target tier?	Promote it into the pool	Keep it out; revisit next release
Does it lower escalation rates vs. the current tier holder?	Consider making it the default	Keep the incumbent

Limitations, and who shouldn't do this

Task tiering is judgment, not measurement. Your "medium" tasks might be another team's "hard." Expect to re-tier after the first few runs when the escalation data disagrees with your guesses.
Weak verifiers corrupt everything. If a task can be "passed" by a patch that doesn't really solve it, the router will happily route garbage. The harness is only as good as your verify command.
Free tiers change. Any free model access or free server option — from anyone — can change terms, capacity, or availability. Build the pool abstraction so losing one provider is a config edit, not a rewrite.
Skip this entirely if your eval suite is under ~15 tasks (just run everything; routing overhead exceeds savings), if you don't yet have per-task verification (build that first), or if you're evaluating a model for a safety-critical use where tiered sampling hides rare failures — those need exhaustive runs, not budget routing.

Closing thought

The model release cycle isn't slowing down, and "just try it on your stuff" doesn't scale when your stuff has a real test suite and real costs. Tier your tasks, escalate on failure, gate every pool change on a frozen subset, and let your own data — not the launch-day thread — decide which model earns which tier.

If you build a variant of this, I'd be curious what your escalation rates look like — that number turned out to be the most honest metric in my harness.