Let the Test Suite Decide Which Model Answers: A Verification-Gated Model Ladder

There's a line item in my AI spend that bothered me once I actually looked at it: a large share of the prompts I fire at my strongest (and priciest) model are things like "rename this field across these files" or "write a parser for this log format — here are the tests it must pass." Those tasks have something in common that has nothing to do with difficulty: a machine can already tell me whether the answer worked.

That observation turned into the workflow in this post. Instead of picking a model per prompt by gut feel, I let my existing tooling — the test runner, the linter, the type checker — act as a gatekeeper. Weak-but-free model attempts the task first; the verifier grades it; only a failed grade buys a ticket to the expensive model. I think of it as a model ladder with the rungs ordered by price, and a turnstile between each rung.

This builds on the eval harness and disposable sandbox setups I've written about before, but it stands alone. Nothing here is tied to a specific vendor: I'll call the rungs tier_a (free/cheap) and tier_b (strong), and you should check current model names, pricing, and limits in each provider's own documentation before wiring anything up — that landscape shifts monthly.

Why "verifiable" beats "easy" as the routing criterion

Early on I tried routing by perceived difficulty: trivial prompts down, hard prompts up. It didn't work, because my difficulty guesses were consistently wrong in both directions — and more importantly, difficulty isn't what makes a cheap model safe to use.

What makes it safe is feedback. Consider two tasks of similar effort:

• Extract all hard-coded UI strings into an i18n catalog, where a script then checks that every string in the source now has a catalog entry. Cheap model fumbles one file? The checker fails, you escalate. Cost of a wrong attempt: one wasted API call.
• Look at a sporadic production deadlock and hypothesize the cause. A weak model produces a confident, plausible, wrong theory — and nothing in your toolchain flags it. You only find out after you've burned an afternoon chasing it.

Same ballpark of difficulty, completely different risk profile. So the ladder's admission rule is: a task may enter at the cheap rung only if an automated, objective check exists for its output. Subjective work — design judgment, root-causing weird bugs, security-sensitive code — skips the ladder entirely and goes straight to the strong model, because that's where a wrong answer costs far more than the tokens do.

A concrete example: the i18n extraction task

To make this less abstract, here's the exact task shape I've been running on the cheap rung:

1. Prompt: "In src/components/, replace every user-facing string literal with a t('key') call and append the key/value pairs to locales/en.json."
2. The model returns a patch, applied inside a throwaway container (same disposable-sandbox pattern as my earlier post — never let an unverified patch near a real worktree).
3. Verifier runs three checks: tsc --noEmit, eslint src/, and a small script that greps for remaining untranslated literals.
4. All green → accept, done, total cost ≈ zero. Any red → escalate to tier_b, with the checker's error output pasted into the escalation prompt.

That last detail — forwarding the failure evidence — turned out to matter more than the routing itself. In my sandbox runs, handing the strong model the lint errors and the failed patch noticeably reduced the cases where it reproduced the same mistake, compared to just re-asking the original question.

Where the free rung comes from

The bottom rung needs an endpoint that costs nothing and speaks a familiar request/response shape. Recently I've been pointing mine at MonkeyCode's free model access via its free server option, simply because the router only cares that the endpoint exists and accepts a standard-style call — the ladder logic is indifferent to who's hosting it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

To be explicit about what I'm not saying: I haven't benchmarked their current lineup, I'm not quoting quotas or model names, and free-availability terms can change at any time. Architecturally, the free tier is a socket, not a foundation. If it vanished next week, I'd plug in whatever cheap endpoint replaced it and the workflow would be unchanged.

The artifact: a policy file plus a tiny ladder runner

The earlier version of this idea lived in one Python file. I've since split it into two pieces, and the split is the point: the routing policy is data you tune, the runner is code that stays stable.

ladder.yaml — the tunable part:

rungs:
  - name: tier_a
    model_env: CHEAP_MODEL_ENDPOINT   # read from env, never hardcode
  - name: tier_b
    model_env: STRONG_MODEL_ENDPOINT

tasks:
  i18n_extract:
    ladder: true
    verify:
      - "tsc --noEmit"
      - "eslint src/"
      - "python scripts/check_untranslated.py"
  sql_migration:
    ladder: true
    verify:
      - "sqlfluff lint migrations/"
      - "pytest tests/test_migration_roundtrip.py -q"
  incident_hypothesis:
    ladder: false        # judgment work: straight to the top rung
  auth_review:
    ladder: false        # 'compiles and passes tests' is not 'safe'

ladder.py — the stable part (a structural sketch; fill in your client and patch-apply logic before running):

import os, subprocess, sys, yaml

def call_model(endpoint: str, prompt: str) -> str:
    ...  # any client; both rungs expose the same shape

def checks_pass(checks: list[str], workdir: str) -> tuple[bool, str]:
    for cmd in checks:
        r = subprocess.run(cmd, shell=True, cwd=workdir,
                           capture_output=True, text=True, timeout=180)
        if r.returncode != 0:
            return False, f"$ {cmd}\n{r.stdout}\n{r.stderr}"
    return True, ""

def run(task_name: str, prompt: str, workdir: str, policy: dict) -> None:
    spec = policy["tasks"][task_name]
    rungs = policy["rungs"]

    start = 0 if spec["ladder"] and spec.get("verify") else len(rungs) - 1

    for rung in rungs[start:]:
        endpoint = os.environ[rung["model_env"]]
        patch = call_model(endpoint, prompt)
        apply_patch(workdir, patch)          # throwaway container only

        if start == len(rungs) - 1 or not spec.get("verify"):
            print(f"[{rung['name']}] top rung: result needs human review")
            return

        ok, evidence = checks_pass(spec["verify"], workdir)
        if ok:
            print(f"[{rung['name']}] accepted — all checks green")
            return
        # Evidence rides along to the next rung.
        prompt = (f"{prompt}\n\nAn earlier attempt failed these checks. "
                  f"Fix the root cause, don't just silence the checker:\n{evidence}")
        revert_patch(workdir)

def apply_patch(workdir: str, patch: str) -> None: ...
def revert_patch(workdir: str) -> None: ...

if __name__ == "__main__":
    policy = yaml.safe_load(open("ladder.yaml"))
    run(sys.argv[1], open(sys.argv[2]).read(), sys.argv[3], policy)

Three behaviors worth stealing even if you ignore the rest:

• No verifier, no ladder. A task without a verify block starts at the top rung. The guardrail is structural, not a convention you can forget.
• Failure output is an input. Each escalation re-prompts with the checker's actual errors attached.
• Endpoints come from the environment. Swapping the free rung later is a one-line config change, not a code change.

Tuning the policy with real numbers

The ladder: true/false flags in the YAML are initial guesses. After two or three weeks, measure per task type:

• Escalation rate — how often the cheap rung's output fails verification. Anything above roughly half is a signal to flip that task to ladder: false; you're paying the round-trip latency of the cheap attempt and buying the strong call anyway.
• Silent-pass rate — the scarier one: cheap output that passes the checks but is still wrong, caught later by human review. If this is nonzero for a task type, your verifier is too thin for that task, and the fix is a better check, not a better model.

If you already run an eval harness for model comparisons, you can grade cheap-rung outputs offline on archived tasks before trusting the route live — same harness, new question.

Honest limitations

• The ladder is exactly as trustworthy as your checks. Sparse test suites turn the cheap rung into a wrong-answer generator with a green checkmark. Projects with weak verification should invest there first — which, conveniently, pays off even if you never build any of this.
• You're trading latency for money. A failed cheap attempt costs a full round trip before the strong model even starts. For synchronous pairing sessions where you're staring at a spinner, that trade is often bad. This pattern fits batch-style work — background agents, queued tasks, CI-adjacent automation — much better than interactive ones.
• Free capacity is not a plan. Free model endpoints change names, limits, and availability without asking your permission. Keep the bottom rung swappable, and make sure the workflow still makes sense if it ever becomes merely cheap instead of free.
• If your work is mostly judgment, skip this. Design reviews, exploratory debugging, architecture debates — none of these have a verifier, so none of them enter the ladder. One capable model and well-written prompts will outperform any routing scheme for that mix.

Closing thought

The useful mental shift here isn't "use cheaper models" — it's "let your tooling arbitrate model quality." Your test suite already knows how to grade a patch; the ladder just routes spending based on its grades. If you want to probe whether your own workload even has a verifiable layer worth routing, the cheapest possible experiment is a free endpoint on the bottom rung — MonkeyCode's free server is the one I've been using — a handful of your most mechanical tasks, and a week of escalation logs. Whatever you learn, the policy file and the routing discipline are the parts you keep.