Two weeks after I let a coding agent run unattended on backlog chores, the provider invoice told me something my logs hadn't: the agent had spent the majority of its budget on a frontier model doing work like regenerating docstrings, bumping pinned dependency versions, and normalizing import order. Meanwhile, the one genuinely subtle bug it touched — a time-ordering issue in a queue consumer — got the same model as the docstrings, with the same shallow retry behavior, and shipped a patch I reverted four days later.
That inversion is the actual problem. Not "models are expensive," but: an unattended agent makes a spend decision on every single call, and the default decision is always the same.
My previous post argued that agent permissions belong in version control. This is the same instinct applied to model selection: routing is policy, policy should be a file, and files should be reviewable in git. What follows is the setup I run now, with placeholders where model identifiers go — catalogs change too fast for any name I write today to survive your reading of it.
The policy file
Instead of burying tiers in code, the routing rules live in a JSON document that's diffable, reviewable, and rollbackable:
{
"tiers": [
{
"name": "gratis",
"models": ["<free-tier-model-id>"],
"accept_when": {
"any_of": ["single-file edit", "no logic change", "docs or comments", "version bump"]
}
},
{
"name": "standard",
"models": ["<mid-tier-model-id>"],
"accept_when": {
"any_of": ["multi-file edit", "new test", "isolated function change"]
}
},
{
"name": "heavy",
"models": ["<top-tier-model-id>"],
"accept_when": {
"any_of": ["concurrency", "data migration", "public API change", "touches auth or crypto"]
}
}
],
"escalation": {
"attempts_per_tier": 1,
"on_exhaustion": "open_issue_for_human"
},
"gate": {
"command": ["make", "verify"],
"timeout_seconds": 900
}
}
Three design choices do the safety work, and none of them are about which models you pick:
-
The gate is
make verify, full stop. A patch advances only if the repo's real test suite, linter, and typechecker pass. No model ever evaluates another model's output — that path lets the cheap tier rubber-stamp its own mistakes. - Escalation has a ceiling. One shot per tier, then the task becomes a GitHub issue with the failure transcript attached. A retry loop without a bound is just a slower way to reach the expensive model anyway.
- The default direction is down, not up. If the classifier is unsure, the task starts lower and fails upward through the gate, rather than starting high and never discovering the task was easy.
The runner that consumes this is uninteresting plumbing — read policy, classify task shape, call provider, run gate, escalate — maybe 120 lines. The policy file is the artifact worth keeping, because it's the thing you review, diff, and blame.
Validate with shadow mode, not vibes
The risky moment isn't writing the policy; it's trusting it. So before any task actually routes through the cheap tier, I run the classifier in shadow mode: it labels every incoming task, logs what it would have done, and the agent keeps using the strong model as usual. After a week of real traffic, I replay the log and ask:
| Question from the shadow log | What "healthy" looks like |
|---|---|
| How often did the label say "gratis" for a task that later needed rework? | Near zero — cheap-tier mislabels are the expensive kind |
| What fraction of traffic was labeled "gratis" at all? | If it's under ~40%, your tier boundaries are probably too conservative to save anything |
| Did anything labeled "gratis" touch a file on my never-cheap list? | Must be zero; enforce with a path blocklist, not keyword hope |
| Projected spend vs. actual spend that week | Meaningful reduction, or the whole exercise is theater |
Only after the shadow numbers look sane do I let the low tier act on its own labels — and only for task shapes with a clean shadow record. The concurrency-and-auth category never graduates out of the top tier, no matter how good the log looks.
One more discipline: re-run this validation whenever you swap a model in or out of a tier. The same model ID can behave differently after a provider-side update, and last month's reliable mid-tier citizen can quietly become this month's source of reverted patches.
Why the bottom rung being free changes the arithmetic
If the cheapest tier costs actual money per call, there's a real argument that routing overhead isn't worth it for small workloads. If it costs nothing, the economics flip: the expected cost of the whole system converges on the small set of tasks that genuinely need a strong model.
I run this through MonkeyCode, which offers free model access and a free server option — so both the routing process itself and the bottom tier execute without a meter attached. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Two caveats I'd hold regardless of provider: I haven't verified which models sit on the free tier as you read this — pull the live model listing rather than trusting any post, this one included — and any free tier can change limits or vanish. The design above already tolerates that: a dead free model errors out of its tier and the task escalates, which is exactly what the bounded-escalation rule is for.
Where this breaks down
-
Weak test suite, no deal. The entire scheme rests on the mechanical gate. If
make verifycan't catch a broken patch, a cheap tier will happily pass garbage. Strengthen the suite first — that pays off even if you never route anything. - The classifier is the fragile part. Task-shape heuristics are crude by nature. Shadow mode exists precisely because I don't trust mine; if you have enough labeled history, similarity search against past tasks beats keyword matching, but that's a later optimization, not a prerequisite.
- Interactive latency. Walking tiers adds round-trips. A developer staring at an autocomplete shouldn't wait through an escalation chain — keep interactive paths single-model.
- Safety-critical and deeply specialized code. If your domain has no "trivial" tasks — kernel work, medical, formal methods — delete the bottom two rows of the policy file and start everything at the top.
If you've shadow-tested a routing policy on a real repo, I'd genuinely like to hear what your mislabel rate looked like — especially which task shapes fooled the classifier. Mine keeps getting tricked by version bumps that turn out to be breaking changes, and I haven't found a clean signal for those yet.
Top comments (2)
The bit I like is making model choice reviewable instead of treating it as runtime plumbing. I’d also log the rejected tier and the verification command next to the accepted one, because the expensive mistake is usually a cheap task silently escalating after one vague failure.
The interesting part is that model routing itself becomes project knowledge. A “simple version bump” can be trivial in one repo and risky in another depending on constraints, dependencies, and history. So the classifier probably needs more than task shape — it needs enough project context to know what “simple” actually means.