DEV Community

Cover image for I Route Every AI Agent to the Cheapest Model with Quota. Here's What Broke
Gala
Gala

Posted on Edited on

I Route Every AI Agent to the Cheapest Model with Quota. Here's What Broke

I wanted coding with AI agents to be genuinely cheap, so I built a tiny Python router that sends every task to the cheapest model that still has quota left.

Within three days, a cheap model wrote a test that looped forever and ate 9 GB of RAM, another stopped halfway with "Free usage limit reached", and a third dumped 419 lines of duplicated helper functions into a script because it assumed "the grader supplies tests."

Everyone is asking whether coding with AI is actually faster and cheaper. My honest answer? It's fast and nearly free, right up until a cheap model writes something that takes two hours to unwind.

So here's the router, the rules that came out of those crashes, and why it's forbidden from upgrading itself.

Agents have roles; tiers have price tags

The first mistake I made was tying each agent prompt to a model.

Now the role ("writer", "designer", "coder") is just a prompt with rules and skills, and a role never names a model. It only says what size of work it usually does.

Why does that matter? Because the moment a model is hardcoded, you can't route around outages, renamed models or an exhausted rate limit. The tier list does that for you:

{
  "small":  ["free-model-a", "free-model-b", "cheap-model-c"],
  "medium": ["mid-model-a", "mid-model-b"],
  "large":  ["big-model-a", "big-model-b"]
}
Enter fullscreen mode Exit fullscreen mode

Cheapest first, more capable after. Swapping a model is a one-line change, and no prompt has to know.

The 20-line candidate filter

When a task comes in, the router walks the requested tier and filters it:

def select_candidates(tier, tiers, allow_paid=False, forced=None):
    if forced:
        return [forced], []
    to_try, skipped = [], []
    for cand in tiers[tier]:
        if cand.get("paid") and not allow_paid:
            skipped.append((cand, "paid model not allowed"))
            continue
        if (q := quota_status(cand)):
            skipped.append((cand, f"quota: {q}"))
            continue
        to_try.append(cand)
    return to_try, skipped
Enter fullscreen mode Exit fullscreen mode

quota_status() reads the CLI's local usage numbers instead of hitting a rate-limit API. If any window (rolling hours or weekly) is above 90%, that model is out.

Why 90 and not 100? Start a big task at 96% and it tends to run dry mid-stream, leaving half-written files behind. The buffer is cheaper than the cleanup.

And yes, you can force a model by hand. The router just warns you and gets out of the way.

The 60-second escape hatch

What about a model that gets picked and then falls over before doing anything?

AI CLIs fail at startup in creative ways: renamed models, instant 429s, free tiers that cut you off at second zero. So the router watches the first minute:

STARTUP_ERRORS = ("429", "model_not_found", "Free usage limit", "quota")

def is_startup_failure(code, seconds, log):
    if seconds > 60:
        return False
    if any(e.lower() in log.lower() for e in STARTUP_ERRORS):
        return True
    return code != 0
Enter fullscreen mode Exit fullscreen mode

Fail fast, and it rolls to the next candidate in the same tier. Every run also leaves its prompt, raw log, final answer and the choice it made, so I can see why a model was skipped without guessing.

What broke when I trusted the cheap end

Did the router make everything smooth? Not even close. A few of my favourite disasters:

  1. The phantom model. A model showed up in the CLI's own model list and failed instantly with model_not_found. Lesson: only route to models you've actually run.
  2. The quota wall. A free model stopped halfway with "Free usage limit reached", and another refused to run because of where it was hosted.
  3. 419 lines of spaghetti. A free small-tier model was asked for a tiny flag in a script. It added ~20 duplicated helper functions, changed existing signatures and wrote zero tests. Full revert.
  4. The 9 GB loop. A cheap model wrote a pagination test whose mock always returned a full page. The loop never ended and memory climbed to 9 GB before the process got killed. Now every generated test runs under a memory cap and a timeout.
  5. The quota burn. One long high-effort run ate about 30% of a weekly allowance in a single go.

(If you like this kind of thing, my previous post on running a side project with AI agents has more.)

Why the router never escalates

When every small model fails, why not bump the task to a bigger one automatically?

Because the router never decides that a task is bigger. Only I do.

When a small task fails across the board, it's rarely the model being dumb. It's usually:

  • a vague prompt,
  • a task touching too many files,
  • or a broken test setup.

Auto-escalating just pays premium prices to retry a broken prompt. So when a tier runs out, the router exits loudly and lists why it skipped each model. Then I either fix the prompt or bump the size by hand.

The machine executes; the human owns the scope.

So did it actually save anything?

I don't have a dollar figure, since most of this runs on subscriptions and free tiers rather than metered APIs.

What it saved was headroom and attention:

  • Drafts, copy and small single-file edits run on free or cheap models.
  • No more surprise rate-limit lockouts in the middle of the day.
  • Startup failures fall back to another model before I even notice.

The one rule I'd keep if I had to delete everything else:

Cheapest-with-quota is the right default for writing and small scoped edits. For anything with tests, set a higher floor.

Saving ten minutes of quota is worthless if you spend an hour reverting a cheap model's "tests".

Over to you

I use this router to keep a few small data sites running on my own, like a cheapest fuel price map for Spain and a public tenders finder.

The agents are the easy part. Getting people to find and use the sites is the hard one. So, for anyone who's grown a side project:

  • What brought you your first steady search traffic on a site with thousands of auto-generated pages?
  • Ads, paid alerts, API or datasets: what actually paid for a niche data site?
  • Where would you share a site like this besides dev.to?
  • What would you stop doing if you were me?

I read and reply to every comment

Top comments (2)

Collapse
 
ayushdhanani369 profile image
Ayush Dhanani •

Your never escalates rule is the one I would keep. When a tier runs out, the router stops and the human decides, instead of paying premium prices to retry a broken prompt. I build Aladdin, a job agent. It works the same way. A fresh worker reads the actual page and form for each application, and if the form cannot be read or an answer is unknown, it asks the user instead of guessing. The phantom model is the failure I recognize most. A model that only exists in a list is not a model you can route to, the same way a claim from the first agent is not a fact until a fresh read confirms it.

Collapse
 
galadev profile image
Gala •

"A claim from the first agent is not a fact until a fresh read confirms it", oh I'm stealing that line 😄 Asking the user instead of guessing is such a good default for job forms too, one wrong answer there costs way more than a question. Quick one since you're building in public: how did Aladdin get its first real users? I'm at the "agents work, nobody knows the sites exist" stage 👀