DEV Community

pyfile-toolkit
pyfile-toolkit

Posted on

Free LLM Tiers Are Lying to You — Model Rotation Saved My Agent

Free LLM Tiers Are Lying to You — Model Rotation Saved My Agent

I built a small autonomous agent (GitHub issue triage) that runs entirely on free LLM tiers — no API budget, no GPU, no AWS account. The project itself is ~250 lines and works. The interesting part is what the free tier did to me.

The 503 wall

The plan was simple: every open issue → LLM → JSON verdict. First real run on a big repo: almost every call failed. Not because the prompt was bad — because the model endpoint answered with 503 Service Unavailable. Free endpoints on popular models are chronically overloaded; they accept the request and then just… don't.

This isn't hypothetical. My scan of facebook/react (22 items) hit 503s repeatedly; the "free" model that the docs told me to use was effectively down half the time.

The fix: rotation with fallback

The agent now holds an ordered list of models and tries them one by one until one answers:

const MODELS = [
  'liquid/lfm-2.5-2.6b:free',          // fast, usually up
  'thinkingmachines/inkling-small:free', // backup
  'nvidia/nemotron-3.5-lightning:free',
  'cohere/north-mini-code:free',
]
for (const model of MODELS) {
  try { return await call(model, prompt) } catch {}
}
Enter fullscreen mode Exit fullscreen mode

Semantics matter: any non-2xx (or empty) response is treated as a failure and we move to the next model. No retries-with-delay loops — just instant failover. Results before/after:

  • Before: facebook/react scan → large fraction of llm 503 errors, dashboard full of red.
  • After: same repo → 22/22 classified, zero errors, priorities assigned, all in a few minutes.

What else free tiers taught me

  1. Treat the LLM as a parser, not a chat. My prompt demands strict {"category","short","priority"} and my parser takes the first {...} block, tolerating markdown noise. Small models return sloppy JSON — the contract still wins.
  2. Expect rate limits on everything. GitHub unauthenticated allows 60 requests/hour — fine for triage-sized scans; a PAT removes the ceiling. Design for the smaller number.
  3. Rotation is non-negotiable. If your agent depends on one free endpoint, it will be down at the worst moment (demo day). Two backups minimum.

The honest tradeoff

Free models are slower and sometimes dumber than paid ones. For {category, short, priority} — a constrained classification task — the difference is irrelevant. For open-ended reasoning, it isn't. Match the model class to the task.

Takeaway

If you're building agents on free tiers: your first engineering task isn't the prompt — it's failover. Rotate, parse defensively, and design for 503s. Then the free tier becomes genuinely free, and your demo stops breaking mid-scan.

I put the whole thing (agent loop + rotation + no-framework dashboard) in a public repo: github.com/pyfile-toolkit/agent-triage. Have fun.

Top comments (0)