DEV Community

pyfile-toolkit
pyfile-toolkit

Posted on

The Honest State of Free LLM APIs (August 2026)

The Honest State of Free LLM APIs (August 2026)

I run a small autonomous agent (issue triage, ~250 lines, no budget) entirely on free LLM tiers. Two months in, here is the unfiltered picture - what I actually hit, and what I'd tell a friend who wants to build on free models today.

The headline

Free endpoints are fine, but "free" buys you availability problems, not quality problems. The models themselves (open-weight, served at 1-2 steps/s) are good enough for structured tasks. The infrastructure isn't: at peak hours, popular free endpoints return 503 Service Unavailable - they accept your request and then just don't answer.

In my worst run on facebook/react (22 items), a big fraction of calls died with 503. The endpoint the docs told me to use was effectively down half the time.

What actually works

  1. Rotation with instant failover. Keep an ordered list of models; any non-2xx moves to the next one. No retries-with-backoff, just failover: 
const MODELS = [
  'liquid/lfm-2.5-2.6b:free',
  'thinkingmachines/inkling-small:free',
  'nvidia/nemotron-3.5-lightning:free',
  'cohere/north-mini-code:free',
]
for (const model of MODELS) {
  try { return await call(model, prompt) } catch {}
}
Enter fullscreen mode Exit fullscreen mode

After adding this: 22/22 classified, zero errors.

  1. Treat the model as a parser, not a chat. Strict {"category","short","priority"} contracts, take the first {...} block, tolerate markdown noise. Small models return sloppy JSON; the contract wins anyway.

  2. Capacity-class models. LFM-2.5 (26B) and Nemotron 3.5 Lightning are the workhorses: fast, usually up, decent at constrained tasks. Inkling Small is the cheap classifier. These aren't GPT-5 - but for extraction, routing and classification, the gap is invisible.

What still sucks

  • No SLA, ever. The free tier is best-effort by design. Your agent will have a 503 moment; design for it or demo day will find it.
  • Rate limits. Unauthenticated GitHub gives you 60 req/h - fine for triage-sized scans, a PAT removes the ceiling. Design for the smaller number.
  • The long tail of "free" junk. Some trendy endpoints appear free, then start charging per-minute or throttle after 24h. Check the actual pricing page, not the marketing tweet.

The honest tradeoff

Free models are slower and sometimes dumber than paid ones. For a constrained classification task, irrelevant. For open-ended reasoning, this isn't the tool. Match the model class to the task.

Takeaway

Your first engineering job on the free tier isn't the prompt - it's failover. Rotate, parse defensively, expect 503s. Do that and the free tier genuinely holds up.

The full agent (loop + rotation + no-framework dashboard) is in github.com/pyfile-toolkit/agent-triage.

Top comments (0)