In September I shared a few small Node packages. One of them was llm-free-cascade. The idea is simple: send a chat request to Groq. If Groq says no, try Gemini. Then Cerebras. And so on, through every free tier you have a key for.
The idea was simple. Getting it right was not. Here is what I learned.
1. Don't keep retrying the same key
My first version always started with key #1. Once key #1 ran out for the day, every request still tried it first, failed, and only then moved on. More keys made it slower, not faster.
Fix: take turns between keys. When a key is rate-limited, let it rest and skip it until it's ready again.
2. Limits are usually per model
A 429 rarely means "this key is done." It usually means "this model is done for this key." So try another model on the same key before giving up on the provider.
new LLMCascade({
keys: { groq: process.env.GROQ_API_KEY },
models: { groq: ['openai/gpt-oss-120b', 'openai/gpt-oss-20b'] }, // 2nd is tried if the 1st is limited
});
3. Know how long to wait
A fixed cooldown is wrong. It's too long for a per-minute limit and too short for a daily one.
Providers tell you when to come back, each in their own way:
- a
Retry-Afterheader - a message like "try again in 2m59s"
- words like "per minute" or "per day"
- a known daily reset time (Gemini resets at midnight Pacific)
The cascade reads all of these. A per-minute limit waits about a minute. A daily limit waits until that provider resets.
4. Not every error means the provider is broken
| Error | What the cascade does |
|---|---|
| Prompt too long | Skips that provider for this one request only |
| 429 / quota | Rests that key and model until the reset |
| 401 / 403 | Stops using that one key |
| 404 (model removed) | Stops using that model |
| Timeouts, 5xx | After 2 in a row, skips the provider for a minute |
That last one matters. Without it, a provider that is down makes every request wait for the full timeout.
5. One trick I chose not to use
You could send the same request to two providers at once and keep the faster answer. On free tiers that spends twice the quota for one answer, so I left it out.
6. Free tiers change all the time
This year alone, Cerebras' free tier became a 30-day trial, SambaNova started asking for a card, and GitHub Models shut down. Model names change even more often.
So the provider list lives in one data file, providers.json. Each entry says when it was last checked and where the numbers came from. It also warns you about things like "non-commercial use only" or "needs a phone number". A small script checks every model name against each provider's live list.
One note: many providers don't allow several free accounts to get around their limits. Check each provider's terms.
7. Use it from any language
Someone opened an issue asking to use it like the OpenAI API, from their own tools. So now you can run it as a local server:
npx llm-free-cascade serve # http://127.0.0.1:8787/v1
Then point any OpenAI client at it, for example from Python:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="unused")
reply = client.chat.completions.create(
model="cascade", # or "groq" to use only Groq
messages=[{"role": "user", "content": "Explain photosynthesis in one sentence."}],
)
print(reply.choices[0].message.content)
It works with streaming too. The server only listens on your own machine by default, since it holds all your keys.
8. A local backup with Ollama
If every free tier is busy at the same time, Ollama can answer from your own computer. No limits, just slower.
new LLMCascade({
custom: {
ollama: {
baseUrl: 'http://localhost:11434/v1',
model: 'llama3.2',
apiKey: 'ollama', // any text works, but it can't be empty
tier: 5, // try it last
},
},
});
9. You can see what it's doing
npx llm-free-cascade keys # which providers you're missing, and where to sign up
npx llm-free-cascade status # your providers, keys and limits
npx llm-free-cascade probe # quick live test of every key
In code, cascade.stats() shows calls, failures, speed and cooldowns for each provider and key.
llm-free-cascade is free (MIT), has zero dependencies, and runs on Node 18+.
Found a model name that no longer works? Fixing it is a one-file PR. Want tool calling in the server? Open an issue and tell me what you're using it with.
Top comments (0)