DEV Community

orca_forge
orca_forge

Posted on Edited on Originally published at forge.workstyle.tech

LiteLLM Fallback: There Are Effective Failures and Ineffective Ones — A Real-World Setup Bundling 7 Free Tiers

#litellm #llm #api

📝 Originally published (in Japanese) at forge.workstyle.tech.

Introduction to LiteLLM and LLM API Pipeline

We have been operating a pipeline for generating articles using LiteLLM, which bundles the free tiers of 7 LLM API providers. The idea is that "even if one provider goes down, the pipeline won't stop because it can switch to another provider."

After about half a year of operation, we measured the success rate and found it to be 6 out of 10.

Upon investigating the cause, we discovered that there are failures that can be mitigated by fallbacks and those that cannot. This article documents our investigation and recovery process. All numerical values are based on actual measurements taken on 2026-09-07.

What We're Building

Our goal is simple: to hide multiple providers with free tiers behind a single model name.

The caller only needs to specify model: "free".

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-xxx" \
  -H "Content-Type: application/json" \
  -d '{"model":"free","messages":[{"role":"user","content":"\u6307\u63a8\u3057\u3066"}]}'
Enter fullscreen mode Exit fullscreen mode

Behind the scenes, LiteLLM selects one entry from multiple entries with the same model_name: free and makes the request.

model_list:
  - model_name: free
    litellm_params:
      model: groq/qwen/qwen3.8-27b
      api_key: os.environ/GROQ_API_KEY
  - model_name: free
    litellm_params:
      model: gemini/gemini-flash-latest
      api_key: os.environ/GEMINI_API_KEY
  - model_name: free
    litellm_params:
      model: cloudflare/@cf/meta/llama-3.3-70b-instruct-fp8-fast
      api_key: os.environ/CLOUDFLARE_API_KEY
  # …and so on
Enter fullscreen mode Exit fullscreen mode

The calling code does not know about the providers. This is one of the best features of LiteLLM: provider replacement can be done just by modifying the configuration file. In fact, the recovery work mentioned at the end of this article did not involve modifying the application code at all.

How router_settings Works

Simply bundling providers is not enough; the behavior is determined by router_settings. Here is the actual configuration we are using:

router_settings:
  routing_strategy: simple-shuffle   # Select from available ones using shuffle
  num_retries: 5                     # Retry up to 5 times if it fails
  allowed_fails: 1                   # If it fails once, cool down the provider
  cooldown_time: 60                  # Cool down for 60 seconds, then automatically recover

litellm_settings:
  drop_params: true                  # Automatically drop parameters not supported by the provider
Enter fullscreen mode Exit fullscreen mode

We will explain each item in the order of their effectiveness in actual operation:

  • num_retries is essentially the core of fallbacks. If the first attempt fails, LiteLLM selects another entry with the same model_name and retries. With a setting of 5, it can try up to 5 providers.
  • allowed_fails and cooldown_time are mechanisms to temporarily exclude a provider that is on the verge of dying. If it fails once, it is excluded from candidates for 60 seconds, and after 60 seconds, it silently recovers. This works well with rate limits (429).
  • drop_params is a subtle but necessary feature. Free models have varying supported parameters, and if you throw a parameter that a model does not support (like seed or response_format), it will fail with a 400. This feature automatically drops such parameters, so the caller does not need to worry about the least common denominator.
  • routing_strategy: simple-shuffle means "randomly select from living candidates." Since free tiers have thin rate limits, shuffling is suitable for dispersing the load. There is also latency-based-routing for selecting based on latency, but we are not using it because, with free tiers, faster providers will be overloaded and exhausted first.

Failure: 6-Provider Configuration Was Essentially Operating as 1 Provider

Now, let's get to the main point. With this configuration, when we made the same request 10 times, the result was:

Success: 6 / Failure: 4
Enter fullscreen mode Exit fullscreen mode

The error when it failed was:

litellm.APIError: APIError: CerebrasException -
  Payment required to access this resource. Visit your billing tab.
  Received Model Group=free
  Available Model Group Fallbacks=None
Enter fullscreen mode Exit fullscreen mode

The last line caught our attention:

Available Model Group Fallbacks=None
Enter fullscreen mode Exit fullscreen mode

It says there are no fallbacks, despite having 6 providers bundled.

We checked the life status of each provider's API individually and found the reason:

Provider Result
Groq × 401 Invalid API Key
Cerebras × 402 Payment required
OpenRouter × 401 API key expired
Gemini × 404 model no longer available
Mistral × 429 Rate limit exceeded
Cohere ○ 200

Only 1 provider was alive.

The reason it succeeded 6 out of 10 times was that num_retries: 5 retried and eventually landed on Cohere. However, even when it succeeded, it was after several failures, making it slower.

Cause: There Are Failures That Fallbacks Cannot Handle

This was the most significant lesson learned:

Fallbacks and retries assume failures that will recover if waited for.

  • 429 (Rate Limit): Waiting 60 seconds will restore the quota.
  • Timeout / 5xx (Temporary Failure): Retrying will make it go through.

These can be handled by cooling down the provider with cooldown_time and then automatically recovering. However, there are failures that will not recover even if waited for:

  • 402 (Payment Required): The free tier has ended. Waiting won't restore it.
  • 404 (Model Not Found): The model has been removed or renamed. Waiting won't bring it back.
  • 401 (Invalid Key): The key has expired. Waiting won't make it valid again.

These failures are not targets for retries. The moment they are encountered, the request fails. This means that if a provider is permanently dead, leaving it in the pool will continue to generate failures at a probability proportional to its presence in the pool.

Recovery: What Worked

We recorded how the success rate changed as we fixed issues:

① Updated Model Names

Groq's error changed from 401 to 404 after reissuing the key.

404 The model `llama-3.3-70b-versatile` does not exist or you do not have access to it.
Enter fullscreen mode Exit fullscreen mode

The free tier was still available, but the model had been removed. We replaced it with qwen/qwen3.8-27b.

Gemini also had a similar issue:

404 NOT_FOUND: This model models/gemini-2.0-flash is no longer available.
    Please update your code to use models/gemini-3.6-flash
Enter fullscreen mode Exit fullscreen mode

We updated it to gemini-flash-latest, an alias that automatically follows the latest generation, so we won't repeat the same mistake.

At this point, measuring again showed 12 out of 12, but this was still not a significant improvement because the main issue was not the model names but the dead providers.

② Removed Dead Providers from the Pool

Cerebras had ended its free tier and was returning a 402 Payment Required error. The key was valid, but it was demanding payment.

We commented it out and removed it from the pool.

  # 2026-09-07: Cerebras has ended its free tier and returns 402 Payment required, so it's disabled.
  # 402/404 are not targets for LiteLLM's retries (= will fail immediately without fallback),
  # so leaving a dead provider in the pool will increase the overall failure rate.
  # - model_name: free
  #   litellm_params:
  #     model: cerebras/gemma-4-31b
  #     api_key: os.environ/CEREBRAS_API_KEY
Enter fullscreen mode Exit fullscreen mode

The result was:

Cerebras removed   15/15 (100%)
Enter fullscreen mode Exit fullscreen mode

Removing the dead provider was more effective than updating model names. This was the most significant takeaway from our experience.

③ Added More Providers

After that, we added Cloudflare Workers AI, HuggingFace, and NVIDIA NIM, eventually forming a 7-provider configuration.

Success rate progression: 6/10 (60%) → 15/15 → 20/20 → 24/24 → 28/28
Enter fullscreen mode Exit fullscreen mode

Another Pitfall: Mixing Inference Models Can Silently Break the System

When adding providers, there's another point to watch out for:

Mixing inference models (which output reasoning processes) can cause the system to fail silently with empty responses.

This is the result of throwing the same question ("What is 2+2? Answer with just the number.") at different models:

Model content
qwen/qwen3.8-27b "4"
qwen/qwen3.6-27b "" ← Empty
openai/gpt-oss-20b "" ← Empty
nvidia/nemotron-3-super-120b-a12b "User asks 2+2..." ← Reasoning is leaking

The answer is in reasoning_content, and content is empty. Since the HTTP response is 200, retries and fallbacks do not kick in. Code that only reads choices[0].message.content will silently receive an empty string.

For multi-turn agent use cases, this breaks more visibly. Because conversation history is accumulated and sent back, the next turn will fail.

400 property 'messages.*.assistant.reasoning_content' is unsupported
Enter fullscreen mode Exit fullscreen mode

The problematic part is that this cannot be determined by the model name series. As shown in the table, even within the same Qwen series, 3.6 outputs reasoning, while 3.8 does not.

So, before adding a model to the pool, we test it once to confirm that reasoning_content is not attached. We also note this in the configuration file.

# ※This pool is for 【single-turn generation only】. Since it includes inference models,
#   do not use it for multi-turn coding agents.
Enter fullscreen mode Exit fullscreen mode

Current Configuration and Measured Latency

Our final configuration looks like this. All of these are credit card-free.

Provider Model Measured Latency
Groq qwen/qwen3.8-27b 220 ms
Cloudflare Workers AI @cf/meta/llama-3.3-70b-instruct-fp8-fast 570 ms
Cohere command-a-03-2025 748 ms
HuggingFace meta-llama/Llama-3.3-70B-Instruct 937 ms
OpenRouter google/gemma-4-26b-a4b-it:free 957 ms
NVIDIA NIM google/diffusiongemma-26b-a4b-it 1,070 ms
Gemini gemini-flash-latest —
Mistral mistral-small-latest 429 frequent

When adding providers, we encountered two issues:

  • Cloudflare Workers AI requires a token with the Account > Workers AI > Read permission. The similarly named AI Gateway permission does not work. 
A token that holds only an AI Gateway permission returns 401 with error code 10000.
Enter fullscreen mode Exit fullscreen mode
  • HuggingFace requires a token with the inference.serverless.write permission (displayed as "Make calls to Inference Providers" in the UI). A token with only repository read permission will result in a 403. Issuing a key and the key being usable are separate issues.

Conclusion: Monitor Provider Health Before Bundling

For those planning to set up a similar configuration, we offer three practical conclusions:

  1. Removing dead providers from the pool is more effective than adding more providers.

    Our experience showed that fixing model names (6/12) was less effective than removing a single dead provider (15/15) in improving the success rate. We should have created a monitoring system before bundling providers.

  2. Distinguish between failures that can be mitigated by fallbacks and those that cannot.

    Failures that can be mitigated include 429 (Rate Limit) and timeouts. Failures that cannot be mitigated include 402 (Payment Required), 404 (Model Not Found), and 401 (Invalid Key). These will continue to cause failures until a human notices and removes them from the configuration.

  3. Do not hard-code model names; test them before adding to the pool.

    Use latest aliases if available. Otherwise, before adding a model to the pool, confirm that it responds and does not output reasoning_content. Even within the same model series, different versions may behave differently.

Note: The numbers in this article are based on measurements taken on 2026-09-07. The conditions, models, and rate limits of free tiers change over a few months. When implementing, please check the latest information on each provider's official documentation.

Top comments (0)