DEV Community

M.T.Ramkrushna
M.T.Ramkrushna

Posted on

LiteLLM Router — Stop Hardcoding One Provider Into Your App

#ai #llm #python #devops

Your support bot is fine until OpenAI returns 429s for twenty minutes.

If the app called openai.OpenAI() with a single model string, those twenty minutes are an outage. Not because the product broke — because you married one vendor in the client code.

LiteLLM gives you one OpenAI-compatible interface over 100+ providers, plus a Router that retries inside a model group and falls back to another group when the primary keeps failing. Your feature code keeps talking "chat completions." Reliability lives in the router config.

What you're solving

Most teams start with:

from openai import OpenAI
client = OpenAI()
client.chat.completions.create(model="gpt-4o-mini", messages=...)
Enter fullscreen mode Exit fullscreen mode

That is fine for a prototype. In production you eventually want:

  • A second provider when the first is rate-limited or down
  • The same message shape no matter which model answers
  • Retries that do not require rewriting every call site
  • A place to log cost and failures without sprinkling try/except everywhere

LiteLLM's answer is: keep the OpenAI client shape, move provider choice into a model_list + Router.

Install and make one call 

pip install litellm
# or: uv add litellm
Enter fullscreen mode Exit fullscreen mode

SDK style (no proxy yet):

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "..."
os.environ["ANTHROPIC_API_KEY"] = "..."

# Same completion() shape for every provider
r = completion(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize this ticket in one line."}],
)
print(r.choices[0].message.content)

r2 = completion(
    model="anthropic/claude-sonnet-4-5",
    messages=[{"role": "user", "content": "Summarize this ticket in one line."}],
)
Enter fullscreen mode Exit fullscreen mode

Every response is OpenAI-shaped. Exceptions map to OpenAI-style errors (RateLimitError, AuthenticationError, …), so existing handlers keep working.

Docs: Getting Started.

The workflow that matters: Router + fallbacks

This is the part you actually ship.

1. Define deployments under shared aliases

import os
from litellm import Router

model_list = [
    {
        "model_name": "support-summarizer",  # app-facing alias
        "litellm_params": {
            "model": "openai/gpt-4o-mini",
            "api_key": os.environ["OPENAI_API_KEY"],
        },
    },
    {
        "model_name": "support-summarizer-backup",
        "litellm_params": {
            "model": "anthropic/claude-haiku-4-5",
            "api_key": os.environ["ANTHROPIC_API_KEY"],
        },
    },
]
Enter fullscreen mode Exit fullscreen mode

2. Wire fallbacks and retries

router = Router(
    model_list=model_list,
    num_retries=2,
    fallbacks=[{"support-summarizer": ["support-summarizer-backup"]}],
)
Enter fullscreen mode Exit fullscreen mode

Retries stay inside the same model_name group. Fallbacks leave that group for the next one in the list. That split is deliberate: don't burn your backup provider on a flaky single request if another deployment of the primary can still answer. See Fallbacks and Router load balancing.

3. Call it like OpenAI

response = router.completion(
    model="support-summarizer",
    messages=[
        {"role": "system", "content": "One-line summary. No preamble."},
        {"role": "user", "content": ticket_body},
    ],
)
summary = response.choices[0].message.content
Enter fullscreen mode Exit fullscreen mode

Your ticket pipeline never imports Anthropic. If OpenAI rate-limits after retries, LiteLLM moves to the backup alias.

4. Log what still fails

Catch litellm.RateLimitError / APIError after fallbacks exhaust. Those inputs become tomorrow's eval set and a signal to raise budgets or add another region.

Optional: run the proxy instead of embedding the Router

If more than one service needs the same failover rules, run the LiteLLM AI Gateway and point every OpenAI client at it:

from openai import OpenAI

client = OpenAI(
    api_key="sk-your-virtual-key",
    base_url="http://localhost:4000",
)
client.chat.completions.create(
    model="support-summarizer",
    messages=[{"role": "user", "content": "hello"}],
)
Enter fullscreen mode Exit fullscreen mode

Config YAML holds model_list, fallbacks, budgets, and virtual keys. Apps stay dumb; the gateway owns reliability.

When not to bother

  • One hobby script, one key, one model — a plain SDK is enough.
  • You already have a mature internal gateway with the same features.
  • Latency budgets are so tight that an extra hop or cross-provider fallback is worse than a brief 429.

Everyone else who has been burned by a single-vendor outage: put the fallback config next to the model list, not inside twelve call sites.

Checklist

  1. Replace scattered provider SDKs with LiteLLM completion or an OpenAI client pointed at the proxy.
  2. Give production jobs an app-facing model_name, not a raw vendor id.
  3. Set num_retries for transient failures inside the group.
  4. Set fallbacks to a different provider (or region) for sustained failures.
  5. Log exhausted fallbacks; treat them as product incidents, not "weird LLM noise."

References

What do you use today when your primary LLM provider rate-limits — hope, a second hardcoded client, or a real router?

Top comments (0)