toolfreebie

Posted on Aug 20 Originally published at toolfreebie.com

LiteLLM: One Free API for OpenAI, Claude & 100+ LLMs

#ai #api #opensource

Quick answer: LiteLLM is a free, open-source (MIT) layer that lets you call 100+ LLM providers through one OpenAI-compatible interface — swap providers by changing a single model string. It ships as a Python SDK (runs in your app) or a self-hosted Proxy/gateway (virtual keys, budgets, fallbacks, spend tracking). Both are genuinely free; you pay only the underlying providers, which on free tiers is often $0.

Once your .env holds free keys from Groq, Gemini, Together AI, and a local Ollama server, each wants a different SDK, request shape, and response format. LiteLLM (MIT, from BerriAI, ~40,000 GitHub stars) deletes that friction: one function, one response shape, a routing prefix that selects the backend.

SDK or Proxy: Two Distinct Things

“LiteLLM” refers to two tools, and people constantly conflate them:

The LiteLLM Python SDK — a library you pip install and import. You call litellm.completion(...) and it runs inside your process. Right for a single app or script.
The LiteLLM Proxy (AI Gateway) — a standalone server (usually Docker) that sits between your apps and the providers. Every app points at it as if it were OpenAI; it handles routing, virtual keys, spend tracking, budgets, rate limits, and fallbacks centrally. Right for a team or multiple apps.

Both are MIT-licensed. The SDK is the entry point; the proxy is where LiteLLM becomes infrastructure.

Is LiteLLM Really Free?

Yes. Both the SDK and Proxy are MIT open source — no per-request fees, token quotas, or log limits from LiteLLM itself. You pay only the underlying providers (often $0 on free tiers). A paid Enterprise tier exists, but it gates organizational governance, not the core:

Capability	Open Source (free)	Enterprise (paid)
100+ provider support, OpenAI format	Yes	Yes
Virtual keys, teams, per-key budgets	Yes	Yes
Load balancing, fallbacks, retries	Yes	Yes
Spend tracking & cost logging	Yes	Yes
RPM/TPM limits, caching, guardrails	Yes	Yes
SSO / SAML, JWT auth, audit logs	—	Yes
Prometheus metrics, enterprise admin UI	—	Yes
SLA support, SOC2/HIPAA help	—	Yes

The entire routing-and-gateway core — including virtual keys and budgets, the features people assume are paywalled — ships free. Enterprise (from around $250/month) is about SSO, audit trails, and compliance. See the enterprise docs for the exact split. The one honest caveat: “free software” is not “free to run” — self-hosting the proxy means you own the server and its ops. The SDK has no such cost.

The SDK: Your First Call in 60 Seconds

pip install litellm

Call any provider with the same function. Only the model string and the environment key change:

from litellm import completion
import os

os.environ["GROQ_API_KEY"] = "gsk-..."

response = completion(
    model="groq/llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Explain LiteLLM in one sentence."}],
)

print(response.choices[0].message.content)

That response shape — response.choices[0].message.content — is the OpenAI format, identical whether the call went to Groq, Gemini, Anthropic, or a local model. To switch providers, change one string:

# Same code, three providers — only the model string changes
completion(model="gemini/gemini-2.5-flash", messages=msgs)        # Google
completion(model="together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo", messages=msgs)
completion(model="ollama/llama3", messages=msgs, api_base="http://localhost:11434")

The provider/model prefix tells LiteLLM where to route. Drop it and it defaults to OpenAI. Streaming works the same way — add stream=True and you get an OpenAI-style chunk iterator regardless of provider.

Automatic Fallbacks: The Killer Feature in Two Lines

Free tiers rate-limit you. When a request fails (a 429, a timeout), you want it to retry elsewhere, not crash. The SDK does this with a fallbacks list:

from litellm import completion

response = completion(
    model="groq/llama-3.3-70b-versatile",        # try the fast one first
    messages=[{"role": "user", "content": "Summarize this contract..."}],
    fallbacks=[
        "gemini/gemini-2.5-flash",               # then Google
        "together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo",  # then Together
    ],
)

If Groq is rate-limited, the same request re-runs against Gemini, then Together — no try/except ladder. It turns three separate free quotas into one resilient pool.

The Proxy: A Free, Self-Hosted AI Gateway

The moment you have multiple apps or need to track and cap spending, you graduate to the Proxy — a server presenting a single OpenAI-compatible endpoint. See the AI Gateway docs. Configure it with a config.yaml:

model_list:
  - model_name: fast            # the name your apps will call
    litellm_params:
      model: groq/llama-3.3-70b-versatile
      api_key: os.environ/GROQ_API_KEY
  - model_name: long-context
    litellm_params:
      model: gemini/gemini-2.5-flash
      api_key: os.environ/GEMINI_API_KEY
  - model_name: local
    litellm_params:
      model: ollama/llama3
      api_base: http://localhost:11434

litellm_settings:
  num_retries: 2
  fallbacks: [{"fast": ["long-context"]}]   # if 'fast' fails, use 'long-context'

Run it with Docker:

docker run -p 4000:4000 \
  -v $(pwd)/config.yaml:/app/config.yaml \
  -e GROQ_API_KEY=$GROQ_API_KEY \
  -e GEMINI_API_KEY=$GEMINI_API_KEY \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml

Now any OpenAI-compatible tool points at http://localhost:4000 and calls the model named fast — the proxy decides which real provider answers:

from openai import OpenAI                       # the real OpenAI SDK

client = OpenAI(base_url="http://localhost:4000", api_key="sk-1234")
resp = client.chat.completions.create(
    model="fast",                               # a name from config.yaml
    messages=[{"role": "user", "content": "Hello"}],
)

Every app speaks plain OpenAI; you change models, providers, and routing in one YAML file instead of redeploying every service.

Virtual Keys and Per-Key Budgets

The standout free feature is virtual keys. Set one master key, then mint child keys each with its own budget, rate limit, and allowed-model list (docs):

curl http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-MASTER-KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["fast", "long-context"],
    "max_budget": 5.00,
    "budget_duration": "30d",
    "rpm_limit": 60
  }'

That returns a key scoped to two models, hard-capped at $5 over 30 days, throttled to 60 RPM. When the budget is hit, requests are rejected automatically, with spend tracked per key (spend tracking docs). For anyone burned by a leaked key or a runaway agent loop, this is the feature that pays for the setup time.

Load Balancing Across Free Tiers

List the same logical model multiple times with different keys and LiteLLM’s Router spreads traffic across them — legitimately stretching several free quotas into one pool:

model_list:
  - model_name: workhorse
    litellm_params:
      model: groq/llama-3.3-70b-versatile
      api_key: os.environ/GROQ_KEY_A
  - model_name: workhorse        # same logical name, second backend
    litellm_params:
      model: together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo
      api_key: os.environ/TOGETHER_KEY

router_settings:
  routing_strategy: usage-based-routing-v2   # send load to the least-busy backend

LiteLLM vs OpenRouter

Both “let you call many models through one OpenAI-style interface,” but the difference is fundamental:

Dimension	LiteLLM	OpenRouter
What it is	Open-source library / self-hosted gateway	Hosted, managed routing service
Who holds keys	You — your own provider keys (BYOK)	OpenRouter — one funded balance
Cost model	$0 software; pay providers directly	Provider price + small routing margin
Local / Ollama models	Yes — proxy your own Ollama/vLLM	No — only hosted models
Virtual keys & budgets	Yes, self-managed	Yes, in dashboard
Setup effort	Run a server (or import the SDK)	Sign up, get one key, done
Best for	Owning your stack, mixing free tiers & local	Zero-ops access to many models fast

OpenRouter is a product; LiteLLM is infrastructure. OpenRouter is the fastest way to reach 300+ models with one key. LiteLLM is what you reach for to keep your own free-tier keys, route to a local Ollama model, and govern spend yourself. They compose well: add OpenRouter as one backend inside a LiteLLM proxy.

Logging and Observability

A unified gateway is the perfect place to capture every LLM call. LiteLLM integrates natively with Langfuse, OpenTelemetry, Prometheus, and others in one line:

# In config.yaml — trace every call through the proxy to Langfuse
litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]
# then set LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY in the environment

Every call across all providers then shows up in Langfuse as a traced, costed, debuggable event — one of the most common free, self-hosted LLMOps setups in 2026.

Common Gotchas

Model names need the provider prefix. completion(model="llama-3.3-70b-versatile") without groq/ defaults to OpenAI and throws a confusing auth error. Always use provider/model (providers list).
The proxy needs a database for persistent keys. Virtual keys, budgets, and spend are stored in Postgres; point it at a DATABASE_URL. A free Postgres tier works fine.
Cost tracking needs correct pricing data. LiteLLM computes spend from a built-in price map; a brand-new or self-hosted model may be tracked as $0. Verify pricing if you rely on budget enforcement.
Fallbacks need compatible models. Group fallbacks by capability — fast-text together, long-context together — rather than one mixed list.

Frequently Asked Questions

Is LiteLLM free to use?

Yes. Both the SDK and the Proxy are MIT open source, with no per-request fees, token quotas, or log limits from LiteLLM itself. You pay only the underlying providers — often nothing on free tiers. A paid Enterprise tier exists for SSO, audit logs, and SLA support, but the entire routing, virtual-key, and budgeting core is free.

What is the difference between the SDK and the Proxy?

The SDK is a Python library you import into one app — it translates completion() calls to 100+ providers inside your own process. The Proxy is a standalone server (usually Docker) that sits between many apps and the providers, adding centralized virtual keys, budgets, spend tracking, and load balancing. Use the SDK for a single app; use the Proxy when multiple apps or people need shared, governed access.

Can LiteLLM call local models like Ollama?

Yes. LiteLLM proxies local runtimes such as Ollama and vLLM through the same OpenAI-compatible interface — use the ollama/model prefix and point api_base at your local server. This is a key advantage over hosted routers, which only reach cloud models. You can mix a local model and a hosted free tier behind one gateway.

Is the LiteLLM proxy production-ready?

Yes — it is widely deployed as a Docker service in front of production traffic, with retries, fallbacks, caching, rate limiting, and Postgres-backed key and spend management. Organizations needing SSO, audit logging, and formal support can layer on the Enterprise tier, but the open-source proxy handles the core gateway workload on its own.

Bottom Line

LiteLLM gives you one OpenAI-compatible interface to 100+ providers, free and MIT-licensed, in two forms:

Building one app? Import the SDK, change a model string to swap providers, add a two-line fallbacks list to make your free tiers resilient.
Running multiple apps or a team? Stand up the proxy as a self-hosted gateway with virtual keys, per-key budgets, load balancing, and spend tracking — all free.
Want zero ops instead? Reach for OpenRouter, or drop it in as one backend behind LiteLLM.

Pair it with the free keys you already have — Groq, Gemini, Together AI — wire in Langfuse for tracing, and you have a unified, observable, multi-provider LLM backend that costs nothing but the server it runs on.

DEV Community