One AI request through LiteLLM: the key check, the router, the translation, the cost, and the choices that shape each step.
A note on how this was written: an AI writing assistant drafted it from the sources listed at the end, and every claim was checked against those sources before it went out.
At a glance
- LiteLLM sits between a company's apps and 100+ AI providers, and every app talks to it in OpenAI's format.
- It checks the key and the budget, picks where to send each call, translates it, and writes down the cost.
- Prices come from a public file read at start, so costs can drift until someone reloads it.
What LiteLLM is for
LiteLLM is an open source (public code anyone can read and reuse) AI gateway (a server every AI request passes through on its way to the real service). It sits between a company's apps and 100+ AI providers (the companies that run the models, such as OpenAI, Anthropic, Gemini, Bedrock and Azure).
Every app talks to it in one format: OpenAI's API (a fixed way for one program to ask another for something). LiteLLM checks the key and the budget, picks where to send the call, translates it for that provider, and writes down what it cost.
Think of a company mailroom. The clerk checks who sent each parcel, picks a courier, repacks the parcel the way that courier wants, and logs the postage. Nobody upstairs has to learn each courier's rules.
It comes as a Python SDK (a library you call from your own code) or as the gateway, also called the proxy (a program that sits between apps and a service and passes requests along), shared by a team or a company. Per the project's design notes, ARCHITECTURE.md, the gateway uses the SDK inside and adds a check of who is asking, usage limits, budgets and routing (choosing where each call goes). This post follows the gateway.
One request, from the app to the provider
Say an app sends 'Hello!' to gpt-4o through the gateway. I'll follow it through the four lanes of my Atlas sheet: what comes in, what does the work, what it keeps and what goes out.
What comes in. The app calls the gateway as it would call OpenAI, say at /chat/completions on port 4000 (a numbered door on the server). In the Authorization header (the line that carries the key), it sends a virtual key (a key the gateway makes for a team or an app, so the real provider keys stay hidden).
Two other inputs matter: a public price list, read at start, and config.yaml (the settings file). The config maps each model name to a model group (one or more deployments, each a place the model runs: another key, region or provider).
What does the work. First, the key and its budget come from a cache (a saved copy kept so slow work isn't redone) in memory or in Redis (a very fast in memory store). The database is read only when the cache does not have the key. Next come rate limits (caps per minute) on requests and on tokens (the pieces of text a model reads and writes), for the server, key, user and team.
Then the router (the part that decides where a call goes) picks one deployment from the group, by default with a simple shuffle. Last, a translation file for that provider turns OpenAI's format into the provider's, and the answer back into OpenAI's.
After the answer: the cost, the logs, the records
Before the answer leaves, the gateway works out the cost: tokens in and out, times the prices in its price list. ARCHITECTURE.md shows that cost going back to the app in a header (extra data sent along with the answer) named x-litellm-response-cost.
What goes out. The app gets the answer in OpenAI's shape. Logs go to tools such as Langfuse and Datadog (services that record each call), in the background. An admin screen shows keys, teams and spend.
What it keeps. Postgres (a widely used open source database) holds keys, teams, budgets and spend logs (records of what each call cost), in tables set up with Prisma (a tool that defines database tables in one schema file). Redis is optional. When present, it holds caches, the rate limit counters and a spend buffer.
The spend for 'Hello!' waits in a queue (a waiting line) until a background job (code that runs by itself at set times) writes it to Postgres in batches (many small writes saved up and written together), every 10 to 15 seconds by default. So no database write sits in the path of the request.
Four choices, and what each one costs
One shape for every provider. The makers chose OpenAI's format over each provider's own SDK, so an app changes providers by changing a model name, not its code. The cost: a feature one provider has and OpenAI's shape lacks needs extra parameters (added settings).
Keep the database out of the request. A gateway sits in front of every call, so it must not add waiting time. The cost: spend totals in the database run behind, and budget checks use live counters that copies share only through Redis.
Prices as a public file. Model prices sit in one JSON file (a plain text format for data) in the repo (the project's public code folder), with a copy inside the installed package. The makers picked a file over a pricing service or each provider's billing API because anyone can read it, fix it and reuse it.
The cost is like a menu printed in the morning. If the kitchen raises a price at noon, the menu on the table shows the old one until someone reprints it. A running proxy keeps the prices it loaded until an admin reloads the list or sets a schedule, so costs can drift. My BillCheck is for that drift: it checks a gateway's spend logs against the price of each day, in the browser.
Redis only when you scale out (run more copies side by side). One copy keeps caches and counters in memory and needs nothing extra. The cost: without Redis, limits and caches are per copy.
What changes at ten times the size
Picture ten copies of the gateway behind a load balancer (a server that spreads requests across the copies). Limits and caches move to Redis, so the copies agree on what each key has used. Spend updates can wait there too, before the batch write. The repo's Terraform modules (files that set up cloud servers from code) go further and run the gateway and the admin screen as separate services.
And if a provider goes down? Think of calling a shop. If one clerk is busy, the call rolls to another clerk in the same shop. If the shop stays closed, you call the next shop on your list. In LiteLLM, a retry asks the same model group again, which may pick another deployment. A fallback moves to the next model group in the config when the first keeps failing.
Check it yourself
Here is how I'd see this on my own computer, using the README (the project's front page).
Step 1. Install the gateway with uv tool install 'litellm[proxy]' (uv installs Python packages). Put your OpenAI key in OPENAI_API_KEY, as the README's SDK example does, and start it with litellm --model gpt-4o.
Step 2. Point any OpenAI client (the code library an app uses to call OpenAI) at port 4000 on your machine (the README uses 0.0.0.0:4000) and send 'Hello!'. With curl (a command line tool for web requests), use /v1/chat/completions and print the response headers.
Step 3. Find the x-litellm-response-cost header: the cost the gateway worked out for that call.
Step 4. Open the price file, model_prices_and_context_window.json, in the repo and find your model. Multiply its prices by the tokens in and out, from the usage part of the answer. If the two numbers differ, I'd first check whether the file changed after the gateway loaded it.
The full sheet
The full sheet, with every box and decision, is on the Atlas. If you want one for a project you use or are building, ask for a sheet. The makers' own walk through a request is Life of a Request.
Do you run LiteLLM as a shared gateway, or as a library inside one app?
Sources
More free tools for data and AI: tabrizianalytics.com
Top comments (0)