DEV Community: Misbah Syed

Set up LiteLLM Auto Router: send each prompt to the right model

Misbah Syed — Mon, 14 Sep 2026 21:41:20 +0000

Most AI apps let you pick one model, and then every query goes to it. "What's the capital of France?" goes to that model. "Find the race condition in our billing service" goes to the same one. Your simplest and hardest queries cost the same, so you pay frontier prices for the easy ones.

Auto Router fixes that at the gateway. You add it to LiteLLM as a model, give it a name like smart-router, and your app calls that name like any other model. For each request, the router reads the prompt, puts it in one of four tiers (SIMPLE, MEDIUM, COMPLEX, REASONING) and sends it to the model you mapped to that tier. Simple prompts go to a cheap model and complex ones go to a frontier model.

Your app doesn't change. The routing lives in one block of config.yaml, so when a newer model comes out, swapping it in is a one-line change.

This guide builds that config step by step and tests it with four curl requests.

If you would like to watch this tutorial as a video, please follow the YouTube link.

Prerequisites

uv
An OpenAI API key and an Anthropic API key
curl and openssl
Port 4000 free on your machine

Step 1: install LiteLLM

If you don't have uv yet:

curl -LsSf https://astral.sh/uv/install.sh | sh

Install the gateway as a uv tool. This guide was tested on 1.100.1, so pin it:

uv tool install 'litellm[proxy]==1.100.1'
litellm --version

If the shell can't find litellm, run uv tool update-shell and open a new terminal.

Step 2: create the project and a strong master key

mkdir auto-router && cd auto-router

The master key is what every client uses to call your gateway, so treat it like a password. The docs use sk-1234 as a placeholder. Don't use that, or anything a person or an agent could guess. Generate a random one:

echo "sk-$(openssl rand -hex 32)"

Create .env and paste your values in:

# .env
OPENAI_API_KEY="your-openai-key"
ANTHROPIC_API_KEY="your-anthropic-key"
LITELLM_MASTER_KEY="sk-<the 64 hex characters you generated>"

Lock it down, keep it out of git, and load it into your shell:

chmod 600 .env
echo ".env" >> .gitignore
set -a && source .env && set +a

Step 3: add your models

Create config.yaml with a plain LiteLLM model list:

model_list:
  - model_name: gpt-5.6-luna
    litellm_params:
      model: openai/gpt-5.6-luna
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-5.6-terra
    litellm_params:
      model: openai/gpt-5.6-terra
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: claude-opus-5
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY

os.environ/NAME reads the key from the environment, so no secret lives in the YAML.

Step 4: add the router and map the tiers

Append a fifth entry to model_list:

  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE: gpt-5.6-luna
          MEDIUM: gpt-5.6-terra
          COMPLEX: claude-sonnet-5
          REASONING: claude-opus-5

This alone is a working router. It scores each prompt locally from signals like length, code and reasoning phrases, then picks a tier. No extra model call is made. Each tier value is a model_name from step 3.

Step 5: add keyword rules

Some requests need a strong model no matter how they score. Add this under complexity_router_config:

        keyword_tier_rules:
          - keywords: ["security review", "incident"]
            tier: COMPLEX

Any prompt containing "security review" or "incident" now goes to COMPLEX without scoring. Set tier: REASONING if you want the top tier instead.

Step 6: let an LLM classify the unclear prompts

Add under complexity_router_config:

        classifier_type: heuristic_first
        heuristic_first_max_tier: SIMPLE
        classifier_llm_config:
          model: gpt-5.6-luna
          timeout_ms: 3000
          classification_rubric: agentic
        classifier_fallback: heuristic
        classifier_context_window_size: 3

heuristic_first with heuristic_first_max_tier: SIMPLE: if the local scorer is confident a prompt is SIMPLE, it routes right away. Anything else goes to the classifier.
classifier_llm_config.model: the small model that does the classifying.
classifier_fallback: heuristic: if the classifier call fails or times out, the router uses the local score.
classifier_context_window_size: 3: the classifier sees the last three turns, so a follow-up like "now do the same for refunds" is judged in context.

Step 7: default model, escalation and housekeeping

Add under complexity_router_config:

        default_model: gpt-5.6-terra
        escalation_keywords: ["LITELLM ESCALATE"]
        route_housekeeping_to_cheapest_tier: true

default_model: used when there is nothing to classify.
escalation_keywords: a user or agent can include this phrase to push a request up one tier. It is case-sensitive.
route_housekeeping_to_cheapest_tier: coding agents rename sessions by sending the whole conversation and asking for a title. That text looks like hard work to a scorer. This sends those calls to the cheapest tier.

Step 8: gateway settings

At the top level of config.yaml, below model_list:

litellm_settings:
  drop_params: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

drop_params: true drops request parameters a provider doesn't support instead of failing the call. Coding agents often send those. The master key is read from the environment, not written in the file.

The full config.yaml

model_list:
  - model_name: gpt-5.6-luna
    litellm_params:
      model: openai/gpt-5.6-luna
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-5.6-terra
    litellm_params:
      model: openai/gpt-5.6-terra
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: claude-opus-5
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE: gpt-5.6-luna
          MEDIUM: gpt-5.6-terra
          COMPLEX: claude-sonnet-5
          REASONING: claude-opus-5

        keyword_tier_rules:
          - keywords: ["security review", "incident"]
            tier: COMPLEX

        classifier_type: heuristic_first
        heuristic_first_max_tier: SIMPLE
        classifier_llm_config:
          model: gpt-5.6-luna
          timeout_ms: 3000
          classification_rubric: agentic
        classifier_fallback: heuristic
        classifier_context_window_size: 3

        default_model: gpt-5.6-terra
        escalation_keywords: ["LITELLM ESCALATE"]
        route_housekeeping_to_cheapest_tier: true

litellm_settings:
  drop_params: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Step 9: start the gateway

LITELLM_LOG=INFO litellm --config config.yaml --port 4000

LITELLM_LOG=INFO prints a routing decision line for every request, so you can watch the router work in this terminal.

Step 10: send four prompts

Open a second terminal in the auto-router folder and load the key there too:

cd auto-router
set -a && source .env && set +a

The response body still says "model": "smart-router". The model that answered comes back in the x-litellm-model-name header, with its cost in x-litellm-response-cost. Each command below prints only those headers. Remove the | grep ... part to see the full response.

The outputs are from one run. Your costs will differ.

A simple question

curl -si http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "smart-router", "messages": [{"role": "user", "content": "What is the capital of Japan? Keep it brief."}]}' \
  | grep -iE '^x-litellm-(model-name|response-cost|classifier-cost):'


x-litellm-model-name: openai/gpt-5.6-luna
x-litellm-response-cost: 9.4e-06

In the gateway terminal:

ComplexityRouter: routing decision cause=heuristic_first_short_circuit, tier=SIMPLE, score=-0.150, signals=('short (11 tokens)', 'simple (what is, brief)'), routed_model=gpt-5.6-luna

The scorer was confident, so it went to the cheap model with no classifier call.

A keyword match

curl -si http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "smart-router", "messages": [{"role": "user", "content": "Do a security review of this login handler: it compares the password with == and logs the request body."}]}' \
  | grep -iE '^x-litellm-(model-name|response-cost|classifier-cost):'


x-litellm-model-name: anthropic/claude-sonnet-5
x-litellm-response-cost: 0.010412000000000001


ComplexityRouter: routing decision cause=literal_keyword_match, escalated=False, tier=COMPLEX, routed_model=claude-sonnet-5

"security review" matched the rule from step 5.

A prompt the classifier decides

curl -si http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "smart-router", "messages": [{"role": "user", "content": "Our billing service double charges about one customer a week. Figure out where retries and webhooks could race, and propose a fix we can ship without downtime."}]}' \
  | grep -iE '^x-litellm-(model-name|response-cost|classifier-cost):'


x-litellm-model-name: anthropic/claude-sonnet-5
x-litellm-response-cost: 0.019232000000000003
x-litellm-classifier-cost: 0.0002036


ComplexityRouter: routing decision cause=llm_classifier, tier=COMPLEX, score=n/a, signals=('llm-classifier:COMPLEX',), routed_model=claude-sonnet-5

No keyword matched and the scorer wasn't sure, so gpt-5.6-luna classified it as COMPLEX. That's the only request with a classifier cost. The classifier is a model call, so another run can pick a different tier.

A housekeeping call

curl -si http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "smart-router", "messages": [{"role": "user", "content": "You are coming up with a succinct title for a coding session. Session: refactor the distributed payment reconciliation architecture, fix concurrency bugs in the async webhook handler, add benchmarks."}]}' \
  | grep -iE '^x-litellm-(model-name|response-cost|classifier-cost):'


x-litellm-model-name: openai/gpt-5.6-luna
x-litellm-response-cost: 2.38e-05


ComplexityRouter: routing decision cause=housekeeping, tier=SIMPLE, score=n/a, signals=('housekeeping',), routed_model=gpt-5.6-luna

The prompt is full of words that would score high, but it's a session title request, so it went to the cheapest tier.

Start simple

Try the router with the prompts you send from your AI tools and coding agents. Start with only the tiers from step 4. Add keyword rules for the requests you care about, then add the LLM classifier once you need better decisions on unclear prompts.

The full option reference is on the Auto Routing docs page, and the code is in BerriAI/litellm.

Additional resources

LiteLLM docs: https://docs.litellm.ai
AutoRouter tutorial page: https://docs.litellm.ai/docs/proxy/auto_routing
LiteLLM on GitHub: https://github.com/BerriAI/litellm
Video walkthrough: Watch it on YouTube

How to set up Cursor with LiteLLM

Misbah Syed — Thu, 10 Sep 2026 18:36:50 +0000

This guide connects Cursor to a LiteLLM proxy, one you run yourself or one your team already hosts. Cursor takes one base URL and one API key in its settings.

The tldr version

Cursor Settings, then Models, then expand API Keys
Turn on Override OpenAI Base URL and set it to https://your-gateway.example.com/cursor
Turn on OpenAI API Key, paste a LiteLLM key, confirm the prompt
Add Custom Model, and type the model_name from your config.yaml
Pick that model in the model picker and send a message

If you prefer to watch the tutorial instead:

Watch it on YouTube

The steps below are the ones in the video, so you can move between the two.

Requirements

You need Cursor recent enough to have a Models page under Cursor Settings, and a LiteLLM gateway. Docker builds that gateway in section 6 if you do not have one; skip Docker entirely if your team already hosts one.

The one that catches people out. The gateway has to be reachable from the public internet. Cursor does not call your model from your laptop, its own backend makes the request, so http://localhost:4000 is not reachable. Section 7 provides tips on how to make a publicly available URL.

Versions used: LiteLLM main-stable, UI v1.99.0, DB Postgres 16-alpine.

Let's get started!

If a gateway is already running somewhere your Cursor can reach, sections 1 to 5 are enough. If you don't have a gateway running, jump to section 6, build the gateway, then come back to section 1.

1. Open the Models page in Cursor

The gear icon in the top right opens Cursor Settings. Everything in this guide lives under Models.

2. Point Cursor at the gateway

Scroll down to API Keys and expand it. Then put your gateway URL in the base URL field with /cursor on the end, and the per-user API key from the LiteLLM gateway(more on it below). Turn on: OpenAI API Key , and Override OpenAI Base URL toggles.

https://your-gateway.example.com/cursor

/cursor . Is the route LiteLLM exposes for Cursor specifically.

3. Add your key and enable it

Paste a LiteLLM key into the OpenAI API Key field and save it. Use a virtual key from the gateway's Virtual Keys page if you have one.

Cursor then asks you to confirm. Say yes.

The dialog warns that Tab, Apply from Chat and Agent run on Cursor's own models and cannot be billed to your key. Whereas, Chat and the model picker route through your gateway.

4. Add the model names your gateway serves

Cursor's model list is its own. If a model_name from your config happens to match something Cursor already lists, it shows up on its own. In some case, it might not allow to use the same model names as the built-in model names in Cursor. In that case, you will have to add a custom model name.

Scroll to the bottom of the model list, click Add Custom Model , and type the name exactly as it appears in config.yaml or public model names from the created models in the gateway.

5. Send a message and check the logs

Open a chat, click the model picker, and pick one of your models.

Now go to the gateway UI, open Logs under Observability, and narrow the time filter to the last fifteen minutes. Your request is there with a cost against it.

Click a row for the detail and now you can see which key sent it, how many tokens went out and came back, what it cost, how long it took, and how much of the prompt was served from cache.

The Tags row shows User-Agent: Cursor, which indicates that it was Cursor traffic.

6. No gateway yet? Let's run one

Start a new project directory, or create these three files in your existing project directory.

config.yaml

model_list:
  - model_name: litellm-gpt-5.6-terra
    litellm_params:
      model: openai/gpt-5.6-terra
      api_key: os.environ/OPENAI_API_KEY
  - model_name: litellm-claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

litellm_settings:
  drop_params: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

  # Coding turns stream for a long time. Do not cut them off.
  request_timeout: 600

router_settings:
  num_retries: 3
  cooldown_time: 30

model_name . This is the name Cursor sends and the name you type into Add Custom Model. Prefixing them with litellm- keeps your model names from colliding with Cursor's built-in list, and when you see one in a log you know where it came from.

model . The provider's own name for the model, with the provider in front. LiteLLM uses this to decide which API to call and how to shape the request. You can find the full list of models supported here.

drop_params: true . Coding agents send parameters that not every provider accepts. This line drops them instead of failing the request.

request_timeout: 600 . An agentic turn can stream for minutes. The default cuts it off well before that.

.env

LITELLM_MASTER_KEY=sk-your-own-long-random-string
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...

The master key is the only required variable, along with any provider api key needed for the models listed in the config.yaml.

docker-compose.yml

name: litellm-coding-agents

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    restart: unless-stopped
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    ports:
      - "4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml:ro
    env_file:
      - .env
    environment:
      DATABASE_URL: postgresql://litellm:litellm@postgres:5432/litellm
      STORE_MODEL_IN_DB: "True"
    depends_on:
      postgres:
        condition: service_healthy
    healthcheck:
      test: ["CMD-SHELL", "python -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:4000/health/liveliness').status==200 else 1)\""]
      interval: 10s
      timeout: 5s
      retries: 12
      start_period: 30s

  postgres:
    image: postgres:16-alpine
    restart: unless-stopped
    # Deliberately not published to the host. Nothing outside this network
    # needs it, and a 5432 collision is an annoying afternoon.
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: litellm
      POSTGRES_DB: litellm
    volumes:
      - litellm-pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm -d litellm"]
      interval: 5s
      timeout: 5s
      retries: 20

volumes:
  litellm-pgdata:

Postgres DB allows UI access to make virtual keys, budgets and the logs we saw in section 5.

STORE_MODEL_IN_DB: "True" . Lets you add models from the UI. Models added that way live in the database.

Bring it up:

docker compose up -d

Then check it is actually serving:

curl -s http://localhost:4000/health/liveliness

You want "I'm alive!" back.

A quick tip: if something else already holds port 4000, find it with lsof -nP -iTCP:4000 -sTCP:LISTEN and either stop it or change the published port in docker-compose.yml.

7. Give the local gateway a public URL

Cursor's backend has to reach the gateway, so a local port needs a tunnel in front of it. You can assign a public URL to your locally deployed gateway using either ngrok or cloudflared.

ngrok:

ngrok http 4000 --log=stdout

cloudflared:

cloudflared tunnel --url http://localhost:4000

Either one prints an HTTPS URL. That URL, plus /cursor goes into the base URL field from section 2.

A free tunnel URL changes every time you restart the tunnel, and Cursor will need the new one. For anything beyond trying this out, put the gateway on a host with a real domain.

8. Adding a model from the LiteLLM UI

With STORE_MODEL_IN_DB on in docker-compose.yml, Models + Endpoints has an Add Model tab that writes to the database instead.

LiteLLM Model Name is the provider's name for the model, the string LiteLLM will send upstream. Public Model Name is what we can assing and what Cursor sends. Mapping litellm-gpt-5.6-luna to gpt-5.6-luna is the UI equivalent of the model_name / model pair in config.yaml.

Scroll down for the credentials.

Save it, add the public model name to Cursor with Add Custom Model the way you did in section 4. Test a short prompt in the chat and the model sends the response.

Advanced options

Manage usage with virtual keys

Create a key per developer under Virtual Keys , give each one a budget and a list of models it is allowed to use, and hand that to your teammates. The Key Alias column in the logs gives you details of the usage.

Using different models

Anything LiteLLM supports works here, not just OpenAI. Add a block to model_list, give it a model_name that starts with litellm-, restart, and add that name to Cursor. Cursor is talking to an OpenAI-shaped endpoint, and it could Claude or Gemini or a local model.

No base URL field? Use the Azure OpenAI panel

Some Cursor builds do not show Override OpenAI Base URL. The Azure OpenAI panel underneath it takes the same three pieces of information and gets you to the same place.

One difference, and it is easy to miss: do not append /cursor here. Base URL is the bare gateway URL. Deployment Name is your public model name, and API Key is the LiteLLM key.

Additional resources

LiteLLM docs: https://docs.litellm.ai
Cursor tutorial page: https://docs.litellm.ai/docs/tutorials/cursor_integration
Gateway quick start: https://docs.litellm.ai/docs/proxy/quick_start
Docker quick start: https://docs.litellm.ai/docs/proxy/docker_quick_start
LiteLLM on GitHub: https://github.com/BerriAI/litellm
Video walkthrough: Watch it on YouTube

How to set up OpenAI Codex with LiteLLM

Misbah Syed — Wed, 09 Sep 2026 17:20:51 +0000

This guide connects OpenAI Codex to a LiteLLM proxy, one you run yourself or one your team already hosts. Codex reads one base URL out of a config file, so pointing it at LiteLLM gives you:

Any model LiteLLM can reach, from the same Codex CLI and desktop app
A log line and a cost for every request, in one place
Per-developer keys with their own budgets and their own model access
One provider key on the server instead of a copy on every machine

The five-minute walkthrough this guide is written from:

The steps below are the ones in the video, so you can move between the two.

Requirements

Docker, for the setting up LiteLLM gateway. Not needed if you are pointing at a gateway already hosted.
Node, for the Codex CLI
One provider key. One is enough to start

Everything below runs on the open-source image, ghcr.io/berriai/litellm:main-stable. The admin UI, virtual keys and spend tracking come from having a database attached, which the compose file does.

Versions used here: litellm 1.99.0, current @openai/codex.

Let's go!

If your team already hosts a LiteLLM gateway, skip steps 2 and 3 below.

1. Install OpenAI Codex

Install the Codex CLI globally:

npm install -g @openai/codex

2. Configure LiteLLM for model routing

Make a directory for the LiteLLM proxy and put config.yaml in it. This file is the model list: the names your clients ask for, and where each one goes.

config.yaml

model_list:
  # `codex -m <id>` works with any of them.
  - model_name: gpt-5.6-terra
    litellm_params:
      model: openai/gpt-5.6-terra
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-5.6-luna
    litellm_params:
      model: openai/gpt-5.6-luna
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-5.6-sol
    litellm_params:
      model: openai/gpt-5.6-sol
      api_key: os.environ/OPENAI_API_KEY

  # Anything from another provider goes in the same list.
  - model_name: claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

  # Catch-all, so `codex -m <anything>` reaches OpenAI without editing
  # this file. Delete this entry to restrict a fleet to the list above.
  - model_name: "*"
    litellm_params:
      model: openai/*
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  drop_params: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  # Coding turns stream for a long time. Do not cut them off.
  request_timeout: 600

You could as many model names to this list from the ones supported by LiteLLM

drop_params: true . Coding agents sometimes send parameters that not every provider accepts, which is dropped by this line.

master_key . This is the key to access the proxy.

3. Start the LiteLLM proxy

We'll start the LiteLLM gateway along with a postgres database, so that we can access the dashboard UI for the gateway..

docker-compose.yml

name: litellm-coding-agents

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    restart: unless-stopped
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    ports:
      - "4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml:ro
    env_file:
      - .env
    environment:
      DATABASE_URL: postgresql://litellm:litellm@postgres:5432/litellm
      STORE_MODEL_IN_DB: "True"
    depends_on:
      postgres:
        condition: service_healthy
    healthcheck:
      test: ["CMD-SHELL", "python -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:4000/health/liveliness').status==200 else 1)\""]
      interval: 10s
      timeout: 5s
      retries: 12
      start_period: 30s

  postgres:
    image: postgres:16-alpine
    restart: unless-stopped
    # Deliberately not published to the host. Nothing outside this
    # network needs it, and 5432 is usually already taken.
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: litellm
      POSTGRES_DB: litellm
    volumes:
      - litellm-pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm -d litellm"]
      interval: 5s
      timeout: 5s
      retries: 20

volumes:
  name: litellm-pgdata

Provider keys go in .env next to the compose file:

OPENAI_API_KEY=...
ANTHROPIC_API_KEY=...
LITELLM_MASTER_KEY=sk-your-strong-string

sk-your-strong-string is the placeholder master key used in the LiteLLM docs. Replace it with a real secret key.

Bring it up

Start the proxy and its database:

docker compose up -d

First boot runs the database migrations and takes about eighty seconds. Every boot after that is about fifteen (depending on the machine spec).

Check that it is alive:

curl -s http://localhost:4000/health/liveliness

And that it is serving the model list:

curl -s http://localhost:4000/v1/models -H "Authorization: Bearer sk-your-strong-string" | head -c 400

A quick tip: If something else is running on port 4000, you can check using this command before running docker compose. You could either stop that process or run LiteLLM gateway on a different port but will have to adjust rest of the tutorial accordingly.

lsof -nP -iTCP:4000 -sTCP:LISTEN

The proxy is up. Now we will look at the Codex side.

4. Configure Codex to use LiteLLM

~/.codex/config.toml

It's a good idea to back up the existing codex config file first.

cp ~/.codex/config.toml ~/.codex/config.toml.bak 2>/dev/null || true

Then edit the codex config file as below:

model = "gpt-5.6-terra"
model_provider = "litellm"
model_reasoning_effort = "medium"

[model_providers.litellm]
name = "LiteLLM Gateway"
base_url = "http://localhost:4000/v1"
# More on setting up LITELLM_API_KEY towards the end
env_key = "LITELLM_API_KEY"
# Codex talks the Responses API. 
wire_api = "responses"
# A coding turn can stream for long time, hence the setting below
stream_idle_timeout_ms = 7200000
stream_max_retries = 5
request_max_retries = 4
# Codex cli complains if not in a trusted folder
[projects."/path/to/your/project"]
trust_level = "trusted"

model_provider = "litellm" . Has to match the [model_providers.<name>] table below it. Any name works, as long as they both match.

wire_api = "responses" . Codex uses the Responses API, whereas the default for a custom provider is chat completions.

5. Run Codex

codex exec "reply with the word CONNECTED and nothing else"

Codex is now runs against your proxy, on your provider key, with your model.

Or you can start a codex session in the terminal with:

codex

Open http://localhost:4000/ui, sign in with the master key, and the request appears in the Logs tab.

6. Use the Codex desktop app

The Codex desktop app reads the same ~/.codex/config.toml. Nothing new to write.

But it will not see your environment variable. macOS hands apps launched from Finder or the Dock the GUI login session's environment, not your shell's. You can run this command to set env variable for the login session:

launchctl setenv LITELLM_API_KEY "sk-your-strong-string"

Then fully quit and reopen the codex app. That holds until you log out. Make it a LaunchAgent if you want it permanent.

Advanced options

Manage usage with virtual keys

Clients get a virtual key, issued from the Virtual Keys page in the admin UI at http://localhost:4000/ui, with its own budget and its own list of allowed models.

Issue one per person, per machine or per project, then set LITELLM_API_KEY to that key instead:

export LITELLM_API_KEY=""

Three things follow from that. Spend in the Logs tab is broken down by key, so you can see which project is costing what. A budget on the key stops it rather than surprising you at the end of the month. And if a key leaks, you can revoke that one key.

Using different models

Any public name in the model list works:

codex -m gpt-5.6-luna

Profiles are saved model and reasoning-effort pairs, that can get you started with quickly.

In your ~/.codex/config.toml add:

# Optional profiles 
# Saved model plus reasoning-effort pairs. `codex --profile deep`.
[profiles.fast]
model = "gpt-5.6-luna"
model_reasoning_effort = "low"

[profiles.deep]
model = "gpt-5.6-sol"
model_reasoning_effort = "high"

Now you can run a codex session with a specific profile.

codex --profile deep

Additional resources

LiteLLM docs: https://docs.litellm.ai
Codex tutorial page: https://docs.litellm.ai/docs/tutorials/openai_codex
LiteLLM on GitHub: https://github.com/BerriAI/litellm
Video walkthrough: Watch it on YouTube