DEV Community

Cover image for Set up LiteLLM Auto Router: send each prompt to the right model
Misbah Syed
Misbah Syed

Posted on Originally published at misbahsy.hashnode.dev on

Set up LiteLLM Auto Router: send each prompt to the right model

#autorouter #litellm #gateway

Most AI apps let you pick one model, and then every query goes to it. "What's the capital of France?" goes to that model. "Find the race condition in our billing service" goes to the same one. Your simplest and hardest queries cost the same, so you pay frontier prices for the easy ones.

Auto Router fixes that at the gateway. You add it to LiteLLM as a model, give it a name like smart-router, and your app calls that name like any other model. For each request, the router reads the prompt, puts it in one of four tiers (SIMPLE, MEDIUM, COMPLEX, REASONING) and sends it to the model you mapped to that tier. Simple prompts go to a cheap model and complex ones go to a frontier model.

Your app doesn't change. The routing lives in one block of config.yaml, so when a newer model comes out, swapping it in is a one-line change.

This guide builds that config step by step and tests it with four curl requests.

If you would like to watch this tutorial as a video, please follow the YouTube link.

Prerequisites

  • uv

  • An OpenAI API key and an Anthropic API key

  • curl and openssl

  • Port 4000 free on your machine

Step 1: install LiteLLM

If you don't have uv yet:

curl -LsSf https://astral.sh/uv/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

Install the gateway as a uv tool. This guide was tested on 1.100.1, so pin it:

uv tool install 'litellm[proxy]==1.100.1'
litellm --version
Enter fullscreen mode Exit fullscreen mode

If the shell can't find litellm, run uv tool update-shell and open a new terminal.

Step 2: create the project and a strong master key 

mkdir auto-router && cd auto-router
Enter fullscreen mode Exit fullscreen mode

The master key is what every client uses to call your gateway, so treat it like a password. The docs use sk-1234 as a placeholder. Don't use that, or anything a person or an agent could guess. Generate a random one:

echo "sk-$(openssl rand -hex 32)"
Enter fullscreen mode Exit fullscreen mode

Create .env and paste your values in:

# .env
OPENAI_API_KEY="your-openai-key"
ANTHROPIC_API_KEY="your-anthropic-key"
LITELLM_MASTER_KEY="sk-<the 64 hex characters you generated>"
Enter fullscreen mode Exit fullscreen mode

Lock it down, keep it out of git, and load it into your shell:

chmod 600 .env
echo ".env" >> .gitignore
set -a && source .env && set +a
Enter fullscreen mode Exit fullscreen mode

Step 3: add your models

Create config.yaml with a plain LiteLLM model list:

model_list:
  - model_name: gpt-5.6-luna
    litellm_params:
      model: openai/gpt-5.6-luna
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-5.6-terra
    litellm_params:
      model: openai/gpt-5.6-terra
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: claude-opus-5
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY
Enter fullscreen mode Exit fullscreen mode

os.environ/NAME reads the key from the environment, so no secret lives in the YAML.

Step 4: add the router and map the tiers

Append a fifth entry to model_list:

  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE: gpt-5.6-luna
          MEDIUM: gpt-5.6-terra
          COMPLEX: claude-sonnet-5
          REASONING: claude-opus-5
Enter fullscreen mode Exit fullscreen mode

This alone is a working router. It scores each prompt locally from signals like length, code and reasoning phrases, then picks a tier. No extra model call is made. Each tier value is a model_name from step 3.

Step 5: add keyword rules

Some requests need a strong model no matter how they score. Add this under complexity_router_config:

        keyword_tier_rules:
          - keywords: ["security review", "incident"]
            tier: COMPLEX
Enter fullscreen mode Exit fullscreen mode

Any prompt containing "security review" or "incident" now goes to COMPLEX without scoring. Set tier: REASONING if you want the top tier instead.

Step 6: let an LLM classify the unclear prompts

Add under complexity_router_config:

        classifier_type: heuristic_first
        heuristic_first_max_tier: SIMPLE
        classifier_llm_config:
          model: gpt-5.6-luna
          timeout_ms: 3000
          classification_rubric: agentic
        classifier_fallback: heuristic
        classifier_context_window_size: 3
Enter fullscreen mode Exit fullscreen mode

  • heuristic_first with heuristic_first_max_tier: SIMPLE: if the local scorer is confident a prompt is SIMPLE, it routes right away. Anything else goes to the classifier.

  • classifier_llm_config.model: the small model that does the classifying.

  • classifier_fallback: heuristic: if the classifier call fails or times out, the router uses the local score.

  • classifier_context_window_size: 3: the classifier sees the last three turns, so a follow-up like "now do the same for refunds" is judged in context.

Step 7: default model, escalation and housekeeping

Add under complexity_router_config:

        default_model: gpt-5.6-terra
        escalation_keywords: ["LITELLM ESCALATE"]
        route_housekeeping_to_cheapest_tier: true
Enter fullscreen mode Exit fullscreen mode

  • default_model: used when there is nothing to classify.

  • escalation_keywords: a user or agent can include this phrase to push a request up one tier. It is case-sensitive.

  • route_housekeeping_to_cheapest_tier: coding agents rename sessions by sending the whole conversation and asking for a title. That text looks like hard work to a scorer. This sends those calls to the cheapest tier.

Step 8: gateway settings

At the top level of config.yaml, below model_list:

litellm_settings:
  drop_params: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
Enter fullscreen mode Exit fullscreen mode

drop_params: true drops request parameters a provider doesn't support instead of failing the call. Coding agents often send those. The master key is read from the environment, not written in the file.

The full config.yaml 

model_list:
  - model_name: gpt-5.6-luna
    litellm_params:
      model: openai/gpt-5.6-luna
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-5.6-terra
    litellm_params:
      model: openai/gpt-5.6-terra
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: claude-opus-5
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE: gpt-5.6-luna
          MEDIUM: gpt-5.6-terra
          COMPLEX: claude-sonnet-5
          REASONING: claude-opus-5

        keyword_tier_rules:
          - keywords: ["security review", "incident"]
            tier: COMPLEX

        classifier_type: heuristic_first
        heuristic_first_max_tier: SIMPLE
        classifier_llm_config:
          model: gpt-5.6-luna
          timeout_ms: 3000
          classification_rubric: agentic
        classifier_fallback: heuristic
        classifier_context_window_size: 3

        default_model: gpt-5.6-terra
        escalation_keywords: ["LITELLM ESCALATE"]
        route_housekeeping_to_cheapest_tier: true

litellm_settings:
  drop_params: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
Enter fullscreen mode Exit fullscreen mode

Step 9: start the gateway 

LITELLM_LOG=INFO litellm --config config.yaml --port 4000
Enter fullscreen mode Exit fullscreen mode

LITELLM_LOG=INFO prints a routing decision line for every request, so you can watch the router work in this terminal.

Step 10: send four prompts

Open a second terminal in the auto-router folder and load the key there too:

cd auto-router
set -a && source .env && set +a
Enter fullscreen mode Exit fullscreen mode

The response body still says "model": "smart-router". The model that answered comes back in the x-litellm-model-name header, with its cost in x-litellm-response-cost. Each command below prints only those headers. Remove the | grep ... part to see the full response.

The outputs are from one run. Your costs will differ.

A simple question 

curl -si http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "smart-router", "messages": [{"role": "user", "content": "What is the capital of Japan? Keep it brief."}]}' \
  | grep -iE '^x-litellm-(model-name|response-cost|classifier-cost):'


x-litellm-model-name: openai/gpt-5.6-luna
x-litellm-response-cost: 9.4e-06
Enter fullscreen mode Exit fullscreen mode

In the gateway terminal:

ComplexityRouter: routing decision cause=heuristic_first_short_circuit, tier=SIMPLE, score=-0.150, signals=('short (11 tokens)', 'simple (what is, brief)'), routed_model=gpt-5.6-luna
Enter fullscreen mode Exit fullscreen mode

The scorer was confident, so it went to the cheap model with no classifier call.

A keyword match 

curl -si http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "smart-router", "messages": [{"role": "user", "content": "Do a security review of this login handler: it compares the password with == and logs the request body."}]}' \
  | grep -iE '^x-litellm-(model-name|response-cost|classifier-cost):'


x-litellm-model-name: anthropic/claude-sonnet-5
x-litellm-response-cost: 0.010412000000000001


ComplexityRouter: routing decision cause=literal_keyword_match, escalated=False, tier=COMPLEX, routed_model=claude-sonnet-5
Enter fullscreen mode Exit fullscreen mode

"security review" matched the rule from step 5.

A prompt the classifier decides 

curl -si http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "smart-router", "messages": [{"role": "user", "content": "Our billing service double charges about one customer a week. Figure out where retries and webhooks could race, and propose a fix we can ship without downtime."}]}' \
  | grep -iE '^x-litellm-(model-name|response-cost|classifier-cost):'


x-litellm-model-name: anthropic/claude-sonnet-5
x-litellm-response-cost: 0.019232000000000003
x-litellm-classifier-cost: 0.0002036


ComplexityRouter: routing decision cause=llm_classifier, tier=COMPLEX, score=n/a, signals=('llm-classifier:COMPLEX',), routed_model=claude-sonnet-5
Enter fullscreen mode Exit fullscreen mode

No keyword matched and the scorer wasn't sure, so gpt-5.6-luna classified it as COMPLEX. That's the only request with a classifier cost. The classifier is a model call, so another run can pick a different tier.

A housekeeping call 

curl -si http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "smart-router", "messages": [{"role": "user", "content": "You are coming up with a succinct title for a coding session. Session: refactor the distributed payment reconciliation architecture, fix concurrency bugs in the async webhook handler, add benchmarks."}]}' \
  | grep -iE '^x-litellm-(model-name|response-cost|classifier-cost):'


x-litellm-model-name: openai/gpt-5.6-luna
x-litellm-response-cost: 2.38e-05


ComplexityRouter: routing decision cause=housekeeping, tier=SIMPLE, score=n/a, signals=('housekeeping',), routed_model=gpt-5.6-luna
Enter fullscreen mode Exit fullscreen mode

The prompt is full of words that would score high, but it's a session title request, so it went to the cheapest tier.

Start simple

Try the router with the prompts you send from your AI tools and coding agents. Start with only the tiers from step 4. Add keyword rules for the requests you care about, then add the LLM classifier once you need better decisions on unclear prompts.

The full option reference is on the Auto Routing docs page, and the code is in BerriAI/litellm.

Additional resources

Top comments (0)