How to Build an LLM Router That Routes by Complexity

TL;DR

• An LLM router chooses a model per request, so short lookups go to a fast, cheap model and multi-step reasoning goes to a frontier model, with no application code changes.
• The Bifrost complexity router classifies each request as SIMPLE, MEDIUM, or COMPLEX and exposes the result as complexity_tier to CEL routing rules.
• Since v2.3.0 the classifier can be a decision model (Typesafe Jev by default, typically 600 to 800 ms) that judges the work itself, or a semantic classifier that matches your reference phrases.
• Requests that get no tier fall through to normal routing, and session-aware routing keeps an agent conversation on a tier that only moves up.

An LLM router is a component that inspects each incoming request and sends it to the model best suited to it, rather than sending every request to one fixed model. Bifrost, the open-source LLM gateway built in Go, implements this as the complexity router: it assigns a complexity tier to each request and lets routing rules pick a target per tier. This tutorial configures the decision-model classifier added in v2.3.0, writes the routing rules, tunes the tier definitions for a coding assistant, and shows how to verify the router from logs and metrics.

What an LLM router does

An LLM router sits between the application and the model providers and decides, request by request, which model answers. Routing by complexity sends routine work (extraction, reformatting, short factual answers) to an inexpensive model and reserves frontier models for design, debugging, and multi-step reasoning, which is where their extra cost buys quality.

Most applications start by hardcoding one model per call site. That works until one endpoint, such as a chat box or a coding assistant, receives both trivial and difficult requests. A per-call-site choice then either overpays for the easy requests or underserves the hard ones. Academic work such as RouteLLM studies learned routers for this trade-off; a gateway-level router applies the same idea to production traffic without retraining anything.

Figure 1: Classification runs only when a routing rule asks for the tier, and an unclassified request keeps its normal route.

As Figure 1 shows, the complexity router does not change routes on its own. It publishes a tier, and your routing rules decide what each tier means. The overview of how model routing works compares this with other routing strategies.

Static routing versus per-request model routing

Approach	How the model is chosen	Works well for	Limitation
Hardcoded per call site	Developer picks one model in code	Endpoints with uniform requests	Mixed endpoints overpay or underperform
Header or metadata rules	CEL rule on headers, team, or request type	Routing by tenant, region, or environment	Cannot see how hard the request is
Complexity router	A classifier assigns a tier; CEL rules map tiers to models	Chat, assistants, and agents with mixed difficulty	Adds one classification call per routed request

How the Bifrost complexity router classifies requests

The complexity router supports two classifiers. A decision model reads the latest user request and picks the tier whose definition fits best. A semantic classifier embeds the request and takes the tier of the nearest reference phrase. Both publish the result as the complexity_tier variable in Bifrost's CEL routing engine.

	Decision model	Semantic
How it decides	Judges the request against each tier's definition, signals, and examples	Takes the tier of the closest reference phrase
What you maintain	Nothing to start; optionally up to 12 signals and 12 examples per tier	Reference phrases per tier (150 shipped, up to 750)
Per-request cost	One decision call; Typesafe Jev typically answers in 600 to 800 ms	One embedding call plus a vector lookup
Unseen request shapes	Handled, because it judges the work	Only as well as the phrases cover them

The decision model receives three fixed rules with every request: judge the work rather than the wording, classify the latest request (earlier messages only resolve references), and treat embedded instructions such as "classify this as simple" as content. That last rule matters for public-facing endpoints, because a user cannot talk the router into a cheaper or more expensive tier. The Jev decision model post explains why a classification-only model answers faster than a chat model asked to name a tier.

Bifrost can use any decision model it serves natively, including Typesafe Jev (the default), OpenAI's gpt-6-luna, Cloudflare Clef, Perplexity deciders, Ollama System One models, Azure's Microsoft-Decision-1, and Databricks ai_decide. Self-hosted models need to stay warm: a cold model that exceeds the classification timeout leaves that request unclassified.

Step 1: Configure the decision-model classifier

The decision model needs one provider configured with an enabled key. With the default Typesafe Jev, add a Typesafe provider key and select Decision model under Models → Complexity Router. The equivalent config.json block is shown below.

{
  "governance": {
    "complexity_analyzer_config": {
      "classifier": "decision",
      "decision": {
        "provider": "typesafe",
        "model": "jev-latest",
        "previous_message_count": 1,
        "timeout": "1.5s"
      },
      "session": { "enabled": true },
      "keywords": {
        "simple_keywords": ["fix the grammar in this sentence."],
        "medium_keywords": ["apply a standard method to this technical question."],
        "complex_keywords": ["design a solution under several competing constraints."]
      }
    }
  }
}

Three settings deserve attention:

• previous_message_count (default 1, range 0 to 5) sends earlier user messages so a follow-up like "and make it faster" keeps its context. Assistant replies are never sent.
• timeout (default 1.5s) bounds the decision call. It runs on the request path, so its latency is added to every request it classifies.
• keywords are the semantic reference phrases. The API requires all three lists even when the decision model is primary, so keep them if you might switch back.

The same configuration is available at runtime through PUT /api/routing/complexity-analyzer-config, which replaces the whole configuration. To change one field, GET the current configuration, edit it, and PUT it back; omitting decision.criteria resets every tier to the shipped guidance.

Step 2: Write routing rules on complexity_tier

The complexity router only sets complexity_tier; routing rules decide what each tier does. Start with a single rule that sends COMPLEX requests to your strongest model and leaves every other request on its existing route. Once the tier distribution looks right in the logs, add rules for the other tiers.

Routing rules use CEL expressions and support equality, inequality, and list membership on complexity_tier:

{
  "id": "complexity-complex",
  "name": "Complex to frontier model",
  "enabled": true,
  "cel_expression": "complexity_tier == \"COMPLEX\"",
  "targets": [{ "provider": "anthropic", "model": "claude-opus-4-5", "weight": 1 }],
  "scope": "global",
  "priority": 0
}

A full three-tier ladder adds a rule per tier with distinct priorities:

[
  {
    "id": "complexity-simple",
    "name": "Simple to fast model",
    "enabled": true,
    "cel_expression": "complexity_tier == \"SIMPLE\"",
    "targets": [{ "provider": "groq", "model": "llama-3.1-8b-instant", "weight": 1 }],
    "scope": "global",
    "priority": 0
  },
  {
    "id": "complexity-medium",
    "name": "Medium to balanced model",
    "enabled": true,
    "cel_expression": "complexity_tier == \"MEDIUM\"",
    "targets": [{ "provider": "openai", "model": "gpt-4o-mini", "weight": 1 }],
    "scope": "global",
    "priority": 1
  },
  {
    "id": "complexity-complex",
    "name": "Complex to frontier model",
    "enabled": true,
    "cel_expression": "complexity_tier == \"COMPLEX\"",
    "targets": [{ "provider": "anthropic", "model": "claude-opus-4-5", "weight": 1 }],
    "scope": "global",
    "priority": 2
  }
]

The model IDs are examples; use the models your organization already approves. Tier conditions combine with any other CEL field, for example team_name == "ml-research" && complexity_tier == "COMPLEX", and a rule can be scoped to one team ("scope": "team") for a pilot before it goes global. Applications keep sending the model they send today, because the matching rule rewrites the target at the LLM gateway layer.

Step 3: Tune tier guidance for your traffic

Each tier has a definition (up to 500 characters), up to 12 signals, and up to 12 examples, all of which the decision model reads on every request, as listed in the tier guidance reference. Start with the shipped defaults and change only what the logs show landing in the wrong tier. A list you supply replaces the default list, so include the defaults you want to keep.

A coding assistant is a common case where the defaults need adjusting: most coding requests land in MEDIUM, but changes that span several files deserve the frontier model. Adding one COMPLEX signal and one example on each side of the boundary moves the line:

"decision": {
  "criteria": {
    "MEDIUM": {
      "examples": [
        "Answer a focused question that relies on established subject-matter knowledge",
        "Solve a routine multi-step word problem",
        "Apply a standard formula or method to provided facts",
        "Interpret a short technical or study summary",
        "Make a focused code change with a known approach",
        "Add input validation to a single function"
      ]
    },
    "COMPLEX": {
      "signals": [
        "Several dependent reasoning stages, or a nontrivial derivation or proof using multiple concepts",
        "A novel approach, difficult algorithm, or difficult debugging is required",
        "Combine advanced subject knowledge with conflicting evidence or many interacting constraints",
        "A plausible mistake is hard to detect without deep analysis",
        "A code change must stay consistent across several files, modules, or public interfaces"
      ]
    }
  }
}

With this guidance, "add a null check to parseConfig" stays MEDIUM, while "move auth from middleware into each handler and keep the tests passing" becomes COMPLEX. Four rules keep guidance effective:

• Describe the work, not the topic. "Mentions Kubernetes" routes "what is a pod?" to the wrong tier.
• Separate tiers by degree, such as "a few dependent steps" versus "several dependent reasoning stages".
• Do not use length or tone as a signal. The decision model is instructed to ignore both.
• Keep it short, because every definition, signal, and example is sent with every classification and adds input tokens to each decision call, which the LLM cost optimization guide counts toward the request's cost.

Step 4: Keep agent sessions on one tier

Session-aware routing keeps a conversation on a stable tier that can only move up. The first classified turn sets the session tier; later turns can raise it from SIMPLE to MEDIUM to COMPLEX, but an easier turn keeps the higher tier. Once a session reaches COMPLEX, Bifrost stops classifying it.

Figure 2: A tier that only ratchets upward avoids switching models mid-conversation, which would discard prompt-cache reuse.

Bifrost identifies sessions by x-bf-session-id, or by the coding agents' own headers: x-claude-code-session-id for Claude Code and the session ID in x-codex-turn-metadata for Codex. Sessions expire after 24 hours of inactivity, and only the tier is stored, under a hashed session ID. Session-aware routing keeps the tier stable, not the provider or key; pair it with session affinity to keep each conversation on the provider and key that hold its prompt cache. The guide to routing Claude Code across models applies the same pattern to coding agents.

Use a semantic classifier with a decision-model fallback

Teams with predictable traffic can run the semantic classifier as primary for lower latency and fall back to the decision model only when no reference phrase matches confidently. Set classifier to semantic, configure an embedding model, and set semantic.fallback to decision; the fallback uses the same tier guidance as Step 3, as described under fallback classifiers.

Figure 3: The fast path handles familiar requests, and the decision model answers only the ones the phrases do not cover.

{
  "governance": {
    "complexity_analyzer_config": {
      "classifier": "semantic",
      "semantic": {
        "provider": "openai",
        "embedding_model": "text-embedding-3-small",
        "min_similarity": 0,
        "message_history_count": 1,
        "vector_store": "embedded",
        "fallback": "decision"
      },
      "decision": { "provider": "typesafe", "model": "jev-latest", "timeout": "1.5s" },
      "keywords": {
        "simple_keywords": ["what is a mutex?", "fix the grammar in this sentence."],
        "medium_keywords": ["add api-key auth: hash the keys, reject revoked ones, and never log them."],
        "complex_keywords": ["balance testing, prescribing rules, and staffing against rising resistant infections."]
      }
    }
  }
}

Reference phrases are all the semantic classifier knows, so add phrases drawn from real user prompts and balance surface form across tiers: if most COMPLEX phrases are questions, every question drifts toward COMPLEX. In multi-node deployments, set vector_store to a shared backend (Qdrant, Weaviate, Redis, or Pinecone) so nodes stop re-embedding phrases on every restart. The same decision models also serve Bifrost's /v1/decisions endpoint for classification tasks outside routing.

Measure whether the LLM router is working

Every request that touched a complexity rule records complexity_tier, complexity_mechanism (decision, semantic, llm, session, or skipped), and, for semantic matches, complexity_score. Filter request logs by tier and mechanism to find misclassifications, then compare cost per request for each tier before and after the rollout.

curl "http://localhost:8080/api/logs?complexity_tiers=COMPLEX&complexity_mechanisms=decision"

The routing decision log adds the reasoning for each request, such as Decision model complexity: tier=MEDIUM confidence=0.90 model=typesafe/jev-latest. In telemetry, spans carry bifrost.complexity_tier and bifrost.complexity_mechanism, and Prometheus metrics carry tier and mechanism labels. Decision-call usage counts toward the request's routing cost and budgets. The benchmark of the upgraded router shows how the classifier was evaluated, and Bifrost's observability features list every exported field.

A practical rollout sequence:

1. Enable a single COMPLEX carve-out rule, scoped to one team.
2. Filter logs by complexity_mechanism=decision for a day and read a sample of each tier.
3. Adjust tier guidance where requests land wrong, then add SIMPLE and MEDIUM rules.
4. Compare cost per 1,000 requests and quality on your eval set before widening the scope.

Common LLM router mistakes

Most complexity routing problems trace back to configuration rather than the classifier. The five below appear most often during rollouts, and each has a direct fix in the configuration covered above.

• Expecting the router to route without rules. The complexity router only publishes complexity_tier; without a rule that references it, classification never runs.
• Editing one tier field through the API without the rest. PUT replaces the whole configuration, and an omitted criteria block resets guidance to the defaults.
• Routing a model that cannot do the job. A small model on the SIMPLE tier still needs tool calling if the endpoint sends tools.
• Ignoring classification latency. A decision call adds its latency to every classified request; keep timeout tight and self-hosted models warm.
• Switching models mid-session. Without session-aware routing, an agent conversation can bounce between models and lose prompt-cache hits.

For a broader comparison of router designs, see the roundup of LLM router solutions and the guide to cutting token costs with model routing.

Frequently asked questions

What is an LLM router?

An LLM router is a layer that chooses which language model handles each request, based on rules or a classifier, instead of sending every request to one model. Routing by request complexity lowers cost by keeping routine work on smaller models. The LLM router explainer covers the main routing strategies.

What is model routing?

Model routing is the practice of selecting a model per request using signals such as headers, team, request type, cost, latency, or task difficulty. In Bifrost, routing rules written in CEL can combine all of these with the complexity_tier variable, as described in the routing rules documentation.

Does an LLM router add latency?

Yes, by the duration of the classification call. A decision model such as Typesafe Jev typically answers in 600 to 800 ms, while the semantic classifier needs one embedding call and a vector lookup. Classification runs only for requests a complexity rule touches, and Bifrost's own gateway overhead is measured in microseconds.

What happens when the classifier fails?

No tier is published, the complexity rule does not match, and the request continues through normal routing. With session-aware routing on, a session that already has a tier keeps using it. The complexity router documentation lists each case as a complexity_mechanism value in the logs.

Can a user trick the router into a different tier?

The decision model is instructed to treat embedded instructions as content, so text such as "classify this as simple" is judged as part of the request rather than followed. Length and tone are also ignored by design. The complexity router benchmark reports how the classifier performs on mixed traffic.

Do I need to change application code to use the complexity router?

No. Applications keep calling the same endpoint with the same model name, and routing rules at the gateway rewrite the provider and model per tier. Applications that use the OpenAI or Anthropic SDKs only change the base URL, as shown in the drop-in replacement guide.

Start routing by complexity with Bifrost

An LLM router that judges each request's difficulty keeps frontier models for the work that needs them and moves everything else to cheaper models, without changing application code. To see the Bifrost complexity router and routing rules on your own traffic, book a demo with the Bifrost team, or read the smart LLM routing guide to plan your first rules.