LLM Routers- Make Good Choices!

We’re all trying to keep our costs down in the world of Generative AI. It’s easy to start utilizing GenAI for more and more of its capabilities. It’s also easy to assume what you are working on is so important that you should use a cutting edge frontier model. But then you get a budget alert, a surprise charge, or hear from your manager about the overage this month. Oops.

How can we keep our costs down? One of the things many companies are experimenting with is an LLM router. These routers intercept the users’ requests and route it to the most appropriate LLM. If the request is simple, a cheaper, less capable model can handle it. If you are asking for some deep research into a complex topic, it may need to go to a more expensive, more capable model. The LLM router can direct traffic, not just based on how simple the request is, but also other considerations. We can base the routing on cost, latency, quality, business rules, or many combinations.

There are a number of available pre-built tools that can perform these tasks. OpenRouter and LiteLLM are two popular tools that can do this. OpenRouter is a hosted tool and LiteLLM can be either hosted or self hosted. They have varying costs, capabilities and limitations.

I wanted to build out an LLM router myself in multiple ways.

Minimalist LLM Router

First, I built a simple LLM router that could analyze the complexity of a user’s question and route to a locally hosted LLM (Ollama) for simple queries or a Bedrock model for more complex queries — https://github.com/mgbec/Local-with-Observability. It uses a classifier that analyzes the users’ prompts and tries to figure out if it can use the local model, or if it needs to escalate to Bedrock.

The classifier scores each prompt on a 0–100 scale using four weighted heuristics. If the score meets or exceeds the threshold (50), the prompt routes to Bedrock; otherwise it goes to Ollama.

The four signals:

1. Token count (30% weight) score map:

   ≤100 tokens → 0–30 (linear)

   100–500 tokens → 30–70 (linear interpolation)

   ≥500 tokens → 70–100

2. Reasoning keywords (25% weight) Checks if the user’s message contains words like “analyze”, “compare”, “explain why”, “step by step”, “trade-offs”, “debug”, “optimize”, “refactor”, “architecture”.

3. Code detection (25% weight) Looks for patterns indicating code-related requests:

   Markdown code fences

   Programming keywords (function, class, def, import, etc.)

   Language names (Python, Javascript, Rust, etc.)

   Generation phrases (“write a function”, “implement a script”, etc.)

4. Context depth (20% weight) Based on conversation length (number of messages in the array):

   1 message → 0

   2–3 messages → 30

   4–6 messages → 60

   7+ messages → 60+ (scales up by 10 per message)

How they combine:

final = 0.30 × token_score + 0.25 × keyword_score + 0.25 × code_score + 0.20 × context_score

This LLM router uses Grafana, Prometheus, Loki, Tempo for observability. The local LLM router and observability stack is great for simple use cases, and will help me save tokens. It has some really basic authentication and it is easy to configure if I want to test a different model, or a different classifier. However, I wanted to add a few more production grade features and my local router(-AKA- decrepit old laptop) will not stand up to that.

I Need More…

To get enhanced features, I built an agent based LLM router using Bedrock AgentCore — https://github.com/mgbec/LLM-Router-deployed-to-AWS I used several components of Bedrock AgentCore- runtime and gateway, as well as some other AWS services. Some of the added functionality is:

AgentCore Gateway

• Inference targets: Routes model requests across multiple providers (Bedrock, SageMaker endpoints, external APIs) through one endpoint. This is the native AWS mechanism for multi-provider model routing.

• Ingress/egress authentication: Handles OAuth2 flows for external providers, IAM for AWS services, and API key management — no credential handling in application code.

• Semantic tool selection: As the router agent’s toolkit grows, Gateway helps it discover the right tools based on task context.

• Protocol translation: Converts between MCP, OpenAPI, and direct HTTP for seamless provider interoperability.

AgentCore Runtime

The Router agent is a containerized agent hosted on AgentCore Runtime. It implements the routing decision logic. The Router Agent itself uses a fast, cheap model (Nova Lite or Haiku) to make routing decisions. The expensive models can be saved for user requests, dependent on routing policies and strategies. There are quite a few different ways to route to LLM models and it would be good to change things around easily.

Routing Policies

In some instances we might be constrained by cost, latency, quality or model choices. The routing policy is stored in DynamoDB and meant for a per-tenant or per-tier differentiators. For example, if I have a project that needs to only use certain models, we could specify that here. Examples:

• Maximum cost per request
• Maximum acceptable latency
• Minimum quality threshold
• Which models are allowed
• Model weights (how much to prefer one model over another)

Routing Strategies

So once we take into consideration the policy, how do we get there? That is where strategy comes into the picture. In my local llm router I used the complexity of the user prompt to decide whether to use the local model or a Bedrock model. Here we can use other tactics, like:

• Complexity-based: Classify the prompt, match to a model tier
• Cost-optimized: Always pick the cheapest model that meets the quality floor
• Latency-optimized: Pick the fastest provider regardless of cost
• Quality-maximized: Always pick the most capable model
• Cascade: Try cheap first, escalate if confidence is low
• Round-robin: Spread traffic for A/B testing

So the relationship is: a policy says “use the cascade strategy, with a $0.05 budget, 0.8 quality floor, and these models are available.” The strategy then runs that algorithm within those bounds.

Different tenants can have different policies pointing to different strategies, or the same strategy with different constraint values.

Authentication

Cognito ends up being the quiet workhorse of this project with both user-facing authentication and service to service authentication. For user-facing authentication it protects all /v1/* endpoints. Only /health is unauthenticated. The service to service authentication is needed between AgentCore Gateway and the Lambda tools to make sure they are being called by an authorized service.

API Gateway and API endpoints

API Gateway validates authentication, applies rate limits, and forwards them to AgentCore Gateway. API Endpoints- we have the typical core endpoints, and also some that help us with our ISO 42001 A.8 and A.9.5 goals.

Transparency & Explainability (A.8)

• Mandatory Headers : Every response includes X-AI-Model, X-AI-Provider, X-AI-Routed, and X-AI-Disclosure headers informing users of AI involvement.

• Explain API : GET /v1/routing/explain/{requestId} returns why a specific model was chosen — classification factors, candidate scores, and human-readable explanation.

• User Audit Log : GET /v1/audit/my-requests lets users see which models served their requests over the past 90 days.

• Model Cards : GET /v1/models/info returns capabilities, limitations, known biases, and data residency info for all models in the pool.

Human Oversight (A.9.5)

• Kill Switch : AppConfig feature flags allow operators to instantly disable the entire system, individual providers, or specific models — no deployment required.

• Override API : Operators can pin models, block models, or require human review for specific categories via POST /v1/admin/override.

• Concern Reporting : Users can report problematic outputs via POST /v1/concerns/report, which queues them for human review with SLA tracking (4h critical, 24h standard).

• Review Queue : DynamoDB + SQS queue for flagged items with CloudWatch alarm if backlog grows beyond threshold.

• Escalation : Critical concerns trigger SNS notification to ops team immediately.

Sync and Async

We need both synchronous and asynchronous since the more complicated questions were hitting a Lambda timeout when going to Opus. The router automatically decides whether to process synchronously or asynchronously:

Auto-detection uses heuristics: prompts with multiple complexity indicators (design, prove, algorithm, pseudocode, etc.) and sufficient length automatically dispatch to the async path.

This particular async implementation was used because it is fast, scalable, simple, and doesn’t need a public port. It’s also cost effective with DynamoDB TTLAutomatic Cleanup. You can use DynamoDB’s TTL feature to automatically delete old job statuses after a few days. For more details, see https://github.com/mgbec/LLM-Router-deployed-to-AWS/blob/main/architecture/async-processing.md

Adding New Models

Part of the fun of using Bedrock is all the different models and model versions you are able to use. Additionally, your model version may become legacy and you will want to move away from it for both functionality and cost concerns. https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle.html

There are five files you will want to edit when you change models. I put more specifics in my GitHub repo, but these are the files:

agent/app.py

terraform/appconfig.tf

terraform/iam.tf

terraform/governance.tf

lambda/transparency_api/index.py

Hot-Swapping Configuration

The router reads feature flags from AWS AppConfig every 30 seconds and changes can take effect without any deployment or restart. There are both routing and kill switch flags you can use. More details available in the GitHub repo.

Kinesis Model Weight Adjustment

After each successful model invocation, the agent puts a record to Kinesis which triggers a Lambda. The Lambda grabs metrics and adjusts model weights in the DynamoDB table.

Observability

AWS makes all things observability really easy and these are some of the items implemented in this project:

• X-Ray: Cross-service distributed tracing with trace group, environment-aware sampling rules, and insights enabled
• CloudWatch Dashboard: 7 panels — requests/model, latency p50/p95/p99, cost/model, quality scores, escalations, circuit breakers, complexity distribution
• Alarms: Error rate, latency (p99 > 5s), cost spikes, circuit breaker opens, human review backlog
• Audit Logs: Routing audit (90-day), data flow log (90-day), human override log (90-day)
• Kinesis: Real-time routing event stream feeding the adaptive weight-adjustment Lambda (both sync and async paths publish)
• AgentCore Native: Built-in OpenTelemetry instrumentation (auto-enabled, no config needed)
• Provenance/Lineage: Every routing decision writes a full lineage record to DynamoDB (see Data Provenance below)

Security and Governance

Security and governance are huge factors and I tried to bake in some elements of ISO 42001 Compliance. There are still pieces that would need to be built out for full compliance. See https://github.com/mgbec/LLM-Router-deployed-to-AWS/blob/main/architecture/iso-42001-gap-analysis.md for the full control mapping and gap analysis.

The /v1/concerns/report API endpoint gives us a reporting channel that meets requirements for ISO 42001 A.3.3. The SQS queue and SNS escalation with SLA tracking and CloudWatch alarms provide data tracking for continuous improvement, as well as investigation, and auditing, more components for A.3.3.

Some example output for the transparency API, which provides components of ISO 42001 A.8:

Data Provenance and Lineage (ISO 42001 A.7.6)

This is something I have never built into a workflow before, so I was very intrigued by the concept. Data and model genealogy, hope we don’t have any awkward surprises.

Every routing decision, both sync and async writes a provenance record to the routing-audit-log DynamoDB table. This provides full lineage tracking for compliance audits.

The data from the provenance log ends up in a DynamoDB and can be queried. There is a script(view-audit-log.py) that produces the following sort of report:

Auditor Access

There is a dedicated read-only IAM role (llm-router-dev-auditor-role) provisioned for ISO 42001 compliance reviews. You can provision the auditor role through Terraform and it will have the following access:

This role cannot:

• Modify any resources or configuration
• Invoke models or send requests through the router
• Access raw user prompts (only SHA-256 hashes stored)
• Change routing policies, feature flags, or kill switch state
• View secrets or credentials

More Cost Savings

One of the ways to save money on token costs is this selective routing process, but there are other ways to save as well.

• Prompt Caching: Many providers support prompt caching, which lets you store stable, frequently used prompts (like system instructions or API documentation) at steep discounts (often up to 90% off). System instructions at the very beginning of the prompt may maximize cache hits.

https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html

https://platform.claude.com/docs/en/build-with-claude/prompt-caching

https://developers.openai.com/api/docs/guides/prompt-caching

• Compress Context: Instead of feeding an entire conversation history or a massive document to the model at each step, try summarization and condense older conversation into short summaries. Replace raw transcripts with structured memory variables (e.g., passing a JSON object of current variables instead of a 50-message chat history).

• Optimize Tools and Outputs: Only load the specific tools needed for the immediate step rather than loading all tools simultaneously. Set a max_tokens limit, and instruct agents to be concise rather than generating wordy explanations.

• Compile Workflows into Code: If your agent repeatedly goes through a fixed workflow, break out the deterministic steps directly in your application code, so you can reduce orchestration and token usage.

So, this is not a perfectly ISO 42001 compliant system, but we’re on the road anyways. The pace of innovation is crazy and we can already see changes that will need to be made as new versions of protocols roll out, as soon as 7/28/2026.
https://blog.modelcontextprotocol.io/posts/2026-07-28-release-candidate/. Enjoy the ride!