How to balance AI cost with user intent (1)

You don't always select the best model for users when they simply want to ask "Hi, what is your model?"

Therefore, you want to build a LLM router inside your AI app, which can save tons of premium tokens output for simple questions.

An AI LLM router is fundamentally a decision engine that maps:

user intent + constraints + economics + risk + context
→ optimal model/tool/workflow execution path

Core Architecture

The core architecture of an AI router is

User Input Gateway

1. authentication

Token Introspection & Cryptographic Verification: Validate incoming JWTs or API keys at the API gateway layer (e.g., Envoy, Kong, or a custom Rust/Go middleware) using RS256/EdDSA public keys.

1. rate limiting

Context Hydration: Upon validation, extract the tenant_id, user_id, and tier (e.g., free, premium, enterprise). Inject these into the request context header (X-Tenant-ID, X-User-Tier). This tiering is crucial for downstream routing.

Distributed Rate Limiting: Implement a Redis-backed Token Bucket or Leaky Bucket algorithm to prevent cascading failures.

Multi-Dimensional Limits: Evaluate rate limits across three concurrent vectors:

IP/User ID: Prevents DDoS and brute-force abuse.

Tier-based Concurrency: Limits maximum active concurrent requests (e.g., 50 concurrent requests for Premium, 5 for Free).Estimated Token Throughput: A rough, pre-routing heuristic calculation (e.g., based on input character count: $\text{characters} \times 0.25$) to drop requests that exceed a tenant's rolling minute-by-minute token quota before hitting expensive models.

1. request normalization

Strict Serialization: Enforce an OpenAPI 3.0/JSON Schema validation layer using high-performance validators (e.g., Pydantic in Python, Ajv in Node.js).

Structure Alignment: Sanitize heterogenous client payloads into a strict, unified internal schema (e.g., mapping various client formats into a standardized {"messages": [{"role": "user", "content": "..."}], "generation_params": {}} structure).

1. streaming

Protocol Management: Support both Server-Sent Events (SSE, text/event-stream) and WebSockets for duplex or persistent connections.

Asynchronous Proxying: The gateway must support HTTP chunked transfer encoding, allowing it to proxy tokens from downstream LLMs to the client with sub-10ms overhead. This requires an asynchronous network stack (e.g., Tokyo/Rust or Asyncio/Python) to hold open connections without thread exhaustion.

1. session tracking

Session Windowing: Extract a session_id or conversation_id. If missing, generate a UUIDv4.

State Hydration: Query a low-latency cache (Redis or Aerospike) to fetch the sliding window history of the conversation.

Context Windows Management: Track total accumulated tokens in the session history. If the history exceeds a threshold, execute a deterministic truncation or summarization strategy before passing it to the preprocessing layer, saving downstream parsing overhead.

1. metadata enrichment

Network & Client Fingerprinting: Inject metadata including client_geo (via MaxMind GeoIP), device_type, request_timestamp_utc, and network latency metrics.

Routing Hints: Append historical model preferences for this specific user (e.g., if the user always toggles "Creative Mode", append a metadata flag preferred_track: creative).

Preprocessing Layer

To prevent noisy, malformed inputs from degrading embedding quality or tricking downstream intent classifiers, the text must be cleaned deterministically.

Typo Correction (Heuristic vs. Light Neural):

Avoid heavy LLMs here due to cost. Use a fast, symmetric delete spelling correction algorithm (SymSpell) or a lightweight, localized character-level sequence-to-sequence model for domain-specific terms.

Purpose: Correcting "fiannce" to "finance" significantly stabilizes downstream embedding vectors.

Language Detection:

Route text through ultra-fast, compiled sub-millisecond classifiers like FastText (Facebook) or CLD3.

Routing Relevance: Non-English inputs might require specialized multilingual models (e.g., Cohere, Qwen, or deep-seek variants) that possess superior tokenization efficiency and semantic understanding for that specific language, minimizing token bloat caused by poor subword tokenization.

Formatting Cleanup:

Regex-based stripping of control characters, unwanted ANSI escape codes, duplicate whitespaces, and systematic markdown/HTML stripping (unless structural layout is explicitly needed).

Normalize unicode normalization forms (e.g., converting to NFKC) to ensure identical characters have identical byte representations.

Domain Detection:

Run the input text through a fast, linear classifier or a lightweight distilled transformer (e.g., a fine-tuned DistilBERT) trained on your target domains (e.g., code, legal, medical, general_chitchat).

Cost Balancing: If the domain is general_chitchat, the router immediately flags this for a sub-billion parameter model ($0.07/M tokens), completely bypassing frontier models.

Entity Recognition (NER):

Deploy a high-throughput spaCy pipeline or a fine-tuned GLiNER model to extract named entities (ORG, PRODUCT, GPE, DATE).

PII Masking & Compliance: If sensitive PII (e.g., SSN, CREDIT_CARD) is detected, trigger token masking/anonymization filters before the request is sent to external, third-party closed-source APIs.

User Intent Hints:

Utilize pattern-matching state machines combined with a multi-label classification head to detect explicit structural intents: requires_calculation, requires_code_generation, complex_reasoning, factual_retrieval.

Example: If the intent matches requires_calculation, the router can flag the request to bypass standard LLMs and route directly to an agent with a Python REPL tool, or a model with native tool-calling optimizations.

Embeddings Generation:Pass the normalized text to a highly efficient, local embedding model running on a dedicated inference server.

Calculate deterministic lexical metrics: Token Count (using specific tokenizers like TikToken for OpenAI, or LlamaTokenizer), character-to-token ratios, sentence count, tree-depth parsing metrics, and the frequency of structural markers (e.g., brackets, assignment operators =, SQL keywords).

Extract abstract conceptual indicators: Sentiment polarity score (via VADER or high-speed transformer), perplexity estimation (using a tiny causal model like GPT (to minimize cost) to evaluate how "predictable" or standard the user's phrasing is), and information density (ratio of content words to stop words).