DEV Community: XJTLU media

How to balance AI cost with user intent (1)

XJTLU media — Thu, 05 Nov 2026 05:55:27 +0000

You don't always select the best model for users when they simply want to ask "Hi, what is your model?"

Therefore, you want to build a LLM router inside your AI app, which can save tons of premium tokens output for simple questions.

An AI LLM router is fundamentally a decision engine that maps:

user intent + constraints + economics + risk + context
→ optimal model/tool/workflow execution path

Core Architecture

The core architecture of an AI router is

User Input Gateway

authentication

Token Introspection & Cryptographic Verification: Validate incoming JWTs or API keys at the API gateway layer (e.g., Envoy, Kong, or a custom Rust/Go middleware) using RS256/EdDSA public keys.

rate limiting

Context Hydration: Upon validation, extract the tenant_id, user_id, and tier (e.g., free, premium, enterprise). Inject these into the request context header (X-Tenant-ID, X-User-Tier). This tiering is crucial for downstream routing.

Distributed Rate Limiting: Implement a Redis-backed Token Bucket or Leaky Bucket algorithm to prevent cascading failures.

Multi-Dimensional Limits: Evaluate rate limits across three concurrent vectors:

IP/User ID: Prevents DDoS and brute-force abuse.

Tier-based Concurrency: Limits maximum active concurrent requests (e.g., 50 concurrent requests for Premium, 5 for Free).Estimated Token Throughput: A rough, pre-routing heuristic calculation (e.g., based on input character count: $\text{characters} \times 0.25$) to drop requests that exceed a tenant's rolling minute-by-minute token quota before hitting expensive models.

request normalization

Strict Serialization: Enforce an OpenAPI 3.0/JSON Schema validation layer using high-performance validators (e.g., Pydantic in Python, Ajv in Node.js).

Structure Alignment: Sanitize heterogenous client payloads into a strict, unified internal schema (e.g., mapping various client formats into a standardized {"messages": [{"role": "user", "content": "..."}], "generation_params": {}} structure).

streaming

Protocol Management: Support both Server-Sent Events (SSE, text/event-stream) and WebSockets for duplex or persistent connections.

Asynchronous Proxying: The gateway must support HTTP chunked transfer encoding, allowing it to proxy tokens from downstream LLMs to the client with sub-10ms overhead. This requires an asynchronous network stack (e.g., Tokyo/Rust or Asyncio/Python) to hold open connections without thread exhaustion.

session tracking

Session Windowing: Extract a session_id or conversation_id. If missing, generate a UUIDv4.

State Hydration: Query a low-latency cache (Redis or Aerospike) to fetch the sliding window history of the conversation.

Context Windows Management: Track total accumulated tokens in the session history. If the history exceeds a threshold, execute a deterministic truncation or summarization strategy before passing it to the preprocessing layer, saving downstream parsing overhead.

metadata enrichment

Network & Client Fingerprinting: Inject metadata including client_geo (via MaxMind GeoIP), device_type, request_timestamp_utc, and network latency metrics.

Routing Hints: Append historical model preferences for this specific user (e.g., if the user always toggles "Creative Mode", append a metadata flag preferred_track: creative).

Preprocessing Layer

To prevent noisy, malformed inputs from degrading embedding quality or tricking downstream intent classifiers, the text must be cleaned deterministically.

Typo Correction (Heuristic vs. Light Neural):

Avoid heavy LLMs here due to cost. Use a fast, symmetric delete spelling correction algorithm (SymSpell) or a lightweight, localized character-level sequence-to-sequence model for domain-specific terms.

Purpose: Correcting "fiannce" to "finance" significantly stabilizes downstream embedding vectors.

Language Detection:

Route text through ultra-fast, compiled sub-millisecond classifiers like FastText (Facebook) or CLD3.

Routing Relevance: Non-English inputs might require specialized multilingual models (e.g., Cohere, Qwen, or deep-seek variants) that possess superior tokenization efficiency and semantic understanding for that specific language, minimizing token bloat caused by poor subword tokenization.

Formatting Cleanup:

Regex-based stripping of control characters, unwanted ANSI escape codes, duplicate whitespaces, and systematic markdown/HTML stripping (unless structural layout is explicitly needed).

Normalize unicode normalization forms (e.g., converting to NFKC) to ensure identical characters have identical byte representations.

Domain Detection:

Run the input text through a fast, linear classifier or a lightweight distilled transformer (e.g., a fine-tuned DistilBERT) trained on your target domains (e.g., code, legal, medical, general_chitchat).

Cost Balancing: If the domain is general_chitchat, the router immediately flags this for a sub-billion parameter model ($0.07/M tokens), completely bypassing frontier models.

Entity Recognition (NER):

Deploy a high-throughput spaCy pipeline or a fine-tuned GLiNER model to extract named entities (ORG, PRODUCT, GPE, DATE).

PII Masking & Compliance: If sensitive PII (e.g., SSN, CREDIT_CARD) is detected, trigger token masking/anonymization filters before the request is sent to external, third-party closed-source APIs.

User Intent Hints:

Utilize pattern-matching state machines combined with a multi-label classification head to detect explicit structural intents: requires_calculation, requires_code_generation, complex_reasoning, factual_retrieval.

Example: If the intent matches requires_calculation, the router can flag the request to bypass standard LLMs and route directly to an agent with a Python REPL tool, or a model with native tool-calling optimizations.

Embeddings Generation:Pass the normalized text to a highly efficient, local embedding model running on a dedicated inference server.

Calculate deterministic lexical metrics: Token Count (using specific tokenizers like TikToken for OpenAI, or LlamaTokenizer), character-to-token ratios, sentence count, tree-depth parsing metrics, and the frequency of structural markers (e.g., brackets, assignment operators =, SQL keywords).

Extract abstract conceptual indicators: Sentiment polarity score (via VADER or high-speed transformer), perplexity estimation (using a tiny causal model like GPT (to minimize cost) to evaluate how "predictable" or standard the user's phrasing is), and information density (ratio of content words to stop words).

How to develop an AI agent application

XJTLU media — Sat, 16 May 2026 09:08:09 +0000

Part 1: The Reality Check

When developers try to move AI agents from a terminal prototype to a robust application, they consistently hit a few major walls:

The "Vibe Coding" Trap and Verification Burden: A major frustration in the community is the reliance on informal, conversational prompting to build logic. This approach, recently studied as "vibe coding," often results in a massive debugging burden. Because developers are forced to manually verify unpredictable outputs rather than relying on structured constraints, code quality and reliability suffer significantly.

In my personal experience, sometimes you have to carefully craft the prompt to guide AI generate a concise output and use formulas to control the quality of AI outputs.

Execution Loops and Environmental Fragility: When agents interact with external tools or dynamic interfaces, they frequently get stuck in repetitive action loops or fail entirely when encountering unexpected noise (e.g., an unpredicted API error or a sudden change in data format). Without strict operational boundaries, an agent lacks the autonomous ability to gracefully break out of these loops and recover.

This could happen especially when the MCP is failing to search online content. Therefore, you have to check the quality of MCP before binding that into AI apps.

The Complexity of Probabilistic Context: Many frameworks push developers toward complex Retrieval-Augmented Generation (RAG) pipelines and vector databases to manage an agent's memory. In practice, developers find this often introduces latency, non-deterministic retrieval failures, and excessive infrastructure overhead for applications that require precise data handling.

Part 2: What Developers and Users Actually Want

Absolute Predictability: Developers want systems that fail gracefully and predictably. They want to know exactly why an agent took a specific action, which requires auditability, deterministic logging, and structured logic. That is why you have to rely on existing packages on programming languages to build an app specifically tailored to control predictability. For example, let LLM write python code to execute, is a good quality control method.

you still need to know what the product should be, what tradeoffs matter, what the architecture should compress around

Zero-Effort Integration: End-users want tools that drop into their existing workflows seamlessly. They do not want to manage or configure the agent; they want the agent to orchestrate the heavy lifting in the background reliably.
Profit-Centric Efficiency: An app should focus on asymmetric risk-reward. The compute cost, latency, and maintenance burden of the agent must be drastically outweighed by the tangible value and time saved by the user.

Part 3: Architectural Blueprint for a High-Value Agent

1. Embrace State-Machine Prompt Engineering

Ditch the open-ended "reasoning loops" that lead to infinite cycles and unpredictable behavior. Instead, design the agent's core workflow as a strict, deterministic state machine. Each state should represent a single, discrete objective powered by highly structured prompt engineering. The LLM's job isn't to figure out the entire application flow on the fly; its job is to process the current state and return a specific output that meets the rigid criteria required to transition to the next node in the machine.

This is a standard tech stack for AI apps

1. The Core AI Layer (Model Providers)

This layer provides the intellectual brainpower of your application. You must choose between managed third-party Application Programming Interfaces (APIs) for rapid development or hosted open-source models for full data control.

Commercial LLMs: OpenAI, Anthropic Claude, and Google Gemini.
Open-Source Models: Deepseek and GLM models can be self-hosted to eliminate third-party API dependencies.

2. AI Orchestration Frameworks

These tools serve as the glue between your application logic, your data sources, and your chosen AI models.

LangChain / LangGraph: Industry standards for chaining multiple prompts together and building complex, stateful multi-agent systems.
LlamaIndex: Optimized specifically for connecting external datasets to LLMs, making it ideal for search engines and document assistants.
Vercel AI SDK: Highly favored for web development to easily stream real-time token responses directly to web interfaces.

3. Data Infrastructure & Vector Storage

Traditional databases cannot search text by its conceptual meaning. AI applications require vector extensions or specialized semantic databases alongside structured storage.

Vector Databases: Pinecone, Milvus, and Qdrant process high-dimensional vector data for lightning-fast retrieval-augmented generation (RAG).
Relational Databases with Vector Support: PostgreSQL using the pgvector extension is widely adopted because it handles standard user tables and AI vectors simultaneously in one system.
Managed Vector Platforms: Supabase and Neon streamline database operations for lean development teams.

4. Backend Orchestration Layer

The backend manages your core business logic, handles secure user authorization, and processes heavy data transformations before handing them off to the AI.

Python (FastAPI / Django): Python remains the dominant language for AI applications because it natively supports data science libraries like PyTorch and NumPy.
TypeScript / Node.js: Excellent for building asynchronous backend systems that must stream text data smoothly to thousands of users simultaneously.

5. Frontend & UI Layer

The user interface must be optimized to render streaming text blocks, handle conversational loops, and manage instant UI state updates. [3]

Web Interfaces: Next.js (React framework) combined with Tailwind CSS and shadcn/ui is the standard configuration for fast, responsive web dashboards.
Mobile Interfaces: Flutter or React Native for cross-platform deployments, or native Swift (iOS) and Kotlin (Android) for hardware-intensive mobile features.

6. MLOps, Evaluation, & Observability

AI inputs and outputs are inherently unpredictable. You need dedicated monitoring to track system costs, check latency, log prompt histories, and detect model errors.

Observability Tools: LangSmith, Helicone, and Phoenix log every single AI prompt and response to help trace errors and measure API spending.
Evaluation Frameworks: Promptflow and Ragas automate test suites to ensure model updates do not introduce system hallucinations.
Hosting & Compute Platforms: Vercel for web frontends, alongside AWS, Google Cloud Vertex AI, or Modal for managing fluctuating backend GPU workloads.

2. Utilize Deterministic State Management

Skip the overhead and unpredictability of vector databases or complex RAG setups when structural integrity is paramount. For robust context and memory tracking, leaning on a lightweight, battle-tested relational database like SQLite ensures that the agent's state history is queryable, immutable, and fully under your control. This allows you to construct highly precise context windows based on hard data rather than algorithmic best guesses.

3. Standardize Tooling and Orchestration

Ground your agent's capabilities using standardized protocols to handle tool execution. By strictly separating the agent's decision logic (the state machine) from its action execution (e.g., utilizing localized Model Context Protocol servers), you create a modular architecture that is significantly easier to debug, test, and scale across different environments.

On what people really need for MCP

XJTLU media — Wed, 13 May 2026 02:19:05 +0000

What is MCP

By definition, MCP (Model Context Protocol) is an open-source standard for connecting AI applications to external systems.
Using MCP, AI applications like Claude or ChatGPT can connect to data sources (e.g. local files, databases), tools (e.g. search engines, calculators) and workflows (e.g. specialized prompts)—enabling them to access key information and perform tasks.

The Three Pillars of MCP

Resources: Think of these as read-only data bridges. Whether it’s a local SQLite database, a Google Drive folder, or a GitHub repo, the AI can now "see" the data it needs to provide accurate, grounded answers.

Tools: This enables the AI to do things. It could be executing a Python script, triggering a web search, or performing complex calculations through a specialized API.

Prompts: MCP allows for standardized "templates" or workflows. This ensures the AI follows specific, high-quality logic consistently across different platforms.

What people really need for MCP

I wrote the markdown formatter (AI-Answer-Copier) months ago. In order to build a bridge between AI answers to saved document, I integrated functions of markdown conversion to multiple format into this MCP.

Here is one of the published version of markdown formatter statistics.

For this sites only, we get at least 6000 usage in the last 30 days, while this is not high. But I can say, this MCP at least serve some part of purposes of what people want.

For HTML online version, see: AI-answer-copier

As you can see, most users use markdown to DOCX. The usage is higher than others. It means most users still prefer to use DOCX for convenient purposes.

It is not hard to understand, markdown is exclusive to computer related workers like developers or AI engineers. Managers still like to use DOCX for their works and modify it at their wish.

Which part of this tool is different to claude chatbox?

Some people might curious, claude.ai already has docx output if you ask them in the chatbox. Why a duplicated tool is needed.

First, claude use python or HTML to generate DOCX, while it is good for one run output of files. It did not, however, serve the purpose for multi run if user want to export every part of their chat into DOCX.

Second, user may want to export to other format, like to PDF, without their manual conversion using MSOffice offline. This is what this tool build for.

How could you build a MCP that people likes?

Build a bridge for what people need

There are still many gaps between what user want between AI output. For example, corporations might deploy LLM with agent to replace human customer services.

One problem existing in here is, AI don't have authority to touch IT system which human customer services can have access to, and AI usually hallucinate, answer different than what people want.

The most important question is, most of client feedback is not about generalized ideas but focusing on specific problems. Like:

"My phone can't receive a verification SMS for your app"

,which can only be solved by a senior engineer.

What is your idea on AI customer service to solve this kind of problem?

Stop Copy-Pasting AI Answers: How I Built an MCP Server for "Classroom-Ready" Exports

XJTLU media — Wed, 04 Feb 2026 03:07:38 +0000

We’ve all been there. You ask Claude or ChatGPT to generate 20 complex physics questions. The output is brilliant, but then the "Final Mile" friction hits.

You spend the next 45 minutes:

Fixing LaTeX formulas that broke during the copy-paste.
Re-formatting bullet points into a CSV template for Kahoot or Quizizz.
Wrestling with Word margins to make a printable PDF.

Generating the content took 10 seconds. Formatting it took an hour. That is a broken developer/educator experience.

The Solution: Model Context Protocol (MCP)

Instead of manually bridging the gap between the AI's "brain" and my local files, I built an MCP Server that allows the AI to handle the export itself.

With the AI Question Copier acting as an MCP server, you don't hit Ctrl+C. You just tell the AI:

"I like these questions. Export them as a Quizizz CSV and a printable PDFHandout."

The AI "sees" the available tools, parses its own reasoning into the correct schema, and triggers the file generation locally.

Tech Stack

Language: TypeScript / Node.js

Protocol: Model Context Protocol (MCP)

Parsing: Intelligent regex-based "Smart-Parse" for Q&A structures.

Outputs: PDF (handouts), XLSX (Quizizz/Kahoot), JSON (LMS), and Markdown.

How to use it in 60 seconds

If you’re using Claude Desktop, you can bind this tool by adding it to your claude_desktop_config.json:

{
  "mcpServers": {
    "question-exporter": {
      "command": "node",
      "args": ["/PATH_TO_PROJECT/mcp-server/dist/index.js"]
    }
  }
}

Once restarted, you'll see the 🔌 icon. Your AI is now an automated education assistant.

Why this matters

We are moving away from "Chatbots" and toward "Actionable Agents." MCP is the "USB-C for AI"—a standardized way to let models interact with the world securely. Whether you are a teacher, a developer, or a technical trainer, removing the "copy-paste tax" is the first step toward true productivity.

You are welcomed to view my project at MCP_Markdown_Formatter