DEV Community: Mudassir Khan

LLM Cost Optimization: Cutting Inference Bills Without Killing Quality

Mudassir Khan — Wed, 01 Jul 2026 19:54:00 +0000

You can cut your LLM API spend by 50 to 90% without switching models or degrading output quality. The techniques exist, the docs are public, and most teams are not using them. Here is what actually moves the needle.

Where your LLM bill actually comes from

Every API call charges you for input tokens plus output tokens. Simple math, but "input tokens" is a bigger footgun than it looks.

Most production workloads send the same system prompt, instructions, or retrieval context on every single request. If your system prompt is 4,000 tokens and you fire 100 requests per minute, that's 400,000 input tokens per minute burning at full price. Before you optimize anything else, map where your token spend is actually going.

Three buckets drive most bills:

Repeated context: system prompts, tool definitions, retrieved documents sent on every call
Frontier model overuse: sending simple classification tasks to GPT-4o or Claude Sonnet when a cheaper model handles them just fine
Bloated outputs: getting paragraph prose back when you need a 10-token structured field

Once you know which bucket hurts most, the fix is obvious. The rest of this article covers each one with concrete numbers.

Prompt caching: the highest ROI optimization nobody uses

If your app sends the same prefix on every call (system prompt, tool definitions, few shot examples), prompt caching is the single highest impact change you can make.

Anthropic's caching API lets you mark a portion of your input as cacheable with a cache_control block. The first call writes the cache at 25% of the normal input token price. Every subsequent cache hit costs 10% of normal. That is a 90% cost reduction on the cached portion.

import anthropic

client = anthropic.Anthropic()

SYSTEM_PROMPT = """
You are a code review assistant. Your job is to identify bugs,
suggest improvements, and flag security issues in submitted code.
... (your 2000-token system prompt here) ...
"""

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": SYSTEM_PROMPT,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {"role": "user", "content": user_code}
    ]
)

On the first call you pay the cache write cost (25% of normal input price). From the second call onward, every hit costs 10% of what you would normally pay. For an app firing 1,000 calls per hour against a 3,000-token system prompt, that is 2.7 million tokens per hour dropping from full price to 10% of full price.

OpenAI has equivalent behavior through automatic prompt caching. The implementation differs slightly but the economics are similar.

Prompt caching helps most when: your system prompt is over 1,000 tokens, you are running a RAG pipeline that injects the same documents repeatedly, or you have an agent framework with large tool definitions reused across turns.

Model routing: stop sending everything to the frontier model

Not every task needs GPT-4o or Claude Sonnet. Intent classification, slot extraction, summarization, and simple Q&A are well within what GPT-4o-mini or Claude Haiku can handle. The cost difference is roughly 10 to 20 times per token.

Routing 40% of traffic to a smaller model can halve your total API spend without touching a single user facing feature. The key is picking the right tasks.

import OpenAI from "openai";

const client = new OpenAI();

type TaskComplexity = "simple" | "complex";

function classifyTask(userMessage: string): TaskComplexity {
  const reasoningKeywords = [
    "explain why",
    "compare",
    "analyze",
    "critique",
    "design architecture"
  ];

  const needsReasoning = reasoningKeywords.some(kw =>
    userMessage.toLowerCase().includes(kw)
  );

  if (!needsReasoning && userMessage.length < 200) {
    return "simple";
  }
  return "complex";
}

async function routedCompletion(userMessage: string) {
  const complexity = classifyTask(userMessage);
  const model = complexity === "simple" ? "gpt-4o-mini" : "gpt-4o";

  const response = await client.chat.completions.create({
    model,
    messages: [{ role: "user", content: userMessage }],
  });

  return { response, model_used: model };
}

This is a naive router based on keywords and message length. Production versions use a small classifier (or another LLM) to score complexity before routing. Projects like LiteLLM and RouteLLM give you prebuilt routers if you don't want to build your own.

The failure mode to watch: tasks that look simple ("what does this error mean?") but need surrounding context to answer well. Build an escalation path so users can retry against the full model when the cheaper one falls short.

Batching and async inference for workloads that can wait

If you are running evals, processing documents, generating reports, or doing any bulk inference where a 24-hour window is acceptable, the OpenAI Batch API cuts costs by 50% versus synchronous pricing.

You submit a JSONL file. OpenAI processes it asynchronously. You pull results when they are ready. No streaming, no realtime SLAs.

import openai
import json
from pathlib import Path

client = openai.OpenAI()

requests = []
for doc_id, text in documents.items():
    requests.append({
        "custom_id": doc_id,
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-4o-mini",
            "messages": [
                {"role": "user", "content": f"Summarize in 3 sentences: {text}"}
            ],
            "max_tokens": 150
        }
    })

batch_path = Path("/tmp/batch_input.jsonl")
batch_path.write_text("\n".join(json.dumps(r) for r in requests))

with open(batch_path, "rb") as f:
    batch_file = client.files.create(file=f, purpose="batch")

batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h"
)

print(f"Batch submitted: {batch.id}")
# Poll client.batches.retrieve(batch.id) to check status

Stack this with model routing: batch jobs on the cheap model with the 50% discount applied on top. That is where the cost math gets seriously interesting.

Anthropic offers similar async batch processing. The API shape differs slightly but the pricing pattern is the same.

Output schema discipline to shrink response tokens

Freeform prose outputs are expensive. When your downstream code parses a JSON blob out of a markdown code block, you are paying for the explanation text, the code fences, and the commentary. None of that is useful to you.

Structured outputs fix this. Both OpenAI and Anthropic support schema constrained generation that returns only the fields you defined, in valid JSON, every time.

from anthropic import Anthropic
import json

client = Anthropic()

response = client.messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=256,
    tools=[{
        "name": "extract_sentiment",
        "description": "Extract sentiment analysis from text",
        "input_schema": {
            "type": "object",
            "properties": {
                "sentiment": {
                    "type": "string",
                    "enum": ["positive", "negative", "neutral"]
                },
                "confidence": {"type": "number"},
                "topics": {
                    "type": "array",
                    "items": {"type": "string"},
                    "maxItems": 3
                }
            },
            "required": ["sentiment", "confidence", "topics"]
        }
    }],
    tool_choice={"type": "tool", "name": "extract_sentiment"},
    messages=[{"role": "user", "content": user_text}]
)

result = json.loads(response.content[0].input)

The schema constraint tells the model to stop once the required fields are populated. No hedging, no markdown formatting, no explanatory prose it doesn't need to write. Output token counts drop noticeably on structured tasks compared to the freeform equivalent.

This one bit me in prod before I switched: an eval pipeline returning full explanation paragraphs for every item when all I needed was a score and a label. Two fields. The verbosity was pure waste.

FAQ

How do you reduce LLM API costs?

The highest ROI levers in order: (1) prompt caching for repeated context, (2) model routing for tasks that don't need frontier models, (3) batching for async workloads, (4) schema constrained outputs to shrink response size. Each one targets a different cost bucket and they stack.

Is prompt caching worth it for LLM cost savings?

Yes, if your system prompt or retrieval context exceeds 1,000 tokens and you send it repeatedly. Anthropic's cache read price is 10% of normal input pricing. On a 4,000-token system prompt fired 500 times per day, you pay full price once and 10% for the other 499. The math is not subtle.

What is the cheapest way to run an LLM in production?

Self hosted open weight models (Llama, Mistral, Qwen) undercut API pricing at scale but require infrastructure you have to operate. For workloads running against vendor APIs: combine routing (cheap model for simple tasks) plus batching (50% off async) plus caching (90% off repeated context). Most teams are not doing all three, which means most teams are overpaying.

I go deeper on production LLM inference optimization on my blog, including a breakdown of what each technique actually saves at different call volumes.

If you want this wired up on your own stack end to end, that is exactly the kind of work I take on.

What routing or caching setup is your team running in production? Drop a comment. Curious what numbers people are actually seeing.

Why LLM Agents Fail Silently and How to Debug Them

Mudassir Khan — Sat, 27 Jun 2026 20:52:01 +0000

Why LLM Agents Fail Silently and How to Debug Them

Your agent returned an empty result. No exception. No error log. No status code that points anywhere useful. Just nothing.

You dig through logs. The LLM call went through. The tool was invoked. The response came back. Everything looks fine and yet the task is incomplete, wrong, or missing entirely.

That's a silent failure. And it's one of the nastiest bugs in AI engineering.

What is a silent failure in an LLM agent?

A silent failure is when your agent completes without raising an exception but produces a wrong or incomplete result. The difference between a noisy failure (a Python traceback, a 5xx from the API) and a silent one is that noisy failures are debuggable. Silent ones require you to instrument the entire agent loop just to notice something went wrong.

They're common because LLMs are designed to always return something. The model won't throw a ValueError when it runs out of context or when your tool schema changes out from under it. It'll return an empty array, a truncated JSON blob, or a confident "I've completed the task" with nothing to show for it.

The result is an agent that appears to work until you look closely at the outputs.

The three root causes: token budget, tool schema drift, and unhandled exceptions

Most silent failures trace back to one of three places.

Token budget exhaustion. OpenAI's function calling API returns an empty choices array when max_tokens is hit in the middle of a tool call. No exception is raised. The call returns 200. Your code checks response.choices[0] and explodes with an IndexError, or worse, your code handles the empty array gracefully and just moves on. The agent continues as if the tool ran, with no output to show for it.

response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    tools=tools,
    max_tokens=512  # too small for a complex tool call
)

# This blows up at runtime — or silently skips if you're defensive
if response.choices:
    tool_call = response.choices[0].message.tool_calls[0]

Fix: always log response.choices, finish_reason, and usage.completion_tokens. If finish_reason == "length", treat it as a hard failure, not a graceful noop.

Tool schema drift. Your tool schema changes. A field gets renamed, a required parameter gets removed, a new enum value gets added. The LLM was tuned against the old schema. It now generates arguments that fail the validator, and your framework silently drops the tool output and continues. LangGraph's StateGraph does exactly this when a tool raises an unhandled exception inside an interrupt: the output gets dropped and the next node receives None.

# Tool raises, StateGraph swallows the exception
@tool
def fetch_user_data(user_id: str) -> dict:
    # KeyError here gets swallowed by the interrupt handler
    return db.fetch(user_id)["profile"]["details"]

Fix: always reraise from your tool handlers, or wrap them in an explicit try/except that returns a structured error payload instead of propagating None downstream.

Unhandled exceptions inside the agent loop. Most agent frameworks catch exceptions at the orchestrator level to keep the loop alive. That's good for reliability, but it means your per step errors get swallowed into a catchall handler that logs nothing useful and lets the next turn proceed. One bad tool call in a 10-step chain silently poisons every step that follows.

Adding per step tracing to catch failures early

The most reliable way to surface silent failures is distributed tracing. OpenTelemetry spans per agent step give you a queryable record of every tool call, its inputs, its outputs, and where it fell over.

from opentelemetry import trace

tracer = trace.get_tracer("agent.loop")

def run_agent_step(step_name: str, messages: list, tools: list):
    with tracer.start_as_current_span(step_name) as span:
        span.set_attribute("step.input_message_count", len(messages))
        span.set_attribute("step.tool_count", len(tools))

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
            tools=tools,
        )

        finish_reason = response.choices[0].finish_reason if response.choices else "empty"
        span.set_attribute("step.finish_reason", finish_reason)
        span.set_attribute("step.completion_tokens", response.usage.completion_tokens)

        if finish_reason == "length" or not response.choices:
            span.set_status(trace.StatusCode.ERROR, "token budget hit or empty response")
            raise RuntimeError(f"Step {step_name} hit token budget before completing")

        return response

Now when something goes wrong, your trace shows exactly which step failed and why. You're not reconstructing the failure from scattered log lines. You have a full span tree.

Plug this into any OpenTelemetry compatible backend (Honeycomb, Jaeger, the OTel Collector) and you get realtime visibility into your agent loop for free.

Structured output validation as a silent failure firewall

If tracing tells you when something went wrong, Pydantic tells you what the model produced that broke your assumption.

Put a Pydantic validation step after every tool call. The model's output schema gets validated before it touches anything downstream. If it fails, you catch a ValidationError with a clear message instead of a silent None that propagates through 5 more steps.

from pydantic import BaseModel, ValidationError

class UserProfile(BaseModel):
    user_id: str
    email: str
    role: str  # "admin" | "viewer" | "editor"

def validate_tool_output(raw: dict) -> UserProfile:
    try:
        return UserProfile(**raw)
    except ValidationError as e:
        # Loud failure here is intentional — better than a silent one later
        raise RuntimeError(f"Tool output failed schema validation: {e}") from e

This is especially powerful for tools that call external APIs. The external schema changes independently of your agent's expectations. Pydantic catches that mismatch at the boundary, before stale data flows into your LLM's next prompt and contaminates the rest of the run.

Building a dead man's switch into long running agent loops

Long running agents (the ones that run for minutes or hours across many tool calls) need a liveness check that fires if the loop goes quiet for too long. If the agent doesn't check in within N seconds, something assumed it was alive when it wasn't.

import threading
import time

class AgentWatchdog:
    def __init__(self, timeout_seconds: int = 60):
        self.timeout = timeout_seconds
        self.last_heartbeat = time.time()
        self._stop = threading.Event()

    def heartbeat(self):
        """Call this after every successful agent step."""
        self.last_heartbeat = time.time()

    def start(self):
        def _watch():
            while not self._stop.is_set():
                if time.time() - self.last_heartbeat > self.timeout:
                    raise RuntimeError("Agent watchdog timeout — loop went silent")
                time.sleep(5)
        threading.Thread(target=_watch, daemon=True).start()

    def stop(self):
        self._stop.set()

# Usage
watchdog = AgentWatchdog(timeout_seconds=90)
watchdog.start()

for step in agent_steps:
    result = run_agent_step(step)
    watchdog.heartbeat()  # prove we're alive after each step

watchdog.stop()

This doesn't replace tracing. It's a last resort: if your instrumentation missed the failure, the watchdog still catches a loop that went silent and gives you something you can alert on.

FAQ

Why do AI agents stop responding without an error?

Usually one of three things: the model hit its token budget in the middle of a tool call and returned an empty choices array, a tool raised an exception that the orchestrator swallowed, or the tool output failed schema validation and got silently dropped. Add finish_reason logging and per step OTel spans and you'll find it fast.

How do you debug an LLM agent that returns empty results?

Start with finish_reason. If it's "length", you hit the token budget. If it's "stop" but the output is empty, check your tool handler for swallowed exceptions. If the tool ran but the downstream state is still None, you have a schema validation gap. Pydantic after every tool call surfaces this immediately.

What causes silent failures in multistep AI agents?

The same bugs that cause noisy failures in any software, except the agent framework is often designed to keep running even when a step fails. That design choice trades reliability for debuggability. You get it back by adding tracing at the framework layer so failures are recorded even when the loop doesn't crash.

If you want a deeper look at agent observability in production, I cover it in more detail on my site.

For the full taxonomy of production failure modes before you build your evaluation harness, this is where I'd start.

If you want this wired up on your own system end to end, that is exactly the kind of work I take on.

Drop a comment if you've hit a different class of silent failure. Curious what variations people are running into in production.

I Measured MCP vs Direct API Calls: The Token Math No One Tells You

Mudassir Khan — Tue, 23 Jun 2026 19:37:17 +0000

MCP is having its moment. Every agent framework is wiring tools through it, and for good reason: standardized tool registration, composable servers, a protocol that doesn't force you to reinvent the plugin system every project. I get the appeal.

What I didn't see coming was the token bill.

Here's the number that stopped me cold: for a simple SerpApi search, an MCP agent used 6,047 tokens per call. A CLI script doing the exact same job used 351 tokens. That's 17x overhead, for the same search result.

I went looking for other data points after that. The range is wider than I thought.

The number that started this: 17x tokens per call

The benchmark comes from a post measuring SerpApi MCP vs a CLI agent with field projection. Both do the same job: search the web, return results. The difference:

Approach	Tokens per call
MCP agent	6,047
CLI script (with field projection)	351
Ratio	~17x

At 10 searches a day in a conversational interface, 6,047 tokens per call is fine. At 1,000 searches a day in an automated pipeline, you're burning 6 million tokens where 351,000 would do.

That's not a rounding error. That's a billing line item.

Why it happens: schema injection on every message

The overhead isn't in the API call itself. It's in what your AI host has to inject into every single message before the call even happens.

When you register an MCP server, every tool definition (name, description, input schema, parameter types) gets serialized and injected into every system prompt or assistant message in your conversation. The LLM needs to "see" all available tools on every turn so it knows what it can call.

A concrete example: a simple repo language check. The task is trivial. The token count is not:

Approach	Tokens
CLI agent (direct API)	1,365
MCP agent (same task)	44,026

The extra 42,661 tokens are 43 tool definitions injected into every message. Tools that had nothing to do with the repo language check. The agent happened to have 43 tools registered on that MCP server, and every one of them rode along for the trip.

The full benchmark breakdown (4x to 32x range)

"17x" is the headline, but the actual range is 4x to 32x depending on:

How many tools are registered on the server
How complex those tool schemas are
How many turns the conversation takes
Whether you're running batch jobs or interactive queries

The 4x end is a small server with simple schemas and a short session. The 32x end is a large server with dozens of complex tools running a multistep pipeline where schema injection compounds across every message.

What matters is that the floor is 4x. Even a minimal MCP setup carries overhead that a direct API call doesn't.

The $270/day surprise: unused tool definitions at scale

Here's where it gets genuinely alarming for production pipelines.

At Claude Sonnet pricing, 90,000 tokens of unused schema overhead costs roughly $0.27 per request. If your pipeline runs 1,000 times a day, that's $270/day on tool definitions your agent never touches.

Not on useful computation. On schemas riding along in the context window because you registered everything upfront.

Most MCP servers are built for flexibility. You register everything because you don't know what the agent will need. That makes sense in a conversational interface where the user might ask anything. In a batch pipeline where the task is deterministic, you're paying the full flexibility tax even when you only need two of the forty registered tools.

The math compounds fast. And it's invisible in your usage dashboard: it just shows up as "input tokens."

When the overhead is absolutely worth it

MCP has real strengths. The overhead is the cost of flexibility, and in these cases it's worth paying.

Conversational agents. The user is asking unpredictable questions. You need all tools available on every turn because you genuinely don't know what comes next. Schema injection is the price of that flexibility.

Governance and auth at the server level. MCP servers can enforce access control, rate limiting, and audit logging in one place for every client that talks to them. Doing that in bespoke scripts for each tool is a footgun waiting to fire. The overhead buys you correctness.

Composable tooling across teams. If five teams are building agents and they all need the same data sources, one MCP server beats five separate integrations. The overhead is shared; the maintenance savings are real.

Rapid prototyping. Getting from zero to a working agent tool takes minutes with MCP. The token overhead doesn't matter when you're still iterating before it runs in production.

When to cut MCP out of the loop

The cases where you should rethink it:

Deterministic batch pipelines. You know exactly what the agent needs to do. Schema injection has zero conversational payoff here. It's pure cost. A job with 50 items takes roughly 50 seconds via direct API vs about 25 minutes via MCP, with 10x to 20x token overhead per step. At that scale, the pipeline eats itself.

High volume single purpose agents. If your agent does one thing 10,000 times a day, a direct API call with a hardcoded schema is faster, cheaper, and easier to debug. MCP's flexibility is a liability in this case.

Latency sensitive paths. More tokens means slower time to first token from the LLM. For realtime applications where response latency matters, every unnecessary token hurts.

Fixed scope, cost optimized pipelines. If you know you'll never need more than three tools and the task is well defined, bespoke beats general purpose every time.

Practical reduction: field projection, selective loading, lazy registration

If you're committed to MCP but want to cut the overhead, three patterns help.

Field projection

Instead of passing the full API response, return only the fields the agent actually needs. The SerpApi CLI example gets most of its efficiency from this: it strips the raw response to title, snippet, and URL before handing anything to the LLM. The agent gets the same information with a fraction of the token cost.

// Instead of passing the full response object:
const raw = await serpapi.search(query);

// Project down to what the agent actually needs:
return {
  results: raw.organic_results.slice(0, 5).map(r => ({
    title: r.title,
    snippet: r.snippet,
    url: r.link,
  }))
};

Selective tool loading

Not every agent needs every tool every session. If you can determine upfront which tools a task requires, register only those. MCP servers support dynamic tool lists, so you don't have to expose everything on every request.

Lazy registration

Register the core tools that handle 90% of your traffic. Add specialized tools only when the conversation signals it needs them: via explicit user intent, a specific task type, or a flag in your system prompt. This keeps the base schema small and only pays for specialized tooling when it's actually relevant.

FAQ

Does MCP use more tokens than direct API?

Yes, always. Schema injection adds a minimum of 4x overhead and can reach 32x for complex multitool servers. The question is whether the flexibility justifies the cost for your use case.

How many tokens does MCP add per request?

It depends on how many tools are registered and how complex their schemas are. A minimal server might add a few hundred tokens per message. A large server with dozens of tools can add tens of thousands per message, per turn.

When should I use MCP vs direct API calls?

MCP for conversational, open ended, governance heavy, or multiteam scenarios. Direct API for batch jobs, high volume single purpose agents, latency sensitive paths, and any pipeline where the task scope is fixed and well defined.

How do I reduce MCP server token overhead?

Field projection on responses, selective tool registration, and lazy loading of specialized tools. For extreme cases, a hybrid approach works: MCP for the orchestration layer, direct API for the hot path inside each tool.

If you want to see how this fits into a production agent architecture, I cover the full decision framework in my post on production architecture.

If you want help building your own MCP setup without the token tax eating your budget, that's the kind of work I take on.

Drop a comment if you've run your own numbers. Curious what the 4x to 32x range looks like in your production setup.

Why Rate Limits Kill Your AI Agents in Production (And the Patterns That Actually Work)

Mudassir Khan — Sun, 21 Jun 2026 19:34:18 +0000

LLM API calls fail between 1% and 5% of the time in production. Not from hallucinations. From 429 errors nobody handled.

You've probably seen this: you ship an agent, everything works in staging, prod hits a burst of traffic, the provider throttles you, and suddenly your agent is retrying forever, burning tokens on every attempt, and the cost graph spikes sideways. The incident isn't model quality. It's the retry loop you forgot to fence.

I've written about production architecture for agentic systems before. Rate limiting is the piece that bites teams hardest, so let me go deep on it here.

Why rate limits catch most AI teams off guard

Most developers come to LLMs from REST APIs where rate limiting is mostly a nuisance you handle with one retry. With LLMs, the shape of the problem is different.

A single agent request isn't one API call. It's potentially dozens: planner calls, tool calls, summarizer calls, verifier calls. They all share the same rate window. One agent serving 10 simultaneous users can hit 200 to 300 API calls per minute before you realize what's happening.

The other thing that surprises teams: LLM rate limits often count tokens, not requests. Two requests at 50 tokens each and one request at 10,000 tokens are not equal, but your requests per minute counter treats them the same. You can stay under RPM and blow right past TPM.

The hidden cost: retry storms in multiagent systems

The most common production incident isn't a model giving the wrong answer. It's an agent that decides to retry, then retries again, each retry being a full provider call with no delay logic to protect the window.

This is the retry storm. You hit a rate limit, your agent retries immediately (or with a fixed 1 second delay), the retried call also hits the limit, all the retries queue up at the rate window boundary and fire at once, and now you've turned a temporary throttle into a sustained overload.

In multiagent systems it compounds. One orchestrator spawning five subagents, each doing their own uncoordinated retries, can turn a single 429 into 50 retry attempts within the same second. This is one of the core failure patterns I covered in why AI agents fail in production.

Proper rate limit handling can cut redundant API costs by 40%. That's not a small rounding error. That's architectural discipline that pays for itself.

Request count vs token count: why you're measuring the wrong thing

Your LLM provider gives you two limits: requests per minute (RPM) and tokens per minute (TPM). Most teams watch RPM. TPM is usually what breaks you.

Here's why: 1 request can be 50 tokens or 10,000 tokens. If you only count requests and stay under RPM, a single heavy prompt (large context, long output) can exhaust your TPM budget while you're still well under RPM. The next 30 requests all get 429s even though you've only made 5 calls that minute.

The fix is to count tokens on the way out, not after the call fails:

interface RateLimiter {
  requestTokens(estimatedTokens: number): Promise<void>;
  recordUsage(actualTokens: number): void;
}

class TokenBucketLimiter implements RateLimiter {
  private bucketTokens: number;
  private lastRefill: number;

  constructor(
    private tpmLimit: number,
    private refillIntervalMs = 60_000
  ) {
    this.bucketTokens = tpmLimit;
    this.lastRefill = Date.now();
  }

  async requestTokens(estimatedTokens: number): Promise<void> {
    this.refillIfNeeded();

    if (this.bucketTokens < estimatedTokens) {
      const waitMs = this.msUntilRefill();
      await new Promise(resolve => setTimeout(resolve, waitMs));
      this.refillIfNeeded();
    }

    this.bucketTokens -= estimatedTokens;
  }

  recordUsage(actualTokens: number): void {
    // Adjust the bucket if the actual usage differed from the estimate.
    // Track pre-reserved vs actual in a real implementation.
    void actualTokens;
  }

  private refillIfNeeded(): void {
    const now = Date.now();
    if (now - this.lastRefill >= this.refillIntervalMs) {
      this.bucketTokens = this.tpmLimit;
      this.lastRefill = now;
    }
  }

  private msUntilRefill(): number {
    return this.refillIntervalMs - (Date.now() - this.lastRefill);
  }
}

Estimate tokens before the call using tiktoken or a rough char/4 heuristic, consume from the bucket, wait if you're over budget. This moves the rate limit behavior from reactive (catch the 429) to proactive (don't send the call that would fail).

The 3-layer architecture that actually works

The teams that handle rate limits cleanly aren't just retrying smarter. They're operating at three layers simultaneously.

Layer 1: token bucket per (user, model). Limit each user's consumption independently. A single heavy user doesn't starve everyone else. Scope the bucket to both the user and the model so a cheap model and an expensive one don't compete for the same budget.

Layer 2: circuit breakers. Three signals should trip a circuit breaker:

Cost velocity: if a user is burning tokens at 3x their rolling average, something is looping
Repeated prompts: the same or very similar prompt fired multiple times in 10 seconds is probably a bug, not intent
Error rate: if 20% of a user's calls are 4xx in a 5 minute window, stop sending and surface the error instead of absorbing it silently

Layer 3: declarative fallback chain. Primary model → cheaper model (e.g. GPT-4o → GPT-4o mini) → semantic cache (return a stored response for similar queries) → 503. The chain is declarative, not imperative. You configure the fallback in one place and every agent inherits it.

TypeScript implementation: exponential backoff with jitter

The reason a fixed retry delay creates storm conditions is that all retried requests fire at the same moment. Jitter desynchronizes them.

interface RetryConfig {
  maxAttempts: number;
  baseDelayMs: number;
  maxDelayMs: number;
  jitterFactor: number; // 0 to 1, how much randomness to add
}

async function withExponentialBackoff<T>(
  fn: () => Promise<T>,
  config: RetryConfig = {
    maxAttempts: 3,
    baseDelayMs: 1_000,
    maxDelayMs: 30_000,
    jitterFactor: 0.3,
  }
): Promise<T> {
  let lastError: Error;

  for (let attempt = 0; attempt < config.maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (error) {
      lastError = error as Error;

      if (!isRateLimitError(error) || attempt === config.maxAttempts - 1) {
        throw error;
      }

      const exponentialDelay = config.baseDelayMs * Math.pow(2, attempt);
      const cappedDelay = Math.min(exponentialDelay, config.maxDelayMs);
      const jitter = cappedDelay * config.jitterFactor * Math.random();
      const finalDelay = cappedDelay + jitter;

      await new Promise(resolve => setTimeout(resolve, finalDelay));
    }
  }

  throw lastError!;
}

function isRateLimitError(error: unknown): boolean {
  if (error instanceof Error) {
    return (
      error.message.includes('429') ||
      error.message.toLowerCase().includes('rate limit')
    );
  }
  return false;
}

The key is Math.random() in the jitter calculation. Two simultaneous retries sleep for different durations and arrive at the provider at different moments. At scale this turns a synchronized wave into a spread distribution.

Also worth checking: the OpenAI 429 response includes a Retry-After header telling you exactly how many seconds to wait. Parse it and honor it directly instead of running your own backoff math.

Wiring in a circuit breaker (code example)

A circuit breaker wraps your LLM client and opens (stops sending) when the error rate crosses a threshold. Here's a minimal implementation:

type CircuitState = 'closed' | 'open' | 'half-open';

class LLMCircuitBreaker {
  private state: CircuitState = 'closed';
  private failureCount = 0;
  private lastFailureTime = 0;
  private readonly failureThreshold = 5;
  private readonly recoveryTimeMs = 60_000;

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() - this.lastFailureTime > this.recoveryTimeMs) {
        this.state = 'half-open';
      } else {
        throw new Error(
          'Circuit open: LLM provider is throttling. Try again shortly.'
        );
      }
    }

    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  private onSuccess(): void {
    this.failureCount = 0;
    this.state = 'closed';
  }

  private onFailure(): void {
    this.failureCount++;
    this.lastFailureTime = Date.now();
    if (this.failureCount >= this.failureThreshold) {
      this.state = 'open';
    }
  }
}

// Wire the two layers together:
const breaker = new LLMCircuitBreaker();
const limiter = new TokenBucketLimiter(100_000);

async function safeLLMCall(prompt: string, estimatedTokens: number) {
  await limiter.requestTokens(estimatedTokens);
  return breaker.call(() =>
    openai.chat.completions.create({
      model: 'gpt-4o',
      messages: [{ role: 'user', content: prompt }],
    })
  );
}

The circuit opens after 5 failures and stays open for 60 seconds. During that window, requests fail fast instead of piling up waiting for a timeout. After 60 seconds it shifts to half-open: one test call goes through, and if it succeeds the circuit closes again. If it fails, the clock resets.

Put this at the call site of every provider interaction and you've got a fence around every agent that uses it.

FAQ

What causes LLM rate limit errors?

Two things. You've hit the provider's RPM or TPM ceiling for your account tier, or a single request exceeded the context window limit. Check both when you see a 429. The error response usually tells you which limit you hit.

How do I handle 429 errors from OpenAI?

The OpenAI 429 response includes a Retry-After header with how many seconds to wait. Parse it and sleep for that duration before retrying. The header value is more reliable than any backoff formula you'll calculate yourself.

What is exponential backoff for APIs?

Instead of retrying immediately or on a fixed interval, each retry waits longer than the previous one: 1 second, then 2 seconds, then 4 seconds, then 8 seconds. Adding random jitter to each wait time prevents all retriers from firing at the same moment and overloading the provider again.

How do I prevent agent retry storms?

Two controls working together. A circuit breaker that opens after N consecutive failures stops new calls from going out while the provider is saturated. A token bucket that estimates usage before sending catches bursts before they hit the API. The combination prevents the feedback loop where retries cause more rate limits cause more retries.

If you're building out the rest of the agent reliability layer, I go deeper into the architectural patterns on my blog.

If you want this wired up on your own stack end to end, agentic AI consulting is exactly the kind of work I take on.

Drop a comment if your rate limit setup looks different. Curious whether people are managing this at the SDK layer or at an API gateway.

Stop letting the prompt be your state machine

Mudassir Khan — Tue, 16 Jun 2026 18:34:17 +0000

Stop letting the prompt be your state machine

You shipped an LLM feature six months ago. Now the same user input produces wildly different outputs depending on... nothing you can point to. Something in the sampling? The time the context filled up and a chunk got dropped? Nobody knows. This is what happens when the prompt becomes your runtime.

The trap: the prompt as an accidental runtime

Here is what the trap looks like in TypeScript:

async function handleUserRequest(input: string): Promise<string> {
  const prompt = `
    You are a helpful assistant.
    The user said: ${input}
    Previous context: ${someGlobalContext}

    Decide what to do, gather any information you need,
    format the response, and return it.
  `;
  return llm.complete(prompt);
}

The model is doing everything here: deciding the intent, gathering data, formatting output, choosing what to persist. That is a footgun. You handed the runtime to a stochastic function.

Gartner attributes many failed agentic AI projects to unclear value and inadequate risk controls. Deterministic, testable workflows address both. The fix is not a better prompt. The fix is to stop using the prompt as an architecture.

What "deterministic" can and cannot mean here

Be honest about what you can and cannot control.

You cannot control: the model's exact output. It is probabilistic by design.

You can control:

The shape of the output (structured output plus schema validation)
The steps that run before and after the model call
What data enters the model
What happens when the output fails validation
Whether a human reviews the result before it commits to anything irreversible

Determinism here means: the same inputs, the same workflow steps, the same guardrails every time. Not the same tokens every time. That is a realistic and achievable target. It is also the thing teams skip when they are moving fast.

Typed workflow steps around the model call

Break the work into discrete typed steps. Each step has a clear input type and a clear output type. The model call is one step in the pipeline, not the whole thing.

type WorkflowInput = {
  userId: string;
  rawRequest: string;
};

type EnrichedInput = WorkflowInput & {
  userContext: UserContext;
  relevantDocs: string[];
};

type ModelOutput = {
  intent: "summarize" | "search" | "draft" | "unknown";
  confidence: number;
  payload: string;
};

type WorkflowResult = {
  response: string;
  audit: {
    intent: string;
    humanReviewed: boolean;
  };
};

async function enrich(input: WorkflowInput): Promise<EnrichedInput> {
  const [userContext, relevantDocs] = await Promise.all([
    fetchUserContext(input.userId),
    fetchRelevantDocs(input.rawRequest),
  ]);
  return { ...input, userContext, relevantDocs };
}

async function classify(enriched: EnrichedInput): Promise<ModelOutput> {
  // Model call is isolated here, not scattered everywhere
  const raw = await llm.complete(buildClassificationPrompt(enriched));
  return parseAndValidate(raw);
}

async function respond(output: ModelOutput): Promise<WorkflowResult> {
  const response = await generateResponse(output);
  return {
    response,
    audit: { intent: output.intent, humanReviewed: false },
  };
}

async function runWorkflow(input: WorkflowInput): Promise<WorkflowResult> {
  const enriched = await enrich(input);
  const classified = await classify(enriched);
  return respond(classified);
}

Each step is independently unit testable. You can mock classify to return a fixed ModelOutput and test respond in complete isolation. That was impossible when the prompt was the runtime.

Structured output + schema validation as a contract

The model call step should never return a raw string when you need structured data. Use JSON mode, tool calling, or a schema constrained completion, then validate immediately.

import { z } from "zod";

const ModelOutputSchema = z.object({
  intent: z.enum(["summarize", "search", "draft", "unknown"]),
  confidence: z.number().min(0).max(1),
  payload: z.string().min(1),
});

async function classify(enriched: EnrichedInput): Promise<ModelOutput> {
  const raw = await llm.complete(buildClassificationPrompt(enriched), {
    response_format: { type: "json_object" },
  });

  const parsed = JSON.parse(raw);
  const result = ModelOutputSchema.safeParse(parsed);

  if (!result.success) {
    throw new ClassificationValidationError(result.error, raw);
  }

  return result.data;
}

Zod gives you a contract. If the model drifts, the validation catches it before the rest of your app sees the output. The answer to "how do you validate LLM responses?" is: schema validation on parse, not on trust.

Retries, idempotency, and failure gates

Validation failures should not crash silently. Wrap the model call with a retry budget and a typed failure signal:

type ClassifyResult =
  | { ok: true; data: ModelOutput }
  | { ok: false; reason: "validation" | "timeout" | "rate_limit"; raw?: string };

async function classifySafe(
  enriched: EnrichedInput,
  maxAttempts = 2
): Promise<ClassifyResult> {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      const data = await classify(enriched);
      return { ok: true, data };
    } catch (err) {
      if (err instanceof ClassificationValidationError && attempt < maxAttempts) {
        continue; // one retry on schema failure
      }
      if (err instanceof RateLimitError) {
        return { ok: false, reason: "rate_limit" };
      }
      return { ok: false, reason: "validation", raw: (err as any).raw };
    }
  }
  return { ok: false, reason: "validation" };
}

Idempotency matters when retries touch external state. If your workflow calls an API inside the model step, wrap it in an idempotency key so a retry does not double the side effect. The workflow layer controls this. The model itself cannot.

Where a human gate belongs

A hybrid memory and retrieval approach (automatic retrieval at request start plus explicit storage) keeps agent state predictable. So does knowing when not to automate the final step.

High impact or irreversible steps should route to a human via a control gate before committing. Not because LLMs are bad. Because some decisions carry real consequences and the cost of a wrong one outweighs the automation gain.

async function runWorkflow(input: WorkflowInput): Promise<WorkflowResult> {
  const enriched = await enrich(input);
  const classifyResult = await classifySafe(enriched);

  if (!classifyResult.ok) {
    return queueForHumanReview(enriched, classifyResult.reason);
  }

  const { data: classified } = classifyResult;

  // Irreversible or low-confidence intent routes to human review
  if (classified.intent === "draft" && classified.confidence < 0.85) {
    return queueForHumanReview(enriched, "low confidence on draft intent");
  }

  return respond(classified);
}

The control gate is a typed branch in your workflow, not a prompt instruction. "Only do this if you are sure" is not a guardrail. A typed branch is.

If you want to go deeper on how this fits into a full system, I wrote up the production architecture for agents including how to wire these patterns together at scale.

FAQ

How do you make LLM output deterministic?
You cannot make the model itself deterministic. You make the system deterministic around it. Schema validated structured output, typed workflow steps, and retry gates with failure signals are the practical levers. The model is one isolated black box step in an otherwise typed, testable pipeline.

What is structured output?
Structured output means the model returns data in a schema you define rather than freeform prose. Most providers support JSON mode or function calling. You parse and validate the result immediately with a schema library. If it does not match the schema, treat it as a failed call, not a soft warning.

How do you validate LLM responses?
Parse the response as JSON, then run it through a schema validator. Zod is a common choice in TypeScript projects. A safeParse call gives you a typed result: success with data or failure with an error you can act on. Failure is an exception to handle, not a case to log and move on.

If you want a deeper look at how deterministic workflows fit into a full production system, I cover the complete production architecture for agents on my site.

If you want Next.js for AI products wired up end to end, that is exactly the kind of work I take on.

Drop a comment below. Curious what patterns people use to keep LLM features testable in production.

One AGENTS.md for every coding agent: stop maintaining CLAUDE.md and GEMINI.md separately

Mudassir Khan — Thu, 11 Jun 2026 10:32:17 +0000

You have a CLAUDE.md in your repo. You have a GEMINI.md. If you use Cursor, you have .cursorrules. They all say roughly the same thing: run pnpm test before committing, don't touch the generated files in src/generated/, keep components colocated with their tests.

Then you update one and forget the other two. The agent you're using today reads the stale file. Bugs follow.

There's a spec for this now. It's called AGENTS.md and it's worth knowing about.

The problem: one repo, five instruction files

Every coding agent has its own convention for where it looks for instructions. Claude Code reads CLAUDE.md. Codex reads AGENTS.md. Cursor reads .cursorrules. Gemini Code Assist reads GEMINI.md. Some tools read from multiple places and merge them.

The result is that a shared codebase ends up with a pile of overlapping instruction files, each slightly out of date. Whoever added the note about the database migration last month updated CLAUDE.md. Nobody updated the Cursor file. Now your teammate is using Cursor and the agent keeps trying to use the old migration pattern.

This is a solved problem in human tooling. We don't maintain separate README-for-alice.md and README-for-bob.md. We have one README. We need the same thing for agents.

What AGENTS.md is and where it came from

AGENTS.md is an open format that works as a "README for agents." It loads automatically into an agent's context and holds the information that would clutter a human README: build steps, test commands, project conventions, and things the agent should never do.

The format emerged from collaboration across the ecosystem. OpenAI Codex, Google Jules, Cursor, Amp, and Factory were all involved in the early development. A growing set of tools now supports it: Aider, goose, opencode, Zed, Warp, VS Code, and Devin are all on board.

The spec lives at agents.md and on GitHub. It's intentionally minimal. The goal is to be readable by both humans and tools, not to define a complex schema.

What goes in it (and what stays in the README)

Think of it as the context your agent needs that a human already knows from working in the repo. A good AGENTS.md usually covers:

# AGENTS.md

## Build
pnpm install
pnpm run build

## Test
pnpm test          # unit tests (Vitest)
pnpm test:e2e      # Playwright, requires local dev server

## Conventions
- Components live next to their tests: src/components/Foo/Foo.tsx + Foo.test.tsx
- Never edit files in src/generated/ — they are codegen outputs
- Use Zod for all input validation, not manual typeof checks
- CSS modules only, no inline styles in components

## Architecture notes
- Auth is handled in src/middleware.ts — do not add auth logic to route handlers
- Database access goes through src/lib/db.ts only — never import Prisma client directly

What stays in the README: human onboarding context, project history, how to get a dev environment running the first time, links to docs. That's for humans. The agent doesn't need your origin story.

The split sounds obvious once you name it. In practice, most READMEs are a mix of both, which is why agents end up reading things that don't help them and missing things that do.

Tool support across the ecosystem

Here's where things stand. This is worth bookmarking because it's moving fast:

Tool	Reads AGENTS.md
OpenAI Codex	Yes (native)
Google Jules	Yes
Cursor	Yes
Amp	Yes
Factory (Droids)	Yes
Aider	Yes
goose	Yes
opencode	Yes
Zed	Yes
Warp	Yes
VS Code (Copilot)	Yes
Devin	Yes
Claude Code	Reads `CLAUDE.md` natively; AGENTS.md support in progress

Claude Code currently reads CLAUDE.md as its primary file. If you're using Claude Code alongside other agents, you still want AGENTS.md as your canonical source and CLAUDE.md as a thin wrapper (more on that in the next section).

Deriving CLAUDE.md from one source

The pattern that works: put everything shared in AGENTS.md. In CLAUDE.md, include a short note pointing to it, then add only configuration specific to Claude.

<!-- CLAUDE.md -->

# Claude Code Instructions

See AGENTS.md for shared project conventions. The notes below are Claude-specific.

## Extended thinking
Use extended thinking for architecture decisions and debugging sessions where context depth matters.

## MCP tools
Local dev uses the Filesystem MCP mounted at /tmp/project. Don't use the Bash tool for file operations when the Filesystem MCP is available.

This way AGENTS.md is the truth. CLAUDE.md is an extension. When a convention changes (new test runner, new linting rule), you update one file.

Same pattern for GEMINI.md or any other agent file: start with "see AGENTS.md" and add only the things that are genuinely specific to that tool.

Gotchas: hierarchy, extensions, and edge cases

Codex extends the spec with a hierarchy. OpenAI Codex supports a hierarchical AGENTS.md system where you can place AGENTS.md files in subdirectories and they compose together. This is an extension specific to Codex, not part of the base spec. Other tools read the root AGENTS.md only (or whatever their lookup path is). Don't rely on the hierarchy pattern if you're targeting the broader ecosystem.

Keep it evergreen. The same rule about avoiding dates that applies to your README applies here. Agent instructions that reference a specific sprint or quarter confuse the agent when that sprint is long past. Write instructions for the current state of the codebase, not for what you were working on when you wrote them.

Test descriptions, not test output. Some teams paste in example test output to tell the agent "this is what passing tests look like." That output goes stale fast. Instead, describe the test runner invocation and what a passing run looks like in plain terms. If you're wiring up more rigorous evaluation for LLM agents in production, this framing carries over.

Tool lookup paths vary. Most tools look for AGENTS.md at the repo root, but some support workspace configurations. Check the docs for whichever tool you're using if you have a monorepo setup. Codex in particular has well documented monorepo support via the hierarchical extension.

FAQ

What is AGENTS.md?
It's an open format for a single instruction file that multiple coding agents can read automatically. It holds project conventions, build commands, and test steps that you want every agent working in the repo to know.

Which tools support AGENTS.md?
OpenAI Codex, Google Jules, Cursor, Amp, Factory, Aider, goose, opencode, Zed, Warp, VS Code (Copilot), and Devin. Claude Code reads CLAUDE.md natively and AGENTS.md support is in progress.

How is it different from a README?
A README is written for humans. AGENTS.md is written for agents. It covers build and test mechanics, codebase conventions, and architecture rules. Human onboarding context belongs in the README.

If you want to go deeper on structuring AI systems architecture in your codebase, that is exactly the kind of work I take on.

More writing on agentic patterns and developer tooling on my blog.

Curious which agents you're targeting. Drop a comment with your stack and whether you've hit the problem of maintaining five instruction files.

Claude Fable 5: Features, Pricing, and Fallbacks

Mudassir Khan — Wed, 10 Jun 2026 17:37:38 +0000

Claude Fable 5: Features, Pricing, and Fallbacks

Anthropic just shipped claude-fable-5, its most capable widely released model. It is a Mythos class model made safe for general use, with a 1M token context window, adaptive thinking, and a safety classifier that quietly hands risky prompts to Claude Opus 4.8. Here is what actually matters when you wire it into a product.

Fable 5 and Mythos 5 are the same model

The launch came in two names. claude-fable-5 and claude-mythos-5 are the same underlying model. The only difference is the safeguards.

Mythos 5 runs without the safety classifiers and is limited to a small group of cyber defenders through Project Glasswing. Fable 5 is the version everyone gets, on the Claude API, Amazon Bedrock, Vertex AI, and Microsoft Foundry. A Mythos class model sits a tier above the Opus class in raw capability, so when your request is not flagged, you are talking to the strongest model Anthropic has put in general hands. Their own data says more than 95% of Fable sessions never hit a fallback.

The specs you plan around

Spec	Claude Fable 5	Claude Opus 4.8
API id	`claude-fable-5`	`claude-opus-4-8`
Context window	1M tokens	up to 1M tokens
Max output	128k tokens	lower
Input price / M tokens	$10	$5
Output price / M tokens	$50	$25
Thinking mode	adaptive only	configurable

Fable 5 costs twice as much per token as Opus 4.8. That premium buys the strongest reasoning and the longest autonomous runs, but it means routing matters. Send the hard, long horizon tasks to Fable and keep routine work on Opus or Sonnet. The 1M context is on by default, and the 128k output ceiling is enough to return whole files or a long migration in one response.

Where it actually leads

The headline is long horizon autonomy. The longer the task, the bigger Fable 5's lead over earlier models.

Software engineering. Stripe ran a codebase wide migration across 50 million lines of Ruby in a single day. Fable 5 also tops Cognition's FrontierCode eval, even at medium effort.
Knowledge work. Highest score of any model on Hebbia's finance benchmark, with real gains on document reasoning and chart and table interpretation.
Vision. New state of the art. It reads precise numbers off scientific figures and rebuilds a web app's source from screenshots alone.
Memory. It stays focused across millions of tokens and improves using notes it writes to a file based memory tool.

The safety classifier is the thing to design for

This is the part that will surprise you in production. A separate classifier model sits in front of Fable 5. When a prompt looks like it touches cybersecurity, biology, chemistry, or distillation, Fable does not answer. The request falls back to Claude Opus 4.8, and the user is told it happened.

The classifiers are tuned conservatively, so they sometimes catch harmless prompts. Anthropic says the fallback fires in under 5% of sessions on average. For builders, the move is to expect the occasional fallback on benign cyber or bio adjacent prompts and handle it gracefully instead of treating it as an error.

What changes on the API

The biggest one is refusals. When Fable 5 declines, the Messages API does not throw. You get a successful HTTP 200 with stop_reason set to refusal, and it tells you which classifier declined. You are not billed for a request refused before any output is generated.

// A refused request comes back as a normal 200 response
{
  "type": "message",
  "role": "assistant",
  "stop_reason": "refusal",
  "content": [],
  "usage": { "input_tokens": 412, "output_tokens": 0 }
}

A refused request can usually be served by another model. Pass the fallbacks parameter and the API retries for you, or use the SDK middleware to retry from the client. Fallback credit refunds the prompt cache cost so you do not pay it twice.

{
  "model": "claude-fable-5",
  "fallbacks": ["claude-opus-4-8"],
  "messages": [ ]
}

Three more behaviors that are specific to Fable 5 and Mythos 5:

Adaptive thinking is always on. It is the only thinking mode. You cannot disable thinking. Use the effort parameter to control depth and spend.
Raw thinking is never returned. It is omitted by default. Set thinking display to summarized for readable summaries, and pass thinking blocks back unchanged in multiturn conversations on the same model.
30 day data retention. Both models are Covered Models, so zero data retention is not available. The data is used for safety, not training.

Pricing and availability

Pricing is $10 per million input tokens and $50 per million output tokens for both models. On the Claude API and consumption based Enterprise plans, Fable 5 is fully available now. On subscription plans the rollout is staged: included on Pro, Max, Team, and seat based Enterprise for a short window, then drawing on usage credits until capacity allows it back into the standard plans. Mythos 5 stays restricted to Glasswing partners and, soon, a small set of biology researchers.

Three things to check before you ship on Fable 5

Handle stop_reason: "refusal" as a normal response, not an exception.
Decide your fallback model now, either with the fallbacks param or SDK middleware.
Drop the thinking: {"type": "disabled"} path. It is not supported here.

If you want a deeper look at Claude Fable 5, I cover the features, pricing, and fallback behavior in more detail on my site.

If you want a model like this wired into a real product end to end, that is exactly the kind of work I take on.

Drop a comment if your fallback strategy looks different. Curious what people are routing to Opus versus keeping on Fable.

AI agent memory management: beyond the context window

Mudassir Khan — Sat, 06 Jun 2026 10:17:54 +0000

AI agent memory management: beyond the context window

Your agent answered correctly five minutes ago. Now it's asking for the same information again. The context window filled up, the early messages got evicted, and all that history is gone.

This is not a hallucination problem. It's a memory architecture problem.

The symptom: agents that forget during a task

You're debugging an agent that handles multistep workflows. Somewhere around turn 15, it starts contradicting itself. It asks questions you already answered in turn 3. It ignores decisions it made in turn 7.

Context limits are the obvious culprit, but the real issue is deeper. Most agent implementations treat the context window as memory. It isn't. It's working memory that evicts old data as soon as the window fills. The moment something scrolls out of the context, the agent has no path back to it.

That's the gap this article addresses.

Working memory vs longterm memory

A context window is the agent's working memory. It holds the current conversation, the current task state, and whatever you stuffed in at the start. It's fast to read, but it resets. Every new session starts blank.

Longterm memory is what persists across sessions: user preferences, prior decisions, learned facts about the problem domain. Without it, every session is the agent's first session.

The distinction matters because the solutions are different. Working memory problems (forgetting during a task) need context management techniques. Longterm memory problems (cross session amnesia) need a storage and retrieval layer.

Most articles conflate the two. Most agents solve neither properly.

The lost in the middle problem and why position matters

Even inside a single context window, position matters more than you'd think.

Research shows that LLMs exhibit a "lost in the middle" effect: accuracy is highest when relevant information is at the start or end of the context, and drops significantly for information buried in the middle. If you have a 64k token window and you put the most critical retrieved documents at position 30k, you've effectively hidden them from the model.

The practical consequence: in a RAG system, you should not dump all your retrieved chunks in the middle of the prompt and hope the model weighs them equally. It doesn't.

A production fix is to place your highest confidence retrieved documents at the very beginning and end of the context. Treat the context window like a sandwich: critical context at the top, critical context again at the bottom, filler in the middle.

The hybrid pattern: autoretrieve at start, agent driven storage after

The pattern that holds up in production is a combination of two mechanisms.

Autoretrieve at request start. Before every agent turn, automatically retrieve relevant longterm memories based on the current query or task state. This means the agent always starts with the best available context, even in a fresh session.

Agent driven storage. Let the agent decide what is worth keeping. After completing a task or making a significant decision, the agent writes to longterm memory: "User prefers TypeScript strict mode, reminded me three times." That's information worth retrieving on the next session.

You can think of solid agent memory management as a filing cabinet alongside the working scratchpad. Autoretrieve pulls relevant files from the cabinet at the start of each turn. Agent driven storage files things back when they are worth keeping.

Where vector stores fit and how to rerank

Longterm memory at scale lives in vector databases. You embed memories (or document chunks) into a vector space and retrieve by semantic similarity. When a new query arrives, you run a similarity search and pull the top K most relevant chunks.

The problem is that "most similar" and "most useful" are not the same thing. A retrieval system that returns raw cosine similarity scores will sometimes surface tangentially related content over the most relevant hit. That's where reranking earns its place.

A reranker takes the top K retrieved chunks and scores them again using a more expensive crossencoder model. It's slower than vector similarity search, but it runs on a small candidate set (K is usually 10 to 20), so the latency stays manageable. The output is a reordered list where the most genuinely useful chunks end up at the top.

If you're picking a stack, you can compare vector databases by latency, filtering support, and managed hosting options before committing.

A minimal memory loop in code

Here's a realistic Python sketch of the hybrid pattern. It uses a hypothetical memory client wrapping a vector store.

from memory_client import MemoryClient
from llm_client import LLMClient

memory = MemoryClient(user_id="user_123")
llm = LLMClient()

def agent_turn(query: str, agent_id: str) -> str:
    # Step 1: autoretrieve relevant memories before every turn
    memories = memory.search(
        query=query,
        agent_id=agent_id,
        top_k=5,
        rerank=True,
    )

    # Step 2: inject top memories at start, rest at end (sandwich pattern)
    top_memories = memories[:2]    # highest confidence at the very top
    bottom_memories = memories[2:] # remaining near the bottom

    context_parts = []
    if top_memories:
        context_parts.append(format_memories(top_memories))

    context_parts.append(query)

    if bottom_memories:
        context_parts.append(format_memories(bottom_memories))

    response = llm.complete("\n\n".join(context_parts))

    # Step 3: agent driven storage — write what is worth keeping
    if response.should_store:
        memory.add(
            content=response.memory_payload,
            agent_id=agent_id,
            user_id="user_123",
        )

    return response.text

The should_store flag is where the interesting decisions happen. You can implement it as a second LLM call ("Is this response something worth remembering for future sessions?"), a simple heuristic (decisions over a certain length, or responses containing explicit preferences), or a structured output field the main LLM populates.

Start simple. A naive heuristic beats no memory at all, and you can upgrade the storage logic once you see what your agent actually needs to keep.

FAQ

What is the difference between a context window and memory?

A context window is temporary working memory. It holds the current turn's information and is cleared between sessions. Memory is a persistent store that survives session resets, backed by a database the agent reads from and writes to explicitly.

How do agents remember across sessions?

They don't automatically. You wire up explicit storage (usually a vector database or a structured keyvalue store) and retrieval logic. The agent writes important information to the store at the end of a task and retrieves relevant entries at the start of the next session.

What is the lost in the middle problem?

LLMs pay less attention to information in the middle of a long context window than to information at the start or end. If you place your most critical retrieved documents in the center of a large prompt, the model may effectively ignore them. The fix is to place highest confidence chunks at the very beginning and end of the context.

If you want a deeper look at agent memory architecture, I cover it in more detail on my site.

If you want this wired up on your own system end to end, that is exactly the kind of work I take on.

Drop a comment if your setup looks different. Curious what memory stacks people are running in production.

How to test MCP servers in TypeScript before they break in production

Mudassir Khan — Mon, 01 Jun 2026 10:20:52 +0000

Your MCP server works on your laptop. The tool calls return the right shapes, the client connects cleanly, the session behaves. Then you deploy it and a client reconnects after a network hiccup and the session state is gone. Or you scale to two instances and half the requests fail because session IDs resolve to the wrong process. Or someone sends two concurrent requests and the tool handler corrupts shared state.

Testing catches these before your users do. This is a testing playbook for TypeScript MCP servers built on the official SDK.

The demo-to-production gap for MCP servers

The official TypeScript SDK makes it easy to get something working. A few tool registrations, an McpServer instance, a transport, and you are serving. The problem is that "working" in the demo sense and "working" in the production sense are different things.

A demo tests one happy path. Production tests edge cases that emerge from real clients: reconnects, concurrent tool calls, malformed inputs, slow downstream APIs, and the transport contract itself. None of those show up in a single manual run against your local instance.

The gap is not a criticism of the SDK. It is a consequence of how easy the SDK makes it to build a server without thinking about what breaks it. A test suite closes the gap before you ship.

What actually breaks: transport, sessions, tool contracts

Three categories fail most often.

Transport behavior. The SDK added Streamable HTTP support in version 1.10.0. Under this transport, the server exposes a single HTTP endpoint that handles both POST and GET. Clients use POST for tool calls and GET to open a streaming connection via server sent events. Tests that only exercise stdio miss this entirely.

Session state. The StreamableHTTPServerTransport is stateful per session. If you store anything in process memory keyed by session ID, a restart or a second instance will drop it. Tests that do not simulate reconnects miss this failure mode.

Tool contracts. Each tool you register has an input schema and an expected output shape. Tests that call tools with valid inputs only miss the cases where a client sends a slightly wrong shape or a downstream API returns something unexpected.

Unit-testing tools and resources in isolation

The cleanest place to start is the tool handler itself, before any transport is involved.

Each tool handler is a function that takes validated input and returns a result. Extract the handler logic from the server.tool() registration so you can call it directly in tests.

// tool-handlers.ts
export async function getItemHandler(input: { id: string }) {
  const item = await fetchItem(input.id);
  return { content: [{ type: "text", text: JSON.stringify(item) }] };
}

// getItem.test.ts
import { getItemHandler } from "./tool-handlers";

test("returns item content", async () => {
  const result = await getItemHandler({ id: "abc" });
  expect(result.content[0].type).toBe("text");
});

This pattern makes each handler independently testable without spinning up the full server. Stub the downstream calls with your test framework's mocking utilities. Cover the happy path, malformed input, and downstream failure.

Resource handlers work the same way. Extract, test in isolation, stub dependencies.

Contract assertions against the MCP schema

After unit tests, the next layer checks that your tool registrations conform to the MCP protocol. A contract test instantiates the real server and sends actual protocol requests, then asserts on the response shape.

import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { InMemoryTransport } from "@modelcontextprotocol/sdk/inMemory.js";
import { createServer } from "./server";

test("list tools returns registered tools", async () => {
  const server = createServer();
  const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair();
  await server.connect(serverTransport);

  const client = new Client({ name: "test", version: "1.0.0" }, {});
  await client.connect(clientTransport);

  const result = await client.listTools();
  const toolNames = result.tools.map((t) => t.name);
  expect(toolNames).toContain("get-item");
});

The InMemoryTransport in the SDK is designed exactly for this. It lets you run client and server in the same process without any network, which keeps tests fast and deterministic.

Assert on the full response shape for each tool: input schema, output content types, error response format. This is the layer that catches the gap between what your server claims to support and what it actually returns.

Testing Streamable HTTP behavior

The in memory transport covers the protocol layer. Testing the HTTP transport layer catches a different class of failure: auth middleware, session header handling, and the streaming path.

Stand up a real HTTP server on a random port, run your requests against it, and shut it down after each test.

import { createServer as createHttpServer } from "http";
import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js";

let httpServer: ReturnType<typeof createHttpServer>;
let baseUrl: string;

beforeAll(async () => {
  const transport = new StreamableHTTPServerTransport({
    sessionIdGenerator: () => crypto.randomUUID(),
  });
  const mcpServer = createServer();
  await mcpServer.connect(transport);
  httpServer = createHttpServer((req, res) => transport.handleRequest(req, res));
  await new Promise<void>((r) => httpServer.listen(0, () => r()));
  const addr = httpServer.address() as { port: number };
  baseUrl = `http://localhost:${addr.port}`;
});

afterAll(() => httpServer.close());

test("POST returns a valid MCP response", async () => {
  const res = await fetch(`${baseUrl}/mcp`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ jsonrpc: "2.0", method: "tools/list", params: {}, id: 1 }),
  });
  expect(res.status).toBe(200);
  const json = await res.json();
  expect(json.result.tools).toBeDefined();
});

Add a test that opens GET on the same endpoint and confirms the SSE connection accepts. Add a test that sends an invalid session ID and checks the server handles it without crashing.

A CI setup that catches regressions

A test suite only helps if it runs consistently. For MCP servers, the minimal CI setup is:

Unit tests on every commit (fast, no network)
Contract tests via InMemoryTransport on every commit (still fast)
HTTP transport tests on pull requests and on merge to main

If you deploy across multiple instances, add a test that starts two server processes and verifies that a session created on instance one can be resumed on instance two. This tests your external session store. It is slower, so running it on pull requests is reasonable.

If you are building an AI product that depends on your MCP server, the deployment and observability patterns in Next.js for AI products apply directly to the production layer above the server. For the enterprise session model, see MCP for enterprise agents.

FAQ

What is an MCP server?
An MCP server exposes tools, resources, and prompts to LLM clients via the Model Context Protocol. Clients connect to it and call tools by name, receiving structured results.

How do you test an MCP server?
Start with unit tests on each tool handler in isolation. Add contract tests using InMemoryTransport. Add HTTP transport tests against a local server instance for the full network path.

What transport does MCP use?
MCP supports stdio for local use and Streamable HTTP for networked servers. Streamable HTTP uses a single endpoint that handles POST requests for tool calls and GET requests for SSE streaming. The TypeScript SDK supports Streamable HTTP from version 1.10.0 onward.

Building production AI systems? I consult on agentic AI and AI product engineering at mudassirkhan.me.

Per User OAuth in a Next.js MCP Server (Step by Step)

Mudassir Khan — Fri, 29 May 2026 13:32:36 +0000

Per User OAuth in a Next.js MCP Server (Step by Step)

Your MCP server is using one shared API key for every caller. That works in a demo. The second you need each user to call a tool with their credentials (their GitHub token, their Notion workspace, their Stripe key) a shared key blows up. Here's how to wire per user OAuth into a Next.js MCP server so each session gets its own scoped token.

The problem with shared keys

When you build an MCP server the usual way, tools look like this:

server.tool("get-my-repos", {}, async () => {
  const octokit = new Octokit({ auth: process.env.GITHUB_TOKEN });
  const { data } = await octokit.repos.listForAuthenticatedUser();
  return { repos: data.map((r) => r.full_name) };
});

That GITHUB_TOKEN is yours. Every user who connects to this server gets results from your account. In a multitenant setup, that's a footgun. Alice calls get-my-repos and sees Bob's repos. Bob calls create-issue and writes to Alice's project.

You need each MCP session to carry its own OAuth token. Here's how to do it without reinventing the session layer.

How the token flow works

Three pieces fit together:

NextAuth holds the user's OAuth access token in their session cookie.
A Next.js Route Handler acts as the MCP transport. It can read the session before any tool runs.
MCP tool context receives the token via closure (no global state, no env vars).

The key insight: your Route Handler runs in a request context, so it has full access to the session. You create the MCP server instance inside that handler, which means every tool closure captures the authenticated user's token before it runs.

Step 1: Set up NextAuth with token persistence

Install what you need:

npm install next-auth @auth/core @octokit/rest

Configure NextAuth to store the OAuth access token in the session. Critically, you need the jwt and session callbacks. By default NextAuth does not expose the raw access token to your session object:

// src/app/api/auth/[...nextauth]/route.ts
import NextAuth from "next-auth";
import GitHub from "next-auth/providers/github";

export const { handlers, auth } = NextAuth({
  providers: [
    GitHub({
      clientId: process.env.GITHUB_CLIENT_ID!,
      clientSecret: process.env.GITHUB_CLIENT_SECRET!,
      authorization: {
        params: { scope: "repo read:user" },
      },
    }),
  ],
  callbacks: {
    async jwt({ token, account }) {
      // `account` is only present on the initial sign-in
      if (account) {
        token.accessToken = account.access_token;
      }
      return token;
    },
    async session({ session, token }) {
      session.accessToken = token.accessToken as string;
      return session;
    },
  },
});

export const { GET, POST } = handlers;

And extend the session type so TypeScript doesn't complain:

// src/types/next-auth.d.ts
import { DefaultSession } from "next-auth";

declare module "next-auth" {
  interface Session extends DefaultSession {
    accessToken: string;
  }
}

After a user signs in with GitHub, session.accessToken is their personal GitHub token scoped to repo read:user. It's stored in their encrypted session cookie, not anywhere on your server.

Step 2: The MCP Route Handler with per user context

This is the important part. Your MCP transport lives inside a Route Handler at /api/mcp:

// src/app/api/mcp/route.ts
import { createMcpHandler } from "@vercel/mcp-adapter";
import { auth } from "@/app/api/auth/[...nextauth]/route";

export const POST = async (request: Request) => {
  // Read the session for THIS request
  const session = await auth();

  if (!session?.accessToken) {
    return new Response("Unauthorized", { status: 401 });
  }

  // Create a new MCP handler instance for this request.
  // The tool registrations close over `session` — each call gets its own.
  const handler = createMcpHandler(
    (server) => {
      server.tool("get-my-repos", {
        description: "Lists repositories the current user has access to",
        inputSchema: {
          type: "object",
          properties: {
            visibility: {
              type: "string",
              enum: ["all", "public", "private"],
              default: "all",
            },
          },
        },
      }, async ({ visibility = "all" }) => {
        const response = await fetch(
          `https://api.github.com/user/repos?visibility=${visibility}&per_page=20`,
          {
            headers: {
              Authorization: `Bearer ${session.accessToken}`,
              Accept: "application/vnd.github+json",
            },
          }
        );

        if (!response.ok) {
          throw new Error(`GitHub API returned ${response.status}`);
        }

        const repos = await response.json();
        return {
          content: [
            {
              type: "text",
              text: repos
                .map((r: { full_name: string; private: boolean }) =>
                  `${r.full_name} (${r.private ? "private" : "public"})`
                )
                .join("\n"),
            },
          ],
        };
      });
    },
    {},
    { redisUrl: process.env.REDIS_URL }
  );

  return handler(request);
};

The session.accessToken in the tool handler is captured from the outer POST function scope. It's the token for whoever made this request. Two users hitting /api/mcp at the same moment get two separate MCP handler instances, each with their own token.

No global state. No shared credentials. Each request is isolated.

Step 3: Adding a second tool

Once the pattern is in place, adding tools is just adding more server.tool calls inside the same closure. They all get session for free:

server.tool("create-issue", {
  description: "Creates a GitHub issue in a repo the current user owns",
  inputSchema: {
    type: "object",
    required: ["repo", "title"],
    properties: {
      repo: { type: "string", description: "owner/repo format" },
      title: { type: "string" },
      body: { type: "string" },
    },
  },
}, async ({ repo, title, body }) => {
  const [owner, repoName] = repo.split("/");

  const response = await fetch(
    `https://api.github.com/repos/${owner}/${repoName}/issues`,
    {
      method: "POST",
      headers: {
        Authorization: `Bearer ${session.accessToken}`,
        Accept: "application/vnd.github+json",
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ title, body }),
    }
  );

  if (!response.ok) {
    const err = await response.json();
    throw new Error(err.message || `GitHub API returned ${response.status}`);
  }

  const issue = await response.json();
  return {
    content: [{ type: "text", text: `Created: ${issue.html_url}` }],
  };
});

This tool creates an issue on the calling user's behalf. If you'd wired this with a shared token, every issue would be created by your service account. With per user tokens, it's the actual user: their name on the issue, their rate limits, their permissions.

Verifying credential isolation

Here's a quick check before you ship. Sign in as two different GitHub accounts and grab each session cookie from the browser DevTools. Then run both calls at the same time:

# Alice's session
curl -s -X POST http://localhost:3000/api/mcp \
  -H "Cookie: next-auth.session-token=<alice-token>" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"tools/call","id":1,"params":{"name":"get-my-repos","arguments":{}}}' \
  | jq '.result.content[0].text' &

# Bob's session (run at the same time)
curl -s -X POST http://localhost:3000/api/mcp \
  -H "Cookie: next-auth.session-token=<bob-token>" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"tools/call","id":1,"params":{"name":"get-my-repos","arguments":{}}}' \
  | jq '.result.content[0].text'

You should get Alice's repos and Bob's repos in the two outputs. If both return the same list, your session isn't threading through. Check that auth() in the Route Handler is returning the right session for each request.

What to watch out for

Token expiry mid session. OAuth tokens expire. If your MCP client holds a long running session, the token can go stale between calls. Handle this by catching 401 responses from the upstream API inside your tool handlers and returning a clear error message the AI can relay to the user: "Your GitHub session expired, please sign in again."

Stateless vs stateful transports. The example above uses @vercel/mcp-adapter with a Redis URL for stateful SSE sessions. If you're using a stateless transport (plain POST with no SSE), you don't need Redis, but you also can't hold multiturn conversations over the same session. For most auth use cases, stateless is fine. You get a fresh authenticated context on every tool call.

Avoid the "just use NextAuth" trap. That's going to be the first reply to this article. Yes, NextAuth is handling the session layer here. The hard part that NextAuth does not solve for you is threading that session token into the MCP tool context specifically. That's what the closure pattern above does.

Three things to verify right now

console.log(session.accessToken) inside the Route Handler: should print a real token, not undefined.
Hit /api/mcp without a session cookie: you should get a 401, not a 500 or a 200 with the server token.
Two concurrent users, same tool, different results. That's the definitive proof the isolation is working.

If you want a deeper look at production MCP patterns at scale, I wrote about the full enterprise architecture in my post on MCP for enterprise agents. The NebulaDesk case study in my agentic AI product work is also a good read if you are building a multitenant Next.js product on top of AI tooling.

If you want this wired up on your own stack end to end, that is exactly the kind of work I take on via my Next.js for AI products service. Alternatively if you want strategic help on the broader agentic AI architecture, my consulting engagement covers that.

Drop a comment if your setup looks different. Curious what OAuth providers people are pairing with MCP in production. Notion? Linear? Stripe? All of the above?

Your LLM Is Wrong. Your Codebase Is Why.

Mudassir Khan — Tue, 26 May 2026 21:53:18 +0000

It happened on a Tuesday. I asked my AI coding assistant to explain a function I'd written three months earlier. It described a function that doesn't exist.

Not a total hallucination. The function did exist. Just not by that name, not with those parameters, not doing what the model confidently told me it was doing. The model had assembled a plausible story from vague signals and filled the gaps with fiction.

My first instinct was to blame the model. My second instinct, the one that actually helped, was to look at the code itself.

The model wasn't broken. My codebase was.

What is comprehension debt?

Technical debt is code that's hard to change. Comprehension debt is code that's hard to understand. Not just by future developers. By anything that has to read it cold: a new hire, a rubber duck, and increasingly, an AI assistant.

You've probably heard "write code as if the next maintainer is a serial killer who knows where you live." The LLM version is more forgiving. But not by much.

Comprehension debt shows up when the intent of your code isn't captured in your code. The logic works. The tests pass. But nothing in the source tells a reader why a function does what it does, what its constraints are, or what it absolutely should not do. That knowledge lives in someone's head, in a Slack thread from two months ago, or nowhere at all.

LLMs don't have access to the Slack thread. They only have your source.

Five signals your LLM is showing you

When your AI assistant gets your own codebase wrong, it's not random. The errors cluster around specific failure modes, and each one points to a real gap.

1. It invents function names.
The model calls functions that don't exist, or calls existing functions by the wrong name. This usually means your naming is inconsistent or your barrel exports are incomplete. The model is pattern matching across conventions that don't agree with each other.

2. It gets parameter types wrong.
It passes a string where you want a typed enum, or a plain object where you've defined a specific interface. This almost always means missing or implicit type annotations in your function signatures. The model is guessing.

3. It imports packages you don't use.
It reaches for lodash or axios when you've got utility wrappers that wrap those already. Your actual internal abstractions aren't legible to the model because they aren't documented anywhere they can be found. The model falls back to what it knows from training.

4. It uses patterns you've deprecated.
It calls the old version of your API, the one you stopped using eight months ago. Your codebase still contains those old patterns (maybe for backward compatibility, maybe just because cleanup hasn't happened yet) and the model doesn't know which version is current. Deprecation comments cost thirty seconds to write. Their absence costs you five minutes of confusion per assistant interaction.

5. It doesn't know the business rule.
It gives you the technically valid version of a function, not the version that accounts for the actual constraint. "This user lookup should always check the soft delete flag first" lives in a comment in no file. It was decided in a call. The model can't know what was never written down.

Each of these errors is a free audit item. You didn't have to run a tool to find it. The model found it for you.

The five minute audit

You don't need a formal process for this. You just need to treat your LLM's confusion as a signal instead of noise.

Pick a module. Any module that's been around for more than a few months. Feed it to your AI assistant and ask these questions:

"What does this module do? Describe it in two sentences."
"List all exported functions, their parameters, and their return types."
"What would break if I deleted this module?"
"What should I never pass to this module's main function?"
"Is there anything in this module that looks like it was deprecated but never removed?"

Don't correct the model when it gets something wrong. Write down what it got wrong. That list is your comprehension debt register.

For a healthy module, the model will get most of this right. For a module with comprehension debt, you'll see the five signals show up fast.

I ran this on an internal TypeScript service last quarter. Twelve exported functions. The model hallucinated the names of three of them, got the return type wrong on two others, and had no idea what the rate limit parameter was for. That's a 41% wrong answer rate on a module I thought was well maintained. It wasn't. It just worked.

Working and legible are not the same thing.

What to do about it

The instinct is to reach for RAG (chunk your codebase, embed it, retrieve relevant context before each LLM call). That helps. I cover the full approach in my production RAG guide if you want the implementation details.

But RAG retrieves your documentation. If your documentation is the code itself and the code is opaque, RAG gives the model better access to opaque code. The underlying problem doesn't change.

The actual fix is cheaper than you think:

Write the intent, not the implementation. A JSDoc comment that says "Validates and normalizes a user object. Always call this before persisting to the database. Does NOT check permissions." gives the model something to retrieve. A comment that says "validates user" does not.

Mark your deprecations inline. @deprecated Use getUserV2 instead takes five seconds. It means the model stops confidently recommending the old API.

Put your business rules in the file that enforces them. Not in the ticket. Not in Confluence. In the file. A comment above the rate limit parameter that says "this is hardcoded per the billing agreement with enterprise customers, do not make it configurable" is documentation that actually travels with the code.

The goal isn't to write documentation for humans. It's to write documentation that your LLM assistant can parse so it can help you correctly. The secondary effect is that it also helps the next human on your team. That's free.

For teams working on larger AI agent systems, the memory and context patterns that help here are the same ones I break down in my post on AI agent memory management. Comprehension debt in your codebase and context gaps in your agents come from the same root cause: undocumented intent.

You can also get a quick read on your current exposure with this LLM hallucination risk estimator. It won't diagnose specific debt, but it gives you a calibrated starting point for where to focus.

The model is the test

Your LLM assistant is, right now, the most honest reader your codebase has. It doesn't know the context you carry in your head. It doesn't remember the decision you made in 2024. It reads what's there and tries to make sense of it.

When it gets something wrong, that's signal. The model isn't failing. It's showing you exactly what a reader without your context has to work with.

That's a gift. Most code never gets that kind of external read until the next engineer joins and asks the same confused questions.

Use it.

If you want a deeper look at production AI systems, I cover it on mudassirkhan.me.

How wrong does your LLM get your own codebase? Drop a number in the comments. Curious what percentage of wrong answers people are seeing in production.

Building MCP Servers in TypeScript That Don't Fall Apart

Mudassir Khan — Sun, 24 May 2026 11:18:35 +0000

Building MCP Servers in TypeScript That Don't Fall Apart

Your MCP server works great at tool number three. By tool number twelve it is a pile of switch cases you are afraid to touch. Here is the TypeScript architecture that keeps it clean as it grows — borrowed straight from Domain-Driven Design.

The Problem with Flat MCP Servers

Most MCP server tutorials start with something like this:

import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";

const server = new Server(
  { name: "my-server", version: "1.0.0" },
  { capabilities: { tools: {} } }
);

server.setRequestHandler(ListToolsRequestSchema, async () => ({
  tools: [
    { name: "get_user", description: "\"Get a user by ID\", inputSchema: { ... } },"
    { name: "create_order", description: "\"Create an order\", inputSchema: { ... } },"
    { name: "send_email", description: "\"Send an email\", inputSchema: { ... } },"
    { name: "get_product", description: "\"Get product details\", inputSchema: { ... } },"
    // ...8 more tools
  ],
}));

server.setRequestHandler(CallToolRequestSchema, async (request) => {
  switch (request.params.name) {
    case "get_user": return handleGetUser(request.params.arguments);
    case "create_order": return handleCreateOrder(request.params.arguments);
    case "send_email": return handleSendEmail(request.params.arguments);
    // ...
  }
});

Familiar? The problem is not that this code is wrong. It is that it scales to exactly one person maintaining it for exactly three weeks.

Once you hit real production requirements — different owners for user logic vs order logic, different auth contexts, stateful resources — you need structure. That is where Domain-Driven Design earns its keep.

Three DDD Concepts, One MCP Mapping

You do not need to read a 600 page book to apply the ideas that matter here. Three concepts cover 90% of what you will encounter:

DDD Concept	What it means	MCP equivalent
Bounded context	A namespace where terms have a consistent meaning	Tool name prefix (`users__get`, `orders__create`)
Aggregate root	The single entry point for a cluster of related state	A typed server context object passed to each handler
Domain event	A fact that happened in the system	An MCP notification emitted after a mutation

The translation rule for this niche: never write an "architecture" article — write a TypeScript tutorial that teaches architecture through code. So let us build the thing.

Bounded Contexts as Tool Namespaces

In DDD, a bounded context is a boundary within which a particular domain model applies. In an MCP server, that maps cleanly to a tool naming convention plus a factory function that groups related tools together.

Here is a createDomainTools factory:

// src/domains/types.ts

import { Tool, CallToolResult } from "@modelcontextprotocol/sdk/types.js";

export interface DomainTool<TInput = unknown> {
  definition: Tool;
  handler: (input: TInput) => Promise<CallToolResult>;
}

export interface DomainModule {
  namespace: string;
  tools: DomainTool[];
}

export function createDomainModule(
  namespace: string,
  tools: DomainTool[]
): DomainModule {
  return {
    namespace,
    tools: tools.map((tool) => ({
      ...tool,
      definition: {
        ...tool.definition,
        name: `${namespace}__${tool.definition.name}`,
      },
    })),
  };
}

Now define the users domain:

// src/domains/users.ts

import { createDomainModule } from "./types.js";

export const usersDomain = createDomainModule("users", [
  {
    definition: {
      name: "get",
      description: "Get a user by ID",
      inputSchema: {
        type: "object",
        properties: {
          id: { type: "string", description: "User UUID" },
        },
        required: ["id"],
      },
    },
    handler: async ({ id }: { id: string }) => {
      const user = await db.users.findById(id);
      return {
        content: [{ type: "text", text: JSON.stringify(user) }],
      };
    },
  },
  {
    definition: {
      name: "list",
      description: "List users with optional filters",
      inputSchema: {
        type: "object",
        properties: {
          limit: { type: "number" },
          role: { type: "string" },
        },
      },
    },
    handler: async ({ limit = 20, role }: { limit?: number; role?: string }) => {
      const users = await db.users.findMany({ limit, role });
      return {
        content: [{ type: "text", text: JSON.stringify(users) }],
      };
    },
  },
]);

The resulting tool names are users__get and users__list. The double underscore is a common MCP convention for namespacing — it is readable and easy to split on in client code.

I cover the agent design patterns that consume these namespaced tools in my multi-agent design patterns post — the namespace boundary maps directly to the "agent as specialist" pattern.

Aggregate Root for Server State

The aggregate root is the single class responsible for coordinating all writes to a domain's state. In an MCP server, "state" usually means cached data, auth tokens, connection pools, or feature flags that multiple tools share.

The anti-pattern: each handler closes over a different reference to the same underlying data, and you end up with stale reads mid-session.

The fix is a typed ServerContext that every handler receives as an injected dependency:

// src/context.ts

export interface UserContext {
  currentUserId: string | null;
  permissions: Set<string>;
  refresh: () => Promise<void>;
}

export interface ServerContext {
  users: UserContext;
  requestId: string;
  startedAt: number;
}

export function createServerContext(): ServerContext {
  const users: UserContext = {
    currentUserId: null,
    permissions: new Set(),
    async refresh() {
      const session = await auth.getCurrentSession();
      this.currentUserId = session?.userId ?? null;
      this.permissions = new Set(session?.permissions ?? []);
    },
  };

  return {
    users,
    requestId: crypto.randomUUID(),
    startedAt: Date.now(),
  };
}

Then update the DomainTool interface to carry context:

export interface DomainTool<TInput = unknown> {
  definition: Tool;
  handler: (input: TInput, ctx: ServerContext) => Promise<CallToolResult>;
}

Now each handler has a type safe path to shared state with no globals:

handler: async ({ id }: { id: string }, ctx: ServerContext) => {
  if (!ctx.users.permissions.has("users:read")) {
    return {
      content: [{ type: "text", text: "Forbidden" }],
      isError: true,
    };
  }
  const user = await db.users.findById(id);
  return { content: [{ type: "text", text: JSON.stringify(user) }] };
},

A stateful MCP server is also an injection surface worth thinking about carefully. I cover the specific attack patterns you should harden against in my post on prompt injection hardening for agents.

Domain Events as MCP Notifications

Domain events record facts that have happened ("OrderPlaced", "UserCreated"). MCP has a notification mechanism that maps naturally to this pattern.

Here is a typed notification emitter that the publisher can subscribe to:

// src/events.ts

import { Server } from "@modelcontextprotocol/sdk/server/index.js";

type EventName =
  | "users.created"
  | "orders.placed"
  | "orders.failed";

interface DomainEvent<TData = unknown> {
  name: EventName;
  data: TData;
  occurredAt: number;
}

export function createEventEmitter(server: Server) {
  return {
    async emit<TData>(name: EventName, data: TData): Promise<void> {
      const event: DomainEvent<TData> = {
        name,
        data,
        occurredAt: Date.now(),
      };

      // MCP notification — any connected client sees this
      await server.notification({
        method: "notifications/message",
        params: {
          level: "info",
          logger: name,
          data: JSON.stringify(event),
        },
      });
    },
  };
}

Use it inside a handler after a successful mutation:

handler: async ({ email, role }: CreateUserInput, ctx: ServerContext) => {
  const user = await db.users.create({ email, role });

  await events.emit("users.created", { userId: user.id, email });

  return {
    content: [{ type: "text", text: JSON.stringify(user) }],
  };
},

The client (your AI agent or IDE) now receives a realtime notification whenever a user is created — without polling. For testing these notification paths, the evaluation patterns I describe in evaluating LLM agents in production apply directly: set up an observer client, trigger the mutation, assert the notification fired within your SLA window.

Wiring It All Together

Here is a complete server.ts that composes the domain modules, context, and events:

// src/server.ts

import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import {
  ListToolsRequestSchema,
  CallToolRequestSchema,
} from "@modelcontextprotocol/sdk/types.js";

import { createServerContext } from "./context.js";
import { createEventEmitter } from "./events.js";
import { usersDomain } from "./domains/users.js";
import { ordersDomain } from "./domains/orders.js";

const server = new Server(
  { name: "my-structured-server", version: "1.0.0" },
  { capabilities: { tools: {}, logging: {} } }
);

const events = createEventEmitter(server);
const domains = [usersDomain, ordersDomain];

// Flat list of all tool definitions across all domains
const allTools = domains.flatMap((d) =>
  d.tools.map((t) => t.definition)
);

// Lookup map: tool name → handler
const handlerMap = new Map(
  domains.flatMap((d) =>
    d.tools.map((t) => [t.definition.name, t.handler])
  )
);

server.setRequestHandler(ListToolsRequestSchema, async () => ({
  tools: allTools,
}));

server.setRequestHandler(CallToolRequestSchema, async (request) => {
  const ctx = createServerContext();
  await ctx.users.refresh();

  const handler = handlerMap.get(request.params.name);
  if (!handler) {
    return {
      content: [{ type: "text", text: `Unknown tool: ${request.params.name}` }],
      isError: true,
    };
  }

  return handler(request.params.arguments ?? {}, ctx);
});

const transport = new StdioServerTransport();
await server.connect(transport);

The handlerMap lookup is O(1). Adding a new domain means writing one file and adding one import — the server.ts does not change.

Is This Overkill?

For three tools? Yes. For a server that a team will ship, maintain, and extend over six months? No.

The patterns here add about 40 lines of infrastructure. In exchange you get: namespaced tools that cannot collide, a single typed context object instead of scattered globals, and notifications that fire without wiring up polling clients.

If you are not sure whether your use case actually needs an MCP server or a simpler agent framework, the AI agent framework chooser on my site walks through the decision.

If you want a deeper look at how stateful MCP servers fit into larger agent architectures, I cover it in more detail on my site.

If you want this wired up on your own production system end to end, that is exactly the kind of work I take on.

Drop a comment if your setup looks different — curious what variations people are running in production.