DEV Community: Mudassir Khan

Next.js Server Actions vs tRPC: how to actually choose

Mudassir Khan — Tue, 21 Jul 2026 19:20:23 +0000

Next.js Server Actions vs tRPC: how to actually choose

If you are building a Next.js app with the App Router and trying to decide between Server Actions and tRPC for your mutations, here is the short version: reach for Server Actions when you have simple forms tied to a single web client, and reach for tRPC once you need request batching, query caching, or a shared API layer across web and mobile. Both solve the same underlying problem, getting typed data from the browser to the server without hand rolling REST endpoints. The right pick depends less on which one is trending and more on what your app actually needs to scale.

The mutation problem both tools solve

Every full stack app eventually answers the same question: how do you move data from a client component to a database, safely, with types that do not lie to you. In the Pages Router era that meant API toutes, fetch calls, and manually keeping request and response types in sync by hand. React 19 changed the equation by building Server Actions directly into React itself. Around the same time, tRPC matured into the default choice for teams who wanted end to end type safety without giving up a real API layer.

Both approaches let you call server code from the client like it is a regular function. The difference shows up once your app grows past a single form.

How Server Actions work, and what React 19 added

A Server Action is a function marked with the "use server" directive. You call it directly from a form or an event handler, and Next.js handles the network request, serialization, and cache revalidation for you.

// app/actions/create-post.ts
"use server";

import { revalidatePath } from "next/cache";
import { db } from "@/lib/db";

export async function createPost(formData: FormData) {
  const title = formData.get("title") as string;
  const body = formData.get("body") as string;

  if (!title || title.length < 3) {
    return { error: "Title needs at least 3 characters" };
  }

  await db.post.create({ data: { title, body } });
  revalidatePath("/posts");
  return { success: true };
}

Wire it to a form and you get a working mutation with zero client side JavaScript required for the base case:

// app/posts/new/page.tsx
import { createPost } from "@/app/actions/create-post";

export default function NewPostPage() {
  return (
    <form action={createPost}>
      <input name="title" placeholder="Title" required />
      <textarea name="body" placeholder="Body" required />
      <button type="submit">Publish</button>
    </form>
  );
}

Forms built this way work without any client side JavaScript at all, which is a real win for accessibility, resilience on flaky connections, and Core Web Vitals. React 19 adds two hooks that close most of the gap Server Actions used to have against client state libraries: useActionState for tracking the pending, error, and result state of a mutation, and useOptimistic for showing the new state immediately while a mutation is still in flight.

"use client";
import { useActionState } from "react";
import { createPost } from "@/app/actions/create-post";

export function PostForm() {
  const [state, formAction, isPending] = useActionState(createPost, null);

  return (
    <form action={formAction}>
      <input name="title" placeholder="Title" required />
      <textarea name="body" placeholder="Body" required />
      <button type="submit" disabled={isPending}>
        {isPending ? "Publishing..." : "Publish"}
      </button>
      {state?.error && <p role="alert">{state.error}</p>}
    </form>
  );
}

That closes most of the reason people used to bolt on a separate API layer just to get loading and error states on a form.

How tRPC works, routing, type safety, batching

tRPC takes a different approach. Instead of individual server functions sprinkled through your app, you define a router of typed procedures, and the client gets full autocomplete and type checking without writing a schema or generating code.

// server/routers/post.ts
import { z } from "zod";
import { publicProcedure, router } from "../trpc";
import { db } from "@/lib/db";

export const postRouter = router({
  create: publicProcedure
    .input(z.object({ title: "z.string().min(3), body: z.string() }))"
    .mutation(async ({ input }) => {
      return db.post.create({ data: input });
    }),

  list: publicProcedure.query(async () => {
    return db.post.findMany({ orderBy: { createdAt: "desc" } });
  }),
});

On the client, calling that mutation looks like a normal function call, fully typed end to end:

"use client";
import { trpc } from "@/lib/trpc-client";

export function PostForm() {
  const utils = trpc.useUtils();
  const createPost = trpc.post.create.useMutation({
    onSuccess: () => utils.post.list.invalidate(),
  });

  return (
    <form
      onSubmit={(e) => {
        e.preventDefault();
        const form = new FormData(e.currentTarget);
        createPost.mutate({
          title: "form.get(\"title\") as string,"
          body: form.get("body") as string,
        });
      }}
    >
      <input name="title" placeholder="Title" required />
      <textarea name="body" placeholder="Body" required />
      <button type="submit" disabled={createPost.isPending}>
        {createPost.isPending ? "Publishing..." : "Publish"}
      </button>
    </form>
  );
}

tRPC has request batching and query caching built in, neither of which Server Actions gives you on its own. If your app has a dashboard firing several queries at once, or you are serving both a web client and a mobile client from the same backend, that caching and batching layer stops being optional pretty fast.

Key differences: progressive enhancement vs supporting multiple clients

The two tools optimize for different failure modes.

	Server Actions	tRPC
Works without JavaScript	Yes, forms submit natively	No, needs the client runtime
Type safety	Yes, through TypeScript function signatures	Yes, through inferred router types
Request batching	Not built in	Built in
Query caching	You wire it yourself with `revalidatePath` or a library	Built in, backed by React Query
Best for	Web only apps, forms, simple mutations	Apps serving web and mobile, complex dashboards
Setup cost	Near zero, it is just a function	A router, a client, and some wiring

Server Actions win on simplicity and progressive enhancement. tRPC wins the moment your data layer needs to serve more than one kind of client, or your queries get complex enough that manual cache invalidation turns into a chore.

The hybrid pattern: combining both

You do not have to pick exactly one. A pattern that holds up well in production: use Server Actions for the simple, form driven mutations that benefit from progressive enhancement, and keep tRPC around for anything read heavy, batched, or shared across clients.

// app/actions/create-post.ts
"use server";

import { appRouter } from "@/server/routers/_app";
import { createCallerFactory } from "@/server/trpc";

const createCaller = createCallerFactory(appRouter);

export async function createPost(formData: FormData) {
  const caller = createCaller({});
  return caller.post.create({
    title: formData.get("title") as string,
    body: formData.get("body") as string,
  });
}

This gives you a Server Action as the entry point, so the form still works without client side JavaScript, while the actual mutation logic lives in exactly one place, the tRPC router. You get progressive enhancement on the surface and a reusable, structured API underneath that your mobile app, or any other client, can call directly.

Decision table: pick based on your app type

Your situation	Pick
Marketing site with a contact form	Server Actions
Internal admin panel, web only	Server Actions, add tRPC later if it grows
SaaS product with a web app and a mobile app	tRPC, or the hybrid pattern
Dashboard with many simultaneous queries	tRPC
You want the smallest possible setup	Server Actions
You already have a tRPC router from a previous project	Keep it, add Server Actions only where forms need progressive enhancement

None of this is permanent. Plenty of teams start with Server Actions because it is the fastest way to ship a form, then bring in tRPC once the query side of the app gets complicated enough to need caching.

FAQ

Are Server Actions replacing tRPC?

No. Server Actions replace a lot of the small API routes that used to exist only to handle form submissions. tRPC still does something Server Actions do not: give you a typed, cacheable, batchable API surface that more than one client can call.

Can you use Server Actions with tRPC?

Yes, and it works well together. Call your tRPC router directly from inside a Server Action using a server side caller, as shown above. You keep one source of truth for your mutation logic and still get progressive enhancement on the form itself.

What is the difference between tRPC and Server Actions?

Server Actions are a React and Next.js primitive, a function that runs on the server and can be called from a form or event handler. tRPC is a full API layer with routing, input validation, batching, and caching, callable from anywhere, not only from React components.

If you want a deeper look at wiring Server Actions into real production forms, with validation and redirects, I cover it in more detail in Server Actions in Next.js.

If you want this wired up on your own AI product end to end, that is exactly the kind of work I take on.

Drop a comment if your team landed somewhere different. Curious what tipped the decision for you.

Vercel AI SDK vs calling model APIs directly: what you actually gain

Mudassir Khan — Fri, 17 Jul 2026 19:18:19 +0000

You can ship a working LLM feature with nothing but fetch and a model's REST endpoint. People do it every day, and for a one shot completion it is honestly fine. So the real question is not "can I call the API directly" but "what am I giving up by not adding the Vercel AI SDK, and is that tradeoff worth a dependency?"

I have wired both approaches into production Next.js apps, and the answer is not the blanket "always use the SDK" you see on Twitter. Here is what the abstraction actually buys you, where it earns its footprint, and when reaching for raw fetch is the smarter call. If you are already deep into building AI products with Next.js, some of this will be familiar, but the middleware section is where most people leave value on the table.

What raw API calls look like (and why they're fine for simple cases)

Here is a bare completion against a chat endpoint. No SDK, no wrapper, just the request.

async function ask(prompt: string) {
  const res = await fetch("https://api.openai.com/v1/chat/completions", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
    },
    body: JSON.stringify({
      model: "gpt-4o",
      messages: [{ role: "user", content: prompt }],
    }),
  });

  const data = await res.json();
  return data.choices[0].message.content;
}

That's the whole thing. If your feature is a single server side completion (a summarizer, a classifier, a one shot rewrite) this is clean and you should not feel bad about it. No bundle cost, no new API to learn, and you can read every byte on the wire.

The pain starts the moment you want any of these: streaming tokens to the browser, swapping to a different provider, handling tool calls, or attaching caching and logging without touching your feature code. Each of those is solvable by hand. The question is whether you want to solve them by hand every time.

What Vercel AI SDK adds: streaming, provider swap, tool call handling

The same completion, streamed, through the SDK:

import { streamText } from "ai";
import { openai } from "@ai-sdk/openai";

export async function POST(req: Request) {
  const { prompt } = await req.json();

  const result = streamText({
    model: openai("gpt-4o"),
    prompt,
  });

  return result.toDataStreamResponse();
}

Three things happened here that are annoying to do with raw fetch.

First, streaming. streamText gives you a properly framed data stream that the client hook consumes with no manual parsing of server sent event chunks. If you have ever hand parsed data: lines out of a readable stream, you know why this matters.

Second, provider agnosticism. Swapping OpenAI for Anthropic is a single line:

import { anthropic } from "@ai-sdk/anthropic";
// model: openai("gpt-4o")  ->
model: anthropic("claude-sonnet-4"),

The unified API spans OpenAI, Anthropic, Google, Mistral, and dozens of other providers, so the rest of your code does not change. With raw calls, each provider has its own request shape, its own streaming format, and its own tool call schema, and you end up writing an adapter layer yourself. That adapter layer is basically the SDK, except untested and maintained by you.

Third, tool calls. Streaming partial tool calls as part of the data stream is now stable, which means the model can call your functions and you can render the arguments as they arrive instead of waiting for the full response. Doing this by hand against multiple providers is where the footgun count really climbs.

Here is the honest comparison:

Concern	Raw fetch	Vercel AI SDK
One shot completion	Trivial	Trivial (slight overhead)
Token streaming to browser	Manual SSE parsing	Built in
Switch provider	Rewrite request + parsing	One line
Tool calls across providers	Per provider schema	Unified
Caching / logging / guardrails	Wrap every call site	Middleware, one place
Bundle footprint	Zero	Real, but modest

Language model middleware: the real unlock

This is the part people underrate. Middleware lets you intercept and modify model calls without changing a single line of application code. Guardrails, caching, and logging all live in one wrapper instead of being smeared across every call site.

import { wrapLanguageModel } from "ai";
import { openai } from "@ai-sdk/openai";

const cachingMiddleware = {
  wrapGenerate: async ({ doGenerate, params }) => {
    const key = hash(params);
    const hit = await cache.get(key);
    if (hit) return hit;

    const result = await doGenerate();
    await cache.set(key, result);
    return result;
  },
};

const model = wrapLanguageModel({
  model: openai("gpt-4o"),
  middleware: cachingMiddleware,
});

Now every call through model is cached, and your route handlers do not know or care. Swap the middleware for a logging wrapper and you get full request tracing with zero call site edits. Chain several and you have guardrails plus caching plus logging composed cleanly.

The caching case connects directly to your bill. Repeated identical prompts are pure waste, and a cache in front of the model is one of the highest leverage moves for LLM inference cost. With raw fetch you would bolt this onto each call site and hope nobody adds a new one that forgets the cache. Middleware makes the wrapper the only path in.

useChat and client side state management

On the frontend, useChat handles the loop you would otherwise build by hand: appending messages, streaming the assistant reply token by token, tracking loading and error state.

"use client";
import { useChat } from "ai/react";

export function Chat() {
  const { messages, input, handleInputChange, handleSubmit } = useChat();

  return (
    <form onSubmit={handleSubmit}>
      {messages.map((m) => (
        <div key={m.id}>{m.role}: {m.content}</div>
      ))}
      <input value={input} onChange={handleInputChange} />
    </form>
  );
}

That is a streaming chat UI with optimistic input and error handling in about ten lines. Building the same thing on raw fetch means owning the readable stream reader, the incremental state updates, and the reconnection edge cases. It is doable. It is also exactly the kind of undifferentiated plumbing that eats a sprint.

When NOT to add the SDK

The abstraction is not free, so here are the cases where I reach for raw fetch instead.

You are doing a single server side completion with no streaming and no provider switching. A classifier, a batch summarizer, a cron job that rewrites text. The SDK adds a dependency for capabilities you will never call.

You are not in a JavaScript or TypeScript frontend context at all. If your LLM calls live in a Python service or an edge function in another runtime, the SDK's frontend ergonomics (useChat, the data stream protocol) are irrelevant, and you are paying for surface area you cannot use.

You need byte level control over the raw request for a provider feature the SDK has not surfaced yet. Abstractions lag the underlying APIs by design. When you hit that edge, drop down to fetch for that one call and keep the SDK for the rest. A mixed codebase is completely valid.

Bundle footprint matters for you and the feature is trivial. On a marketing site with one tiny AI widget, shipping the SDK to every visitor for a single completion is a poor trade.

Decision tree: should you use it?

Walk it top to bottom and stop at the first yes.

Do you need token streaming to the browser?      -> yes: use the SDK
Will you switch providers or A/B two models?     -> yes: use the SDK
Do you need tool calls, especially streamed?     -> yes: use the SDK
Do you want caching/logging/guardrails in one    -> yes: use the SDK (middleware)
  place instead of per call site?
Is it a single server side completion, one       -> yes: raw fetch is fine
  provider, no streaming?
Not a JS/TS frontend runtime at all?             -> raw fetch (or that runtime's client)

Most real product features hit one of the first four within a month. That is why the SDK is the default recommendation. It is not that raw calls are wrong. It is that the features you will inevitably want are the ones the SDK already handles, tested and maintained, so you are not the one paying to build and debug them.

FAQ

What does the Vercel AI SDK do?
It gives you a unified API for calling LLMs across many providers, with built in token streaming, tool call handling, language model middleware for caching and logging and guardrails, and React hooks like useChat for the frontend loop.

Is the Vercel AI SDK free?
Yes, the SDK itself is open source and free. You still pay each model provider for the tokens you use, and if you deploy on Vercel you pay for that hosting, but the library carries no license fee.

Does the Vercel AI SDK work with Claude and Gemini?
Yes. It supports Anthropic (Claude), Google (Gemini), OpenAI, Mistral, and dozens of other providers through the same interface. Switching between them is a single line change in most code.

If you want a deeper look at cutting model spend, I cover it in more detail on my site.

If you want this wired up on your own site end to end, that is exactly the kind of work I take on.

Drop a comment if your setup looks different — curious what variations people are running in production.

Stop Using LangChain for Simple LLM Tasks: The Raw Anthropic SDK Is Cheaper and Clearer

Mudassir Khan — Sun, 12 Jul 2026 19:11:59 +0000

If you reach for LangChain every time you wire up a new LLM call, you are not alone. Most teams do it. The problem is that for simple tasks, that reach costs you in tokens, in debugging time, and in a monthly bill from LangSmith that you forgot you enabled.

The Default Reach for Abstraction

LangChain became the default choice because it made the first demo fast. You imported a model wrapper, chained a couple of prompts, and had something running in 20 minutes. That early speed creates a gravitational pull. When the next feature comes up, you reach for the same toolkit.

But demos and production are different environments. In production, you need to know exactly what is going into every API call. You need to trace a failure back to the token that caused it. You need to reason about cost at the individual request level.

LangChain's abstraction layers make all of that harder. The framework handles prompt templating, message formatting, retry logic, and model selection (sometimes with opinions you did not ask for). That is genuinely useful when your pipeline has six steps, conditional routing, and tool calls that need orchestration. It is a footgun when your use case is a single turn call to summarize a support ticket.

Same Task, Two Ways

Here is the same task implemented both ways. The task: take a support ticket string and return a one sentence summary.

// LangChain approach — @langchain/anthropic + @langchain/core
import { ChatAnthropic } from "@langchain/anthropic";
import { HumanMessage } from "@langchain/core/messages";

const model = new ChatAnthropic({
  model: "claude-3-5-sonnet-20241022",
  apiKey: process.env.ANTHROPIC_API_KEY,
});

const response = await model.invoke([
  new HumanMessage("Summarize this support ticket in one sentence: " + ticket),
]);

// response is an AIMessage — content is a string here, but
// the shape changes across invoke / stream / generate / batch
console.log(response.content);

// Raw Anthropic SDK — @anthropic-ai/sdk
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

const response = await client.messages.create({
  model: "claude-3-5-sonnet-20241022",
  max_tokens: 256,
  messages: [
    {
      role: "user",
      content: "Summarize this support ticket in one sentence: " + ticket,
    },
  ],
});

// response.content is a typed ContentBlock array — no shape surprises
console.log(response.content[0].text);

The raw SDK version tells you everything: the model name, max_tokens (LangChain has a default you have to look up), and the exact message structure being sent. When something blows up, you know exactly where to look.

The LangChain version hides the request behind invoke, and the response shape silently changes depending on which invocation method you used. That is the kind of thing that bites you at 2am when one code path calls stream instead of invoke and suddenly response.content is a generator.

The Cost Reality

For LLM cost optimization, the raw SDK gives you direct control over prompt caching breakpoints. Anthropic offers a 90% discount on cached tokens vs OpenAI's 50%. When you set cache control points explicitly in the raw SDK, you capture that discount precisely. LangChain's message wrappers do not expose this at the same granularity, so you are often leaving money on the table.

The more concrete cost story is LangSmith. LangSmith adds $39 per month for 10,000 traces, with $0.50 per 1,000 after that. For a pipeline running 50,000 calls a month, that is an extra $59 on top of your LLM API bill.

// Rough monthly cost model for a simple summarization pipeline
// Scenario: 10,000 calls/month, ~2,000 tokens each

const llmCostPerMonth = 10_000 * 0.000003 * 2_000; // roughly $60/month in LLM costs

// Raw SDK only — pay the API cost
const costWithoutLangSmith = llmCostPerMonth; // $60

// With LangSmith at 10K traces/month (exactly at the base cap)
const langsmithBase = 39; // $39/month for 10K traces
const costWithLangSmith = llmCostPerMonth + langsmithBase; // $99

// At 50K calls/month, overage kicks in
const langsmithAt50K = 39 + 40 * 0.5; // $59 extra = $119 total non-LLM tooling cost

Teams that have moved to the native SDK report roughly a 40% reduction in API expenses. Another case study found LangChain incurred 2.7x higher costs compared to a manual implementation for a simple RAG system. That delta is not from the LLM provider. It is from auxiliary tooling and inefficient prompt routing that disappears when you own the code path directly.

When LangChain IS the Right Call

This is not a LangChain hit piece. The framework earns its weight in specific scenarios.

If you are building a multistep agent that calls tools, decides which tool based on the last output, maintains memory across a conversation, and routes to different subchains based on intent, the raw SDK will make you hand roll all of that. For anything with tool use at that depth, check the function calling guide and then honestly evaluate whether LangChain's abstractions pay off for your pipeline shape.

Similarly, if your team is switching between OpenAI, Anthropic, and Gemini and you want a single interface, LangChain's model agnostic layer is worth it. The Vercel AI SDK is also worth a look if you are in a Next.js context and want native streaming and RSC integration at a lighter weight than LangChain.

The rough rule: fewer than three steps and a single model? Reach for the raw SDK. Conditional routing, tool use, or multiprovider switching? Evaluate the abstraction cost honestly before committing.

Making the Call

The teams I have seen migrate away from LangChain consistently name the same two friction points: debugging and production stability. When a call fails inside a LangChain chain, the stack trace gives you LangChain internals rather than the raw request that was sent. You end up adding log statements at multiple layers just to figure out what actually went to the model.

With the raw SDK, there is nothing between you and the API. The request body is a plain object you can log, diff, and replay. The response is typed and predictable. When costs spike, you can trace it to a specific call site without parsing through abstraction layers.

Start with the raw SDK for anything simple. You will understand your token usage better, debug faster, and avoid the LangSmith subscription until you genuinely need the observability it provides. Reach for an abstraction layer when your pipeline complexity demands it, not as the default starting point.

If you want a deeper look at LLM cost optimization, I cover it in more detail on my site.

If you want this wired up on your own site end to end, that is exactly the kind of work I take on.

Drop a comment if you have made the same switch. Curious what the migration looked like for your team.

I Replaced My Entire Backend With MCP Tools — Here Is What Broke (And How to Fix It)

Mudassir Khan — Thu, 09 Jul 2026 19:12:58 +0000

I Replaced My Entire Backend With MCP Tools — Here Is What Broke (And How to Fix It)

MCP tools are genuinely useful. I shipped half my backend through them in about two weeks. Then they started failing silently in production and I had no idea why.

This is the debugging playbook I built from that experience.

The failures nobody warns you about

When an MCP tool fails, it does not always throw an error you can catch. Before v1.27.1, transport errors were often swallowed completely. The onerror callback simply did not fire. Your agent would hang, retry, or silently skip the tool call. No exception, no log, nothing in your monitoring.

That behavior got fixed in v1.27.1 so errors now surface properly. But schema drift is still entirely on you.

A 2026 taxonomy paper on MCP fault classes puts API usage issues at number one: incorrect request construction, missing parameters, and schema contract violations. In practice this means your tool's input schema gets bumped in a server update, your agent does not pull the new schema, and calls start failing in ways that look like flaky network behavior.

You spend an hour chasing the wrong thing.

Stdout: the footgun hiding in plain sight

If you run a local MCP server over stdio transport, here is one of the easiest mistakes to make: logging debug output to stdout.

Do not do this.

The stdio transport uses stdout to carry the protocol messages. Anything you write there corrupts the stream. Your server breaks silently and the error surfaces three steps removed from the actual cause.

Use stderr for all logging in stdio servers:

// Wrong — corrupts stdio transport
console.log("tool called:", toolName);

// Right — stderr is safe and does not touch the protocol stream
console.error("[mcp-server] tool called:", toolName);
process.stderr.write(
  JSON.stringify({ tool: toolName, ts: Date.now() }) + "\n"
);

This one rule would have saved me about six hours across two separate incidents.

Add a correlation ID before anything else

The single most valuable change you can make to an MCP setup is adding a correlation ID to every tool invocation. Without it you cannot trace a failed call back to the agent decision that triggered it, especially once you have multiple tools calling each other in a chain.

Here is the pattern I settled on:

import { randomUUID } from "crypto";

async function callTool(
  client: MCPClient,
  toolName: string,
  params: Record<string, unknown>
) {
  const correlationId = randomUUID();
  const startMs = Date.now();

  process.stderr.write(
    JSON.stringify({
      event: "tool_call_start",
      correlationId,
      tool: toolName,
      params,
    }) + "\n"
  );

  try {
    const result = await client.callTool({ name: toolName, arguments: params });
    process.stderr.write(
      JSON.stringify({
        event: "tool_call_end",
        correlationId,
        tool: toolName,
        durationMs: Date.now() - startMs,
        success: true,
      }) + "\n"
    );
    return result;
  } catch (err) {
    process.stderr.write(
      JSON.stringify({
        event: "tool_call_error",
        correlationId,
        tool: toolName,
        durationMs: Date.now() - startMs,
        error: err instanceof Error ? err.message : String(err),
      }) + "\n"
    );
    throw err;
  }
}

Every log line gets the same correlationId. When something breaks you can grep for the ID and reconstruct the full call sequence in about thirty seconds.

Schema drift: log the hash before it bites you

The other silent killer is schema drift between server versions. Your agent negotiates the tool list once at connection time. If the server updates its schema after connection, your agent keeps using the old one.

Log the server SHA and the tool schema hash at bind time:

import { createHash } from "crypto";

async function bindServer(client: MCPClient, serverLabel: string) {
  const tools = await client.listTools();
  const schemaHash = createHash("sha256")
    .update(JSON.stringify(tools))
    .digest("hex")
    .slice(0, 8);

  process.stderr.write(
    JSON.stringify({
      event: "server_bound",
      server: serverLabel,
      toolCount: tools.length,
      schemaHash,
    }) + "\n"
  );

  return tools;
}

Now when a deployment goes out and calls start behaving differently, you can diff the schemaHash between the old and new logs. It takes five minutes instead of an hour.

Use the MCP Inspector before you ship

The official debugging tool is the MCP Inspector. It connects to stdio and Streamable HTTP servers directly, letting you invoke tools interactively without an agent in the loop.

Run it against your server before you plug it into any agent. You will catch schema issues, transport problems, and unexpected tool behaviors before they show up in production at midnight.

The observability needs go deeper once you have multiple servers in a fan out topology. The correlation ID approach above is the foundation; from there you wire the logs into whatever APM you already use. I wrote about the broader agentic AI production architecture patterns separately, and there is also a dedicated piece on LLM observability in production if you want the full monitoring picture.

Three things to check right now

Search every stdio server for console.log calls. Replace them with process.stderr.write or a logger bound to stderr.
Wrap every client.callTool() call with a correlationId. If you are not logging the ID today you are flying blind.
Run the MCP Inspector against each server before the next deployment. Invoke every tool once manually and look at the raw responses.

If you want a deeper look at production observability for AI systems, I cover it in more detail on my blog.

If you want this wired up on your own stack end to end, that is exactly the kind of work I take on.

Drop a comment if your MCP setup looks different — curious what debugging approaches people are running in production.

Next.js App Router Caching: What Actually Gets Cached (and What Doesn't)

Mudassir Khan — Sun, 05 Jul 2026 19:30:35 +0000

You deployed a Next.js App Router app, called revalidatePath() after a mutation, and the page still shows stale data. Or you disabled caching on a fetch call and the page is somehow still cached. Or you migrated from Next.js 14 to 15 and everything that was working broke silently.

All of these happen because the App Router has four separate caching layers, and they each have their own invalidation story. Developers hit these bugs constantly because the mental model of "there is one cache" is wrong.

Here's what actually exists.

The four caching layers in Next.js App Router

Next.js App Router does not have a single cache. It has four distinct layers, each running in a different context with different lifetimes and different invalidation APIs. (Next.js caching docs)

Layer	Where it runs	Lifetime	Invalidated by
Request Memoization	Server, per request	Request lifetime	Automatic
Data Cache	Server, persistent	Until revalidated	`revalidateTag`, `revalidatePath`
Full Route Cache	Server, persistent	Until revalidated	`revalidatePath`, redeploy
Router Cache	Browser, per session	Short TTL or navigation	`router.refresh()`, full reload

You need to know all four. Knowing only one is what causes the "I called revalidatePath and nothing happened" bugs.

Request Memoization vs Data Cache: why they're different

These two are the most commonly confused.

Request Memoization is automatic. If you call the same fetch() URL twice during a single server render, Next.js deduplicates it. The second call returns the already fetched result without hitting the network. It's scoped to a single request and disappears when the render is done. You cannot configure it, disable it, or persist it. It just works.

// Both of these calls in the same render only hit the network once
const user = await fetch('/api/user/123')
const userAgain = await fetch('/api/user/123') // deduped automatically

Data Cache is different. It persists across requests. When you do a fetch with caching:

const data = await fetch('https://api.example.com/posts', {
  next: { revalidate: 60 }
})

That cached result survives across multiple users and multiple requests until the 60-second TTL expires or you explicitly call revalidateTag(). This is the cache you need to think about for interal data.

The key difference: Request Memoization handles the "same fetch called twice in one render" problem. Data Cache handles the "this data should not hit the database on every request" problem. They are solving different things.

Full Route Cache: when your page stops being static

The Full Route Cache stores the rendered HTML and React Server Component payload for a page. Next.js generates this at build time for static routes and serves it to every visitor without rerunning the server component.

Your page is static (cached in Full Route Cache) when:

It has no dynamic functions (cookies(), headers(), searchParams)
All its fetch calls use cache: 'force-cache' or a revalidate value

Your page opts out of Full Route Cache when:

You use cookies() or headers() inside a server component
You set export const dynamic = 'force-dynamic'
A fetch call inside it uses cache: 'no-store'

One dynamic fetch in a layout component can silently make the entire route dynamic. This is a footgun. I've seen production apps serving a fully static marketing page do a database round trip on every request because a single deeply nested component called cookies() for analytics.

revalidatePath() and redeployment both invalidate the Full Route Cache. A revalidateTag() call only invalidates it if the route's fetch calls are tagged with the same tag.

Router Cache: the client side cache that surprises everyone

The Router Cache is the one that trips up nearly everyone who calls revalidatePath() and wonders why the page hasn't updated.

It lives in the browser, not the server. Next.js caches prefetched and visited route segments in memory for the duration of the user's session. This means even after you invalidate the Data Cache and Full Route Cache on the server, the user may still see stale content because their browser is serving the cached client side payload.

Two things clear it:

router.refresh() called in a client component
A full browser page reload

'use client'

import { useRouter } from 'next/navigation'

function SubmitButton() {
  const router = useRouter()

  async function handleSubmit() {
    await updateData()
    router.refresh() // clears Router Cache for current route, triggers revalidation
  }

  return <button onClick={handleSubmit}>Submit</button>
}

Important: revalidatePath() called in a server action does not clear the Router Cache in the same request. (Next.js revalidatePath docs) You still need router.refresh() on the client if you want an immediate update.

On demand revalidation with revalidatePath and revalidateTag

Time based revalidation (the revalidate: 60 pattern) is fine for data that can be a minute stale. For data that must update immediately after a mutation, you want on demand revalidation.

revalidatePath(path) invalidates the Full Route Cache and Data Cache for all fetch calls associated with that path. Use it after mutations that affect a whole page.

revalidateTag(tag) is more surgical. Tag your fetch calls, then invalidate only what changed.

// Tag the fetch at data-fetch time
const posts = await fetch('https://api.example.com/posts', {
  next: { tags: ['posts'] }
})

// Invalidate only posts-tagged data after a mutation
import { revalidateTag } from 'next/cache'

async function createPost(data: FormData) {
  'use server'
  await savePost(data)
  revalidateTag('posts') // only posts-tagged fetches are invalidated
}

For unstable_cache (the server side equivalent of SWR, useful when you're caching database queries or other data that does not go through fetch), invalidation works the same way with revalidateTag. (Next.js unstable_cache docs)

import { unstable_cache } from 'next/cache'

const getProducts = unstable_cache(
  async () => db.query.products.findMany(),
  ['products'],
  { tags: ['products'] }
)

Next.js 15 caching changes: the breaking change you need to know

If you migrated from Next.js 14 to 15 and your data stopped being cached, this is why.

In Next.js 14, fetch() requests were cached by default. The default was cache: 'force-cache'. You had to explicitly opt out with cache: 'no-store'.

In Next.js 15, fetch requests are not cached by default. (Next.js 15 release blog) The default is now cache: 'no-store'. You have to explicitly opt in with cache: 'force-cache' or a revalidate value.

This is a breaking change with no deprecation warning during migration.

// Next.js 14 default — cached
const data = await fetch('https://api.example.com/data')

// Next.js 15 default — NOT cached (same code, different behavior)
const data = await fetch('https://api.example.com/data')

// To opt into caching in Next.js 15
const data = await fetch('https://api.example.com/data', {
  next: { revalidate: 60 }
})
// or
const data = await fetch('https://api.example.com/data', {
  cache: 'force-cache'
})

The migration path: audit every fetch call in your app and add explicit caching configuration where you actually want it. Rely on the default only for data you intentionally want fresh on every request.

FAQ

How does caching work in Next.js App Router?

Next.js App Router uses four caching layers: Request Memoization (deduplicates fetch calls within a single server render), Data Cache (persists fetch results across requests until revalidated), Full Route Cache (stores rendered HTML for static routes), and Router Cache (browser memory cache for prefetched route segments). Each layer has its own invalidation mechanism.

How do I disable caching in Next.js 14?

To opt a specific fetch call out of caching, add { cache: 'no-store' } as the second argument: fetch(url, { cache: 'no-store' }). To make an entire route always dynamic, add export const dynamic = 'force-dynamic' at the top of the route file. Note that in Next.js 15, fetches are not cached by default so you may not need to do anything if you have already upgraded.

What is the difference between fetch cache and route segment config in Next.js?

The fetch cache (cache option on fetch() or next.revalidate) controls the Data Cache for that specific data fetch. Route segment config (export const dynamic, export const revalidate) sets the default behavior for the entire route. Route segment config is a blunt instrument. Individual fetch options are more precise and composable.

What does router.refresh() do?

It triggers a soft navigation to the current route, clearing the Router Cache for the current route and refetching server component data. It does not do a full browser reload. Use it in client components after server mutations when you need the UI to reflect the updated data immediately.

Why is unstable_cache still prefixed with unstable_?

The API was marked unstable while the team iterated on its interface. The behavior is production ready and widely used. The prefix signals that the API shape may still change in a future major version. Use it, but be prepared for a migration if the API changes.

I've wired up Next.js caching for AI products with complex data fetching requirements — the four layer model is especially important when mixing realtime data with statically cached content.

If you want to see a practical example of caching in a Next.js project, I walk through building a calculator in Next.js where fetch caching decisions matter.

Drop a comment if you've hit a caching bug that isn't covered here — curious what edge cases people are running into in production.

LLM Cost Optimization: Cutting Inference Bills Without Killing Quality

Mudassir Khan — Wed, 01 Jul 2026 19:54:00 +0000

You can cut your LLM API spend by 50 to 90% without switching models or degrading output quality. The techniques exist, the docs are public, and most teams are not using them. Here is what actually moves the needle.

Where your LLM bill actually comes from

Every API call charges you for input tokens plus output tokens. Simple math, but "input tokens" is a bigger footgun than it looks.

Most production workloads send the same system prompt, instructions, or retrieval context on every single request. If your system prompt is 4,000 tokens and you fire 100 requests per minute, that's 400,000 input tokens per minute burning at full price. Before you optimize anything else, map where your token spend is actually going.

Three buckets drive most bills:

Repeated context: system prompts, tool definitions, retrieved documents sent on every call
Frontier model overuse: sending simple classification tasks to GPT-4o or Claude Sonnet when a cheaper model handles them just fine
Bloated outputs: getting paragraph prose back when you need a 10-token structured field

Once you know which bucket hurts most, the fix is obvious. The rest of this article covers each one with concrete numbers.

Prompt caching: the highest ROI optimization nobody uses

If your app sends the same prefix on every call (system prompt, tool definitions, few shot examples), prompt caching is the single highest impact change you can make.

Anthropic's caching API lets you mark a portion of your input as cacheable with a cache_control block. The first call writes the cache at 25% of the normal input token price. Every subsequent cache hit costs 10% of normal. That is a 90% cost reduction on the cached portion.

import anthropic

client = anthropic.Anthropic()

SYSTEM_PROMPT = """
You are a code review assistant. Your job is to identify bugs,
suggest improvements, and flag security issues in submitted code.
... (your 2000-token system prompt here) ...
"""

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": SYSTEM_PROMPT,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {"role": "user", "content": user_code}
    ]
)

On the first call you pay the cache write cost (25% of normal input price). From the second call onward, every hit costs 10% of what you would normally pay. For an app firing 1,000 calls per hour against a 3,000-token system prompt, that is 2.7 million tokens per hour dropping from full price to 10% of full price.

OpenAI has equivalent behavior through automatic prompt caching. The implementation differs slightly but the economics are similar.

Prompt caching helps most when: your system prompt is over 1,000 tokens, you are running a RAG pipeline that injects the same documents repeatedly, or you have an agent framework with large tool definitions reused across turns.

Model routing: stop sending everything to the frontier model

Not every task needs GPT-4o or Claude Sonnet. Intent classification, slot extraction, summarization, and simple Q&A are well within what GPT-4o-mini or Claude Haiku can handle. The cost difference is roughly 10 to 20 times per token.

Routing 40% of traffic to a smaller model can halve your total API spend without touching a single user facing feature. The key is picking the right tasks.

import OpenAI from "openai";

const client = new OpenAI();

type TaskComplexity = "simple" | "complex";

function classifyTask(userMessage: string): TaskComplexity {
  const reasoningKeywords = [
    "explain why",
    "compare",
    "analyze",
    "critique",
    "design architecture"
  ];

  const needsReasoning = reasoningKeywords.some(kw =>
    userMessage.toLowerCase().includes(kw)
  );

  if (!needsReasoning && userMessage.length < 200) {
    return "simple";
  }
  return "complex";
}

async function routedCompletion(userMessage: string) {
  const complexity = classifyTask(userMessage);
  const model = complexity === "simple" ? "gpt-4o-mini" : "gpt-4o";

  const response = await client.chat.completions.create({
    model,
    messages: [{ role: "user", content: userMessage }],
  });

  return { response, model_used: model };
}

This is a naive router based on keywords and message length. Production versions use a small classifier (or another LLM) to score complexity before routing. Projects like LiteLLM and RouteLLM give you prebuilt routers if you don't want to build your own.

The failure mode to watch: tasks that look simple ("what does this error mean?") but need surrounding context to answer well. Build an escalation path so users can retry against the full model when the cheaper one falls short.

Batching and async inference for workloads that can wait

If you are running evals, processing documents, generating reports, or doing any bulk inference where a 24-hour window is acceptable, the OpenAI Batch API cuts costs by 50% versus synchronous pricing.

You submit a JSONL file. OpenAI processes it asynchronously. You pull results when they are ready. No streaming, no realtime SLAs.

import openai
import json
from pathlib import Path

client = openai.OpenAI()

requests = []
for doc_id, text in documents.items():
    requests.append({
        "custom_id": doc_id,
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-4o-mini",
            "messages": [
                {"role": "user", "content": f"Summarize in 3 sentences: {text}"}
            ],
            "max_tokens": 150
        }
    })

batch_path = Path("/tmp/batch_input.jsonl")
batch_path.write_text("\n".join(json.dumps(r) for r in requests))

with open(batch_path, "rb") as f:
    batch_file = client.files.create(file=f, purpose="batch")

batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h"
)

print(f"Batch submitted: {batch.id}")
# Poll client.batches.retrieve(batch.id) to check status

Stack this with model routing: batch jobs on the cheap model with the 50% discount applied on top. That is where the cost math gets seriously interesting.

Anthropic offers similar async batch processing. The API shape differs slightly but the pricing pattern is the same.

Output schema discipline to shrink response tokens

Freeform prose outputs are expensive. When your downstream code parses a JSON blob out of a markdown code block, you are paying for the explanation text, the code fences, and the commentary. None of that is useful to you.

Structured outputs fix this. Both OpenAI and Anthropic support schema constrained generation that returns only the fields you defined, in valid JSON, every time.

from anthropic import Anthropic
import json

client = Anthropic()

response = client.messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=256,
    tools=[{
        "name": "extract_sentiment",
        "description": "Extract sentiment analysis from text",
        "input_schema": {
            "type": "object",
            "properties": {
                "sentiment": {
                    "type": "string",
                    "enum": ["positive", "negative", "neutral"]
                },
                "confidence": {"type": "number"},
                "topics": {
                    "type": "array",
                    "items": {"type": "string"},
                    "maxItems": 3
                }
            },
            "required": ["sentiment", "confidence", "topics"]
        }
    }],
    tool_choice={"type": "tool", "name": "extract_sentiment"},
    messages=[{"role": "user", "content": user_text}]
)

result = json.loads(response.content[0].input)

The schema constraint tells the model to stop once the required fields are populated. No hedging, no markdown formatting, no explanatory prose it doesn't need to write. Output token counts drop noticeably on structured tasks compared to the freeform equivalent.

This one bit me in prod before I switched: an eval pipeline returning full explanation paragraphs for every item when all I needed was a score and a label. Two fields. The verbosity was pure waste.

FAQ

How do you reduce LLM API costs?

The highest ROI levers in order: (1) prompt caching for repeated context, (2) model routing for tasks that don't need frontier models, (3) batching for async workloads, (4) schema constrained outputs to shrink response size. Each one targets a different cost bucket and they stack.

Is prompt caching worth it for LLM cost savings?

Yes, if your system prompt or retrieval context exceeds 1,000 tokens and you send it repeatedly. Anthropic's cache read price is 10% of normal input pricing. On a 4,000-token system prompt fired 500 times per day, you pay full price once and 10% for the other 499. The math is not subtle.

What is the cheapest way to run an LLM in production?

Self hosted open weight models (Llama, Mistral, Qwen) undercut API pricing at scale but require infrastructure you have to operate. For workloads running against vendor APIs: combine routing (cheap model for simple tasks) plus batching (50% off async) plus caching (90% off repeated context). Most teams are not doing all three, which means most teams are overpaying.

I go deeper on production LLM inference optimization on my blog, including a breakdown of what each technique actually saves at different call volumes.

If you want this wired up on your own stack end to end, that is exactly the kind of work I take on.

What routing or caching setup is your team running in production? Drop a comment. Curious what numbers people are actually seeing.

Why LLM Agents Fail Silently and How to Debug Them

Mudassir Khan — Sat, 27 Jun 2026 20:52:01 +0000

Why LLM Agents Fail Silently and How to Debug Them

Your agent returned an empty result. No exception. No error log. No status code that points anywhere useful. Just nothing.

You dig through logs. The LLM call went through. The tool was invoked. The response came back. Everything looks fine and yet the task is incomplete, wrong, or missing entirely.

That's a silent failure. And it's one of the nastiest bugs in AI engineering.

What is a silent failure in an LLM agent?

A silent failure is when your agent completes without raising an exception but produces a wrong or incomplete result. The difference between a noisy failure (a Python traceback, a 5xx from the API) and a silent one is that noisy failures are debuggable. Silent ones require you to instrument the entire agent loop just to notice something went wrong.

They're common because LLMs are designed to always return something. The model won't throw a ValueError when it runs out of context or when your tool schema changes out from under it. It'll return an empty array, a truncated JSON blob, or a confident "I've completed the task" with nothing to show for it.

The result is an agent that appears to work until you look closely at the outputs.

The three root causes: token budget, tool schema drift, and unhandled exceptions

Most silent failures trace back to one of three places.

Token budget exhaustion. OpenAI's function calling API returns an empty choices array when max_tokens is hit in the middle of a tool call. No exception is raised. The call returns 200. Your code checks response.choices[0] and explodes with an IndexError, or worse, your code handles the empty array gracefully and just moves on. The agent continues as if the tool ran, with no output to show for it.

response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    tools=tools,
    max_tokens=512  # too small for a complex tool call
)

# This blows up at runtime — or silently skips if you're defensive
if response.choices:
    tool_call = response.choices[0].message.tool_calls[0]

Fix: always log response.choices, finish_reason, and usage.completion_tokens. If finish_reason == "length", treat it as a hard failure, not a graceful noop.

Tool schema drift. Your tool schema changes. A field gets renamed, a required parameter gets removed, a new enum value gets added. The LLM was tuned against the old schema. It now generates arguments that fail the validator, and your framework silently drops the tool output and continues. LangGraph's StateGraph does exactly this when a tool raises an unhandled exception inside an interrupt: the output gets dropped and the next node receives None.

# Tool raises, StateGraph swallows the exception
@tool
def fetch_user_data(user_id: str) -> dict:
    # KeyError here gets swallowed by the interrupt handler
    return db.fetch(user_id)["profile"]["details"]

Fix: always reraise from your tool handlers, or wrap them in an explicit try/except that returns a structured error payload instead of propagating None downstream.

Unhandled exceptions inside the agent loop. Most agent frameworks catch exceptions at the orchestrator level to keep the loop alive. That's good for reliability, but it means your per step errors get swallowed into a catchall handler that logs nothing useful and lets the next turn proceed. One bad tool call in a 10-step chain silently poisons every step that follows.

Adding per step tracing to catch failures early

The most reliable way to surface silent failures is distributed tracing. OpenTelemetry spans per agent step give you a queryable record of every tool call, its inputs, its outputs, and where it fell over.

from opentelemetry import trace

tracer = trace.get_tracer("agent.loop")

def run_agent_step(step_name: str, messages: list, tools: list):
    with tracer.start_as_current_span(step_name) as span:
        span.set_attribute("step.input_message_count", len(messages))
        span.set_attribute("step.tool_count", len(tools))

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
            tools=tools,
        )

        finish_reason = response.choices[0].finish_reason if response.choices else "empty"
        span.set_attribute("step.finish_reason", finish_reason)
        span.set_attribute("step.completion_tokens", response.usage.completion_tokens)

        if finish_reason == "length" or not response.choices:
            span.set_status(trace.StatusCode.ERROR, "token budget hit or empty response")
            raise RuntimeError(f"Step {step_name} hit token budget before completing")

        return response

Now when something goes wrong, your trace shows exactly which step failed and why. You're not reconstructing the failure from scattered log lines. You have a full span tree.

Plug this into any OpenTelemetry compatible backend (Honeycomb, Jaeger, the OTel Collector) and you get realtime visibility into your agent loop for free.

Structured output validation as a silent failure firewall

If tracing tells you when something went wrong, Pydantic tells you what the model produced that broke your assumption.

Put a Pydantic validation step after every tool call. The model's output schema gets validated before it touches anything downstream. If it fails, you catch a ValidationError with a clear message instead of a silent None that propagates through 5 more steps.

from pydantic import BaseModel, ValidationError

class UserProfile(BaseModel):
    user_id: str
    email: str
    role: str  # "admin" | "viewer" | "editor"

def validate_tool_output(raw: dict) -> UserProfile:
    try:
        return UserProfile(**raw)
    except ValidationError as e:
        # Loud failure here is intentional — better than a silent one later
        raise RuntimeError(f"Tool output failed schema validation: {e}") from e

This is especially powerful for tools that call external APIs. The external schema changes independently of your agent's expectations. Pydantic catches that mismatch at the boundary, before stale data flows into your LLM's next prompt and contaminates the rest of the run.

Building a dead man's switch into long running agent loops

Long running agents (the ones that run for minutes or hours across many tool calls) need a liveness check that fires if the loop goes quiet for too long. If the agent doesn't check in within N seconds, something assumed it was alive when it wasn't.

import threading
import time

class AgentWatchdog:
    def __init__(self, timeout_seconds: int = 60):
        self.timeout = timeout_seconds
        self.last_heartbeat = time.time()
        self._stop = threading.Event()

    def heartbeat(self):
        """Call this after every successful agent step."""
        self.last_heartbeat = time.time()

    def start(self):
        def _watch():
            while not self._stop.is_set():
                if time.time() - self.last_heartbeat > self.timeout:
                    raise RuntimeError("Agent watchdog timeout — loop went silent")
                time.sleep(5)
        threading.Thread(target=_watch, daemon=True).start()

    def stop(self):
        self._stop.set()

# Usage
watchdog = AgentWatchdog(timeout_seconds=90)
watchdog.start()

for step in agent_steps:
    result = run_agent_step(step)
    watchdog.heartbeat()  # prove we're alive after each step

watchdog.stop()

This doesn't replace tracing. It's a last resort: if your instrumentation missed the failure, the watchdog still catches a loop that went silent and gives you something you can alert on.

FAQ

Why do AI agents stop responding without an error?

Usually one of three things: the model hit its token budget in the middle of a tool call and returned an empty choices array, a tool raised an exception that the orchestrator swallowed, or the tool output failed schema validation and got silently dropped. Add finish_reason logging and per step OTel spans and you'll find it fast.

How do you debug an LLM agent that returns empty results?

Start with finish_reason. If it's "length", you hit the token budget. If it's "stop" but the output is empty, check your tool handler for swallowed exceptions. If the tool ran but the downstream state is still None, you have a schema validation gap. Pydantic after every tool call surfaces this immediately.

What causes silent failures in multistep AI agents?

The same bugs that cause noisy failures in any software, except the agent framework is often designed to keep running even when a step fails. That design choice trades reliability for debuggability. You get it back by adding tracing at the framework layer so failures are recorded even when the loop doesn't crash.

If you want a deeper look at agent observability in production, I cover it in more detail on my site.

For the full taxonomy of production failure modes before you build your evaluation harness, this is where I'd start.

If you want this wired up on your own system end to end, that is exactly the kind of work I take on.

Drop a comment if you've hit a different class of silent failure. Curious what variations people are running into in production.

I Measured MCP vs Direct API Calls: The Token Math No One Tells You

Mudassir Khan — Tue, 23 Jun 2026 19:37:17 +0000

MCP is having its moment. Every agent framework is wiring tools through it, and for good reason: standardized tool registration, composable servers, a protocol that doesn't force you to reinvent the plugin system every project. I get the appeal.

What I didn't see coming was the token bill.

Here's the number that stopped me cold: for a simple SerpApi search, an MCP agent used 6,047 tokens per call. A CLI script doing the exact same job used 351 tokens. That's 17x overhead, for the same search result.

I went looking for other data points after that. The range is wider than I thought.

The number that started this: 17x tokens per call

The benchmark comes from a post measuring SerpApi MCP vs a CLI agent with field projection. Both do the same job: search the web, return results. The difference:

Approach	Tokens per call
MCP agent	6,047
CLI script (with field projection)	351
Ratio	~17x

At 10 searches a day in a conversational interface, 6,047 tokens per call is fine. At 1,000 searches a day in an automated pipeline, you're burning 6 million tokens where 351,000 would do.

That's not a rounding error. That's a billing line item.

Why it happens: schema injection on every message

The overhead isn't in the API call itself. It's in what your AI host has to inject into every single message before the call even happens.

When you register an MCP server, every tool definition (name, description, input schema, parameter types) gets serialized and injected into every system prompt or assistant message in your conversation. The LLM needs to "see" all available tools on every turn so it knows what it can call.

A concrete example: a simple repo language check. The task is trivial. The token count is not:

Approach	Tokens
CLI agent (direct API)	1,365
MCP agent (same task)	44,026

The extra 42,661 tokens are 43 tool definitions injected into every message. Tools that had nothing to do with the repo language check. The agent happened to have 43 tools registered on that MCP server, and every one of them rode along for the trip.

The full benchmark breakdown (4x to 32x range)

"17x" is the headline, but the actual range is 4x to 32x depending on:

How many tools are registered on the server
How complex those tool schemas are
How many turns the conversation takes
Whether you're running batch jobs or interactive queries

The 4x end is a small server with simple schemas and a short session. The 32x end is a large server with dozens of complex tools running a multistep pipeline where schema injection compounds across every message.

What matters is that the floor is 4x. Even a minimal MCP setup carries overhead that a direct API call doesn't.

The $270/day surprise: unused tool definitions at scale

Here's where it gets genuinely alarming for production pipelines.

At Claude Sonnet pricing, 90,000 tokens of unused schema overhead costs roughly $0.27 per request. If your pipeline runs 1,000 times a day, that's $270/day on tool definitions your agent never touches.

Not on useful computation. On schemas riding along in the context window because you registered everything upfront.

Most MCP servers are built for flexibility. You register everything because you don't know what the agent will need. That makes sense in a conversational interface where the user might ask anything. In a batch pipeline where the task is deterministic, you're paying the full flexibility tax even when you only need two of the forty registered tools.

The math compounds fast. And it's invisible in your usage dashboard: it just shows up as "input tokens."

When the overhead is absolutely worth it

MCP has real strengths. The overhead is the cost of flexibility, and in these cases it's worth paying.

Conversational agents. The user is asking unpredictable questions. You need all tools available on every turn because you genuinely don't know what comes next. Schema injection is the price of that flexibility.

Governance and auth at the server level. MCP servers can enforce access control, rate limiting, and audit logging in one place for every client that talks to them. Doing that in bespoke scripts for each tool is a footgun waiting to fire. The overhead buys you correctness.

Composable tooling across teams. If five teams are building agents and they all need the same data sources, one MCP server beats five separate integrations. The overhead is shared; the maintenance savings are real.

Rapid prototyping. Getting from zero to a working agent tool takes minutes with MCP. The token overhead doesn't matter when you're still iterating before it runs in production.

When to cut MCP out of the loop

The cases where you should rethink it:

Deterministic batch pipelines. You know exactly what the agent needs to do. Schema injection has zero conversational payoff here. It's pure cost. A job with 50 items takes roughly 50 seconds via direct API vs about 25 minutes via MCP, with 10x to 20x token overhead per step. At that scale, the pipeline eats itself.

High volume single purpose agents. If your agent does one thing 10,000 times a day, a direct API call with a hardcoded schema is faster, cheaper, and easier to debug. MCP's flexibility is a liability in this case.

Latency sensitive paths. More tokens means slower time to first token from the LLM. For realtime applications where response latency matters, every unnecessary token hurts.

Fixed scope, cost optimized pipelines. If you know you'll never need more than three tools and the task is well defined, bespoke beats general purpose every time.

Practical reduction: field projection, selective loading, lazy registration

If you're committed to MCP but want to cut the overhead, three patterns help.

Field projection

Instead of passing the full API response, return only the fields the agent actually needs. The SerpApi CLI example gets most of its efficiency from this: it strips the raw response to title, snippet, and URL before handing anything to the LLM. The agent gets the same information with a fraction of the token cost.

// Instead of passing the full response object:
const raw = await serpapi.search(query);

// Project down to what the agent actually needs:
return {
  results: raw.organic_results.slice(0, 5).map(r => ({
    title: r.title,
    snippet: r.snippet,
    url: r.link,
  }))
};

Selective tool loading

Not every agent needs every tool every session. If you can determine upfront which tools a task requires, register only those. MCP servers support dynamic tool lists, so you don't have to expose everything on every request.

Lazy registration

Register the core tools that handle 90% of your traffic. Add specialized tools only when the conversation signals it needs them: via explicit user intent, a specific task type, or a flag in your system prompt. This keeps the base schema small and only pays for specialized tooling when it's actually relevant.

FAQ

Does MCP use more tokens than direct API?

Yes, always. Schema injection adds a minimum of 4x overhead and can reach 32x for complex multitool servers. The question is whether the flexibility justifies the cost for your use case.

How many tokens does MCP add per request?

It depends on how many tools are registered and how complex their schemas are. A minimal server might add a few hundred tokens per message. A large server with dozens of tools can add tens of thousands per message, per turn.

When should I use MCP vs direct API calls?

MCP for conversational, open ended, governance heavy, or multiteam scenarios. Direct API for batch jobs, high volume single purpose agents, latency sensitive paths, and any pipeline where the task scope is fixed and well defined.

How do I reduce MCP server token overhead?

Field projection on responses, selective tool registration, and lazy loading of specialized tools. For extreme cases, a hybrid approach works: MCP for the orchestration layer, direct API for the hot path inside each tool.

If you want to see how this fits into a production agent architecture, I cover the full decision framework in my post on production architecture.

If you want help building your own MCP setup without the token tax eating your budget, that's the kind of work I take on.

Drop a comment if you've run your own numbers. Curious what the 4x to 32x range looks like in your production setup.

Why Rate Limits Kill Your AI Agents in Production (And the Patterns That Actually Work)

Mudassir Khan — Sun, 21 Jun 2026 19:34:18 +0000

LLM API calls fail between 1% and 5% of the time in production. Not from hallucinations. From 429 errors nobody handled.

You've probably seen this: you ship an agent, everything works in staging, prod hits a burst of traffic, the provider throttles you, and suddenly your agent is retrying forever, burning tokens on every attempt, and the cost graph spikes sideways. The incident isn't model quality. It's the retry loop you forgot to fence.

I've written about production architecture for agentic systems before. Rate limiting is the piece that bites teams hardest, so let me go deep on it here.

Why rate limits catch most AI teams off guard

Most developers come to LLMs from REST APIs where rate limiting is mostly a nuisance you handle with one retry. With LLMs, the shape of the problem is different.

A single agent request isn't one API call. It's potentially dozens: planner calls, tool calls, summarizer calls, verifier calls. They all share the same rate window. One agent serving 10 simultaneous users can hit 200 to 300 API calls per minute before you realize what's happening.

The other thing that surprises teams: LLM rate limits often count tokens, not requests. Two requests at 50 tokens each and one request at 10,000 tokens are not equal, but your requests per minute counter treats them the same. You can stay under RPM and blow right past TPM.

The hidden cost: retry storms in multiagent systems

The most common production incident isn't a model giving the wrong answer. It's an agent that decides to retry, then retries again, each retry being a full provider call with no delay logic to protect the window.

This is the retry storm. You hit a rate limit, your agent retries immediately (or with a fixed 1 second delay), the retried call also hits the limit, all the retries queue up at the rate window boundary and fire at once, and now you've turned a temporary throttle into a sustained overload.

In multiagent systems it compounds. One orchestrator spawning five subagents, each doing their own uncoordinated retries, can turn a single 429 into 50 retry attempts within the same second. This is one of the core failure patterns I covered in why AI agents fail in production.

Proper rate limit handling can cut redundant API costs by 40%. That's not a small rounding error. That's architectural discipline that pays for itself.

Request count vs token count: why you're measuring the wrong thing

Your LLM provider gives you two limits: requests per minute (RPM) and tokens per minute (TPM). Most teams watch RPM. TPM is usually what breaks you.

Here's why: 1 request can be 50 tokens or 10,000 tokens. If you only count requests and stay under RPM, a single heavy prompt (large context, long output) can exhaust your TPM budget while you're still well under RPM. The next 30 requests all get 429s even though you've only made 5 calls that minute.

The fix is to count tokens on the way out, not after the call fails:

interface RateLimiter {
  requestTokens(estimatedTokens: number): Promise<void>;
  recordUsage(actualTokens: number): void;
}

class TokenBucketLimiter implements RateLimiter {
  private bucketTokens: number;
  private lastRefill: number;

  constructor(
    private tpmLimit: number,
    private refillIntervalMs = 60_000
  ) {
    this.bucketTokens = tpmLimit;
    this.lastRefill = Date.now();
  }

  async requestTokens(estimatedTokens: number): Promise<void> {
    this.refillIfNeeded();

    if (this.bucketTokens < estimatedTokens) {
      const waitMs = this.msUntilRefill();
      await new Promise(resolve => setTimeout(resolve, waitMs));
      this.refillIfNeeded();
    }

    this.bucketTokens -= estimatedTokens;
  }

  recordUsage(actualTokens: number): void {
    // Adjust the bucket if the actual usage differed from the estimate.
    // Track pre-reserved vs actual in a real implementation.
    void actualTokens;
  }

  private refillIfNeeded(): void {
    const now = Date.now();
    if (now - this.lastRefill >= this.refillIntervalMs) {
      this.bucketTokens = this.tpmLimit;
      this.lastRefill = now;
    }
  }

  private msUntilRefill(): number {
    return this.refillIntervalMs - (Date.now() - this.lastRefill);
  }
}

Estimate tokens before the call using tiktoken or a rough char/4 heuristic, consume from the bucket, wait if you're over budget. This moves the rate limit behavior from reactive (catch the 429) to proactive (don't send the call that would fail).

The 3-layer architecture that actually works

The teams that handle rate limits cleanly aren't just retrying smarter. They're operating at three layers simultaneously.

Layer 1: token bucket per (user, model). Limit each user's consumption independently. A single heavy user doesn't starve everyone else. Scope the bucket to both the user and the model so a cheap model and an expensive one don't compete for the same budget.

Layer 2: circuit breakers. Three signals should trip a circuit breaker:

Cost velocity: if a user is burning tokens at 3x their rolling average, something is looping
Repeated prompts: the same or very similar prompt fired multiple times in 10 seconds is probably a bug, not intent
Error rate: if 20% of a user's calls are 4xx in a 5 minute window, stop sending and surface the error instead of absorbing it silently

Layer 3: declarative fallback chain. Primary model → cheaper model (e.g. GPT-4o → GPT-4o mini) → semantic cache (return a stored response for similar queries) → 503. The chain is declarative, not imperative. You configure the fallback in one place and every agent inherits it.

TypeScript implementation: exponential backoff with jitter

The reason a fixed retry delay creates storm conditions is that all retried requests fire at the same moment. Jitter desynchronizes them.

interface RetryConfig {
  maxAttempts: number;
  baseDelayMs: number;
  maxDelayMs: number;
  jitterFactor: number; // 0 to 1, how much randomness to add
}

async function withExponentialBackoff<T>(
  fn: () => Promise<T>,
  config: RetryConfig = {
    maxAttempts: 3,
    baseDelayMs: 1_000,
    maxDelayMs: 30_000,
    jitterFactor: 0.3,
  }
): Promise<T> {
  let lastError: Error;

  for (let attempt = 0; attempt < config.maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (error) {
      lastError = error as Error;

      if (!isRateLimitError(error) || attempt === config.maxAttempts - 1) {
        throw error;
      }

      const exponentialDelay = config.baseDelayMs * Math.pow(2, attempt);
      const cappedDelay = Math.min(exponentialDelay, config.maxDelayMs);
      const jitter = cappedDelay * config.jitterFactor * Math.random();
      const finalDelay = cappedDelay + jitter;

      await new Promise(resolve => setTimeout(resolve, finalDelay));
    }
  }

  throw lastError!;
}

function isRateLimitError(error: unknown): boolean {
  if (error instanceof Error) {
    return (
      error.message.includes('429') ||
      error.message.toLowerCase().includes('rate limit')
    );
  }
  return false;
}

The key is Math.random() in the jitter calculation. Two simultaneous retries sleep for different durations and arrive at the provider at different moments. At scale this turns a synchronized wave into a spread distribution.

Also worth checking: the OpenAI 429 response includes a Retry-After header telling you exactly how many seconds to wait. Parse it and honor it directly instead of running your own backoff math.

Wiring in a circuit breaker (code example)

A circuit breaker wraps your LLM client and opens (stops sending) when the error rate crosses a threshold. Here's a minimal implementation:

type CircuitState = 'closed' | 'open' | 'half-open';

class LLMCircuitBreaker {
  private state: CircuitState = 'closed';
  private failureCount = 0;
  private lastFailureTime = 0;
  private readonly failureThreshold = 5;
  private readonly recoveryTimeMs = 60_000;

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() - this.lastFailureTime > this.recoveryTimeMs) {
        this.state = 'half-open';
      } else {
        throw new Error(
          'Circuit open: LLM provider is throttling. Try again shortly.'
        );
      }
    }

    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  private onSuccess(): void {
    this.failureCount = 0;
    this.state = 'closed';
  }

  private onFailure(): void {
    this.failureCount++;
    this.lastFailureTime = Date.now();
    if (this.failureCount >= this.failureThreshold) {
      this.state = 'open';
    }
  }
}

// Wire the two layers together:
const breaker = new LLMCircuitBreaker();
const limiter = new TokenBucketLimiter(100_000);

async function safeLLMCall(prompt: string, estimatedTokens: number) {
  await limiter.requestTokens(estimatedTokens);
  return breaker.call(() =>
    openai.chat.completions.create({
      model: 'gpt-4o',
      messages: [{ role: 'user', content: prompt }],
    })
  );
}

The circuit opens after 5 failures and stays open for 60 seconds. During that window, requests fail fast instead of piling up waiting for a timeout. After 60 seconds it shifts to half-open: one test call goes through, and if it succeeds the circuit closes again. If it fails, the clock resets.

Put this at the call site of every provider interaction and you've got a fence around every agent that uses it.

FAQ

What causes LLM rate limit errors?

Two things. You've hit the provider's RPM or TPM ceiling for your account tier, or a single request exceeded the context window limit. Check both when you see a 429. The error response usually tells you which limit you hit.

How do I handle 429 errors from OpenAI?

The OpenAI 429 response includes a Retry-After header with how many seconds to wait. Parse it and sleep for that duration before retrying. The header value is more reliable than any backoff formula you'll calculate yourself.

What is exponential backoff for APIs?

Instead of retrying immediately or on a fixed interval, each retry waits longer than the previous one: 1 second, then 2 seconds, then 4 seconds, then 8 seconds. Adding random jitter to each wait time prevents all retriers from firing at the same moment and overloading the provider again.

How do I prevent agent retry storms?

Two controls working together. A circuit breaker that opens after N consecutive failures stops new calls from going out while the provider is saturated. A token bucket that estimates usage before sending catches bursts before they hit the API. The combination prevents the feedback loop where retries cause more rate limits cause more retries.

If you're building out the rest of the agent reliability layer, I go deeper into the architectural patterns on my blog.

If you want this wired up on your own stack end to end, agentic AI consulting is exactly the kind of work I take on.

Drop a comment if your rate limit setup looks different. Curious whether people are managing this at the SDK layer or at an API gateway.

Stop letting the prompt be your state machine

Mudassir Khan — Tue, 16 Jun 2026 18:34:17 +0000

Stop letting the prompt be your state machine

You shipped an LLM feature six months ago. Now the same user input produces wildly different outputs depending on... nothing you can point to. Something in the sampling? The time the context filled up and a chunk got dropped? Nobody knows. This is what happens when the prompt becomes your runtime.

The trap: the prompt as an accidental runtime

Here is what the trap looks like in TypeScript:

async function handleUserRequest(input: string): Promise<string> {
  const prompt = `
    You are a helpful assistant.
    The user said: ${input}
    Previous context: ${someGlobalContext}

    Decide what to do, gather any information you need,
    format the response, and return it.
  `;
  return llm.complete(prompt);
}

The model is doing everything here: deciding the intent, gathering data, formatting output, choosing what to persist. That is a footgun. You handed the runtime to a stochastic function.

Gartner attributes many failed agentic AI projects to unclear value and inadequate risk controls. Deterministic, testable workflows address both. The fix is not a better prompt. The fix is to stop using the prompt as an architecture.

What "deterministic" can and cannot mean here

Be honest about what you can and cannot control.

You cannot control: the model's exact output. It is probabilistic by design.

You can control:

The shape of the output (structured output plus schema validation)
The steps that run before and after the model call
What data enters the model
What happens when the output fails validation
Whether a human reviews the result before it commits to anything irreversible

Determinism here means: the same inputs, the same workflow steps, the same guardrails every time. Not the same tokens every time. That is a realistic and achievable target. It is also the thing teams skip when they are moving fast.

Typed workflow steps around the model call

Break the work into discrete typed steps. Each step has a clear input type and a clear output type. The model call is one step in the pipeline, not the whole thing.

type WorkflowInput = {
  userId: string;
  rawRequest: string;
};

type EnrichedInput = WorkflowInput & {
  userContext: UserContext;
  relevantDocs: string[];
};

type ModelOutput = {
  intent: "summarize" | "search" | "draft" | "unknown";
  confidence: number;
  payload: string;
};

type WorkflowResult = {
  response: string;
  audit: {
    intent: string;
    humanReviewed: boolean;
  };
};

async function enrich(input: WorkflowInput): Promise<EnrichedInput> {
  const [userContext, relevantDocs] = await Promise.all([
    fetchUserContext(input.userId),
    fetchRelevantDocs(input.rawRequest),
  ]);
  return { ...input, userContext, relevantDocs };
}

async function classify(enriched: EnrichedInput): Promise<ModelOutput> {
  // Model call is isolated here, not scattered everywhere
  const raw = await llm.complete(buildClassificationPrompt(enriched));
  return parseAndValidate(raw);
}

async function respond(output: ModelOutput): Promise<WorkflowResult> {
  const response = await generateResponse(output);
  return {
    response,
    audit: { intent: output.intent, humanReviewed: false },
  };
}

async function runWorkflow(input: WorkflowInput): Promise<WorkflowResult> {
  const enriched = await enrich(input);
  const classified = await classify(enriched);
  return respond(classified);
}

Each step is independently unit testable. You can mock classify to return a fixed ModelOutput and test respond in complete isolation. That was impossible when the prompt was the runtime.

Structured output + schema validation as a contract

The model call step should never return a raw string when you need structured data. Use JSON mode, tool calling, or a schema constrained completion, then validate immediately.

import { z } from "zod";

const ModelOutputSchema = z.object({
  intent: z.enum(["summarize", "search", "draft", "unknown"]),
  confidence: z.number().min(0).max(1),
  payload: z.string().min(1),
});

async function classify(enriched: EnrichedInput): Promise<ModelOutput> {
  const raw = await llm.complete(buildClassificationPrompt(enriched), {
    response_format: { type: "json_object" },
  });

  const parsed = JSON.parse(raw);
  const result = ModelOutputSchema.safeParse(parsed);

  if (!result.success) {
    throw new ClassificationValidationError(result.error, raw);
  }

  return result.data;
}

Zod gives you a contract. If the model drifts, the validation catches it before the rest of your app sees the output. The answer to "how do you validate LLM responses?" is: schema validation on parse, not on trust.

Retries, idempotency, and failure gates

Validation failures should not crash silently. Wrap the model call with a retry budget and a typed failure signal:

type ClassifyResult =
  | { ok: true; data: ModelOutput }
  | { ok: false; reason: "validation" | "timeout" | "rate_limit"; raw?: string };

async function classifySafe(
  enriched: EnrichedInput,
  maxAttempts = 2
): Promise<ClassifyResult> {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      const data = await classify(enriched);
      return { ok: true, data };
    } catch (err) {
      if (err instanceof ClassificationValidationError && attempt < maxAttempts) {
        continue; // one retry on schema failure
      }
      if (err instanceof RateLimitError) {
        return { ok: false, reason: "rate_limit" };
      }
      return { ok: false, reason: "validation", raw: (err as any).raw };
    }
  }
  return { ok: false, reason: "validation" };
}

Idempotency matters when retries touch external state. If your workflow calls an API inside the model step, wrap it in an idempotency key so a retry does not double the side effect. The workflow layer controls this. The model itself cannot.

Where a human gate belongs

A hybrid memory and retrieval approach (automatic retrieval at request start plus explicit storage) keeps agent state predictable. So does knowing when not to automate the final step.

High impact or irreversible steps should route to a human via a control gate before committing. Not because LLMs are bad. Because some decisions carry real consequences and the cost of a wrong one outweighs the automation gain.

async function runWorkflow(input: WorkflowInput): Promise<WorkflowResult> {
  const enriched = await enrich(input);
  const classifyResult = await classifySafe(enriched);

  if (!classifyResult.ok) {
    return queueForHumanReview(enriched, classifyResult.reason);
  }

  const { data: classified } = classifyResult;

  // Irreversible or low-confidence intent routes to human review
  if (classified.intent === "draft" && classified.confidence < 0.85) {
    return queueForHumanReview(enriched, "low confidence on draft intent");
  }

  return respond(classified);
}

The control gate is a typed branch in your workflow, not a prompt instruction. "Only do this if you are sure" is not a guardrail. A typed branch is.

If you want to go deeper on how this fits into a full system, I wrote up the production architecture for agents including how to wire these patterns together at scale.

FAQ

How do you make LLM output deterministic?
You cannot make the model itself deterministic. You make the system deterministic around it. Schema validated structured output, typed workflow steps, and retry gates with failure signals are the practical levers. The model is one isolated black box step in an otherwise typed, testable pipeline.

What is structured output?
Structured output means the model returns data in a schema you define rather than freeform prose. Most providers support JSON mode or function calling. You parse and validate the result immediately with a schema library. If it does not match the schema, treat it as a failed call, not a soft warning.

How do you validate LLM responses?
Parse the response as JSON, then run it through a schema validator. Zod is a common choice in TypeScript projects. A safeParse call gives you a typed result: success with data or failure with an error you can act on. Failure is an exception to handle, not a case to log and move on.

If you want a deeper look at how deterministic workflows fit into a full production system, I cover the complete production architecture for agents on my site.

If you want Next.js for AI products wired up end to end, that is exactly the kind of work I take on.

Drop a comment below. Curious what patterns people use to keep LLM features testable in production.

One AGENTS.md for every coding agent: stop maintaining CLAUDE.md and GEMINI.md separately

Mudassir Khan — Thu, 11 Jun 2026 10:32:17 +0000

You have a CLAUDE.md in your repo. You have a GEMINI.md. If you use Cursor, you have .cursorrules. They all say roughly the same thing: run pnpm test before committing, don't touch the generated files in src/generated/, keep components colocated with their tests.

Then you update one and forget the other two. The agent you're using today reads the stale file. Bugs follow.

There's a spec for this now. It's called AGENTS.md and it's worth knowing about.

The problem: one repo, five instruction files

Every coding agent has its own convention for where it looks for instructions. Claude Code reads CLAUDE.md. Codex reads AGENTS.md. Cursor reads .cursorrules. Gemini Code Assist reads GEMINI.md. Some tools read from multiple places and merge them.

The result is that a shared codebase ends up with a pile of overlapping instruction files, each slightly out of date. Whoever added the note about the database migration last month updated CLAUDE.md. Nobody updated the Cursor file. Now your teammate is using Cursor and the agent keeps trying to use the old migration pattern.

This is a solved problem in human tooling. We don't maintain separate README-for-alice.md and README-for-bob.md. We have one README. We need the same thing for agents.

What AGENTS.md is and where it came from

AGENTS.md is an open format that works as a "README for agents." It loads automatically into an agent's context and holds the information that would clutter a human README: build steps, test commands, project conventions, and things the agent should never do.

The format emerged from collaboration across the ecosystem. OpenAI Codex, Google Jules, Cursor, Amp, and Factory were all involved in the early development. A growing set of tools now supports it: Aider, goose, opencode, Zed, Warp, VS Code, and Devin are all on board.

The spec lives at agents.md and on GitHub. It's intentionally minimal. The goal is to be readable by both humans and tools, not to define a complex schema.

What goes in it (and what stays in the README)

Think of it as the context your agent needs that a human already knows from working in the repo. A good AGENTS.md usually covers:

# AGENTS.md

## Build
pnpm install
pnpm run build

## Test
pnpm test          # unit tests (Vitest)
pnpm test:e2e      # Playwright, requires local dev server

## Conventions
- Components live next to their tests: src/components/Foo/Foo.tsx + Foo.test.tsx
- Never edit files in src/generated/ — they are codegen outputs
- Use Zod for all input validation, not manual typeof checks
- CSS modules only, no inline styles in components

## Architecture notes
- Auth is handled in src/middleware.ts — do not add auth logic to route handlers
- Database access goes through src/lib/db.ts only — never import Prisma client directly

What stays in the README: human onboarding context, project history, how to get a dev environment running the first time, links to docs. That's for humans. The agent doesn't need your origin story.

The split sounds obvious once you name it. In practice, most READMEs are a mix of both, which is why agents end up reading things that don't help them and missing things that do.

Tool support across the ecosystem

Here's where things stand. This is worth bookmarking because it's moving fast:

Tool	Reads AGENTS.md
OpenAI Codex	Yes (native)
Google Jules	Yes
Cursor	Yes
Amp	Yes
Factory (Droids)	Yes
Aider	Yes
goose	Yes
opencode	Yes
Zed	Yes
Warp	Yes
VS Code (Copilot)	Yes
Devin	Yes
Claude Code	Reads `CLAUDE.md` natively; AGENTS.md support in progress

Claude Code currently reads CLAUDE.md as its primary file. If you're using Claude Code alongside other agents, you still want AGENTS.md as your canonical source and CLAUDE.md as a thin wrapper (more on that in the next section).

Deriving CLAUDE.md from one source

The pattern that works: put everything shared in AGENTS.md. In CLAUDE.md, include a short note pointing to it, then add only configuration specific to Claude.

<!-- CLAUDE.md -->

# Claude Code Instructions

See AGENTS.md for shared project conventions. The notes below are Claude-specific.

## Extended thinking
Use extended thinking for architecture decisions and debugging sessions where context depth matters.

## MCP tools
Local dev uses the Filesystem MCP mounted at /tmp/project. Don't use the Bash tool for file operations when the Filesystem MCP is available.

This way AGENTS.md is the truth. CLAUDE.md is an extension. When a convention changes (new test runner, new linting rule), you update one file.

Same pattern for GEMINI.md or any other agent file: start with "see AGENTS.md" and add only the things that are genuinely specific to that tool.

Gotchas: hierarchy, extensions, and edge cases

Codex extends the spec with a hierarchy. OpenAI Codex supports a hierarchical AGENTS.md system where you can place AGENTS.md files in subdirectories and they compose together. This is an extension specific to Codex, not part of the base spec. Other tools read the root AGENTS.md only (or whatever their lookup path is). Don't rely on the hierarchy pattern if you're targeting the broader ecosystem.

Keep it evergreen. The same rule about avoiding dates that applies to your README applies here. Agent instructions that reference a specific sprint or quarter confuse the agent when that sprint is long past. Write instructions for the current state of the codebase, not for what you were working on when you wrote them.

Test descriptions, not test output. Some teams paste in example test output to tell the agent "this is what passing tests look like." That output goes stale fast. Instead, describe the test runner invocation and what a passing run looks like in plain terms. If you're wiring up more rigorous evaluation for LLM agents in production, this framing carries over.

Tool lookup paths vary. Most tools look for AGENTS.md at the repo root, but some support workspace configurations. Check the docs for whichever tool you're using if you have a monorepo setup. Codex in particular has well documented monorepo support via the hierarchical extension.

FAQ

What is AGENTS.md?
It's an open format for a single instruction file that multiple coding agents can read automatically. It holds project conventions, build commands, and test steps that you want every agent working in the repo to know.

Which tools support AGENTS.md?
OpenAI Codex, Google Jules, Cursor, Amp, Factory, Aider, goose, opencode, Zed, Warp, VS Code (Copilot), and Devin. Claude Code reads CLAUDE.md natively and AGENTS.md support is in progress.

How is it different from a README?
A README is written for humans. AGENTS.md is written for agents. It covers build and test mechanics, codebase conventions, and architecture rules. Human onboarding context belongs in the README.

If you want to go deeper on structuring AI systems architecture in your codebase, that is exactly the kind of work I take on.

More writing on agentic patterns and developer tooling on my blog.

Curious which agents you're targeting. Drop a comment with your stack and whether you've hit the problem of maintaining five instruction files.

Claude Fable 5: Features, Pricing, and Fallbacks

Mudassir Khan — Wed, 10 Jun 2026 17:37:38 +0000

Claude Fable 5: Features, Pricing, and Fallbacks

Anthropic just shipped claude-fable-5, its most capable widely released model. It is a Mythos class model made safe for general use, with a 1M token context window, adaptive thinking, and a safety classifier that quietly hands risky prompts to Claude Opus 4.8. Here is what actually matters when you wire it into a product.

Fable 5 and Mythos 5 are the same model

The launch came in two names. claude-fable-5 and claude-mythos-5 are the same underlying model. The only difference is the safeguards.

Mythos 5 runs without the safety classifiers and is limited to a small group of cyber defenders through Project Glasswing. Fable 5 is the version everyone gets, on the Claude API, Amazon Bedrock, Vertex AI, and Microsoft Foundry. A Mythos class model sits a tier above the Opus class in raw capability, so when your request is not flagged, you are talking to the strongest model Anthropic has put in general hands. Their own data says more than 95% of Fable sessions never hit a fallback.

The specs you plan around

Spec	Claude Fable 5	Claude Opus 4.8
API id	`claude-fable-5`	`claude-opus-4-8`
Context window	1M tokens	up to 1M tokens
Max output	128k tokens	lower
Input price / M tokens	$10	$5
Output price / M tokens	$50	$25
Thinking mode	adaptive only	configurable

Fable 5 costs twice as much per token as Opus 4.8. That premium buys the strongest reasoning and the longest autonomous runs, but it means routing matters. Send the hard, long horizon tasks to Fable and keep routine work on Opus or Sonnet. The 1M context is on by default, and the 128k output ceiling is enough to return whole files or a long migration in one response.

Where it actually leads

The headline is long horizon autonomy. The longer the task, the bigger Fable 5's lead over earlier models.

Software engineering. Stripe ran a codebase wide migration across 50 million lines of Ruby in a single day. Fable 5 also tops Cognition's FrontierCode eval, even at medium effort.
Knowledge work. Highest score of any model on Hebbia's finance benchmark, with real gains on document reasoning and chart and table interpretation.
Vision. New state of the art. It reads precise numbers off scientific figures and rebuilds a web app's source from screenshots alone.
Memory. It stays focused across millions of tokens and improves using notes it writes to a file based memory tool.

The safety classifier is the thing to design for

This is the part that will surprise you in production. A separate classifier model sits in front of Fable 5. When a prompt looks like it touches cybersecurity, biology, chemistry, or distillation, Fable does not answer. The request falls back to Claude Opus 4.8, and the user is told it happened.

The classifiers are tuned conservatively, so they sometimes catch harmless prompts. Anthropic says the fallback fires in under 5% of sessions on average. For builders, the move is to expect the occasional fallback on benign cyber or bio adjacent prompts and handle it gracefully instead of treating it as an error.

What changes on the API

The biggest one is refusals. When Fable 5 declines, the Messages API does not throw. You get a successful HTTP 200 with stop_reason set to refusal, and it tells you which classifier declined. You are not billed for a request refused before any output is generated.

// A refused request comes back as a normal 200 response
{
  "type": "message",
  "role": "assistant",
  "stop_reason": "refusal",
  "content": [],
  "usage": { "input_tokens": 412, "output_tokens": 0 }
}

A refused request can usually be served by another model. Pass the fallbacks parameter and the API retries for you, or use the SDK middleware to retry from the client. Fallback credit refunds the prompt cache cost so you do not pay it twice.

{
  "model": "claude-fable-5",
  "fallbacks": ["claude-opus-4-8"],
  "messages": [ ]
}

Three more behaviors that are specific to Fable 5 and Mythos 5:

Adaptive thinking is always on. It is the only thinking mode. You cannot disable thinking. Use the effort parameter to control depth and spend.
Raw thinking is never returned. It is omitted by default. Set thinking display to summarized for readable summaries, and pass thinking blocks back unchanged in multiturn conversations on the same model.
30 day data retention. Both models are Covered Models, so zero data retention is not available. The data is used for safety, not training.

Pricing and availability

Pricing is $10 per million input tokens and $50 per million output tokens for both models. On the Claude API and consumption based Enterprise plans, Fable 5 is fully available now. On subscription plans the rollout is staged: included on Pro, Max, Team, and seat based Enterprise for a short window, then drawing on usage credits until capacity allows it back into the standard plans. Mythos 5 stays restricted to Glasswing partners and, soon, a small set of biology researchers.

Three things to check before you ship on Fable 5

Handle stop_reason: "refusal" as a normal response, not an exception.
Decide your fallback model now, either with the fallbacks param or SDK middleware.
Drop the thinking: {"type": "disabled"} path. It is not supported here.

If you want a deeper look at Claude Fable 5, I cover the features, pricing, and fallback behavior in more detail on my site.

If you want a model like this wired into a real product end to end, that is exactly the kind of work I take on.

Drop a comment if your fallback strategy looks different. Curious what people are routing to Opus versus keeping on Fable.