DEV Community: Mudassir Khan

AGENTS.md vs CLAUDE.md: Where Agent Context Actually Lives

Mudassir Khan — Tue, 11 Aug 2026 21:59:40 +0000

AGENTS.md vs CLAUDE.md: Where Agent Context Actually Lives

If you have opened three different repos this month and found three different context files (AGENTS.md in one, CLAUDE.md in another, both in a third, out of sync), you are not imagining the mess. AGENTS.md is now an open, vendor neutral standard that most major coding agents read, but CLAUDE.md has not gone away, and knowing which file wins where saves you from an agent quietly following stale instructions.

What AGENTS.md actually is

AGENTS.md started as a proposal from Sourcegraph's Amp team to fix a specific problem: every coding agent invented its own context file, so teams ended up maintaining CLAUDE.md, .cursorrules, .windsurfrules, and whatever else, all describing the same project. OpenAI and Google backed the standard, and it has since moved under the Linux Foundation's Agentic AI Foundation. Guides tracking adoption report 28+ supporting tools and more than 60,000 open source repos containing the file (secondary source, treat the exact counts as approximate, not audited).

The pitch is simple: one Markdown file, one format, every agent reads the same source of truth instead of you hand syncing five files that drift within a week.

Which tools actually read it

This is the part that matters when you are deciding whether to migrate. Tools with native AGENTS.md support include:

GitHub Copilot coding agent
Cursor
Amp
Factory
RooCode
Zed
Warp

Notice what is not confirmed on that list. Reports that Claude Code reads AGENTS.md natively circulate in comparison guides, but I could not verify this against Anthropic's own changelog, so I am stating it qualitatively here rather than as fact: treat it as unconfirmed until you see it in Anthropic's own docs, and keep CLAUDE.md in place as your safety net if you rely on Claude Code specifically.

AGENTS.md vs CLAUDE.md vs the well known directory

Three layers get conflated constantly, and they solve different problems.

Layer	What it is	Scope
AGENTS.md	Vendor neutral project context file	Repo or monorepo package level
CLAUDE.md	Anthropic specific context file for Claude Code	Repo or monorepo package level
well known agents directory	Emerging discovery layer for agents to find capabilities at a domain, similar in spirit to robots.txt	Domain or service level

AGENTS.md and CLAUDE.md compete for the same job (project context for a coding agent). The well known directory is not competing with either, it is a discovery mechanism, closer to how a search engine finds a sitemap than to how an agent reads project instructions. Do not treat these as three versions of the same thing.

Monorepo precedence, the part everyone gets wrong

In a monorepo, the nearest AGENTS.md wins. A file at packages/api/AGENTS.md overrides anything set at the repo root for that package, the same pattern you already know from .eslintrc or .gitignore cascading. If you have context that applies everywhere (coding style, commit conventions), put it at root. If a package has its own build tooling or test runner that the root context does not know about, give that package its own file.

CLAUDE.md follows the same nested pattern in Claude Code specifically. If you keep both files, keep the precedence rules identical across them, otherwise you get an agent that behaves differently depending on which tool opened the repo.

Migrating without breaking your existing setup

You do not need to pick one file and delete the other overnight. The recommended migration is a rename plus a symlink, so legacy tools that only look for the old filename keep working:

mv CLAUDE.md AGENTS.md
ln -s AGENTS.md CLAUDE.md

Now AGENTS.md is your source of truth, tools that support the open standard read it directly, and Claude Code (or any tool still hardcoded to look for CLAUDE.md) follows the symlink and gets identical content. No duplicate maintenance, no drift between two files that were supposed to say the same thing.

For a monorepo, run this per package that has its own context file, not just at root.

What to actually put in the file

Keep it operational, not aspirational. An AGENTS.md that reads like a mission statement is dead weight to an agent. What earns its place:

# AGENTS.md

## Setup
npm install
cp .env.example .env

## Test
npm run test -- --watch=false

## Build
npm run build

## Conventions
- All API routes live in src/app/api, not src/pages/api
- Use the shared Zod schemas in src/lib/schemas, do not redefine types inline
- Never commit generated files in dist/

Commands the agent can run verbatim, project specific conventions it cannot infer from the code alone, and nothing that is already obvious from package.json or the folder structure. If your AGENTS.md is longer than your README, you are probably explaining things the agent should be reading from the code directly.

Three things to check right now

Run find . -iname "AGENTS.md" -o -iname "CLAUDE.md" from your repo root and see how many context files you actually have, and whether any two of them disagree.
If you run a monorepo, confirm precedence is doing what you think by putting a deliberately wrong instruction in a nested AGENTS.md and watching whether your agent picks up the nested version or the root one.
If you are not ready to fully migrate, do the symlink move above on one low risk repo first and confirm your existing tooling still resolves CLAUDE.md correctly before touching anything that matters.

If you want a deeper look at how agent context and enterprise agent stacks fit together, I cover it in more detail on my site.

Writing the actual instructions inside AGENTS.md well is its own skill. I built a free agent system prompt builder if you want a starting structure instead of a blank file.

If you want this wired up across a real monorepo end to end, that is exactly the kind of work I take on.

Curious how many context files you are all secretly juggling right now. Drop your count in the comments, and whether they agree with each other.

Spec Driven Development: What It Fixes (and Breaks)

Mudassir Khan — Mon, 10 Aug 2026 20:40:38 +0000

Spec Driven Development: What It Fixes (and Breaks)

AI coding agents are great, but they have a knack for drifting. Left alone, they reinterpret vague prompts, quietly expand scope, and ship code that technically works but nobody actually asked for. Spec driven development is the attempt to fix that by making the specification, not the prompt, the source of truth the agent has to work from. It works. Here's the catch: a lot of teams adopting it are just rebuilding waterfall with a chatbot bolted on. This post breaks down which parts of the seven phase workflow earn their keep and which parts are pure ceremony.

What spec driven development actually changes

Spec driven development shifts the source of truth from the prompt to the specification. The spec, not the agent's interpretation of your Slack message, becomes the authoritative document the agent has to work from. In practice that means a seven phase pipeline: constitution, specify, clarify, plan, tasks, implement, analyze. Each phase has a human review gate between it and the next one, and that gate is the whole mechanism. It's what stops an agent from confidently running off with a bad interpretation for three hours before anyone notices.

Here's roughly what each phase is doing:

Constitution: the standing rules for the project (conventions, constraints, things that are always true).
Specify: the actual requirement, written in enough detail that two people would build the same thing from it.
Clarify: turning anything vague into a testable, unambiguous acceptance criterion. This is where EARS notation earns its keep. You want to write something an agent literally cannot misread. I built the agent system prompt builder partly because watching agents misread plain English requirements over and over got old fast.
Plan: breaking the specify plus clarify output into an actual sequence of work.
Tasks: the individual units the agent (or you) will execute.
Implement: the agent writes the code.
Analyze: someone checks the output against the spec before it ships, not after. This is the phase most teams skip or rubber stamp, and it's exactly where good testing strategy actually pays for itself, because analyze is worthless if nobody is verifying against real behavior instead of eyeballing a diff.

A vague requirement like "users shouldn't be able to log in too often" becomes something like this once you clarify it into an EARS style acceptance criterion:

Given a user has attempted to log in 5 times within 1 minute
When the user attempts to log in again
Then the system shall block the login attempt and return a rate limit error

That's the whole trick. An agent can't argue with that sentence the way it can argue with "add some rate limiting."

Diagram: A horizontal flowchart showing the seven-phase Spec-Driven Development pipeline: constitution → specify → clarify → plan → tasks → implement → analyze, with a checkmark review gate between each phase.

The tooling landscape: Spec Kit vs Kiro vs nothing at all

GitHub Spec Kit is open source and MIT licensed, a CLI first toolkit that treats specifications as the actual executable source of truth for an agent. It's a good fit if your team wants to own the workflow and is already comfortable stitching CLI tools into whatever setup you're running.

AWS Kiro takes the opposite approach: it's a full agentic IDE built around spec driven development from the ground up, not a CLI you bolt on. Kiro reached international general availability on May 7, 2026, shipping with team plans, a CLI, and property based spec testing. It had already pulled in over 250,000 developers during its preview and more than 100,000 waitlist signups in roughly 90 days before that GA date, which tells you the demand for this workflow was real before the tooling caught up. Kiro is the better fit if you want the discipline enforced by the IDE itself rather than assembled from parts you have to maintain.

And then there's option three: no dedicated tooling at all, just a well written AGENTS.md or equivalent context file plus manual review discipline. This genuinely works for smaller projects. It just doesn't scale the enforcement Kiro or Spec Kit give you for free once more than two or three people are touching the same agent workflows.

Where this actually breaks down

Here's the part most writeups skip: Thoughtworks places spec driven development in the Assess ring of its Technology Radar, not Adopt. That's a "proceed with caution," not an endorsement. The real criticism behind that placement is concrete: this practice can double documentation overhead on a project when every phase gets treated as mandatory ceremony instead of a tool you reach for when the task actually warrants it.

Here's the tell. If your team is writing exhaustive specs for a two hour task, running every phase gate on a one line CSS fix, and treating the human review step as a rubber stamp instead of an actual check, you're not doing spec driven development anymore. You're doing waterfall with an AI coding agent attached, and you've made your process slower without making it safer.

The phases are supposed to scale down for small tasks and scale up for genuinely risky or ambiguous ones. Teams that apply identical ceremony to everything are the ones who end up hating this workflow within a month, and honestly, they're right to hate it. That's a process problem, not a spec driven development problem.

How to run this without turning it into paperwork

Scale the phases to the actual risk of the task. A trivial change gets a one line spec and goes straight to implement. A genuinely ambiguous or high risk change gets the full seven phases, gates and all. Don't run the same checklist for both.

Keep the constitution file short and opinionated. It should be the rules you actually enforce day to day, not an aspirational wishlist nobody reads past the first week.

Never skip the human review gates. Everything else in this workflow is process around that one mechanism, and it's the actual safety net. This is also where wiring your agent workflow correctly matters more than people expect: if the gate is a Slack notification nobody reads until the next morning, you've built the workflow but skipped the safety mechanism it exists to provide.

Write acceptance criteria in EARS style specifically at the clarify and plan phases. That's where it earns the most, because it forces ambiguity to surface before the agent starts generating code instead of after you're already reviewing a pull request you don't fully understand.

Where this leaves you

Spec driven development is a real fix for a real problem: agents drifting on vague prompts and shipping scope nobody asked for. It's not a fad, and it's not going away. But it's also not a substitute for judgment about when process is worth the cost. The teams getting the most out of this treat the seven phases as a dial they turn up or down based on risk, not a checklist they run unconditionally on every ticket.

If you're setting this up for a team and want a second pair of eyes on the constitution file or the gate design before it calcifies into ceremony, that's exactly the kind of work I take on.

Drop a comment if your team's setup looks different, curious what variations people are actually running in production.

React Server Components and AI Streaming in Next.js: Patterns That Actually Work

Mudassir Khan — Sun, 09 Aug 2026 19:39:46 +0000

A user hits submit. Your model thinks for eight seconds. The screen shows nothing, then everything appears at once.

That is almost never a model problem. It is a boundary problem. Where you put your Suspense boundaries decides whether the page feels instant or broken, and most of the RSC advice floating around treats this as an architecture question when it is really a placement question. Here is what I actually reach for when wiring LLM output into a Next.js UI, and the mistakes that cost me the most time.

The RSC and streaming mental model you need

Most of us carry the SSR model in our heads: the server does its work, produces one HTML blob, and ships it when the whole thing is ready. React Server Components do not work that way. RSC sends a serialized React component tree over the React Flight protocol via streaming HTTP, not raw HTML.

That distinction sounds academic until you notice what it buys you. Because the payload is a stream of tree updates instead of one document, the server can flush the parts that already resolved and keep the connection open for the parts that have not. Your nav, your heading, your empty state, and your input box can be on screen while the model is still generating token one.

This is why the App Router pairs so naturally with streaming LLM output, and why it is a common first choice for AI heavy apps. You are not bolting streaming onto a render model that fights it.

Here is the shape everyone writes first:

// app/answer/page.tsx
export default async function AnswerPage() {
  const answer = await getAnswer();     // 8s
  return <article>{answer}</article>;
}

Correct, and terrible. That await sits above everything, so the whole route waits on the slowest thing in it. Nothing about RSC saves you here. You have to tell React which part is allowed to arrive late.

Suspense boundary placement for LLM streaming responses

This is the section that matters. Everything else is detail.

With multiple Suspense boundaries, the total wait equals the slowest fetch, not the sum of all fetches. React sends the shell immediately, then streams each boundary's content in as it resolves. On the model side, streaming in Next.js delivers tokens to the browser as the model produces them, so the reader sees output within the first network round trip instead of after the full generation.

The rule I follow: one boundary around the slow model call. Not one around the whole page, not one around every element.

Push the boundary too high and you have rebuilt the blank screen you were trying to fix, because everything inside it (including the static parts) waits on the model. Push it too low and you get a wall of spinners that pop in at different times, each one resizing its container, and the reader's eye chases the layout around the viewport.

The fallback deserves real attention too. Whatever you render while waiting should reserve roughly the space the finished content will occupy. A twelve pixel spinner replaced by four hundred pixels of answer is a layout jump, and it reads as jank even when the actual timing is good.

// app/chat/page.tsx
import { Suspense } from "react";

export default async function ChatPage({
  searchParams,
}: {
  searchParams: Promise<{ q?: string }>;
}) {
  const { q = "" } = await searchParams;

  return (
    <main className="mx-auto max-w-2xl p-6">
      {/* renders immediately, never waits on the model */m
      <PromptHeader query={q} />

      <Suspense fallback={<AnswerSkeleton />}>
        <ModelAnswer query={q} />
      </Suspense>
    </main>
  );
}

async function ModelAnswer({ query }: { query: string }) {
  const answer = await getAnswer(query);
  return <article className="prose">{answer}</article>;
}

function AnswerSkeleton() {
  // same footprint as a typical answer, so nothing shifts on swap
  return <div className="min-h-64 animate-pulse rounded-xl bg-neutral-100" />;
}

PromptHeader is on screen in the first flush. ModelAnswer arrives when it arrives. That is the entire trick.

Partial Prerendering in Next.js 15 and when it helps AI apps

Partial Prerendering in Next.js 15 combines a static shell with dynamic server component streaming, which improves Time to First Byte. For a chat or answer page this fits almost too neatly, because the shell genuinely is static: nav, heading, input, empty state, footer. Only the answer changes.

When it helps: your page has a real static skeleton and one clearly dynamic hole. The shell can be served from the edge cache while the model output streams into the boundary.

When it buys you nothing: the page is dynamic top to bottom. A personalised dashboard where the greeting, the usage counters, the recent items and the answer are all user specific has no static shell to prerender, so you are paying configuration complexity for a cached wrapper that is mostly empty.

Be honest about which one you have. I have watched people enable PPR on a fully dynamic route, measure nothing, and conclude the feature does not work. It worked fine. There was just nothing static to hoist.

The sibling component pattern for parallel data fetching

Once one boundary works, the instinct is to keep adding awaits to the same component. Resist it. Sequential awaits in one parent are the most common source of "why is this page four seconds slower than the slowest call in it".

// sequential: every await blocks the next one, and the whole tree below
export default async function Page() {
  const profile = await getProfile();   // 400ms
  const history = await getHistory();   // 300ms
  const answer  = await getAnswer();    // 6s
  return <Layout profile={profile} history={history} answer={answer} />;
}

Split it into siblings instead. Each one owns its own fetch, each one gets its own boundary, and React streams whichever finishes first:

export default function Page() {
  return (
    <main>
      <Suspense fallback={<Skeleton height={120} />}>
        <Profile />
      </Suspense>

      <Suspense fallback={<Skeleton height={180} />}>
        <History />
      </Suspense>

      <Suspense fallback={<AnswerSkeleton />}>
        <ModelAnswer />
      </Suspense>
    </main>
  );
}

Profile and history land almost instantly, the answer takes its six seconds, and the reader has something to look at the whole time. Total wait is the slowest fetch, not the sum. Note that the parent is no longer async: the moment you put an await back at the top, you have quietly serialised everything under it again.

Common mistakes: blocking RSC with client side AI calls

Three mistakes account for most of the streaming bugs I have had to unpick.

Calling the model from useEffect. The page gets marked as a client component, so nothing happens until the browser downloads the JavaScript, parses it, hydrates the tree, and runs the effect. Only then does the request to your model even start. You have added the entire hydration cost in front of a call that was already the slowest thing on the page. Move it to a server component and let the request start while the HTML is still streaming.

Buffering the whole response before rendering. Collecting chunks into a string and setting state once at the end is a very natural thing to write, and it throws away streaming completely. The tokens arrived early. You chose to sit on them. Render each chunk as it lands.

Putting use client too high in the tree. That directive is contagious downward: everything imported below it becomes client code. One interactive button at the top of a layout can drag the entire route to the client, and then your careful server side boundaries stop meaning anything. Keep the directive on the smallest interactive leaf you can, and pass server rendered children into it rather than importing them below it.

Testing streaming components without mocking the LLM

Mocking the SDK tests your mock. What you actually want is a fake stream sitting behind the same interface the real call uses, emitting chunks on a timer.

// test/fake-stream.ts
export async function* fakeStream(chunks: string[], delayMs = 20) {
  for (const chunk of chunks) {
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    yield chunk;
  }
}

Point your data function at that generator in tests and assert on what the reader sees over time, not just on the final string:

it("shows the first chunk before the stream finishes", async () => {
  render(await ModelAnswer({ query: "hello" }));
  expect(await screen.findByText(/Streaming/)).toBeInTheDocument();
  // stream is still open here
});

Test the loading state with the same seriousness as the resolved state. The fallback is what the reader stares at for six seconds, so a broken skeleton is a more visible bug than a mistimed final render.

FAQ

How do I stream AI responses in Next.js?

Put the model call in an async server component, wrap that component in a Suspense boundary with a realistic fallback, and let the App Router stream the rest of the page immediately. Tokens reach the browser as the model produces them, so the reader sees output within the first network round trip rather than after the full generation completes.

What is the best pattern for AI streaming with React Server Components?

One boundary around the slow model call, and siblings for anything else that fetches. Keeping each fetch in its own component with its own boundary means the total wait equals the slowest fetch instead of the sum of all of them, and it stops one slow call from holding the rest of the page hostage.

How does Partial Prerendering work with AI apps?

Partial Prerendering serves a static shell and streams the dynamic server components into it, which improves Time to First Byte. It pays off when your page has a genuinely static skeleton around one dynamic hole. On a route that is dynamic all the way down, there is no shell to prerender and you will not see much.

Most of this came out of shipping Next.js for AI products, where the streaming UI is usually the part that decides whether the whole thing feels fast.

If your slow call is a retrieval step rather than the model itself, the production RAG guide covers where that latency actually goes.

Drop a comment if your setup looks different. Curious how people are handling boundaries on pages with more than one model call in flight.

Your LLM Context Window Is Lying to You: How Token Budgets Actually Work

Mudassir Khan — Fri, 07 Aug 2026 19:47:30 +0000

The context window number in your model's docs is a capacity spec, not a promise. A model that advertises 200K tokens will happily accept 200K tokens, but the quality of what it does with them starts sliding long before you reach that ceiling. If you build production features assuming the whole window works equally well, you will ship bugs that only show up on long inputs. Here is how token budgets actually behave, and how to stop the window from quietly lying to you.

The number on the box is not the number you get

Advertised context windows describe capacity, not usable quality. Work on long context behavior shows models paying far more attention to the beginning and end of the input, while content buried in the middle gets missed even when you stay well inside the stated limit. People call this the lost in the middle problem, and it comes from how attention works, not from a bug you can prompt your way around.

The gap is wider than most teams expect. Models claiming a 200K window show measurable quality degradation around 130K tokens in practice. That is not the model refusing to answer. It is the model getting quietly worse at using the tokens you paid to send. If a critical instruction sits in the middle of a huge prompt, treat it as maybe read, not definitely read.

Everything shares one budget

The biggest misconception is that the context window is only for your input. It is not. The limit applies to the total of input and output tokens combined. Your system prompt, the conversation history, any retrieved documents, the user query, and the model's own response all draw from the same pool.

That has a consequence people hit constantly. A generous system prompt plus a long chat history can leave almost no room for the answer. The model does not warn you first. It runs out of budget mid thought and the response gets truncated, or the API rejects the request outright. Once you accept that output competes with input for the same space, you stop being surprised by cut off answers on your longest sessions.

Why big contexts get slow and expensive

Long prompts do not just risk quality. They cost you time and money on every call. The attention step compares every token to every other token, so the core computation grows with the square of the input length. The QK^T matrix is n by n, which means doubling your context roughly quadruples the work the model has to do. One study measured a 7x latency increase at 15,000 words of context. That is the difference between a snappy reply and a spinner your users abandon.

Cost follows the same curve, because LLM APIs charge per token for both input and output. Every extra token of history or retrieved context is money you spend on every single request, whether or not it earned its place. If your bills keep climbing, oversized prompts are usually part of the story, and trimming them is one of the fastest ways to reduce inference costs without changing your model.

Count tokens before you send them

You cannot manage a budget you never measure. Before firing a request, count what each part of the prompt actually costs. This one habit surfaces the system prompt bloat and runaway history that silently eat your window.

import { encoding_for_model } from "tiktoken";

const enc = encoding_for_model("gpt-4o");
const count = (text) => enc.encode(text).length;

const parts = {
  system: count(systemPrompt),
  history: count(conversation.map((m) => m.content).join("\n")),
  docs: count(retrievedChunks.join("\n\n")),
  query: count(userQuery),
};

const inputTokens = Object.values(parts).reduce((a, b) => a + b, 0);
console.log(parts, "input total:", inputTokens);

Run this once against a real session and you will usually find one part hogging the budget. Nine times out of ten it is either a bloated system prompt nobody has trimmed in months, or an unbounded history that grows every turn.

Watch the budget in production

Local counting is step one. In production you want the budget checked on the hot path and an alert before you hit the wall, not after. The rule that works: log token usage on every call, and fire an alert when usage crosses 80 percent of the context limit. That gives you room to react before requests start failing.

const CONTEXT_LIMIT = 128000; // set this to your model's real limit

function checkBudget({ inputTokens, maxOutputTokens }, requestId) {
  const projected = inputTokens + maxOutputTokens;
  const usage = projected / CONTEXT_LIMIT;

  console.log(JSON.stringify({
    requestId,
    inputTokens,
    maxOutputTokens,
    usagePct: Math.round(usage * 100),
  }));

  if (usage > 0.8) {
    notifyOncall(`Context at ${Math.round(usage * 100)}% on ${requestId}`);
  }
  return usage <= 1;
}

When you cross the line, the fix is rarely a bigger model. It is sending less. Summarize old turns, drop stale retrieved chunks, and lean on retrieval so you fetch only the passages a query needs instead of stuffing everything into the prompt. If you are wiring up that retrieval layer, a solid RAG to manage context setup does most of that trimming for you.

Three things to verify right now

Log the token count of your system prompt today. If it runs over a couple thousand tokens, it is probably carrying instructions you no longer need.
Put the beginning and end of your prompt to work. Move the single most important instruction to the very top or the very bottom, never the middle.
Add the 80 percent usage alert before your next deploy, so the window tells you it is nearly full instead of failing silently.

None of this needs a fancy platform. A token counter, one budget check, and an 80 percent alert will catch most context problems before your users do. If you want to see what oversized prompts are doing to your bill, run the numbers through the LLM pipeline cost calculator and size the budget against real pricing.

If you want a deeper look at cutting your model bill, I cover it in more detail on my site.

If you want this wired up on your own site end to end, that is exactly the kind of work I take on.

Drop a comment if your setup looks different. Curious what token budgets people are actually running in production.

Building an LLM API Gateway in Node.js

Mudassir Khan — Sun, 02 Aug 2026 20:33:27 +0000

Your LLM bill just tripled and nobody on the team can say why. Sound familiar? If you're calling OpenAI or Anthropic directly from every service in your stack, you don't have an integration problem, you have a governance problem. This is where an LLM API gateway earns its keep, and setting one up in Node.js takes less time than debugging your next surprise invoice.

Why you need a gateway layer (and what breaks without one)

Here's what actually happens without a gateway. Every service that calls an LLM hardcodes its own API key, its own retry logic, its own model name. Then one day you want to switch from GPT to Claude for cost reasons, and you're grepping through six repos. Or a junior dev's script goes into an infinite retry loop against a rate limited endpoint and your bill spikes overnight. Or, worse, you have zero visibility into which team or feature is actually driving spend.

A gateway sits between your app code and the model providers. Every request flows through one chokepoint, which means you get centralized routing, centralized cost tracking, centralized rate limiting, and a single place to swap providers without touching application code. Enterprise model API spend has already climbed past 8.4 billion dollars and keeps climbing, and most of that spend is going through zero centralized control. That's not a scale problem you'll deal with later, it's a scale problem you're already in.

The gotcha nobody warns you about: adding a gateway late in a project is way more painful than adding it on day one, because by then every service has its own bespoke calling convention baked in. If you're greenfield, wire this up before you write your first prompt call.

LiteLLM vs Portkey vs Cloudflare AI Gateway: quick comparison

You've got three solid options here and picking the wrong one for your situation is an easy footgun.

Gateway	Best for	Deployment	Standout feature
LiteLLM	Teams who want full control, self hosted	Your own infra (Docker, VM, k8s)	Massive provider support, drop in OpenAI compatible API
Portkey	Teams who want guardrails without building them	Cloud or self hosted	Fully open source under Apache 2.0, over 1600 models across 250 plus providers, 40 plus prebuilt guardrails
Cloudflare AI Gateway	Teams already on Cloudflare's edge	Cloudflare's network	Near zero added latency since it's already at the edge, built in analytics

If you're the kind of team that wants to own every knob and doesn't mind running another service, reach for LiteLLM. If you want guardrails (PII detection, prompt injection checks, content moderation) without writing that logic yourself, Portkey's prebuilt set is hard to beat, especially now that it's fully open source. If your traffic already runs through Cloudflare, the AI Gateway is the path of least resistance since there's no new infra to stand up.

For this walkthrough I'm going with LiteLLM, since it gives you the clearest picture of what's actually happening under the hood, and that understanding transfers even if you end up choosing Portkey or Cloudflare later.

Setting up LiteLLM as your Node.js proxy in 15 minutes

LiteLLM runs as a proxy server with an OpenAI compatible API, so your Node code barely changes, you just point it at a different base URL.

# litellm-config.yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  routing_strategy: least-busy
  fallbacks:
    - gpt-4o: [claude-sonnet]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

# spin up the proxy locally (Docker keeps this reproducible across environments)
docker run -d \
  -v $(pwd)/litellm-config.yaml:/app/config.yaml \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -e LITELLM_MASTER_KEY=$LITELLM_MASTER_KEY \
  -p 4000:4000 \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml

Now your Node.js code calls the proxy instead of the provider directly. If you're already using the OpenAI SDK, this is a one line change to the base URL:

// lib/llm-client.ts
import OpenAI from "openai";

const gateway = new OpenAI({
  apiKey: process.env.LITELLM_MASTER_KEY,
  baseURL: "http://localhost:4000/v1",
});

export async function askModel(prompt: string) {
  const response = await gateway.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: prompt }],
  });
  return response.choices[0].message.content;
}

That's it. Your application code has no idea it's talking to a proxy instead of OpenAI directly, and swapping providers later is a config change, not a code change.

Semantic caching: the cost cut you're leaving on the table

Here's a pattern that bit me early on. If your app answers common questions, "what's your refund policy," "how do I reset my password," you're paying full price to regenerate nearly identical answers over and over. Semantic caching fixes this by checking if a semantically similar prompt has already been answered, instead of requiring an exact string match.

Semantic caching alone can cut costs 30 to 50 percent for repetitive workloads, and routing strategies combined with caching cut costs 40 to 70 percent overall. Stack a few optimization techniques together and you're looking at 50 to 80 percent reduction on production workloads. That's not a marginal tweak, that's the difference between a sustainable AI feature and one finance keeps asking you to justify. I cover more of this cost math in my LLM inference cost optimization writeup if you want the full breakdown.

// lib/semantic-cache.ts
import { createClient } from "redis";
import OpenAI from "openai";

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

const embeddings = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const SIMILARITY_THRESHOLD = 0.95;

async function getEmbedding(text: string): Promise<number[]> {
  const res = await embeddings.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
  });
  return res.data[0].embedding;
}

function cosineSimilarity(a: number[], b: number[]): number {
  const dot = a.reduce((sum, val, i) => sum + val * b[i], 0);
  const magA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
  const magB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
  return dot / (magA * magB);
}

export async function cachedAsk(prompt: string, askFn: (p: string) => Promise<string>) {
  const promptEmbedding = await getEmbedding(prompt);
  const cachedKeys = await redis.keys("semcache:*");

  for (const key of cachedKeys) {
    const entry = JSON.parse((await redis.get(key)) ?? "{}");
    if (!entry.embedding) continue;
    if (cosineSimilarity(promptEmbedding, entry.embedding) >= SIMILARITY_THRESHOLD) {
      return entry.response; // cache hit, skip the model call entirely
    }
  }

  const response = await askFn(prompt);
  await redis.set(
    `semcache:${Date.now()}`,
    JSON.stringify({ embedding: promptEmbedding, response }),
    { EX: 86400 }
  );
  return response;
}

Scanning all cached keys with a linear similarity check is fine at small scale, but reach for a vector database (Pinecone, Qdrant, or pgvector) once you've got more than a few thousand cached entries. The pattern stays identical, only the lookup mechanism changes.

Circuit breakers and fallback routing in production

Providers go down. Rate limits get hit. Latency spikes happen at the worst possible time. If your gateway doesn't handle this gracefully, one flaky provider takes your whole feature down with it.

A circuit breaker watches for repeated failures and, once a threshold is crossed, "opens" and stops sending requests to the failing provider for a cooldown period, instead immediately failing over to a backup. This is exactly what LiteLLM's fallback config handles at the proxy level, but it's worth understanding the pattern so you can reason about it, and so you can add the same protection at the application layer for anything that bypasses the gateway.

// lib/circuit-breaker.ts
import CircuitBreaker from "opossum";
import { gateway } from "./llm-client";

async function callPrimary(prompt: string) {
  const res = await gateway.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: prompt }],
  });
  return res.choices[0].message.content;
}

const options = {
  timeout: 8000,           // fail fast instead of hanging
  errorThresholdPercentage: 50,
  resetTimeout: 30000,     // try again after 30 seconds
};

const breaker = new CircuitBreaker(callPrimary, options);

breaker.fallback(async (prompt: string) => {
  // circuit is open, reroute to the fallback model instead of failing the request
  const res = await gateway.chat.completions.create({
    model: "claude-sonnet",
    messages: [{ role: "user", content: prompt }],
  });
  return res.choices[0].message.content;
});

breaker.on("open", () => console.warn("Circuit opened, primary model degraded"));
breaker.on("close", () => console.info("Circuit closed, primary model recovered"));

export async function resilientAsk(prompt: string) {
  return breaker.fire(prompt);
}

Bifrost, one of the newer gateway options, adds just 11 microseconds of overhead per request at 5000 requests per second, worth knowing if you're comparing raw proxy performance, since Python based gateways commonly add hundreds of microseconds by comparison. Node and Go based proxies tend to sit closer to the low end of that range.

Cost caps and per user quotas with middleware

Routing and caching stop runaway costs at the model layer. But you also want a hard ceiling that stops a single user, tenant, or buggy script from blowing your monthly budget before anyone notices. This is where per user quotas earn their place, right at the middleware layer, before a request ever reaches the gateway.

// middleware/cost-cap.ts
import { Request, Response, NextFunction } from "express";
import { createClient } from "redis";

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

const DAILY_TOKEN_BUDGET = 100_000;

export async function costCapMiddleware(req: Request, res: Response, next: NextFunction) {
  const userId = req.headers["x-user-id"] as string;
  if (!userId) return res.status(401).json({ error: "Missing user identifier" });

  const key = `quota:${userId}:${new Date().toISOString().slice(0, 10)}`;
  const used = parseInt((await redis.get(key)) ?? "0", 10);

  if (used >= DAILY_TOKEN_BUDGET) {
    return res.status(429).json({
      error: "Daily token budget exceeded",
      resetAt: "midnight UTC",
    });
  }

  res.locals.recordUsage = async (tokensUsed: number) => {
    await redis.incrBy(key, tokensUsed);
    await redis.expire(key, 86400);
  };

  next();
}

Wire this in ahead of your route handler, and have the handler call res.locals.recordUsage(tokensUsed) once it gets the token count back from the model response. Combine this with the gateway's own governance layer (this is where Portkey's guardrails or LiteLLM's budget settings do double duty) and you've got protection at both the edge and the middleware. I go deeper into the governance side of this in AI governance in production if quotas alone don't cover your compliance requirements.

FAQ

What is an LLM API gateway?
It's a proxy layer that sits between your application and LLM providers (OpenAI, Anthropic, and others), handling routing, caching, rate limiting, and cost tracking in one centralized place instead of scattering that logic across every service that calls a model.

How do I reduce LLM API costs?
Combine semantic caching (cuts 30 to 50 percent on repetitive workloads), smart routing between cheaper and more capable models depending on task complexity, and hard per user quotas. Stacked together, these techniques commonly deliver 50 to 80 percent cost reduction versus calling providers directly with no controls.

What is the best LLM gateway for Node.js?
LiteLLM if you want a self hosted, fully controllable proxy with an OpenAI compatible API. Portkey if you want prebuilt guardrails without building them yourself. Cloudflare AI Gateway if your traffic already runs through Cloudflare's edge and you want the lowest added latency.

If you want a deeper look at LLM cost optimization, I cover it in more detail on my site.

If you want this wired up on your own site end to end, that is exactly the kind of work I take on.

Drop a comment if your setup looks different, curious what routing or caching strategies people are actually running in production.

Claude Fable 5 vs GPT-4o for TypeScript Codegen

Mudassir Khan — Wed, 29 Jul 2026 19:39:17 +0000

Which model actually writes better TypeScript, and does it matter enough to change what you reach for? I ran Claude Fable 5 and GPT-4o through the same set of TypeScript generation tasks to find out, and the answer is more useful than a leaderboard number: each model has a different failure mode, and which one bites you depends on what you are building.

I do a fair amount of AI systems architecture work, and picking the right model for codegen inside a pipeline is a decision that compounds. Get it wrong and you are paying for review time on every generated PR. Here is what actually held up under test.

How I set up the test (apples to apples methodology)

I kept the test as fair as I could make it. Both models got the same prompts, the same grading rubric, and no retries. The rubric had three gates:

Does the generated code compile with no edits.
Does it pass under strict: true in tsconfig without new errors.
Does it handle errors gracefully instead of letting a promise reject silently.

Each task was a well defined, realistic TypeScript problem: shaping a typed API response, modeling a tagged union for a state machine, and writing an async function with proper error boundaries. I ran every prompt through both models and scored the output against the rubric, so any difference in output is attributable to the model, not to the prompt or the grading.

Test dimension	What I checked
Compiles clean	Zero manual fixes needed
Strict mode	No new errors under `strict: true`
Error handling	No silent unhandled rejections
Schema validation	Runtime safety at the API boundary, not just compile time types

TypeScript type inference: where each model trips up

Both models are competent at basic type inference, but they trip up in different spots once things get less trivial.

Claude Fable 5 is generally stronger with generics and utility types, and it tends to produce output that survives strict mode without hand holding. Where it struggles more is deeply nested discriminated unions. It occasionally misses an edge case in the union or narrows a branch incorrectly, so you still want a human pass on anything with more than two or three variants.

GPT-4o is consistent on simple and intermediate inference, but it is more likely to fall over on advanced patterns like conditional types or heavily generic utility helpers. Here is a discriminated union pattern that is a decent stress test for either model:

type RequestState<T> =
  | { status: "idle" }
  | { status: "loading" }
  | { status: "success"; data: T }
  | { status: "error"; message: string };

function render<T>(state: RequestState<T>): string {
  switch (state.status) {
    case "idle":
      return "Waiting to start";
    case "loading":
      return "Loading";
    case "success":
      return `Loaded: ${JSON.stringify(state.data)}`;
    case "error":
      return `Failed: ${state.message}`;
  }
}

If a model drops the exhaustiveness check on that switch, strict mode will not catch a missing case unless you add a never assertion at the end. That is exactly the kind of thing I check for by hand, because both models occasionally skip it.

Error handling patterns: who generates safer async code

This is where the gap is most visible in day to day generated code. Claude Fable 5 is more thorough about wrapping awaited calls in try or catch and tends to type the error branch instead of leaving it as any or unknown without narrowing. GPT-4o is fine on the common path but more likely to leave a promise chain that can reject without a handler, especially inside Promise.all calls or event callbacks.

A pattern I now check for explicitly in any generated async code:

async function fetchUser(id: string): Promise<User> {
  try {
    const res = await fetch(`/api/users/${id}`);
    if (!res.ok) {
      throw new Error(`Request failed with status ${res.status}`);
    }
    return (await res.json()) as User;
  } catch (err) {
    if (err instanceof Error) {
      throw new Error(`fetchUser failed: ${err.message}`);
    }
    throw new Error("fetchUser failed with an unknown error");
  }
}

Small thing, but the err instanceof Error narrowing before you touch .message is exactly the kind of detail that gets skipped under a fast generation pass. Review for it either way, but expect to catch it more often in GPT-4o output.

Tool calling and Zod schema generation compared

Structured tool calling with Zod schema validation is the production standard for LLM apps right now, and this is a category where the SDK you wire the model into matters as much as the model itself. If you want the fuller picture on picking models for agentic pipelines, I go deeper in this LLM comparison for agentic AI.

On raw schema generation, Claude Fable 5 tends to produce Zod schemas that are stricter by default, catching optional versus required fields correctly more often. GPT-4o's schemas are usable but more often need a manual tightening pass, particularly around nullable fields.

import { z } from "zod";

const ToolInput = z.object({
  query: z.string().min(1),
  maxResults: z.number().int().positive().max(50).default(10),
  filters: z
    .object({
      category: z.enum(["docs", "code", "issues"]).optional(),
    })
    .optional(),
});

type ToolInput = z.infer<typeof ToolInput>;

One footprint note worth knowing before you pick your SDK layer: the Vercel AI SDK is 67.5kB gzipped versus the OpenAI SDK at 34.3kB, with LangChain JS coming in heaviest at 101.2kB. If bundle size matters for your deployment target, that is a real tradeoff independent of which model you are calling through it.

Real latency and cost per 1000 tokens in 2026

I do not have clean head to head pricing for the full Claude Fable 5 and GPT-4o tiers, so I will not fake precision there. What I can say with real numbers: at the smaller tiers, GPT-4o-mini runs about $0.15 per 1M input tokens, while Claude Haiku runs closer to $0.80 per 1M input tokens. That gap matters once you are running codegen at scale across a CI pipeline instead of a one off request.

The broader trend helps regardless of which model you land on. LLM prices dropped roughly 80 percent between early 2025 and early 2026, so the cost conversation looks very different than it did two years ago. Cost sensitive teams running high volume, low complexity generation are in a much better spot than they used to be, even on the pricier tier.

Tier	Approx input cost per 1M tokens
GPT-4o-mini	$0.15
Claude Haiku	$0.80

When to pick Claude Fable 5, when to pick GPT-4o

I will take an actual position instead of hedging. If you are prototyping, doing cost sensitive high volume generation, or the output is going to get a human review pass anyway, GPT-4o is the practical default. It is cheaper at the mini tier and good enough for common patterns.

If you are generating code that ships closer to production with less review, especially anything involving nested unions, async error paths, or Zod schemas at an API boundary, Claude Fable 5 is worth the extra cost. It fails less often in the places that are expensive to catch in code review, which is where the real cost of a codegen pipeline actually lives.

Neither is the universally correct pick. Match the model to how much human review sits between the generated code and production.

FAQ

Is Claude better than GPT-4 for TypeScript?
For strict mode compliance and safer async error handling, Claude Fable 5 comes out ahead in this test. GPT-4o is close on common patterns and cheaper at the mini tier, so the better choice depends on your review process and budget.

Which LLM writes the best TypeScript code?
In this comparison, Claude Fable 5 produced more reliable output on discriminated unions, async error handling, and Zod schema generation. GPT-4o is a solid, cheaper option for simpler generation tasks.

How does Claude Fable 5 compare to GPT-4o for developers?
Claude Fable 5 needs less manual cleanup on complex type patterns and error handling, which matters if less human review sits between generation and production. GPT-4o remains a strong, more budget friendly option for prototyping and simpler codegen work.

If you want a deeper look at picking models for production pipelines, I cover it in more detail on my site.

If you want this wired up on your own site end to end, that is exactly the kind of work I take on.

Drop a comment if your setup looks different — curious what variations people are running in production.

MCP in enterprise: access control and audit logging

Mudassir Khan — Sat, 25 Jul 2026 19:14:16 +0000

MCP in enterprise: access control and audit logging

Ship an MCP server at a company with compliance requirements and you will hit the same wall every team hits: the protocol connects your AI agent to tools beautifully, and it gives you almost nothing to control what that agent is allowed to touch once it is connected.

That gap is not a bug. MCP was built to solve connectivity, not governance. But if your agent touches a database, a ticketing system, or a support inbox with real customer data in it, connectivity without governance is exactly what gets you a very uncomfortable meeting with security.

What MCP is and what it connects (quick recap)

MCP (Model Context Protocol) standardizes how an AI agent talks to external tools and data sources. Instead of writing custom glue code for every integration, you stand up an MCP server that exposes a set of tools, and any MCP compatible client (an agent, an IDE, a chat app) can call them the same way.

That is the whole pitch, and it works. What it does not include is anything resembling per user permissions.

The default security posture: one service account for everything

Here is the part that catches teams off guard. Default MCP implementations authenticate with a single service account, and that account gets whatever access it was granted at setup. There is no concept of "this specific user should only see their own records" baked into the protocol itself.

So if your MCP server connects to a CRM using one API key with read and write access to every record, every agent call through that server inherits the full blast radius of that key. It does not matter if the human on the other end of the request is a support rep who should only ever touch their own five open tickets. The agent, and by extension anyone who can reach the agent, effectively has the same reach as the service account.

This is fine for a weekend project talking to your own sandbox. It is not fine for a production system touching HIPAA, GDPR, or SOX regulated data, and it is the single most common reason enterprise MCP rollouts stall in security review.

The six enterprise controls MCP does not ship with

Enterprise MCP governance needs, at minimum, six things the base protocol leaves as an exercise for the implementer:

OAuth 2.0 authentication. Real user identity flowing through the request, not just a static key.
Per operation RBAC or ABAC. Role based or attribute based access control scoped to the specific tool call, not the whole server.
Attribution level audit logging. Every call tied to the person who triggered it, not just the service account that executed it.
Path and scope controls. Limits on which resources a given call can reach, even within a tool the agent is otherwise allowed to use.
Rate limiting. Protection against an agent that gets stuck in a loop and hammers a downstream system.
Sensitivity label evaluation. A check on whether the data being requested is even the kind of thing this caller should see, independent of whether the underlying credential technically permits it.

None of this comes free. You either build it yourself around your MCP servers, or you put something in front of them that already has.

What an MCP gateway gives you, and when you need one

An MCP gateway sits between your agents and your MCP servers as a central broker. Instead of every agent talking directly to every server, calls route through the gateway first, which enforces policy and records a single unified audit log of everything that happened.

Practically, this means the gateway is where you implement the six controls above once, instead of rebuilding them inside every MCP server you stand up. New server, same gateway policy layer in front of it. That is the point.

You need one the moment more than one team is deploying MCP servers independently, or the moment a single MCP server touches anything regulated. If you are running one internal tool for one small team with nothing sensitive behind it, you can probably get away without one for a while. That window closes fast in a real company.

The minimum viable audit log capture set

If you are building this yourself before a gateway exists, here is the floor, not the ceiling, for what a defensible audit log needs to capture on every call:

Identity of the caller (not just the service account)
Tool name invoked
Full arguments passed, not a summary
Execution outcome (success, failure, denied)
Authorization context under which the call was allowed
Lineage back to the parent LLM request that triggered it
A cryptographic integrity hash so the log itself cannot be quietly edited after the fact

Skip any of these and you have a log that looks reassuring in a demo and falls apart the moment an auditor asks a follow up question.

Compliance implications: HIPAA, GDPR, SOX gaps in naive MCP

MCP's native logging is minimal and ephemeral. That phrase does a lot of work. Minimal means it was never designed to answer "who accessed this specific record and why." Ephemeral means even the logging it does produce may not persist long enough to satisfy a retention requirement.

For HIPAA, that is a problem the moment protected health information passes through a tool call. For GDPR, it is a problem the moment a data subject access request forces you to reconstruct exactly what an automated system touched on their behalf. For SOX, it is a problem the moment financial data flows through an agent that has no attribution trail back to a specific authorized user.

None of these frameworks care that the gap came from the protocol rather than your code. The audit still has to happen, and the burden of proof still lands on whoever deployed the system.

FAQ

Is MCP secure for enterprise?
MCP itself is a connectivity protocol, not a security framework. It can be run securely in an enterprise, but only once you add authentication, authorization, and audit logging around it. Deployed with the default single service account setup, it is not enterprise ready on its own.

What is an MCP gateway?
A broker layer that sits between agents and MCP servers, enforcing access policy and centralizing audit logging so you are not rebuilding those controls inside every individual server.

How do you audit MCP tool calls?
Capture caller identity, tool name, full arguments, execution outcome, authorization context, and request lineage on every call, with a cryptographic hash so the log cannot be silently altered later. A gateway is the practical place to implement this once instead of per server.

If you want a deeper look at securing agent deployments in production, I cover it in more detail on my blog.

If you want this wired up on your own stack end to end, that is exactly the kind of work I take on.

Curious how other teams are handling MCP access control in production. Drop a comment if your setup looks different, especially if you have landed on a gateway pattern that works well.

Next.js Server Actions vs tRPC: how to actually choose

Mudassir Khan — Tue, 21 Jul 2026 19:20:23 +0000

Next.js Server Actions vs tRPC: how to actually choose

If you are building a Next.js app with the App Router and trying to decide between Server Actions and tRPC for your mutations, here is the short version: reach for Server Actions when you have simple forms tied to a single web client, and reach for tRPC once you need request batching, query caching, or a shared API layer across web and mobile. Both solve the same underlying problem, getting typed data from the browser to the server without hand rolling REST endpoints. The right pick depends less on which one is trending and more on what your app actually needs to scale.

The mutation problem both tools solve

Every full stack app eventually answers the same question: how do you move data from a client component to a database, safely, with types that do not lie to you. In the Pages Router era that meant API toutes, fetch calls, and manually keeping request and response types in sync by hand. React 19 changed the equation by building Server Actions directly into React itself. Around the same time, tRPC matured into the default choice for teams who wanted end to end type safety without giving up a real API layer.

Both approaches let you call server code from the client like it is a regular function. The difference shows up once your app grows past a single form.

How Server Actions work, and what React 19 added

A Server Action is a function marked with the "use server" directive. You call it directly from a form or an event handler, and Next.js handles the network request, serialization, and cache revalidation for you.

// app/actions/create-post.ts
"use server";

import { revalidatePath } from "next/cache";
import { db } from "@/lib/db";

export async function createPost(formData: FormData) {
  const title = formData.get("title") as string;
  const body = formData.get("body") as string;

  if (!title || title.length < 3) {
    return { error: "Title needs at least 3 characters" };
  }

  await db.post.create({ data: { title, body } });
  revalidatePath("/posts");
  return { success: true };
}

Wire it to a form and you get a working mutation with zero client side JavaScript required for the base case:

// app/posts/new/page.tsx
import { createPost } from "@/app/actions/create-post";

export default function NewPostPage() {
  return (
    <form action={createPost}>
      <input name="title" placeholder="Title" required />
      <textarea name="body" placeholder="Body" required />
      <button type="submit">Publish</button>
    </form>
  );
}

Forms built this way work without any client side JavaScript at all, which is a real win for accessibility, resilience on flaky connections, and Core Web Vitals. React 19 adds two hooks that close most of the gap Server Actions used to have against client state libraries: useActionState for tracking the pending, error, and result state of a mutation, and useOptimistic for showing the new state immediately while a mutation is still in flight.

"use client";
import { useActionState } from "react";
import { createPost } from "@/app/actions/create-post";

export function PostForm() {
  const [state, formAction, isPending] = useActionState(createPost, null);

  return (
    <form action={formAction}>
      <input name="title" placeholder="Title" required />
      <textarea name="body" placeholder="Body" required />
      <button type="submit" disabled={isPending}>
        {isPending ? "Publishing..." : "Publish"}
      </button>
      {state?.error && <p role="alert">{state.error}</p>}
    </form>
  );
}

That closes most of the reason people used to bolt on a separate API layer just to get loading and error states on a form.

How tRPC works, routing, type safety, batching

tRPC takes a different approach. Instead of individual server functions sprinkled through your app, you define a router of typed procedures, and the client gets full autocomplete and type checking without writing a schema or generating code.

// server/routers/post.ts
import { z } from "zod";
import { publicProcedure, router } from "../trpc";
import { db } from "@/lib/db";

export const postRouter = router({
  create: publicProcedure
    .input(z.object({ title: "z.string().min(3), body: z.string() }))"
    .mutation(async ({ input }) => {
      return db.post.create({ data: input });
    }),

  list: publicProcedure.query(async () => {
    return db.post.findMany({ orderBy: { createdAt: "desc" } });
  }),
});

On the client, calling that mutation looks like a normal function call, fully typed end to end:

"use client";
import { trpc } from "@/lib/trpc-client";

export function PostForm() {
  const utils = trpc.useUtils();
  const createPost = trpc.post.create.useMutation({
    onSuccess: () => utils.post.list.invalidate(),
  });

  return (
    <form
      onSubmit={(e) => {
        e.preventDefault();
        const form = new FormData(e.currentTarget);
        createPost.mutate({
          title: "form.get(\"title\") as string,"
          body: form.get("body") as string,
        });
      }}
    >
      <input name="title" placeholder="Title" required />
      <textarea name="body" placeholder="Body" required />
      <button type="submit" disabled={createPost.isPending}>
        {createPost.isPending ? "Publishing..." : "Publish"}
      </button>
    </form>
  );
}

tRPC has request batching and query caching built in, neither of which Server Actions gives you on its own. If your app has a dashboard firing several queries at once, or you are serving both a web client and a mobile client from the same backend, that caching and batching layer stops being optional pretty fast.

Key differences: progressive enhancement vs supporting multiple clients

The two tools optimize for different failure modes.

	Server Actions	tRPC
Works without JavaScript	Yes, forms submit natively	No, needs the client runtime
Type safety	Yes, through TypeScript function signatures	Yes, through inferred router types
Request batching	Not built in	Built in
Query caching	You wire it yourself with `revalidatePath` or a library	Built in, backed by React Query
Best for	Web only apps, forms, simple mutations	Apps serving web and mobile, complex dashboards
Setup cost	Near zero, it is just a function	A router, a client, and some wiring

Server Actions win on simplicity and progressive enhancement. tRPC wins the moment your data layer needs to serve more than one kind of client, or your queries get complex enough that manual cache invalidation turns into a chore.

The hybrid pattern: combining both

You do not have to pick exactly one. A pattern that holds up well in production: use Server Actions for the simple, form driven mutations that benefit from progressive enhancement, and keep tRPC around for anything read heavy, batched, or shared across clients.

// app/actions/create-post.ts
"use server";

import { appRouter } from "@/server/routers/_app";
import { createCallerFactory } from "@/server/trpc";

const createCaller = createCallerFactory(appRouter);

export async function createPost(formData: FormData) {
  const caller = createCaller({});
  return caller.post.create({
    title: formData.get("title") as string,
    body: formData.get("body") as string,
  });
}

This gives you a Server Action as the entry point, so the form still works without client side JavaScript, while the actual mutation logic lives in exactly one place, the tRPC router. You get progressive enhancement on the surface and a reusable, structured API underneath that your mobile app, or any other client, can call directly.

Decision table: pick based on your app type

Your situation	Pick
Marketing site with a contact form	Server Actions
Internal admin panel, web only	Server Actions, add tRPC later if it grows
SaaS product with a web app and a mobile app	tRPC, or the hybrid pattern
Dashboard with many simultaneous queries	tRPC
You want the smallest possible setup	Server Actions
You already have a tRPC router from a previous project	Keep it, add Server Actions only where forms need progressive enhancement

None of this is permanent. Plenty of teams start with Server Actions because it is the fastest way to ship a form, then bring in tRPC once the query side of the app gets complicated enough to need caching.

FAQ

Are Server Actions replacing tRPC?

No. Server Actions replace a lot of the small API routes that used to exist only to handle form submissions. tRPC still does something Server Actions do not: give you a typed, cacheable, batchable API surface that more than one client can call.

Can you use Server Actions with tRPC?

Yes, and it works well together. Call your tRPC router directly from inside a Server Action using a server side caller, as shown above. You keep one source of truth for your mutation logic and still get progressive enhancement on the form itself.

What is the difference between tRPC and Server Actions?

Server Actions are a React and Next.js primitive, a function that runs on the server and can be called from a form or event handler. tRPC is a full API layer with routing, input validation, batching, and caching, callable from anywhere, not only from React components.

If you want a deeper look at wiring Server Actions into real production forms, with validation and redirects, I cover it in more detail in Server Actions in Next.js.

If you want this wired up on your own AI product end to end, that is exactly the kind of work I take on.

Drop a comment if your team landed somewhere different. Curious what tipped the decision for you.

Vercel AI SDK vs calling model APIs directly: what you actually gain

Mudassir Khan — Fri, 17 Jul 2026 19:18:19 +0000

You can ship a working LLM feature with nothing but fetch and a model's REST endpoint. People do it every day, and for a one shot completion it is honestly fine. So the real question is not "can I call the API directly" but "what am I giving up by not adding the Vercel AI SDK, and is that tradeoff worth a dependency?"

I have wired both approaches into production Next.js apps, and the answer is not the blanket "always use the SDK" you see on Twitter. Here is what the abstraction actually buys you, where it earns its footprint, and when reaching for raw fetch is the smarter call. If you are already deep into building AI products with Next.js, some of this will be familiar, but the middleware section is where most people leave value on the table.

What raw API calls look like (and why they're fine for simple cases)

Here is a bare completion against a chat endpoint. No SDK, no wrapper, just the request.

async function ask(prompt: string) {
  const res = await fetch("https://api.openai.com/v1/chat/completions", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
    },
    body: JSON.stringify({
      model: "gpt-4o",
      messages: [{ role: "user", content: prompt }],
    }),
  });

  const data = await res.json();
  return data.choices[0].message.content;
}

That's the whole thing. If your feature is a single server side completion (a summarizer, a classifier, a one shot rewrite) this is clean and you should not feel bad about it. No bundle cost, no new API to learn, and you can read every byte on the wire.

The pain starts the moment you want any of these: streaming tokens to the browser, swapping to a different provider, handling tool calls, or attaching caching and logging without touching your feature code. Each of those is solvable by hand. The question is whether you want to solve them by hand every time.

What Vercel AI SDK adds: streaming, provider swap, tool call handling

The same completion, streamed, through the SDK:

import { streamText } from "ai";
import { openai } from "@ai-sdk/openai";

export async function POST(req: Request) {
  const { prompt } = await req.json();

  const result = streamText({
    model: openai("gpt-4o"),
    prompt,
  });

  return result.toDataStreamResponse();
}

Three things happened here that are annoying to do with raw fetch.

First, streaming. streamText gives you a properly framed data stream that the client hook consumes with no manual parsing of server sent event chunks. If you have ever hand parsed data: lines out of a readable stream, you know why this matters.

Second, provider agnosticism. Swapping OpenAI for Anthropic is a single line:

import { anthropic } from "@ai-sdk/anthropic";
// model: openai("gpt-4o")  ->
model: anthropic("claude-sonnet-4"),

The unified API spans OpenAI, Anthropic, Google, Mistral, and dozens of other providers, so the rest of your code does not change. With raw calls, each provider has its own request shape, its own streaming format, and its own tool call schema, and you end up writing an adapter layer yourself. That adapter layer is basically the SDK, except untested and maintained by you.

Third, tool calls. Streaming partial tool calls as part of the data stream is now stable, which means the model can call your functions and you can render the arguments as they arrive instead of waiting for the full response. Doing this by hand against multiple providers is where the footgun count really climbs.

Here is the honest comparison:

Concern	Raw fetch	Vercel AI SDK
One shot completion	Trivial	Trivial (slight overhead)
Token streaming to browser	Manual SSE parsing	Built in
Switch provider	Rewrite request + parsing	One line
Tool calls across providers	Per provider schema	Unified
Caching / logging / guardrails	Wrap every call site	Middleware, one place
Bundle footprint	Zero	Real, but modest

Language model middleware: the real unlock

This is the part people underrate. Middleware lets you intercept and modify model calls without changing a single line of application code. Guardrails, caching, and logging all live in one wrapper instead of being smeared across every call site.

import { wrapLanguageModel } from "ai";
import { openai } from "@ai-sdk/openai";

const cachingMiddleware = {
  wrapGenerate: async ({ doGenerate, params }) => {
    const key = hash(params);
    const hit = await cache.get(key);
    if (hit) return hit;

    const result = await doGenerate();
    await cache.set(key, result);
    return result;
  },
};

const model = wrapLanguageModel({
  model: openai("gpt-4o"),
  middleware: cachingMiddleware,
});

Now every call through model is cached, and your route handlers do not know or care. Swap the middleware for a logging wrapper and you get full request tracing with zero call site edits. Chain several and you have guardrails plus caching plus logging composed cleanly.

The caching case connects directly to your bill. Repeated identical prompts are pure waste, and a cache in front of the model is one of the highest leverage moves for LLM inference cost. With raw fetch you would bolt this onto each call site and hope nobody adds a new one that forgets the cache. Middleware makes the wrapper the only path in.

useChat and client side state management

On the frontend, useChat handles the loop you would otherwise build by hand: appending messages, streaming the assistant reply token by token, tracking loading and error state.

"use client";
import { useChat } from "ai/react";

export function Chat() {
  const { messages, input, handleInputChange, handleSubmit } = useChat();

  return (
    <form onSubmit={handleSubmit}>
      {messages.map((m) => (
        <div key={m.id}>{m.role}: {m.content}</div>
      ))}
      <input value={input} onChange={handleInputChange} />
    </form>
  );
}

That is a streaming chat UI with optimistic input and error handling in about ten lines. Building the same thing on raw fetch means owning the readable stream reader, the incremental state updates, and the reconnection edge cases. It is doable. It is also exactly the kind of undifferentiated plumbing that eats a sprint.

When NOT to add the SDK

The abstraction is not free, so here are the cases where I reach for raw fetch instead.

You are doing a single server side completion with no streaming and no provider switching. A classifier, a batch summarizer, a cron job that rewrites text. The SDK adds a dependency for capabilities you will never call.

You are not in a JavaScript or TypeScript frontend context at all. If your LLM calls live in a Python service or an edge function in another runtime, the SDK's frontend ergonomics (useChat, the data stream protocol) are irrelevant, and you are paying for surface area you cannot use.

You need byte level control over the raw request for a provider feature the SDK has not surfaced yet. Abstractions lag the underlying APIs by design. When you hit that edge, drop down to fetch for that one call and keep the SDK for the rest. A mixed codebase is completely valid.

Bundle footprint matters for you and the feature is trivial. On a marketing site with one tiny AI widget, shipping the SDK to every visitor for a single completion is a poor trade.

Decision tree: should you use it?

Walk it top to bottom and stop at the first yes.

Do you need token streaming to the browser?      -> yes: use the SDK
Will you switch providers or A/B two models?     -> yes: use the SDK
Do you need tool calls, especially streamed?     -> yes: use the SDK
Do you want caching/logging/guardrails in one    -> yes: use the SDK (middleware)
  place instead of per call site?
Is it a single server side completion, one       -> yes: raw fetch is fine
  provider, no streaming?
Not a JS/TS frontend runtime at all?             -> raw fetch (or that runtime's client)

Most real product features hit one of the first four within a month. That is why the SDK is the default recommendation. It is not that raw calls are wrong. It is that the features you will inevitably want are the ones the SDK already handles, tested and maintained, so you are not the one paying to build and debug them.

FAQ

What does the Vercel AI SDK do?
It gives you a unified API for calling LLMs across many providers, with built in token streaming, tool call handling, language model middleware for caching and logging and guardrails, and React hooks like useChat for the frontend loop.

Is the Vercel AI SDK free?
Yes, the SDK itself is open source and free. You still pay each model provider for the tokens you use, and if you deploy on Vercel you pay for that hosting, but the library carries no license fee.

Does the Vercel AI SDK work with Claude and Gemini?
Yes. It supports Anthropic (Claude), Google (Gemini), OpenAI, Mistral, and dozens of other providers through the same interface. Switching between them is a single line change in most code.

If you want a deeper look at cutting model spend, I cover it in more detail on my site.

If you want this wired up on your own site end to end, that is exactly the kind of work I take on.

Drop a comment if your setup looks different — curious what variations people are running in production.

Stop Using LangChain for Simple LLM Tasks: The Raw Anthropic SDK Is Cheaper and Clearer

Mudassir Khan — Sun, 12 Jul 2026 19:11:59 +0000

If you reach for LangChain every time you wire up a new LLM call, you are not alone. Most teams do it. The problem is that for simple tasks, that reach costs you in tokens, in debugging time, and in a monthly bill from LangSmith that you forgot you enabled.

The Default Reach for Abstraction

LangChain became the default choice because it made the first demo fast. You imported a model wrapper, chained a couple of prompts, and had something running in 20 minutes. That early speed creates a gravitational pull. When the next feature comes up, you reach for the same toolkit.

But demos and production are different environments. In production, you need to know exactly what is going into every API call. You need to trace a failure back to the token that caused it. You need to reason about cost at the individual request level.

LangChain's abstraction layers make all of that harder. The framework handles prompt templating, message formatting, retry logic, and model selection (sometimes with opinions you did not ask for). That is genuinely useful when your pipeline has six steps, conditional routing, and tool calls that need orchestration. It is a footgun when your use case is a single turn call to summarize a support ticket.

Same Task, Two Ways

Here is the same task implemented both ways. The task: take a support ticket string and return a one sentence summary.

// LangChain approach — @langchain/anthropic + @langchain/core
import { ChatAnthropic } from "@langchain/anthropic";
import { HumanMessage } from "@langchain/core/messages";

const model = new ChatAnthropic({
  model: "claude-3-5-sonnet-20241022",
  apiKey: process.env.ANTHROPIC_API_KEY,
});

const response = await model.invoke([
  new HumanMessage("Summarize this support ticket in one sentence: " + ticket),
]);

// response is an AIMessage — content is a string here, but
// the shape changes across invoke / stream / generate / batch
console.log(response.content);

// Raw Anthropic SDK — @anthropic-ai/sdk
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

const response = await client.messages.create({
  model: "claude-3-5-sonnet-20241022",
  max_tokens: 256,
  messages: [
    {
      role: "user",
      content: "Summarize this support ticket in one sentence: " + ticket,
    },
  ],
});

// response.content is a typed ContentBlock array — no shape surprises
console.log(response.content[0].text);

The raw SDK version tells you everything: the model name, max_tokens (LangChain has a default you have to look up), and the exact message structure being sent. When something blows up, you know exactly where to look.

The LangChain version hides the request behind invoke, and the response shape silently changes depending on which invocation method you used. That is the kind of thing that bites you at 2am when one code path calls stream instead of invoke and suddenly response.content is a generator.

The Cost Reality

For LLM cost optimization, the raw SDK gives you direct control over prompt caching breakpoints. Anthropic offers a 90% discount on cached tokens vs OpenAI's 50%. When you set cache control points explicitly in the raw SDK, you capture that discount precisely. LangChain's message wrappers do not expose this at the same granularity, so you are often leaving money on the table.

The more concrete cost story is LangSmith. LangSmith adds $39 per month for 10,000 traces, with $0.50 per 1,000 after that. For a pipeline running 50,000 calls a month, that is an extra $59 on top of your LLM API bill.

// Rough monthly cost model for a simple summarization pipeline
// Scenario: 10,000 calls/month, ~2,000 tokens each

const llmCostPerMonth = 10_000 * 0.000003 * 2_000; // roughly $60/month in LLM costs

// Raw SDK only — pay the API cost
const costWithoutLangSmith = llmCostPerMonth; // $60

// With LangSmith at 10K traces/month (exactly at the base cap)
const langsmithBase = 39; // $39/month for 10K traces
const costWithLangSmith = llmCostPerMonth + langsmithBase; // $99

// At 50K calls/month, overage kicks in
const langsmithAt50K = 39 + 40 * 0.5; // $59 extra = $119 total non-LLM tooling cost

Teams that have moved to the native SDK report roughly a 40% reduction in API expenses. Another case study found LangChain incurred 2.7x higher costs compared to a manual implementation for a simple RAG system. That delta is not from the LLM provider. It is from auxiliary tooling and inefficient prompt routing that disappears when you own the code path directly.

When LangChain IS the Right Call

This is not a LangChain hit piece. The framework earns its weight in specific scenarios.

If you are building a multistep agent that calls tools, decides which tool based on the last output, maintains memory across a conversation, and routes to different subchains based on intent, the raw SDK will make you hand roll all of that. For anything with tool use at that depth, check the function calling guide and then honestly evaluate whether LangChain's abstractions pay off for your pipeline shape.

Similarly, if your team is switching between OpenAI, Anthropic, and Gemini and you want a single interface, LangChain's model agnostic layer is worth it. The Vercel AI SDK is also worth a look if you are in a Next.js context and want native streaming and RSC integration at a lighter weight than LangChain.

The rough rule: fewer than three steps and a single model? Reach for the raw SDK. Conditional routing, tool use, or multiprovider switching? Evaluate the abstraction cost honestly before committing.

Making the Call

The teams I have seen migrate away from LangChain consistently name the same two friction points: debugging and production stability. When a call fails inside a LangChain chain, the stack trace gives you LangChain internals rather than the raw request that was sent. You end up adding log statements at multiple layers just to figure out what actually went to the model.

With the raw SDK, there is nothing between you and the API. The request body is a plain object you can log, diff, and replay. The response is typed and predictable. When costs spike, you can trace it to a specific call site without parsing through abstraction layers.

Start with the raw SDK for anything simple. You will understand your token usage better, debug faster, and avoid the LangSmith subscription until you genuinely need the observability it provides. Reach for an abstraction layer when your pipeline complexity demands it, not as the default starting point.

If you want a deeper look at LLM cost optimization, I cover it in more detail on my site.

If you want this wired up on your own site end to end, that is exactly the kind of work I take on.

Drop a comment if you have made the same switch. Curious what the migration looked like for your team.

I Replaced My Entire Backend With MCP Tools — Here Is What Broke (And How to Fix It)

Mudassir Khan — Thu, 09 Jul 2026 19:12:58 +0000

I Replaced My Entire Backend With MCP Tools — Here Is What Broke (And How to Fix It)

MCP tools are genuinely useful. I shipped half my backend through them in about two weeks. Then they started failing silently in production and I had no idea why.

This is the debugging playbook I built from that experience.

The failures nobody warns you about

When an MCP tool fails, it does not always throw an error you can catch. Before v1.27.1, transport errors were often swallowed completely. The onerror callback simply did not fire. Your agent would hang, retry, or silently skip the tool call. No exception, no log, nothing in your monitoring.

That behavior got fixed in v1.27.1 so errors now surface properly. But schema drift is still entirely on you.

A 2026 taxonomy paper on MCP fault classes puts API usage issues at number one: incorrect request construction, missing parameters, and schema contract violations. In practice this means your tool's input schema gets bumped in a server update, your agent does not pull the new schema, and calls start failing in ways that look like flaky network behavior.

You spend an hour chasing the wrong thing.

Stdout: the footgun hiding in plain sight

If you run a local MCP server over stdio transport, here is one of the easiest mistakes to make: logging debug output to stdout.

Do not do this.

The stdio transport uses stdout to carry the protocol messages. Anything you write there corrupts the stream. Your server breaks silently and the error surfaces three steps removed from the actual cause.

Use stderr for all logging in stdio servers:

// Wrong — corrupts stdio transport
console.log("tool called:", toolName);

// Right — stderr is safe and does not touch the protocol stream
console.error("[mcp-server] tool called:", toolName);
process.stderr.write(
  JSON.stringify({ tool: toolName, ts: Date.now() }) + "\n"
);

This one rule would have saved me about six hours across two separate incidents.

Add a correlation ID before anything else

The single most valuable change you can make to an MCP setup is adding a correlation ID to every tool invocation. Without it you cannot trace a failed call back to the agent decision that triggered it, especially once you have multiple tools calling each other in a chain.

Here is the pattern I settled on:

import { randomUUID } from "crypto";

async function callTool(
  client: MCPClient,
  toolName: string,
  params: Record<string, unknown>
) {
  const correlationId = randomUUID();
  const startMs = Date.now();

  process.stderr.write(
    JSON.stringify({
      event: "tool_call_start",
      correlationId,
      tool: toolName,
      params,
    }) + "\n"
  );

  try {
    const result = await client.callTool({ name: toolName, arguments: params });
    process.stderr.write(
      JSON.stringify({
        event: "tool_call_end",
        correlationId,
        tool: toolName,
        durationMs: Date.now() - startMs,
        success: true,
      }) + "\n"
    );
    return result;
  } catch (err) {
    process.stderr.write(
      JSON.stringify({
        event: "tool_call_error",
        correlationId,
        tool: toolName,
        durationMs: Date.now() - startMs,
        error: err instanceof Error ? err.message : String(err),
      }) + "\n"
    );
    throw err;
  }
}

Every log line gets the same correlationId. When something breaks you can grep for the ID and reconstruct the full call sequence in about thirty seconds.

Schema drift: log the hash before it bites you

The other silent killer is schema drift between server versions. Your agent negotiates the tool list once at connection time. If the server updates its schema after connection, your agent keeps using the old one.

Log the server SHA and the tool schema hash at bind time:

import { createHash } from "crypto";

async function bindServer(client: MCPClient, serverLabel: string) {
  const tools = await client.listTools();
  const schemaHash = createHash("sha256")
    .update(JSON.stringify(tools))
    .digest("hex")
    .slice(0, 8);

  process.stderr.write(
    JSON.stringify({
      event: "server_bound",
      server: serverLabel,
      toolCount: tools.length,
      schemaHash,
    }) + "\n"
  );

  return tools;
}

Now when a deployment goes out and calls start behaving differently, you can diff the schemaHash between the old and new logs. It takes five minutes instead of an hour.

Use the MCP Inspector before you ship

The official debugging tool is the MCP Inspector. It connects to stdio and Streamable HTTP servers directly, letting you invoke tools interactively without an agent in the loop.

Run it against your server before you plug it into any agent. You will catch schema issues, transport problems, and unexpected tool behaviors before they show up in production at midnight.

The observability needs go deeper once you have multiple servers in a fan out topology. The correlation ID approach above is the foundation; from there you wire the logs into whatever APM you already use. I wrote about the broader agentic AI production architecture patterns separately, and there is also a dedicated piece on LLM observability in production if you want the full monitoring picture.

Three things to check right now

Search every stdio server for console.log calls. Replace them with process.stderr.write or a logger bound to stderr.
Wrap every client.callTool() call with a correlationId. If you are not logging the ID today you are flying blind.
Run the MCP Inspector against each server before the next deployment. Invoke every tool once manually and look at the raw responses.

If you want a deeper look at production observability for AI systems, I cover it in more detail on my blog.

If you want this wired up on your own stack end to end, that is exactly the kind of work I take on.

Drop a comment if your MCP setup looks different — curious what debugging approaches people are running in production.

Next.js App Router Caching: What Actually Gets Cached (and What Doesn't)

Mudassir Khan — Sun, 05 Jul 2026 19:30:35 +0000

You deployed a Next.js App Router app, called revalidatePath() after a mutation, and the page still shows stale data. Or you disabled caching on a fetch call and the page is somehow still cached. Or you migrated from Next.js 14 to 15 and everything that was working broke silently.

All of these happen because the App Router has four separate caching layers, and they each have their own invalidation story. Developers hit these bugs constantly because the mental model of "there is one cache" is wrong.

Here's what actually exists.

The four caching layers in Next.js App Router

Next.js App Router does not have a single cache. It has four distinct layers, each running in a different context with different lifetimes and different invalidation APIs. (Next.js caching docs)

Layer	Where it runs	Lifetime	Invalidated by
Request Memoization	Server, per request	Request lifetime	Automatic
Data Cache	Server, persistent	Until revalidated	`revalidateTag`, `revalidatePath`
Full Route Cache	Server, persistent	Until revalidated	`revalidatePath`, redeploy
Router Cache	Browser, per session	Short TTL or navigation	`router.refresh()`, full reload

You need to know all four. Knowing only one is what causes the "I called revalidatePath and nothing happened" bugs.

Request Memoization vs Data Cache: why they're different

These two are the most commonly confused.

Request Memoization is automatic. If you call the same fetch() URL twice during a single server render, Next.js deduplicates it. The second call returns the already fetched result without hitting the network. It's scoped to a single request and disappears when the render is done. You cannot configure it, disable it, or persist it. It just works.

// Both of these calls in the same render only hit the network once
const user = await fetch('/api/user/123')
const userAgain = await fetch('/api/user/123') // deduped automatically

Data Cache is different. It persists across requests. When you do a fetch with caching:

const data = await fetch('https://api.example.com/posts', {
  next: { revalidate: 60 }
})

That cached result survives across multiple users and multiple requests until the 60-second TTL expires or you explicitly call revalidateTag(). This is the cache you need to think about for interal data.

The key difference: Request Memoization handles the "same fetch called twice in one render" problem. Data Cache handles the "this data should not hit the database on every request" problem. They are solving different things.

Full Route Cache: when your page stops being static

The Full Route Cache stores the rendered HTML and React Server Component payload for a page. Next.js generates this at build time for static routes and serves it to every visitor without rerunning the server component.

Your page is static (cached in Full Route Cache) when:

It has no dynamic functions (cookies(), headers(), searchParams)
All its fetch calls use cache: 'force-cache' or a revalidate value

Your page opts out of Full Route Cache when:

You use cookies() or headers() inside a server component
You set export const dynamic = 'force-dynamic'
A fetch call inside it uses cache: 'no-store'

One dynamic fetch in a layout component can silently make the entire route dynamic. This is a footgun. I've seen production apps serving a fully static marketing page do a database round trip on every request because a single deeply nested component called cookies() for analytics.

revalidatePath() and redeployment both invalidate the Full Route Cache. A revalidateTag() call only invalidates it if the route's fetch calls are tagged with the same tag.

Router Cache: the client side cache that surprises everyone

The Router Cache is the one that trips up nearly everyone who calls revalidatePath() and wonders why the page hasn't updated.

It lives in the browser, not the server. Next.js caches prefetched and visited route segments in memory for the duration of the user's session. This means even after you invalidate the Data Cache and Full Route Cache on the server, the user may still see stale content because their browser is serving the cached client side payload.

Two things clear it:

router.refresh() called in a client component
A full browser page reload

'use client'

import { useRouter } from 'next/navigation'

function SubmitButton() {
  const router = useRouter()

  async function handleSubmit() {
    await updateData()
    router.refresh() // clears Router Cache for current route, triggers revalidation
  }

  return <button onClick={handleSubmit}>Submit</button>
}

Important: revalidatePath() called in a server action does not clear the Router Cache in the same request. (Next.js revalidatePath docs) You still need router.refresh() on the client if you want an immediate update.

On demand revalidation with revalidatePath and revalidateTag

Time based revalidation (the revalidate: 60 pattern) is fine for data that can be a minute stale. For data that must update immediately after a mutation, you want on demand revalidation.

revalidatePath(path) invalidates the Full Route Cache and Data Cache for all fetch calls associated with that path. Use it after mutations that affect a whole page.

revalidateTag(tag) is more surgical. Tag your fetch calls, then invalidate only what changed.

// Tag the fetch at data-fetch time
const posts = await fetch('https://api.example.com/posts', {
  next: { tags: ['posts'] }
})

// Invalidate only posts-tagged data after a mutation
import { revalidateTag } from 'next/cache'

async function createPost(data: FormData) {
  'use server'
  await savePost(data)
  revalidateTag('posts') // only posts-tagged fetches are invalidated
}

For unstable_cache (the server side equivalent of SWR, useful when you're caching database queries or other data that does not go through fetch), invalidation works the same way with revalidateTag. (Next.js unstable_cache docs)

import { unstable_cache } from 'next/cache'

const getProducts = unstable_cache(
  async () => db.query.products.findMany(),
  ['products'],
  { tags: ['products'] }
)

Next.js 15 caching changes: the breaking change you need to know

If you migrated from Next.js 14 to 15 and your data stopped being cached, this is why.

In Next.js 14, fetch() requests were cached by default. The default was cache: 'force-cache'. You had to explicitly opt out with cache: 'no-store'.

In Next.js 15, fetch requests are not cached by default. (Next.js 15 release blog) The default is now cache: 'no-store'. You have to explicitly opt in with cache: 'force-cache' or a revalidate value.

This is a breaking change with no deprecation warning during migration.

// Next.js 14 default — cached
const data = await fetch('https://api.example.com/data')

// Next.js 15 default — NOT cached (same code, different behavior)
const data = await fetch('https://api.example.com/data')

// To opt into caching in Next.js 15
const data = await fetch('https://api.example.com/data', {
  next: { revalidate: 60 }
})
// or
const data = await fetch('https://api.example.com/data', {
  cache: 'force-cache'
})

The migration path: audit every fetch call in your app and add explicit caching configuration where you actually want it. Rely on the default only for data you intentionally want fresh on every request.

FAQ

How does caching work in Next.js App Router?

Next.js App Router uses four caching layers: Request Memoization (deduplicates fetch calls within a single server render), Data Cache (persists fetch results across requests until revalidated), Full Route Cache (stores rendered HTML for static routes), and Router Cache (browser memory cache for prefetched route segments). Each layer has its own invalidation mechanism.

How do I disable caching in Next.js 14?

To opt a specific fetch call out of caching, add { cache: 'no-store' } as the second argument: fetch(url, { cache: 'no-store' }). To make an entire route always dynamic, add export const dynamic = 'force-dynamic' at the top of the route file. Note that in Next.js 15, fetches are not cached by default so you may not need to do anything if you have already upgraded.

What is the difference between fetch cache and route segment config in Next.js?

The fetch cache (cache option on fetch() or next.revalidate) controls the Data Cache for that specific data fetch. Route segment config (export const dynamic, export const revalidate) sets the default behavior for the entire route. Route segment config is a blunt instrument. Individual fetch options are more precise and composable.

What does router.refresh() do?

It triggers a soft navigation to the current route, clearing the Router Cache for the current route and refetching server component data. It does not do a full browser reload. Use it in client components after server mutations when you need the UI to reflect the updated data immediately.

Why is unstable_cache still prefixed with unstable_?

The API was marked unstable while the team iterated on its interface. The behavior is production ready and widely used. The prefix signals that the API shape may still change in a future major version. Use it, but be prepared for a migration if the API changes.

I've wired up Next.js caching for AI products with complex data fetching requirements — the four layer model is especially important when mixing realtime data with statically cached content.

If you want to see a practical example of caching in a Next.js project, I walk through building a calculator in Next.js where fetch caching decisions matter.

Drop a comment if you've hit a caching bug that isn't covered here — curious what edge cases people are running into in production.