DEV Community: Mudassir Khan

GitHub Copilot MCP Usage Metrics Without False Signals

Mudassir Khan — Fri, 30 Oct 2026 18:43:18 +0000

GitHub's latest Copilot usage metrics update finally names things. Instead of lumping every extension into one undifferentiated bucket, the usage metrics API now reports the names of your customer configured skills, custom agents and MCP servers, and each record can carry 25 entries instead of five. That is great news for adoption dashboards, and a trap for anyone who reads "used a lot" as "works well".

TL;DR

The usage metrics API now names your own skills, agents and MCP servers, with 25 entries per record where it used to be five. You can finally see which internal tools people actually touch.
A usage count is an adoption signal, not a quality signal. Join it to evaluation outcomes before you make any rollout call.
Telemetry has gaps (clients that never report, telemetry that gets blocked), and instrumentation does not enforce permissions. Treat the numbers as a floor and keep your guardrails somewhere else.

What changed in the Copilot usage metrics API

Short version: named entries, and more of them.

Before this change, if your org ran three internal MCP servers and a handful of custom agents, the usage data mostly told you that "something custom" got used. Now the record carries the names you configured, which means you can answer questions like "is anyone actually using the read only billing server we shipped last sprint?" without sending a survey nobody fills in.

The jump from five to 25 entries per record matters more than it sounds. With five slots, a busy developer who touched a dozen tools in a day got truncated, and the long tail (usually the internal tools you care about most) fell off the end. At 25, most of that tail survives.

What it does not tell you: whether the call helped. A connection is not a success, and an invocation is not a merged pull request. That gap is the whole point of this post, and it is the same gap I keep running into when evaluating LLM agents in production: activity metrics look fantastic right up to the moment you check outcomes.

Collect named MCP and skills usage

Pull the data on a schedule and land it in your own table. Don't build dashboards straight off the API response. You'll want history, and you'll want to join it with your own data later.

I'm not going to reproduce GitHub's response schema here; read it from the API docs for your plan and map the field names once. What matters is the normalized shape you store. Here is the normalizer I'd write, with the raw record type kept deliberately loose:

// normalize-usage.ts
// Map whatever the usage metrics API returns into one row per named tool, per user, per day.

type RawUsageRecord = {
  day: string; // map from the API's date field
  user: string; // map from the API's user identifier
  entries: Array<{
    kind: "skill" | "agent" | "mcp_server";
    name: string; // the customer configured name, now reported
    interactions: number;
  }>;
};

export type ToolUsageRow = {
  day: string;
  user: string;
  toolKind: "skill" | "agent" | "mcp_server";
  toolName: string;
  interactions: number;
  truncated: boolean; // the record hit the entry limit, so its tail may be missing
};

const ENTRY_LIMIT = 25;

export function normalize(records: RawUsageRecord[]): ToolUsageRow[] {
  return records.flatMap((record) =>
    record.entries.map((entry) => ({
      day: record.day,
      user: record.user,
      toolKind: entry.kind,
      toolName: entry.name.trim().toLowerCase(),
      interactions: entry.interactions,
      truncated: record.entries.length >= ENTRY_LIMIT,
    }))
  );
}

Two details in there are easy to skip and expensive later. Normalize tool names, because people rename servers and casing drifts. And keep the truncated flag: if a record is full at 25, the long tail might still be cut, and your dashboard should say so instead of pretending the count is complete.

Join event counts to evaluation outcomes

This is where usage metrics turn into something you can actually decide with. Counts tell you who touched a tool. Outcomes tell you whether touching it was worth it. You need both on the same row.

The join key is the hard part. GitHub's numbers are aggregated, your evaluation results are per run, so pick a grain both sides can honestly share. Tool name plus day is the one I'd reach for. Per user is tempting, but it quietly turns an adoption report into a surveillance report, and people will route around it.

On your side, emit your own spans for agent operations and their outcomes. OpenTelemetry's semantic conventions (version 1.44.0) include GenAI attributes for operations and for evaluations, which gives you a vendor neutral way to separate "the agent did something" from "what it did was any good":

// record-outcome.ts
import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("agent-evals");

export async function recordToolRun(
  toolName: string,
  run: () => Promise<{ passed: boolean; score: number }>
) {
  return tracer.startActiveSpan(`execute_tool ${toolName}`, async (span) => {
    span.setAttribute("gen_ai.operation.name", "execute_tool");
    span.setAttribute("gen_ai.tool.name", toolName);
    try {
      const result = await run();
      // Check the GenAI attribute registry for the exact evaluation names in your semconv version.
      span.setAttribute("gen_ai.evaluation.name", "task_passed");
      span.setAttribute("gen_ai.evaluation.score.value", result.score);
      return result;
    } finally {
      span.end();
    }
  });
}

Then the join itself is boring SQL, which is exactly what you want. Aggregate each side first, otherwise the join multiplies your counts:

WITH usage AS (
  SELECT day, tool_name,
         SUM(interactions) AS interactions,
         BOOL_OR(truncated) AS any_truncated
  FROM tool_usage
  GROUP BY day, tool_name
),
evals AS (
  SELECT day, tool_name,
         COUNT(*) AS evaluated_runs,
         AVG(CASE WHEN passed THEN 1.0 ELSE 0.0 END) AS pass_rate
  FROM tool_evals
  GROUP BY day, tool_name
)
SELECT u.day, u.tool_name, u.interactions, u.any_truncated,
       e.evaluated_runs, e.pass_rate
FROM usage u
LEFT JOIN evals e USING (day, tool_name)
ORDER BY u.day DESC, u.interactions DESC;

A NULL in evaluated_runs is not a bug in the query. It is a finding: a tool people use that nobody is measuring.

Avoid sampling and telemetry bias

Copilot counts are GitHub telemetry, and telemetry has holes. Three of them will bend your numbers if you ignore them.

Missing clients. If a tool is mostly used from a surface that doesn't report, it looks unpopular when it isn't.
Blocked telemetry. Teams with strict network rules, often the same teams running your most sensitive MCP servers, undercount without anyone noticing.
Counts are not success. Interactions and connections measure activity. Only your evaluations measure whether the activity helped.

So treat every count as a floor, compare a tool against its own trend rather than against other tools, and never turn the data into a team leaderboard. My opinion, from building dashboards like this: the first chart someone screenshots is "most used tools", and it is the chart most likely to mislead.

There is a security angle too. OWASP LLM06 describes excessive agency as coming from three drivers: excessive functionality, excessive permissions and excessive autonomy. Instrumentation alone does not enforce any of those boundaries. A popular MCP server with write access it doesn't need is still a footgun; the usage chart just tells you how many people are holding it. If you're reviewing those boundaries, my production defense guide for AI agent security covers the permission side in detail.

Build a rollout decision table

Once usage and outcomes sit on the same row, the rollout call stops being a vibe. This is the table I'd pin in the platform team's channel:

Usage	Outcomes	Decision
High	Good	Expand. Make it a default and document it.
High	Poor	Fix or restrict. People want it and it is hurting them.
Low	Good	Promote. This is a discoverability problem.
Low	Poor	Retire. Remove the server and its permissions.
Any	No evaluations	Wait. Add evaluation before you decide anything.

The last row is the one teams skip. A tool with heavy usage and zero evaluations feels like a win, and that feeling is exactly the false signal this whole setup exists to catch.

FAQ

How do I implement github copilot mcp usage metrics safely?

Pull the usage metrics API on a schedule into your own table, normalize tool names, and keep a flag for records that hit the 25 entry limit. Emit your own OpenTelemetry spans for agent operations and evaluation results, join both sides on tool name and day, and make rollout decisions from usage and outcomes together. Keep permission boundaries enforced separately, because instrumentation does not enforce them.

What changed with github copilot mcp usage metrics?

GitHub's usage metrics API now reports the names of customer configured skills, custom agents and MCP servers instead of grouping them together, and each record can hold 25 entries instead of five. That makes internal tool adoption visible, but the counts still measure activity, not success.

If you're still deciding which agent stack to measure in the first place, the AI agent framework chooser narrows it down in a few questions.

Drop a comment if your setup looks different. Curious how people are joining usage data to outcomes in production.

Google's UGC Fresh Data Program: Freshness Is a Pipeline

Mudassir Khan — Tue, 27 Oct 2026 18:46:57 +0000

Google just documented its UGC Fresh Data Program, and the interesting part is not who gets in. It is the architecture it admits to: for eligible high volume user generated content platforms, Google now runs a dedicated ingestion pipeline that is separate from ordinary crawling and separate from the Indexing API. If your forum, Q&A site or review platform lives on fresh content, that is a signal worth reading closely, even if you never apply.

TL;DR

The UGC Fresh Data Program is a separate ingestion path into Google for eligible UGC platforms. It is not crawling and it is not the Indexing API, and participation does not guarantee Search visibility.
Google recommends sending new UGC within minutes and updating engagement counters within 72 hours. That is a freshness contract, not a recrawl schedule.
Most sites will never qualify, but the lesson transfers: treat every content change as an event, keep content updates apart from counter noise, and make freshness something your system produces on purpose.

What the Google UGC fresh data program is

Think of it as a push lane next to the crawl lane. Normally Google decides when to come back and look at your pages. In this program, an eligible platform sends its fresh content into a dedicated pipeline instead of waiting for the next visit.

The moment new content gets created is where all of this starts. On most stacks that moment is a form submit, which is why I care so much about handling Next.js form submissions with server actions cleanly: if the write path is a mess, nothing downstream can be fresh on purpose.

Two details from Google's own documentation matter more than the headline:

It is separate. Not a faster crawler, not a new mode of the Indexing API. A different pipe.
It is not a ranking ticket. Participation does not guarantee Search visibility. Getting your content in faster and getting it shown are still two different problems.

Why it is not the Indexing API

This is the part I expect people to get wrong first. The instinct is to read "fresh data" and think "great, another way to force URLs into the index." It is not that.

Think of the Indexing API as a notification channel: you tell Google a URL changed. The UGC Fresh Data Program is an ingestion pipeline for a whole class of platforms, built around how fast content and engagement change on them. The difference shows up in the numbers Google attaches to it.

New UGC: within minutes. That is the recommendation for fresh posts.
Engagement counters: within 72 hours. Votes, replies and similar signals get their own, slower clock.
Application response: roughly 6 to 8 weeks. This is a selective program, not a self serve API you wire up on a Friday.

Here is what that split looks like in a type, which is how I'd model it in any codebase that has to reason about it:

type FreshnessLane = "content" | "engagement";

interface ChangeEvent {
  lane: FreshnessLane;
  url: string;
  entityId: string;
  occurredAt: Date;
}

// The target lag per lane, taken from Google's recommendation for the program.
const TARGET_LAG_MS: Record<FreshnessLane, number> = {
  content: 5 * 60 * 1000,          // "within minutes"
  engagement: 72 * 60 * 60 * 1000, // "within 72 hours"
};

export function isLate(event: ChangeEvent, now = new Date()): boolean {
  return now.getTime() - event.occurredAt.getTime() > TARGET_LAG_MS[event.lane];
}

Five minutes is my own reading of "within minutes", not a number from Google. The point is that content and counters run on different clocks, and your system should know which one a change belongs to.

The eligibility bar

Google says applicants should expect roughly 6 to 8 weeks for a response, and independent coverage from Search Engine Journal and Search Engine Land confirms the program has a selective eligibility process. So treat this as something a small number of large UGC platforms will use, not a growth hack.

My opinion, for what it is worth: the platforms that get in will be the ones that can already answer "what changed in the last ten minutes?" from a single table. If that question needs a full database scan or a recrawl of your own site, you are not ready for a push pipeline, whether it points at Google or anywhere else.

Being ready looks like this. Every write produces an event in the same transaction as the content itself:

// app/threads/[id]/actions.ts
"use server";

import { revalidatePath } from "next/cache";
import { db } from "@/lib/db";

export async function addReply(threadId: string, formData: FormData) {
  const body = String(formData.get("body") ?? "").trim();
  if (body.length < 2) return { ok: false, error: "Reply is too short" };

  await db.$transaction(async (tx) => {
    const reply = await tx.reply.create({ data: { threadId, body } });

    // Content lane: the page really changed.
    await tx.thread.update({
      where: { id: threadId },
      data: { contentUpdatedAt: new Date() },
    });
    await tx.changeEvent.create({
      data: { lane: "content", url: `/threads/${threadId}`, entityId: reply.id },
    });
  });

  revalidatePath(`/threads/${threadId}`);
  return { ok: true };
}

Use it from the page with addReply.bind(null, thread.id) as the form action. The outbox row and the reply commit together or not at all, so a fresh page never exists without an event that says so.

Google UGC fresh data program SEO architecture for ordinary sites

You probably do not qualify. You can still steal the architecture.

Content saved. The write path from the previous section.
Change logged. One outbox row per change, tagged with its lane.
Sitemap lastmod updated. Only from the content lane. A vote count changing is not a content update, and pretending it is just teaches crawlers your lastmod is noise.
Fresh hub pages link to it. New threads show up on pages that already get visited often, so discovery does not depend on one file.

The sitemap is where most teams quietly break step 3. Here is the version I'd ship in a Next.js App Router project:

// app/sitemap.ts
import type { MetadataRoute } from "next";
import { db } from "@/lib/db";

export const revalidate = 300;

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const threads = await db.thread.findMany({
    where: { hidden: false },
    select: { id: true, contentUpdatedAt: true },
    orderBy: { contentUpdatedAt: "desc" },
    take: 45000,
  });

  return threads.map((t) => ({
    url: `https://example.com/threads/${t.id}`,
    lastModified: t.contentUpdatedAt,
  }));
}

lastModified reads contentUpdatedAt, never a generic updatedAt that moves every time someone upvotes. That one column is the difference between a lastmod signal and a random timestamp.

The same push versus pull question shows up when you own the index yourself. If you run hybrid search for RAG with keyword and vector retrieval, the outbox above is exactly what keeps both indexes in sync instead of nightly rebuilds.

Google UGC fresh data program SEO checklist

[ ] Every content write emits a change event in the same transaction
[ ] Content changes and engagement counters live on separate lanes
[ ] Sitemap lastModified comes only from real content changes
[ ] You can measure "time to fresh" per lane from the outbox
[ ] New pages get linked from hub pages that are already crawled often
[ ] You apply to the program only if you are a high volume UGC platform, and you plan for a 6 to 8 week wait

FAQ

What changed in google ugc fresh data program seo?

Google documented the UGC Fresh Data Program: a dedicated ingestion pipeline for eligible high volume UGC platforms, separate from both ordinary crawling and the Indexing API. Google recommends sending new UGC within minutes and updating engagement counters within 72 hours. Participation does not guarantee Search visibility.

How should teams adopt google ugc fresh data program seo?

Large UGC platforms can apply and should expect roughly 6 to 8 weeks for a response. Everyone else should adopt the architecture instead: emit a change event for every content write, keep content updates apart from engagement counters, drive sitemap lastmod only from real content changes, and link new pages from frequently crawled hubs.

If you want a freshness pipeline like this wired into your own product, that is the kind of work I do in Next.js builds for AI products.

Drop a comment if your setup looks different. Curious how people are tracking time to fresh in production.

Persistent Agents Need a Workspace, Not a Bigger Prompt

Mudassir Khan — Thu, 22 Oct 2026 18:41:08 +0000

Vercel Drives just reached general availability: persistent storage that survives individual Sandbox runtimes, with drives that default to 1 TiB and can be configured up to 16 TiB. That release is a good excuse to fix the mistake I see in almost every long running agent I review. Teams treat a persistent agent workspace, sandbox memory and chat history as one thing, then wonder why the agent forgets its own files halfway through a job.

TL;DR

Agents juggle three kinds of state: conversation memory, workflow checkpoints, and files. Each one has its own lifecycle, and mixing them up is where long tasks break.
A bigger prompt does not make an agent persistent. A workspace that outlives the runtime does, and that is the gap storage like Vercel Drives fills.
Give each kind of state its own home, write a small manifest that ties them together, and make resume the default path instead of the exception.

If you want the wider map of agent memory types first, I wrote up the four kinds of AI agent memory every architect should know. This post is narrower: what has to survive when the sandbox goes away.

Three kinds of state agents confuse

Here is the split I use on every agent project.

Conversation memory is what the model has been told. The OpenAI Agents SDK keeps session history outside a single model call, which is great, but its in process MemorySession resets when the process exits. That is fine for a chat. It is a footgun for a job that runs for an hour.

Workflow checkpoints are where the agent is in its plan. LangGraph handles this with a checkpointer plus a thread identifier, so a workflow can resume after an interruption, even long after the original process ended.

Files are what the agent has actually produced: the cloned repo, the generated report, the half finished dataset. Neither of the first two stores holds them, because neither was built to.

The bug pattern is always the same. Someone stuffs file contents into the conversation so the agent "remembers" them. The prompt grows, the context fills up, and the next run starts from a fresh sandbox with none of the files on disk anyway.

What survives a run

Ask one question about every piece of state: what happens to it when the runtime dies?

State	Lives in	Survives the process?	Survives the sandbox?
Conversation memory (in process)	RAM	No	No
Conversation memory (persisted session)	Session store	Yes	Yes
Workflow checkpoint	Checkpointer, keyed by thread id	Yes	Yes
Files on the sandbox disk	Ephemeral disk	No	No
Files on a persistent drive	Drive	Yes	Yes

The row that bites people is the fourth one. The agent wrote real work to disk, the checkpoint says "step 7 of 9 done", and the resumed run finds an empty directory. The checkpoint is now lying.

Why files and memory are different

Memory is about meaning. Files are about bytes. You summarise memory, trim it, and pick what goes back into the prompt. You never summarise a git repo or a parquet file; you either have it or you do not.

That difference changes how you design for persistent agent workspace sandbox memory architecture. Memory gets a budget and a retrieval strategy. Files get a path, a size limit and an owner. With drives that can grow up to 16 TiB, the size limit is rarely what stops you. Knowing which run wrote what is.

The cheapest fix I know is a manifest: one small JSON file on the persistent drive that records what each step produced. The checkpoint points at the manifest, and the manifest points at the files. Here is a version you can run with plain Node:

import { mkdir, readFile, writeFile, stat } from "node:fs/promises";
import { join } from "node:path";

type StepRecord = { step: string; files: string[]; finishedAt: string };
type Manifest = { threadId: string; steps: StepRecord[] };

const WORKSPACE = process.env.AGENT_WORKSPACE ?? "./workspace";

async function loadManifest(threadId: string): Promise<Manifest> {
  try {
    const raw = await readFile(join(WORKSPACE, threadId, "manifest.json"), "utf8");
    return JSON.parse(raw) as Manifest;
  } catch {
    return { threadId, steps: [] };
  }
}

async function recordStep(m: Manifest, step: string, files: string[]) {
  for (const f of files) await stat(join(WORKSPACE, m.threadId, f)); // throws if a file is missing
  m.steps.push({ step, files, finishedAt: new Date().toISOString() });
  await mkdir(join(WORKSPACE, m.threadId), { recursive: true });
  await writeFile(join(WORKSPACE, m.threadId, "manifest.json"), JSON.stringify(m, null, 2));
}

export async function nextStep(threadId: string, plan: string[]) {
  const m = await loadManifest(threadId);
  const done = new Set(m.steps.map((s) => s.step));
  return { manifest: m, next: plan.find((s) => !done.has(s)) ?? null };
}

Point AGENT_WORKSPACE at the mounted persistent drive and the same code works on your laptop and in a fresh sandbox. The stat check is the important bit. A step only counts as done when its files really exist, so a checkpoint can never claim work the disk does not have.

A persistence architecture for long tasks

Put the three stores together and you get an architecture that survives a dead sandbox without a bigger prompt.

The rules I follow:

One thread id, everywhere. The checkpoint key, the manifest folder and the session id share the same id. Debugging a stuck run should start with one string.
Files on the drive, never in the prompt. The prompt gets a short listing of what exists and where. The agent reads a file when it needs it.
Workers get their own folders. When the agent forks parallel workers, each one writes under its own path on the shared drive. The coordinator merges at the end. No two workers ever write the same file.
Resume is the default. Every run starts by loading the manifest and asking for the next unfinished step. A fresh job is just a manifest with no steps.

Which framework you pick for the checkpoint layer matters less than following those rules, but it is still a real decision. The AI agent framework chooser walks through the tradeoffs if you are still deciding between a graph runtime and a lighter SDK.

Persistent agent workspace sandbox memory checklist for production

Before you ship a long running agent, check these:

[ ] Conversation memory is persisted to a session store, not only kept in process
[ ] Workflow state uses a checkpointer keyed by a thread id
[ ] Files live on storage that outlives the sandbox (a persistent drive, not the ephemeral disk)
[ ] A manifest records which step produced which files, and verifies they exist
[ ] Parallel workers write to separate folders
[ ] A killed run resumes at the next unfinished step without a human

My own rule of thumb after a few painful rewrites: if I cannot kill the sandbox in the middle of a job and watch the next run pick up cleanly, the agent is not production ready, no matter how good its answers look.

FAQ

What changed in persistent agent workspace sandbox memory?

Vercel Drives reached general availability as persistent storage that survives individual Sandbox runtimes, with drives that default to 1 TiB and can be configured up to 16 TiB. That gives agents a place for files that outlives the runtime, which conversation memory and workflow checkpoints were never designed to hold.

How should teams adopt persistent agent workspace sandbox memory?

Split agent state into three stores: a persisted session for conversation memory, a checkpointer keyed by a thread id for workflow state, and a persistent drive for files. Tie them together with a small manifest per thread, give parallel workers separate folders, and make every run start by resuming from the manifest.

If you are deciding whether a long running agent is worth building at all, the AI agent ROI calculator is a quick way to sanity check the numbers before you invest in the plumbing.

Drop a comment if your setup looks different. Curious how people are splitting memory and files in production.

Next.js 16.4 Is a Migration Release, Not a Version Bump

Mudassir Khan — Tue, 20 Oct 2026 18:47:16 +0000

Next.js 16.4 made Cache Components the recommended path, switched them on by default for new create-next-app projects, and shipped next upgrade --agent so an AI agent can drive the upgrade for you. Bumping the version number takes a minute. Moving an existing app onto the new caching model is a migration, and nobody runs it for you, not even the agent.

TL;DR

Existing apps do not adopt the new caching model on their own. Upgrading the package and adopting Cache Components are two separate jobs, and the second one changes what your pages are allowed to cache.
Cache Components flip the default question from "which routes are dynamic?" to "which components and functions do I explicitly cache?" That is a contract change, not a config change.
next upgrade --agent is useful for the mechanical edits. It cannot decide how fresh your data has to be, so keep that decision with a human.

What 16.4 actually changes

Three things land in this release that matter for a production app:

Cache Components become the recommended path. New projects created with create-next-app get them by default, so the docs, the examples and the next hire's muscle memory all move to this model.
next upgrade --agent exists. The upgrade tool now has an agent oriented mode built for coding agents to run the migration steps.
Builds got faster for at least one real team. An independent production upgrade write up from Roboto Studio reported roughly 5% less shipped JavaScript after moving to 16.4, and a 1,555 page benchmark went from 83.9s total build time on 16.3 to 49.1s on 16.4.

That third point is a reason to measure your own app, not a promise. Your page count, your data layer and your hosting all change the number.

The part that trips people up is in independent migration guidance from iqx.dev: existing applications do not automatically switch to the new caching model, and teams should review caching, navigation and configuration behavior explicitly. So after npm install next@16.4 your app keeps behaving the way it did. A fresh project scaffolded the same afternoon behaves differently.

The cheapest way to feel that difference is to scaffold a throwaway 16.4 app next to your real one and build something tiny in it, like this small Next.js calculator project. Ten minutes in the new defaults teaches you more than an hour of reading release notes, and you get a clean reference to diff against when your real app does something surprising.

Why Cache Components change the migration contract

In the older App Router model you mostly reasoned about caching per route. A route was static or dynamic, and you nudged it with route segment config and fetch options.

With Cache Components you reason per component and per function. Anything you want cached, you mark with the use cache directive and give it a lifetime and tags. Anything that reads request data (cookies, headers, search params) renders at request time and streams in behind a Suspense boundary.

The opt in is one line of config:

// next.config.ts
import type { NextConfig } from "next";

const nextConfig: NextConfig = {
  cacheComponents: true,
};

export default nextConfig;

The real work is what that line forces you to write next. Here is a product page that mixes shared catalog data with a per visitor cart badge:

// app/products/[id]/page.tsx
import { Suspense } from "react";
import { cookies } from "next/headers";
import { cacheLife, cacheTag } from "next/cache";
import { db } from "@/lib/db";

async function getProduct(id: string) {
  "use cache";
  cacheLife("hours");
  cacheTag(`product:${id}`);
  return db.product.findUnique({ where: { id } });
}

async function CartBadge() {
  const cartId = (await cookies()).get("cart_id")?.value;
  const count = cartId ? await db.cartItem.count({ where: { cartId } }) : 0;
  return <span className="badge">{count}</span>;
}

export default async function ProductPage({
  params,
}: {
  params: Promise<{ id: string }>;
}) {
  const { id } = await params;
  const product = await getProduct(id);

  return (
    <main>
      <h1>{product?.name}</h1>
      <p>{product?.description}</p>
      <Suspense fallback={<span className="badge">0</span>}>
        <CartBadge />
      </Suspense>
    </main>
  );
}

Two decisions live in that file that no tool can make for you. cacheLife("hours") says a product description may be up to hours old. The Suspense boundary says the cart badge is per visitor and must never be cached. Get the first one wrong and customers see stale prices. Get the second one wrong and one visitor sees another visitor's cart.

Invalidation follows the tags. When an admin edits a product, a server action expires exactly that entry:

// app/admin/products/actions.ts
"use server";

import { revalidateTag } from "next/cache";
import { db } from "@/lib/db";

export async function updateProductPrice(id: string, price: number) {
  await db.product.update({ where: { id }, data: { price } });
  revalidateTag(`product:${id}`, "max");
}

The exact signatures of cacheLife profiles and revalidateTag have moved between 16.x releases, so check them against the docs for the version you install before you copy this into a real codebase.

My view after reading through the release and the migration notes: treat every use cache you add as a written promise about freshness. If you cannot say out loud how stale a piece of data is allowed to be, you are not ready to cache it.

What next upgrade --agent can and cannot validate

The agent mode is aimed at a real pain: upgrades are full of mechanical edits that are boring for people and easy for a coding agent. Run it like this, on a branch:

git checkout -b upgrade/next-16-4
npx next upgrade --agent
npm run build

In my opinion the line between what you hand the agent and what you keep is simple: the agent can make the build pass, but only you know whether the passing build is correct.

What the agent is good at:

applying codemods and renaming moved imports
reading build errors and fixing the obvious ones
wrapping request time reads in a Suspense boundary when the build demands it

What it cannot know:

how fresh each page has to be for your business (that is the cacheLife value)
which data belongs to one visitor and must never be shared
whether a navigation or prefetch change breaks a flow your users depend on
when it is safe to ship

This is the same split I argue for with any agent that touches production: give it narrow, reviewable permissions and keep the judgment calls with people. Our guide to MCP architecture for enterprise agents goes deeper on drawing those boundaries.

A practical rule: review the agent's diff with one question per changed file. "Did this edit change what gets cached, or for how long?" If yes, a human signs off on that file.

A production migration checklist

Upgrade the version and adopt Cache Components in two separate deploys. Then run the loop below for each route group until the whole app is on the new model.

Step	What you do	Stop the rollout if
1. Upgrade the version	Install 16.4, keep `cacheComponents` off, deploy	Anything changes that you did not expect
2. Build and read errors	Turn `cacheComponents` on in a branch and run `next build`	You cannot explain an error
3. Mark cache boundaries	Add `use cache`, `cacheLife` and `cacheTag` per component, `Suspense` around request data	A page mixes shared and per visitor data with no boundary
4. Compare in staging	Diff responses and headers against production, test navigation and prefetch	Any per visitor data shows up in a cached response

A quick staging check for the scariest failure, a cached response that leaks visitor data:

# Two different carts, same product URL. The badge must differ.
curl -s -H "Cookie: cart_id=cart_a" https://staging.example.com/products/42 | grep -o 'class="badge">[0-9]*'
curl -s -H "Cookie: cart_id=cart_b" https://staging.example.com/products/42 | grep -o 'class="badge">[0-9]*'

If both lines print the same count for carts that hold different items, something per visitor got cached. Stop and find the missing boundary before you go further.

Then measure. Record build time and shipped JavaScript before and after, the same way the Roboto Studio write up did, so you know whether your app sees anything like their numbers.

FAQ

What changed in the Next.js 16.4 Cache Components upgrade?
Cache Components became the recommended path and are on by default for new create-next-app projects, and the release added next upgrade --agent. Existing apps keep their current caching behavior until you opt in.

How should teams adopt the Next.js 16.4 Cache Components upgrade?
Ship the version bump first with Cache Components off. Then enable them on a branch, mark cache boundaries one route group at a time, and compare against production in staging before each rollout.

Will 16.4 make my build faster?
One independent production write up saw a 1,555 page build drop from 83.9s to 49.1s and roughly 5% less JavaScript. Measure your own app before you count on it.

If you want a team to plan and run this migration on a real product, Next.js development for AI products is exactly the kind of work I take on.

Drop a comment with how you are splitting the upgrade: one big switch, or route group by route group? Curious what is working in production.

Human in the Loop for Computer Use Agents

Mudassir Khan — Fri, 04 Sep 2026 19:43:18 +0000

A computer use agent that hits 62 percent on a hard benchmark is not a product you ship unattended. It's a product you gate. The computer use agent human in the loop question is not philosophical, it's an architecture decision you make per action, and the benchmark numbers tell you exactly where the line goes.

The benchmark nobody puts in the pitch deck

You've seen the demo video. Agent opens a browser, fills a form, books the thing, everyone claps. Now here are the numbers that never make the slide.

On BU Bench V1, a set of 100 hard real world browser tasks, open source browser agents score between 35.2 and 62 percent. The best managed cloud agent scored 78 percent. The top model reached 80 percent, and it burned 580.87 dollars in API cost across that single 100 task run. So the ceiling is roughly four out of five, and the ceiling is expensive.

WebArena is bleaker. The strongest GPT-4 based agent there managed 14.41 percent end to end task success. Humans on the same tasks hit 78.24 percent. That is not a rounding gap, that's a different category of worker.

Browser use agent reliability is the number that decides your architecture. Not the demo, not the model card, not the vendor's cherry picked screen recording. If you design assuming 95 percent and reality hands you 50, every downstream assumption you made about error handling, retries, and user trust is wrong at the same time.

The honest read: these agents are good at proposing actions and bad at guaranteeing outcomes. Build for that and you'll ship something that works. Build for the demo and you'll ship something that quietly corrupts data on a Tuesday.

Why failures compound

Failures don't add up, they multiply. Three agents in a chain, each at a 70 percent success rate, gives you 0.7 times 0.7 times 0.7. That's 34 percent end to end.

// The math that kills long agent chains
const perStepSuccess = 0.7;

function chainSuccess(steps: number, perStep: number): number {
  return perStep ** steps;
}

console.log(chainSuccess(1, perStepSuccess)); // 0.70
console.log(chainSuccess(3, perStepSuccess)); // 0.343
console.log(chainSuccess(5, perStepSuccess)); // 0.168
console.log(chainSuccess(8, perStepSuccess)); // 0.058

Run that and stare at the last line for a second. An agent you'd describe in a standup as "usually works" becomes a coin flip you lose 94 percent of the time once you ask it to do eight things in a row.

Repetition makes it worse independently of chain length. Measured performance drops from about 60 percent on a single run to just 25 percent over 8 consecutive runs. Context fills up, earlier mistakes get treated as ground truth, and the agent starts reasoning from its own bad output.

This is why agentic task success rate is such a misleading metric in isolation. A vendor quotes you a per step number because a per step number always looks respectable. The number you actually care about is the one at the end of your chain, and nobody publishes that because it depends on how long your chain is.

Shorter chains. More checkpoints. That's the whole lesson.

Co pilot vs autonomous: the architecture decision you make once

The co pilot vs autonomous agent choice sounds like a product positioning question. It isn't. Co pilot means the agent proposes and a human confirms before anything lands. Autonomous means the agent acts and you find out afterwards, from a log or from a customer.

Most teams pick one mode for the whole product. That's the mistake. The right granularity is the action, not the application.

Action	Reversible?	Blast radius	Mode
Read a page, extract data	Yes	None	Autonomous
Fill a draft form field	Yes	Local	Autonomous
Send an email	No	External	Gate
Submit a payment	No	Financial	Hard gate
Delete a record	Sometimes	Data loss	Gate
Accept terms of service	No	Legal	Hard gate

Same agent, same session, four different trust levels. Reading a dashboard and wiring money are not the same risk just because the same model requested both.

Once you accept that, the mode stops being a config flag and becomes a classifier that runs before every tool call. Which is more work up front and dramatically less work at 3am.

Building the approval gate

An AI agent approval gate is simpler than it sounds. Before any action executes, you classify it. Safe ones run. Risky ones go into a queue a human can approve or reject. The agent blocks on that queue and continues when the verdict lands.

Three signals are worth classifying on:

Reversibility. Can you undo this in one step without begging anyone? Reading is reversible. Sending is not. Anything that leaves your system boundary is effectively permanent.

Blast radius. How many rows, users, or dollars does this touch? Updating one draft is not the same as updating a table. Scale the gate with the number.

Confidence. How sure is the agent? Low confidence plus low blast radius is fine, let it run and check the log. Low confidence plus high blast radius is the exact case the gate exists for.

type Risk = "auto" | "review" | "hard_gate";

interface ActionMeta {
  reversible: boolean;
  blastRadius: "none" | "local" | "external" | "financial";
  confidence: number; // 0 to 1, from the agent
}

const ALWAYS_HUMAN = new Set([
  "auth.submit_credentials",
  "payment.charge",
  "record.hard_delete",
  "content.publish",
  "legal.accept_terms",
]);

export function classify(tool: string, meta: ActionMeta): Risk {
  if (ALWAYS_HUMAN.has(tool)) return "hard_gate";
  if (!meta.reversible) return "review";
  if (meta.blastRadius === "external" || meta.blastRadius === "financial") return "review";
  if (meta.confidence < 0.75 && meta.blastRadius !== "none") return "review";
  return "auto";
}

Notice that ALWAYS_HUMAN is checked first and never consults confidence. That ordering is the point, and I'll come back to why.

Wire the gate into the tool layer, not the prompt. A prompt instruction to "ask before deleting" is a suggestion. A classifier in front of the executor is a rule.

export async function runTool(tool: string, args: unknown, meta: ActionMeta) {
  const risk = classify(tool, meta);

  if (risk === "auto") return execute(tool, args);

  const decision = await requestHumanApproval({ tool, args, risk });
  if (!decision.approved) {
    return { ok: false, reason: "rejected_by_human", note: decision.note };
  }
  return execute(tool, args);
}

Hybrid HITL: over 95 percent success for 15 seconds of your attention

Here's the number that makes all of this worth building. In a hybrid study combining selective human intervention with autonomous browser agents, success rates improved to over 95 percent across all tested scenarios. Average human intervention time was 15 to 30 seconds per intervention.

Sit with that trade. You go from a ceiling around 80 percent, at 580 dollars per 100 tasks, to over 95 percent, in exchange for half a minute of somebody's attention at the moments that matter.

That is a spectacular exchange rate, and it reframes the whole problem. The engineering challenge was never "make the agent fully autonomous". It's "pick the right moments to interrupt".

Interrupt too often and you've built a slower version of doing it yourself. Users start rubber stamping approvals without reading, which is worse than no gate because now you have the latency and the false confidence. Interrupt too rarely and the compounding failure math from earlier eats you.

The tuning knob is the confidence threshold and the blast radius mapping in classify. Log every gated action along with what the human decided, then move the threshold based on real approval rates. If humans approve 99 percent of a given tool's requests without edits, that tool graduates to auto. If they reject or edit often, tighten it.

Your gate should get quieter over time. If it doesn't, you're not reading your own logs.

What always needs a human

Some actions never earn autonomy, no matter how good the model gets. Security boundaries such as password input required 100 percent human participation regardless of agent capability. That finding held across capability levels, which is the interesting part. Getting a better model does not move this line.

Five categories belong in the hard gate permanently:

Credentials and authentication. Passwords, one time codes, session tokens, recovery flows. An agent that can authenticate as you can do everything you can do.
Payments and money movement. Charges, transfers, refunds, subscription changes. Money out is the definition of irreversible.
Irreversible deletion. Hard deletes, dropped tables, emptied trash. Soft delete is reviewable. Hard delete is not.
Publishing anything public. Posts, comments, emails, anything with an audience. You cannot unsend.
Accepting legal terms. Contracts, terms of service, consent flows. An agent clicking "I agree" is a liability question, not an engineering one.

These are not confidence threshold questions. There is no score high enough. Put them in a set, check the set first, and never let a heuristic override it. That's why ALWAYS_HUMAN sits above every other branch in the classifier.

FAQ

How reliable are computer use agents right now?
Open source browser agents score 35.2 to 62 percent on BU Bench V1's 100 hard tasks. The best managed cloud agent hit 78 percent and the top model reached 80 percent at 580.87 dollars per 100 task run. On WebArena the best GPT-4 based agent managed 14.41 percent versus 78.24 percent for humans. Plan for the benchmark, not the demo.

When should a browser agent pause and ask a human?
When the action is irreversible, when it crosses your system boundary, when its blast radius is large, or when agent confidence is low and the blast radius is anything above none. Credentials, payments, hard deletes, publishing and legal acceptance always pause, regardless of confidence.

What is the difference between co pilot and autonomous agent modes?
Co pilot means the agent proposes and a human confirms before execution. Autonomous means the agent executes and you review afterwards. Pick per action rather than per product, since a single session usually contains both safe reads and unsafe writes.

Does adding a human gate cancel out the speed benefit?
Not at the measured intervention cost. Selective intervention pushed success past 95 percent while costing 15 to 30 seconds per intervention. The failure mode to watch is gating so often that reviewers start approving without reading.

Three things to verify right now

Grep your tool definitions for anything that authenticates, charges, deletes, publishes or accepts terms. Every hit belongs in an ALWAYS_HUMAN set today.
Count the steps in your longest agent chain, raise your measured per step success rate to that power, and compare the result to what you tell users.
Check whether your "ask before doing X" rule lives in a prompt or in code. If it's in the prompt, move it into the executor.

If you want a deeper look at how agents get manipulated into taking actions they were never supposed to take, I cover AI agent security and prompt injection in more detail on my site, along with multi agent approval patterns.

If you want this wired up on your own site end to end, that is exactly the kind of work I take on.

Drop a comment if your setup looks different. Curious which actions other people ended up hard gating after getting burned once.

Why LLM Memory in Production Fails Silently

Mudassir Khan — Wed, 02 Sep 2026 19:58:20 +0000

Your agent's memory layer will not throw. It returns three plausible looking chunks, the model answers confidently from them, and nobody notices for a week. That is the real failure mode of LLM memory in production: retrieval quality drifts while every dashboard stays green, so the only defence that actually holds is asserting on what came back before the model ever sees it.

Here is where memory breaks, what the benchmarks say happens at scale, and the verification hooks I wire around retrieval so the failure gets loud.

The silence before the failure

Start with the distinction most teams collapse. Context is what you put in the prompt this turn. Memory is what you can pull back on turn four hundred, in a session that started three weeks ago. Context is a buffer. Memory is a retrieval system, and retrieval systems fail differently from buffers.

A buffer fails visibly. You blow the window, the API returns an error, you see it in logs. A retrieval system returns something no matter what. Ask it for what the user said about their billing preference and it will hand you the nearest neighbours in embedding space. If nothing relevant exists, the nearest neighbours are still returned, just with lower scores that nobody is reading.

The model then does exactly what it is trained to do. It writes a fluent answer grounded in whatever you gave it. There is no exception, no 500, no alert. Your error rate is zero and your answers are wrong.

That is why "why does my AI agent forget things between sessions" is almost never a forgetting problem. The fact is usually sitting in the store. Episodic recall found it during your demo with fifty documents and stopped finding it at fifty thousand, and nothing in the stack was built to notice the difference.

Why vector only retrieval degrades as the corpus grows

The pattern that gets shipped first is always the same: embed everything, store the vectors, fetch the top k by cosine similarity, stuff them in the prompt. It works beautifully in development. It is also the single most common thing I find at the root of a "the agent got dumber" report.

Analysis of production memory architectures points the same way: vector only retrieval approaches degrade as corpus size grows, and the primary cause is the retrieval architecture rather than the model on top of it (FalkorDB). Swapping to a stronger model does nothing here, which is exactly why teams burn weeks on it.

The mechanics are mundane. Similarity is relative, not absolute, so as you add documents the gap between rank one and rank ten compresses until the ordering carries almost no signal. Vector embedding drift compounds it: the store was built with one embedding model, half of it was reindexed with a newer one, and now two chunks about the same fact live in different neighbourhoods. Nothing errors. Precision just leaks.

Temporal reasoning is where it shows up first, because similarity has no opinion about time. "The user cancelled their subscription" and "the user asked about cancelling" embed almost identically. Both come back. The model picks one.

The benchmark reality: LoCoMo says 92.5, BEAM at 10M says 48.6

The published numbers make the scale problem concrete. On the LoCoMo benchmark, the newer Mem0 algorithm scores 92.5 at roughly 6,956 tokens per retrieval call, with sizeable gains over the previous algorithm on both temporal reasoning and questions that chain several facts together (Mem0).

Then the same work measures BEAM, which pushes the corpus toward production size:

Benchmark	Corpus scale	Leading score
LoCoMo	benchmark scale	92.5
BEAM	1M tokens	64.1
BEAM	10M tokens	48.6

Gain over prior algorithm (LoCoMo)	Points
Temporal reasoning	+29.6
Multi hop reasoning	+23.1

Read the BEAM rows again. The leading system loses roughly a quarter of its score going from 1M tokens to 10M, landing at 48.6. That is the best available system, measured deliberately, not somebody's weekend project. Your store is going to cross 10M tokens faster than you think.

So the honest answer to "what is the best way to add memory to an LLM agent" is not a product name. It is: pick a reasonable store, then instrument the retrieval step, because whatever you pick is going to degrade along this curve and you need to see it happening.

Building verification hooks: what to assert after retrieval

A verification hook is a plain function that runs between the store and the prompt and answers one question: does this result set look like a healthy retrieval, or does it look like the store shrugging?

Four assertions catch most of it. Start with the shape of the result:

// verify-retrieval.ts
export interface MemoryHit {
  id: string;
  text: string;
  score: number;       // cosine similarity, 0 to 1
  createdAt: number;   // epoch ms
  sessionId: string;
}

export interface Assertion {
  name: string;
  pass: boolean;
  detail: string;
}

export interface VerifyOptions {
  minScore: number;
  minHits: number;
  maxAgeDays: number;
}

const DEFAULTS: VerifyOptions = { minScore: 0.35, minHits: 1, maxAgeDays: 365 };

export function verifyRetrieval(
  query: string,
  hits: MemoryHit[],
  opts: Partial<VerifyOptions> = {},
): Assertion[] {
  const o = { ...DEFAULTS, ...opts };
  const now = Date.now();
  const maxAgeMs = o.maxAgeDays * 24 * 60 * 60 * 1000;
  const top = hits[0];

  const uniqueSessions = new Set(hits.map((h) => h.sessionId)).size;
  const stale = hits.filter((h) => now - h.createdAt > maxAgeMs).length;
  const spread = hits.length > 1 ? hits[0].score - hits[hits.length - 1].score : 1;

  return [
    {
      name: "non_empty",
      pass: hits.length >= o.minHits,
      detail: `${hits.length} hit(s) for a ${query.length} char query`,
    },
    {
      name: "top_score_above_floor",
      pass: Boolean(top) && top.score >= o.minScore,
      detail: top ? `top score ${top.score.toFixed(3)}` : "no hits",
    },
    {
      name: "score_spread_is_meaningful",
      pass: spread >= 0.05,
      detail: `spread ${spread.toFixed(3)} across ${hits.length} hits`,
    },
    {
      name: "no_stale_dominance",
      pass: stale <= hits.length / 2,
      detail: `${stale} of ${hits.length} hits older than ${o.maxAgeDays} days`,
    },
  ];
}

The one people skip is score_spread_is_meaningful, and it is the one that catches corpus growth. When every hit scores within a hair of every other hit, ranking has stopped ranking. The store is not broken and the scores are not low. They have simply gone flat, which is the compression problem from the previous section showing up as a number you can alert on.

Then wrap the retriever so nothing calls it raw:

// with-verification.ts
import { verifyRetrieval, type MemoryHit, type Assertion } from "./verify-retrieval";

type Retriever = (query: string, k: number) => Promise<MemoryHit[]>;

export interface Incident {
  query: string;
  latencyMs: number;
  hitCount: number;
  failed: Assertion[];
}

export function withVerification(
  retrieve: Retriever,
  onIncident: (i: Incident) => void,
): Retriever {
  return async (query, k) => {
    const started = Date.now();
    const hits = await retrieve(query, k);
    const failed = verifyRetrieval(query, hits).filter((a) => !a.pass);

    if (failed.length > 0) {
      onIncident({
        query,
        latencyMs: Date.now() - started,
        hitCount: hits.length,
        failed,
      });
    }
    return hits;
  };
}

Note what it does not do: it does not block the request. Retrieval quality is a spectrum, and a hook that throws on a soft signal will page you at 3am for a user asking something genuinely novel. Emit the incident, keep serving, and let the rate tell you the story. A steady 2% incident rate is your baseline. The same metric at 15% next month is your corpus growing past what a flat vector index can rank, and now you can see it in a chart instead of a support ticket.

Wire it up once at the boundary:

const memory = withVerification(rawRetriever, (incident) => {
  metrics.increment("memory.retrieval.incident", {
    assertion: incident.failed.map((f) => f.name).join(","),
  });
  logger.warn({ ...incident, queryPreview: incident.query.slice(0, 80) });
});

That answers "how do you verify LLM memory retrieval accuracy" in the only way that survives contact with production. Not a one time eval run. A continuous assertion on live traffic, with the score distribution recorded so you can compare this week against last.

Memory consolidation: 60% less storage, 22% better precision

Once you can see retrieval health, the highest leverage fix is usually not a better index. It is storing less.

Raw conversational memory is enormously redundant. The same preference gets restated in six sessions, each turn is embedded separately, and the store fills with near duplicates that all compete for the same slots in your result set. Consolidation collapses those into single canonical facts. In tested deployments that cut storage by 60% and raised retrieval precision by 22% (Redis).

The precision gain is the interesting half. Fewer near duplicate vectors means the top results stop being six phrasings of one fact, which directly restores the score spread your hook is watching. Consolidation and verification are the same lever pulled from two ends.

A cheap first pass, before you reach for anything clever:

// consolidation-candidates.ts
import type { MemoryHit } from "./verify-retrieval";

export function findDuplicateClusters(
  hits: MemoryHit[],
  threshold = 0.94,
  similarity: (a: MemoryHit, b: MemoryHit) => number,
): MemoryHit[][] {
  const seen = new Set<string>();
  const clusters: MemoryHit[][] = [];

  for (const hit of hits) {
    if (seen.has(hit.id)) continue;
    const cluster = hits.filter(
      (other) => other.id !== hit.id && !seen.has(other.id) && similarity(hit, other) >= threshold,
    );
    if (cluster.length > 0) {
      [hit, ...cluster].forEach((h) => seen.add(h.id));
      clusters.push([hit, ...cluster]);
    }
  }
  return clusters;
}

Run it over a sample of your store and count what comes back. If a meaningful share of your vectors sit in duplicate clusters, you have found your cheapest precision win, and you will pay less for storage on the way.

Three things to verify right now

Log your score distribution, not just your hits. Record top score, bottom score and spread for every retrieval for one day. If the spread is already flat, your ranking stopped working before you noticed.
Ask your store for something it cannot possibly know. A made up name, a fact never mentioned. If it returns four confident looking chunks instead of nothing, you have no floor and every empty query is silently answered.
Count duplicate clusters in a sample. Pull a thousand vectors, cluster them at high similarity, and see how many collapse. That number is your consolidation headroom.

FAQ

What is the best way to add memory to an LLM agent?
Start with the simplest store that fits your access pattern, then instrument the retrieval step before you tune anything. The choice of store matters far less than whether you can see retrieval quality moving. Published benchmarks show every leading system degrading substantially as the corpus grows, so plan for the curve rather than trying to pick your way around it.

Why does my AI agent forget things between sessions?
Usually it did not forget. The fact is in the store and retrieval is no longer surfacing it, because similarity ranking compresses as the corpus grows and older facts lose to newer near duplicates. Check whether the fact is retrievable by direct lookup first. If it is, this is a ranking problem, not a storage problem.

How do you verify LLM memory retrieval accuracy?
Assert on the result set at request time: non empty, top score above a floor, meaningful spread between best and worst hit, and no domination by stale entries. Emit an incident when an assertion fails, keep serving, and watch the incident rate over weeks. Offline evals tell you how your system did on a fixed set. Only live assertions tell you what it is doing now.

If you want a deeper look at how retrieval fits into a system you actually run, I cover production retrieval architecture in more detail on my site.

I also wrote about evaluating LLM memory systems if you want the evaluation side. And if you want this wired up on your own stack end to end, that is exactly the kind of work I take on.

Drop a comment if your setup looks different. Curious what assertions people are actually running on retrieval, and which ones caught something real.

A TypeScript Gatekeeper for AI Agent Tool Calling

Mudassir Khan — Tue, 01 Sep 2026 20:27:57 +0000

Your agent has a deleteRecord tool. Your OAuth token says it's allowed to call the records API. So when the model decides, halfway through a reasonable looking chain of thought, that cleaning up duplicates means deleting forty rows, nothing in your stack objects. Your token was valid. The tool existed. Off it went.

That's the gap this post is about, and the fix is smaller than you'd think: a gatekeeper that sits between the model's request and the actual side effect, and answers a question OAuth never asks.

Why OAuth scope is not enough for agentic tool safety

Scopes are fine at what they do. A scope answers "is this app allowed to touch the calendar API at all", once, at grant time, for the whole session. What it can't answer is "should this specific call, with these specific arguments, in this specific conversation, run right now". Scopes are static and coarse. AI agent tool calling is dynamic and argument dependent, which is a completely different shape of problem.

The failure data backs this up. Across a survey of 27 benchmark papers and 19 benchmarks, tool invocation and parameter level errors came out as the single biggest failure cluster in LLM agents. Not reasoning. Not retrieval. The call itself, and what got passed into it.

Meanwhile the surface area keeps growing. Autonomous agents now outnumber humans in enterprise environments at a ratio of 82 to 1, and only 22 percent of organizations treat AI agents as identity bearing entities with formal access controls. So most of these callers have no identity, no policy attached to them, and full use of whatever credential the process happens to hold.

What's missing is contextual authorization: a decision made per call, with the arguments in hand, that knows who is acting and what the blast radius is. That's the layer we're building.

The three tier approval model

The pattern that holds up in production is a three tier model. Auto approve for reads and searches. Notify for actions that land somewhere visible but stay recoverable. Block or require human approval for anything irreversible or high stakes.

Here's how that splits for a normal SaaS backend:

Tool	Tier	Why
`searchKnowledgeBase`	auto approve	Read only, no side effect, cheap to get wrong
`readDocument`	auto approve	Same, though watch what the doc contains
`sendEmail`	notify	Visible to a human, embarrassing, survivable
`postToSlack`	notify	Same shape, plus a delete button exists
`createRecord`	notify	Additive, easy to reverse
`deleteRecord`	block	Data loss, and nobody notices until later
`issueRefund`	block	Real money leaves the building
`deployToProduction`	block	You know why

Three questions decide where a tool lands. Can you undo it? How far does the damage travel? And would a human notice in time to react?

Sending a wrong email fails all three gracefully. It's reversible enough (you send a correction), the blast radius is one inbox, and somebody replies within the hour. Deleting a record fails all three badly. It's gone, it quietly breaks whatever referenced it, and you find out during a quarterly report.

Notice that createRecord and deleteRecord sit in different tiers despite hitting the same table. That's the whole point. Tier is about consequence, not about which service you're talking to.

Building a TypeScript gatekeeper middleware

Three pieces: a policy table keyed by tool name, a decision function, and a wrapper that every call has to pass through.

Start with the types and the table.

export type Tier = "allow" | "notify" | "deny";

export interface CallerContext {
  actorId: string;
  actorRole: "owner" | "admin" | "member" | "readonly";
  conversationId: string;
}

export interface Decision {
  tier: Tier;
  reason: string;
}

type Rule = (args: Record<string, unknown>, ctx: CallerContext) => Decision;

const POLICY: Record<string, Rule> = {
  searchKnowledgeBase: () => ({ tier: "allow", reason: "read only" }),

  sendEmail: (args, ctx) => {
    const to = String(args.to ?? "");
    if (!to.endsWith("@yourcompany.com")) {
      return { tier: "notify", reason: "external recipient" };
    }
    if (ctx.actorRole === "readonly") {
      return { tier: "deny", reason: "readonly actor cannot send" };
    }
    return { tier: "allow", reason: "internal recipient" };
  },

  deleteRecord: (args, ctx) => {
    const ids = Array.isArray(args.ids) ? args.ids : [args.id];
    if (ctx.actorRole !== "owner") {
      return { tier: "deny", reason: "owner only" };
    }
    if (ids.length > 1) {
      return { tier: "deny", reason: `bulk delete of ${ids.length} rows` };
    }
    return { tier: "notify", reason: "single row delete by owner" };
  },
};

export function decide(
  tool: string,
  args: Record<string, unknown>,
  ctx: CallerContext,
): Decision {
  const rule = POLICY[tool];
  if (!rule) return { tier: "deny", reason: "tool not in allowlist" };
  return rule(args, ctx);
}

Two things worth calling out. The default is deny, so an unknown tool fails closed instead of sliding through. And every rule gets the arguments, not just the tool name, because sendEmail to a colleague and sendEmail to your entire customer list are the same tool and very different events.

That argument inspection is where most of the value lives. Agent RBAC in TypeScript falls out of the same place: CallerContext carries the acting identity down to the rule, so a member and an owner asking for the same delete get different answers.

Now the wrapper. Nothing calls a tool directly anymore.

type ToolFn = (args: Record<string, unknown>) => Promise<unknown>;

export function gatekeep(tool: string, fn: ToolFn, ctx: CallerContext): ToolFn {
  return async (args) => {
    const decision = decide(tool, args, ctx);
    await auditLog({ tool, args, ctx, decision });

    if (decision.tier === "deny") {
      throw new ToolDeniedError(`${tool} blocked: ${decision.reason}`);
    }
    if (decision.tier === "notify") {
      await notifyHumans({ tool, args, ctx, decision });
    }
    return fn(args);
  };
}

When the gatekeeper denies a call, throw an error the model can read. Agents recover surprisingly well from "you may not delete more than one row at a time" and will usually retry with something narrower. A silent null just makes it try again identically.

Where to put the policy: execution layer, not model layer

Here's the part teams get wrong, and it's the difference between a control and a suggestion.

Policy enforcement has to live at the tool execution layer, not inside the agent's reasoning loop. Writing "never delete more than one record" in your system prompt isn't enforcement. It's a request. The model is a suggestion engine, it can be argued out of a rule by a user or by its own chain of thought, and prompt injected content in a retrieved document can push it around too. If you want the same reasoning on how that attack surface works, I wrote about AI agent security patterns separately.

Enforcement is code that runs after the model has decided and before the side effect happens. That's it. If your rule can be talked around in English, it isn't a rule.

MCP tool access control gets easy here, because an MCP server is already the chokepoint. Every tool call funnels through one request handler, so you have exactly one place to wrap.

server.setRequestHandler(CallToolRequestSchema, async (req, extra) => {
  const ctx = contextFromSession(extra.sessionId);
  const { name, arguments: args = {} } = req.params;

  const decision = decide(name, args, ctx);
  await auditLog({ tool: name, args, ctx, decision });

  if (decision.tier === "deny") {
    return {
      isError: true,
      content: [{ type: "text", text: `Blocked by policy: ${decision.reason}` }],
    };
  }
  if (decision.tier === "notify") await notifyHumans({ tool: name, args, ctx, decision });

  return runTool(name, args);
});

One handler, every tool covered, no way for a new tool to ship without inheriting the policy.

Production patterns: logging, alerting, rollback

Log every decision, including the ones you allowed. Tool name, arguments, tier, reason, actor. That log is the only way you'll ever tune the policy, because on day one you're guessing about tiers and by week three the log tells you which guesses were wrong.

async function auditLog(entry: {
  tool: string;
  args: Record<string, unknown>;
  ctx: CallerContext;
  decision: Decision;
}) {
  await db.insert("agent_tool_audit", {
    tool: entry.tool,
    args: redactSecrets(entry.args),
    tier: entry.decision.tier,
    reason: entry.decision.reason,
    actor_id: entry.ctx.actorId,
    conversation_id: entry.ctx.conversationId,
    at: new Date(),
  });
}

Alert on deny spikes rather than on individual denies. A steady trickle is the policy working. A sudden cluster means either a tier is too tight and you're breaking a real workflow, or something is probing you.

Guard the notify tier from becoming wallpaper. If every notification is routine, people stop reading them, and then notify is just allow with extra steps. Keep the tier small enough that a message in it still means something, and route it somewhere with a response expectation.

For rollback, bias the whole system toward reversible writes. Soft deletes, append only ledgers, staged changes that need a second call to commit. When tier two mistakes are cheap to undo, you can afford a broader tier two, and the agent gets more useful without getting more dangerous.

OWASP's Agentic AI Top 10 names Excessive Agency as a critical vulnerability with three root causes: excessive functionality, excessive permissions, and excessive autonomy. Each one maps to a cut you can make today. Excessive functionality means trimming the tool allowlist, because most agents ship with tools nobody uses. Excessive permissions means giving the agent its own scoped credential instead of the service account that can do everything. Excessive autonomy is the tier three list, and it should feel slightly too long rather than slightly too short.

Once that's running, the audit log doubles as your eval set. Replaying real denied and allowed calls is a much better signal than synthetic cases, which is worth pairing with a proper approach to evaluating agent tool calls.

Three things to verify right now

Call a tool that isn't in your policy table and confirm you get a deny, not an execution. Fail closed or you have nothing.
Send the same tool two argument sets, one benign and one destructive, and confirm they get different tiers.
Grep your system prompt for the word "never". Every rule you find there needs a matching rule in code.

FAQ

What is excessive agency in AI agents?

OWASP's Agentic AI Top 10 lists it as a critical vulnerability, and it shows up when an agent can do more than its job requires. Three root causes: excessive functionality (tools it never needed), excessive permissions (a credential broader than its task), and excessive autonomy (no approval step on actions you can't undo). Cut any one and the damage from a bad tool call drops.

How do you control which tools an AI agent can use?

Put a gatekeeper between the model and the tool runtime. It looks up the tool in a policy table, inspects the actual arguments plus the acting identity, and returns allow, notify, or deny. Unknown tools deny by default. Because it runs at the execution layer, the model can't reason its way past it the way it can with a system prompt instruction.

What is the difference between OAuth scope and tool level authorization?

Scope is granted once and answers whether an app may touch an API at all. Tool level authorization runs on every single call and answers whether this call, with these arguments, from this actor, should proceed. You need both. Scope stops the wrong system connecting, and the gatekeeper stops the right system doing the wrong thing with a perfectly valid token.

If you want a deeper look at securing agent tool calls, I cover it in more detail on my site.

If you want this wired up on your own stack end to end, that is exactly the kind of work I take on.

Drop a comment if your setup looks different. Curious what tier boundaries people are actually running in production.

5 LLM Context Window Myths That Are Costing You Production Bugs

Mudassir Khan — Tue, 01 Sep 2026 14:41:32 +0000

The context window is the most misunderstood number in AI engineering. Developers ship LLM features based on what the model card says, hit weird failures in production, and spend hours debugging something that was never going to work the way they assumed.

I've watched this happen repeatedly. The good news: the failure modes are predictable once you understand what the context window actually does versus what the marketing says it does.

Here are five myths worth retiring.

Myth 1: Advertised Context Size Equals Usable Context

"This model supports 128K tokens" sounds like a promise. It isn't.

The advertised number is a ceiling, not a target. Independent testing consistently shows models start losing reliability well before their stated limit. Think of it the way you think about RAM: your laptop technically has 16GB, but run Chrome with 40 tabs and notice what happens at 12GB.

Before your user's first message even arrives, context is already being consumed. The system prompt takes a chunk. Retrieval results take another. Conversation history from previous turns accumulates fast. Tool call results, if you're building agents, can be enormous. By the time the actual question lands, your "128K context model" might have 15K to 25K tokens of usable headroom left.

Token budget estimation is an active task, not a one time calculation. Measure it every time your infrastructure changes.

Practical fix: instrument your system to log actual token usage per request. Calculate how much context your infrastructure (system prompt plus retrieval plus conversation history) consumes before the user query arrives. Design your budget around what's left, not around the headline number.

Myth 2: Bigger Context Window Always Means Better Performance

Intuitively, you'd expect a model with 200K context to outperform one with 32K on tasks that require broad information synthesis. The benchmarks tell a different story.

In testing across 13 LLMs, 11 of them dropped below 50% of their baseline accuracy at just 32K tokens. GPT-4o specifically fell from 99.3% accuracy to 69.7% once the task required genuine reasoning rather than pattern matching against text that happened to be in the prompt.

Here's why. Attention mechanisms distribute a model's capacity across the entire context. A longer context means each token competes with more other tokens for that attention budget. The model doesn't become smarter as you add tokens. In practice, it often becomes less reliable at locating the specific information you need.

Noisy context compounds the problem. Retrieving 20 documents when 5 are actually relevant introduces 15 documents worth of distraction that the model still has to process.

Practical fix: measure answer quality at different context sizes for your specific workload. Most production teams discover that a well filtered 6K to 10K context outperforms a bloated 50K context. Invest in retrieval precision before investing in window size.

Myth 3: The Lost in the Middle Problem Is Solved

You may have seen vendor posts claiming their models have overcome the lost in the middle limitation. Read the benchmarks carefully before you trust that claim.

The lost in the middle problem is a property of how transformers attend to tokens. Content at the very beginning and the very end of the context window gets more attention. Content buried in the middle gets less. This produces a U shaped performance curve: accuracy is higher for facts placed near the edges and lower for facts placed in the center.

Even at 4K tokens (small by current standards) accuracy drops from around 75% to 55 to 60 percent for information sitting in the middle of a document. At longer contexts, the middle section can be effectively invisible to the model.

Models have improved on this. The problem has not been eliminated. Any claim to the contrary should come with benchmark numbers you can inspect yourself.

If your retrieval pipeline returns documents in relevance order and your most critical document lands at position 3 of 5, you are betting the model will find it. Sometimes it doesn't.

Practical fix: reorder retrieval results so the highest relevance documents appear at position 1 and the final slot, not in the middle. For long conversation histories, consider summarizing old turns rather than appending them verbatim.

Myth 4: Filling the Window Is Always Better

More context means more information means better answers. It seems obvious. It's also wrong in practice.

Latency and cost scale with context length. On virtually every major LLM API, you pay per token in as well as per token out. A 100K context prompt costs significantly more than a 10K context prompt, is slower to process, and doesn't automatically produce better answers.

There's a subtler problem: context poisoning. When retrieval returns documents that are topically adjacent but not precisely relevant, you've injected noise. The model now has to reason over that noise to find the actual signal. Sometimes it anchors confidently on the wrong paragraph. This is how hallucination adjacent errors sneak into RAG pipelines that otherwise look correct in unit tests.

A tight, relevant 8K context frequently outperforms a sprawling 80K context for the same query.

Practical fix: treat context length as a variable to optimize, not a setting to maximize. Start with the minimum context that answers the question correctly in testing. Add more only when you can measure that it improves output quality. Establish a cost and latency budget before you scale.

Myth 5: Every Token Costs the Same

You count total tokens to estimate API cost and stay inside the context limit. But different content tokenizes at very different densities, and some content consumes tokens with almost no reasoning benefit.

Numbers are the classic case. The year "2026" splits into per digit tokens in most tokenizers. A formatted price like "USD 1,234,567.89" produces roughly 10 to 12 tokens. A table of 50 rows of numeric data can consume two to three times the token budget of a prose summary that communicates the same information.

Whitespace and verbose structure add up too. Extra blank lines, deeply nested JSON keys, repeated section headers in a long document you're passing as context: all consume tokens that rarely improve the model's reasoning.

And not all token positions carry equal reasoning weight (see Myth 3). A token deep in the middle of a 100K context contributes less to the output than the same token would near the top.

Practical fix: profile your actual prompts with tiktoken before running them in production. Compress aggressively. Convert raw numeric tables to prose summaries where the model needs to reason about values rather than cite them exactly. Strip decorative whitespace. The goal is information per token, not tokens per request.

Putting It All Together

These myths compound each other in production. A team builds on the advertised 128K limit (Myth 1), fills the context with extra retrieval results because more seems better (Myth 4), ignores document ordering because lost in the middle is "solved" (Myth 3), includes raw numeric tables without checking token cost (Myth 5), then wonders why accuracy is erratic at scale.

The thread connecting all five: treat context as a constrained resource to manage actively. Budget it. Measure quality at different sizes. Prioritize retrieval precision over recall. Optimize for information per token, not tokens per request. Put the most important content where attention is strongest.

FAQ

Does a larger context window improve LLM accuracy?

Not automatically. At long contexts, most models show degraded accuracy compared to their baseline. A smaller but well filtered context tends to outperform a large but noisy one for complex reasoning tasks.

What is the lost in the middle problem?

LLMs pay more attention to content at the start and end of the context window. Information positioned in the middle is attended to less reliably, producing a U shaped accuracy curve across context positions. The effect persists even at short contexts and has not been fully solved by current models.

How many tokens can you actually use reliably?

This varies by model and task type. As a working rule: calculate your system prompt and retrieval overhead first, then design around what's left. The reliable reasoning budget is typically 40 to 60 percent of the advertised limit for complex tasks, less if your content has high numeric density or many nested structures.

If you're building AI agents that manage context across multiple turns, these tradeoffs compound fast. I wrote a practical deep dive on AI agent memory management that covers session context design, working memory patterns, and when to summarize versus when to retrieve.

If you've hit a different context window footgun in production, drop it in the comments. I'd love to build a more complete list of the real failure modes.

Your AI Agent Evaluation Harness Is Lying to You

Mudassir Khan — Wed, 26 Aug 2026 20:18:47 +0000

Your AI Agent Evaluation Harness Is Lying to You

Your eval suite is green and your agent is still doing something dumb in production. Both of those things can be true at the same time, and the reason is uncomfortable: AI agent evaluation that only scores the final answer is measuring the wrong thing. An agent can pass every check you have while accessing unauthorized resources, leaking private context, or triggering side effects nobody can undo. The final response looks fine, so the run gets marked successful.

Here is the part I think most teams get wrong. We ship agents to production with roughly the same evaluation rigor we would apply to a staging demo, then act surprised when the demo grade harness does not catch production grade failures. This one bit me. Below is what a harness has to look at instead, and what to start logging if you log nothing today.

Agents pass evals, then fail in production

That is not a contradiction. It is a trajectory problem.

Gartner expects over 40 percent of agentic AI projects to be canceled by the end of 2027, and 32 percent of organizations name quality as the number one deployment barrier. Those numbers are not about models being dumb. They are about teams not being able to tell a good run from a bad one.

Agent failure is usually trajectory level, not output level. A final answer can look completely acceptable while the intermediate steps show wasted cost, unsafe actions, or planning so brittle it only worked by luck. Your scorer never sees any of that, because your scorer only ever sees the last string.

Picture a support agent asked to summarize a customer's order history. It returns a correct summary. Green check. What the trace would have shown you is that it hit an expensive search endpoint eleven times because its first three queries were malformed, then pulled the record from an internal table it was never scoped to read. Correct answer. Terrible run. Your eval suite calls that a pass and moves on, and it will keep calling it a pass every night until a bill or an audit makes it someone's problem.

Why final answer only evals miss intermediate failures

Call it final answer bias. You grade one output string, so you can only ever detect defects that show up in that string.

Three categories slip straight through:

Failure	What actually happened	Why the output looks fine
Unauthorized resource access	Agent queried a datastore outside its scope	The answer it produced was still correct
Private context leakage	Sensitive context ended up in a tool argument or downstream call	Leakage happened on the way, not in the reply
Irreversible side effects	Agent wrote, deleted, or dispatched something it cannot take back	The confirmation message reads perfectly

None of those are detectable from a response string, no matter how good your judge prompt is. That is the whole point. A regression suite built only on final accuracy is not neutral, it is actively reassuring you about the exact class of failure it cannot observe. Green means "the last message looked right." It has never meant "nothing bad happened."

What a real metric framework actually covers

There is a published 12 metric evaluation framework for production agents drawn from over 100 deployments. I am not going to recite the twelve names here, because I would be reconstructing them from memory and getting one wrong helps nobody. What I can describe is the shape any serious llm agent evaluation metrics setup has to have.

Start with task outcome, since that is the one you already have. Did the agent do the thing. Keep it, just stop treating it as the whole score.

Then trajectory quality, which asks whether the path was sane. Two runs can land on identical answers and deserve wildly different grades.

Tool call correctness is the one I would add next if I could only add one. Right tool, right arguments, right order, correct handling when the call fails. Most bad trajectories are just a pile of bad tool calls wearing a trench coat.

After that: cost and token efficiency, because agents fail quietly by being expensive long before they fail loudly. Safety and permissions, which is where the three miss categories above finally become measurable. Latency, which nobody cares about until a reasoning loop goes from four steps to nineteen. And human judgment, because some qualities genuinely do not reduce to an automatic scorer, and pretending otherwise just moves the lie somewhere else.

Categories, not a checklist. Fill them in with metrics you can actually compute against your own system.

Trace based evals: what to capture

A trace is the honest version of a run. Every tool call, every argument passed, every intermediate step, every retry, every token spent, in order.

Once you have traces, a tool call audit becomes possible: replay the run and ask whether each call should have happened at all, whether the arguments were well formed, and whether anything in that call touched a resource outside the agent's scope. That is the audit your final answer scorer can never run, because it does not have the material.

This is also why the strongest harnesses stack four things rather than one. Traces tell you what happened. An eval dataset tells you what should have happened on cases you care about. Production monitoring tells you whether live behavior still matches either of those. Human feedback catches what all three miss. Accuracy alone gives you one number and no way to explain it.

If you log nothing today, here is Monday morning. Wrap your tool layer so every invocation writes a record before and after:

type ToolCallRecord = {
  runId: string;
  step: number;
  tool: string;
  args: unknown;
  ok: boolean;
  error?: string;
  ms: number;
  tokens?: number;
};

export function traced<A extends unknown[], R>(
  name: string,
  fn: (...args: A) => Promise<R>,
  sink: (r: ToolCallRecord) => void,
) {
  let step = 0;
  return async (runId: string, ...args: A): Promise<R> => {
    const started = Date.now();
    const current = ++step;
    try {
      const result = await fn(...args);
      sink({ runId, step: current, tool: name, args, ok: true, ms: Date.now() - started });
      return result;
    } catch (err) {
      sink({
        runId,
        step: current,
        tool: name,
        args,
        ok: false,
        error: err instanceof Error ? err.message : String(err),
        ms: Date.now() - started,
      });
      throw err;
    }
  };
}

That is it. One wrapper, one sink, and suddenly every run has a trajectory you can grade instead of a single string you can only trust.

The eval dataset is the bottleneck, not the harness

Teams spend weeks picking a harness and an afternoon writing the cases. It should be the other way around. Agent datasets almost never capture real production failure modes, which is exactly why a demo grade suite passes everything.

The fix is unglamorous: harvest. Every production run that went sideways, whether a user complained, a retry storm showed up in the logs, or a trace looked wrong on review, becomes a case. Freeze the inputs, record what the trajectory should have looked like, drop it into the regression suite. Do that for a month and you have an eval dataset your competitors cannot copy, because it is made of your own scar tissue.

If you want the fuller version of how these pieces fit together, I wrote up an AI agent evaluation framework with the layering in more detail, and a companion piece on LLM agent evaluation in production.

FAQ

How do you evaluate AI agents in production?

Score the trajectory, not just the reply. Combine traces of every tool call with an eval dataset built from real production failures, live monitoring of the running system, and periodic human review. Task outcome stays in the mix, it just stops being the only signal. The goal is being able to explain why a run passed, not only that it did.

What metrics should an agent eval harness measure?

Cover seven areas rather than one number: task outcome, trajectory quality, tool call correctness, cost and token efficiency, safety and permissions, latency, and human judgment. If you can only add one thing to an existing accuracy check, add tool call correctness. Most bad trajectories are a sequence of bad tool calls, and that metric surfaces them immediately.

Why do agents pass evals but fail in production?

Because evals grade the last message and production grades everything else. An agent can reach a correct answer through an expensive, unsafe, or barely working path, and a final answer scorer has no way to see any of it. Add the fact that eval datasets rarely contain real production failures, and a green suite tells you very little.

Three things to verify right now

Open your eval config and check whether any assertion reads something other than the final output. If not, that is your gap.
Pick yesterday's most expensive agent run and count its tool calls. If you cannot count them, you have no trace.
Look at your last five production incidents. Count how many exist as cases in your regression suite.

If you want a deeper look at layering traces, datasets, and monitoring together, I cover it in more detail on my site.

If you want this wired up on your own site end to end, that is exactly the kind of work I take on.

Drop a comment if your setup looks different. Curious what variations people are running in production.

Enterprise MCP Gateway: OAuth 2.0 and RBAC in Production

Mudassir Khan — Sat, 22 Aug 2026 19:52:10 +0000

Vibe Coding Reality Check: 41% More Bugs, 2.74x Flaws

Mudassir Khan — Tue, 18 Aug 2026 19:32:17 +0000

Vibe Coding Reality Check: 41% More Bugs, 2.74x Flaws

You ship faster with a model in your editor. You also ship more defects, and the gap is wider than most teams assume: one study put the bug rate increase at 41% after teams adopted AI coding tools, and security flaws turn up roughly 2.74x more often in AI assisted code.

That is not an argument for turning the tools off. Around 95% of code reaching production now contains AI assisted content, so the real question stopped being whether you use them. It became how you stop quality from quietly draining out while you enjoy the speed.

Here is what the data actually says, why it happens, and the five practices that hold the line without putting you back on the slow path.

The vibe coding moment: what actually happened

Vibe coding is the habit of describing what you want, accepting whatever the model writes, and judging the result by whether it runs. Prompt, run, prompt again. No line by line read of the diff.

It feels incredible. A feature that used to eat an afternoon lands in twenty minutes. Cursor, Copilot and Claude Code all got good enough at the same time that the friction of writing code dropped below the friction of reviewing it, and that inversion is the whole story.

Because when writing gets cheap and reviewing stays expensive, people write more and review less. Not out of laziness. Out of arithmetic. A tool that produces 300 lines in one shot does not produce 300 lines of review attention alongside it, and nobody budgeted for the difference.

The Uplevel study: 41% bug rate increase after AI adoption

The number that keeps getting quoted is Uplevel's: teams that adopted AI coding tools saw a 41% increase in bug rates. The mechanism identified was over reliance, developers taking suggestions without enough review behind them.

CodeRabbit's research points the same direction from a different angle. AI assisted codebases carried 1.7x more issues per pull request than human only code. Not per line. Per pull request, which is the unit your team actually reviews, so the extra load lands squarely on whoever is doing the reading.

Sit with that second number for a second, because it explains the first one. If every PR now carries almost twice the issues, and your review process did not change, your review process is now catching a smaller fraction of what is there. The bugs are not appearing out of nowhere. They are walking through a gate that was sized for a different volume.

What makes this hard to notice is that the failures are boring. Not exotic model hallucinations. An error path that swallows the exception. A null check that reads correctly and is placed one branch too late. Code that passes review because it looks like code you would have written, which is exactly what these models are optimised to produce.

Security vulnerabilities: 2.74x more flaws in AI assisted codebases

Static analysis across enterprise repositories found security vulnerabilities roughly 2.74x more prevalent in AI assisted code than in manually written code. That multiplier is the one worth taking to your lead, because security defects have a cost curve nothing else in this list matches.

Why security specifically? A model writes what is statistically typical for the surrounding context. Typical code on the public internet includes a lot of tutorial grade shortcuts: string concatenation into queries, permissive CORS, secrets read straight from a literal, validation that trusts the shape of an object because the type annotation said so. None of it looks wrong. All of it is a footgun in a real deployment.

The model also has no idea where your trust boundary sits. It cannot know that this particular handler is reachable without auth, or that this input crossed the network two frames ago. That context lives in your head and in your architecture, and it is precisely the context that decides whether a piece of code is fine or a hole.

Type systems do not save you here either. Type safe code can be perfectly type safe and still authorise the wrong user.

Why developers still trust AI despite the data (and when that trust breaks)

Ask a room of engineers whether AI tools make them faster and almost every hand goes up. Ask whether the code is better and the hands drop. Both answers are honest, and they are not in conflict.

The trust holds because the failure mode is delayed. You feel the speed instantly, in the same session. You feel the defect three weeks later, in an incident channel, usually attributed to something else entirely. Nothing in that loop connects the two events, so the feedback that would calibrate your trust never arrives.

Trust breaks in exactly one situation: the first time someone traces a production incident back to a block of code nobody on the team can explain. Not because it is complicated, but because no human ever really read it. That moment lands differently than any statistic, and it is the moment most teams finally add gates.

You do not need to wait for it.

Five practices that preserve quality without killing speed

None of these ask you to write less with AI. They move the cost from your future incident channel to your current pipeline, where it is cheaper.

1. Read the diff like it came from a stranger. Not the prompt. Not the explanation the model gave you. The diff. If you would send it back when a contractor you had never met submitted it, send it back now. The strongest version of this rule is a personal one: never merge code you could not defend in an incident review.

2. Make static analysis blocking, not advisory. A SAST tool that posts a comment gets ignored inside a week. One that fails the build gets fixed. Given a 2.74x security multiplier, this is the single highest leverage change on the list.

# .github/workflows/security.yml
name: security
on: pull_request

jobs:
  sast:
    runs-on: ubuntu-latest
    container:
      image: semgrep/semgrep
    steps:
      - uses: actions/checkout@v4
      # --error makes findings exit nonzero, which fails the check
      - run: semgrep ci --config auto --error

3. Cap the blast radius per pull request. Review quality falls off a cliff past a few hundred changed lines, and AI makes large diffs trivially easy to produce. Put a real limit in front of yourself:

#!/usr/bin/env bash
# .git/hooks/pre-push  (chmod +x this file)
# Refuse to push a branch that has grown past a reviewable size.
set -euo pipefail

BASE="${BASE_BRANCH:-origin/main}"
LIMIT="${DIFF_LIMIT:-400}"

changed=$(git diff --numstat "$BASE"...HEAD \
  | awk '{ added += $1; removed += $2 } END { print added + removed + 0 }')

if [ "$changed" -gt "$LIMIT" ]; then
  echo "Branch changes ${changed} lines, limit is ${LIMIT}."
  echo "Split it, or push with DIFF_LIMIT=$((changed + 1)) if you have a reason."
  exit 1
fi

4. Test the boundaries the model cannot see. Auth, authorisation, input validation, error paths, resource cleanup. A model writes the happy path beautifully because the happy path is what most public code demonstrates. Write those tests yourself, or at minimum write the test names yourself so the shape of the contract comes from you.

5. Track one quality signal per pull request. Issues found in review, escaped defects, whatever your team already counts. You cannot manage a 41% drift you never measured, and the whole danger of this failure mode is that it is invisible month to month. A single number, tracked over eight weeks, tells you more than any benchmark someone else published.

The pattern underneath all five: the model generates, and a gate that does not get tired verifies. Human attention is the scarce resource now, so spend it on the parts machines cannot check.

Three things to verify right now

Run semgrep ci --config auto against your main branch and count the findings. That is your current baseline, whether you knew it or not.
Check whether your SAST job is set to fail the build or just comment. If it comments, flip it.
Pull the last ten merged pull requests and check the diff size with git diff --numstat. If the median is over 400 lines, your review process is already running past its limit.

FAQ

Does AI generated code introduce more bugs?
The available data says yes. One study measured a 41% increase in bug rates after teams adopted AI coding tools, and separate research found 1.7x more issues per pull request in AI assisted codebases. The cause identified in both cases is reduced review depth rather than the model producing nonsense.

What is vibe coding?
Describing what you want to a model, accepting the generated code, and validating it by running it rather than reading it. It works well for prototypes and throwaway scripts. It degrades badly once the code has users, because "it runs" and "it is correct" stop being the same claim.

How do you maintain code quality when using AI tools?
Move verification into automation and keep human attention on judgment. Blocking static analysis, small reviewable diffs, tests you wrote for the boundaries the model cannot see, and one tracked quality metric will cover most of the gap. The tools are not the problem. An unchanged review process running at several times its designed volume is.

If you want a deeper look at how these gates fit into an AI system you actually run in production, I cover it in more detail on my site.

If you want this wired up on your own codebase end to end, that is exactly the kind of work I take on.

Drop a comment if your setup looks different. Curious what gates people are actually running, and which ones survived contact with a deadline.

Zod vs Valibot in 2026: Bundle Size and Speed, Compared

Mudassir Khan — Sat, 15 Aug 2026 19:32:25 +0000

# Zod vs Valibot in 2026: Bundle Size and Speed, Compared

For about two years the pitch for Valibot was one line long: it's the tiny one. That was enough. If you were shipping schemas to the browser, the weight difference was impossible to ignore, and plenty of teams migrated on that argument alone.

Then Zod v4 landed with a 14x string parsing speed gain and a stripped down build called Zod Mini, and the tiny one argument suddenly needed a rematch. So here are the actual numbers, where each library still wins, and which one I'd reach for on a new project today.

Why this choice costs you twice

Validation is one of the few pieces of code that gets taxed on both sides of the wire.

On the client it ships. Every user who loads your page downloads your schema library before they can type into a form, and it sits in the same budget you fight over when you're arguing about whether to lazy load a chart component.

On the server it runs hot. Every request that crosses your API boundary gets parsed. Every webhook payload, every query param, every row you pull back from an untrusted source. If your validator is slow, that slowness compounds at exactly the point where you have the least headroom.

Most library choices only hit one of those. A charting library is bundle weight and nothing else. A queue worker is runtime cost and nothing else. Schema validation is both, which is why the argument gets so loud for a dependency that most people write maybe forty lines against.

Bundle size: the three numbers that matter

Here's the comparison people keep getting wrong, because there are three builds in play and not two.

Build	Minified + gzipped	Why
Valibot (core)	~1.37 kB	Modular functions, pruned per import
Zod Mini	~6.88 kB	Tree shakeable subpackage added in v4
Zod Standard	~17.7 kB	Full chainable API, one large surface

Zod Standard is heavy for a reason that isn't sloppiness. The chainable API means z.string() returns an object carrying every refinement method you might call next, so your bundler can't confidently prove that .datetime() is dead code when you never touch it. One big connected surface prunes badly.

Valibot goes the other way. Everything is a standalone function you import by name, so the bundler does what bundlers are good at: it drops what you never referenced. Import three validators, ship three validators.

The mistake I see constantly in 2026 is people quoting the 17.7 kB figure at Valibot. That comparison was fair in the Zod v3 era. It isn't now. If bundle size is your reason for looking at Valibot, the honest matchup is 1.37 kB against 6.88 kB, and that's a much narrower gap than the one that started the migrations.

Zod v4: what actually changed

Two things, and they happen to be the exact two complaints that pushed people out.

Speed came first. String parsing got roughly 14x faster, which is the kind of jump you notice in a load test rather than in a demo. Then Zod Mini arrived as a separate entry point built for tree shaking, which finally gave Zod an answer to the weight question instead of a shrug.

// Zod Standard: chainable, reads well, prunes badly
import { z } from "zod";

const User = z.object({
  id: z.string(),
  age: z.number().int().min(18),
});

const result = User.safeParse(input);

// Zod Mini: same engine, function-first entry point, tree shakeable
import * as z from "zod/mini";

const User = z.object({
  id: z.string(),
  age: z.number(),
});

const result = z.safeParse(User, input);

Be honest about where the speed actually lands, though. A signup form validating one object on submit will never notice 14x. What notices is the API gateway parsing thousands of payloads a minute, or the ingestion job running a schema across a large array of rows. If validation isn't on your hot path, treat the speed number as nice rather than as a reason to migrate.

API ergonomics: where each one wins

Zod wins on familiarity, and familiarity is worth more than developers like to admit. The chained style reads like a sentence, your editor autocompletes you through it, and roughly every tutorial, every form library adapter, and every Stack Overflow answer written since 2022 assumes you're holding a Zod schema. That ecosystem gravity is the real product.

import * as v from "valibot";

const User = v.object({
  email: v.pipe(v.string(), v.email()),
  age: v.pipe(v.number(), v.integer(), v.minValue(18)),
});

const result = v.safeParse(User, input);

Valibot wins on honesty. That import line at the top is a literal bill of materials for what you're about to ship, and after a week the pipe style stops feeling like ceremony and starts feeling like composition. You can pull a validator out into a named constant and reuse it across schemas without wrapping anything.

It has a real learning curve, though, and I'd rather say that plainly than pretend otherwise. Everyone arriving from Zod finds it verbose at first, error message customisation takes a minute to locate, and if a teammate is debugging a schema at 2am they will be slower in the style they've never written. That cost is temporary but it's not zero, and on a team of eight it's eight times whatever it is.

When to pick which

Situation	Pick	Why
Server only code, no client bundle	Zod Standard	Weight is irrelevant, ergonomics and ecosystem win
Existing Zod codebase	Zod Mini	Same engine, no rewrite, most of the size back
Greenfield, client heavy, tight budget	Valibot	Smallest footprint, and no migration cost to pay
Publishing a library	Valibot	Your kilobytes become your users' kilobytes
Team optimising for velocity	Zod	Ecosystem answers questions before you ask them

The one I'd push back on hardest is migrating a large existing Zod codebase purely for kilobytes. Switching a few hundred schemas costs you review time, a window of subtle behaviour differences in error shapes, and every integration that expects a Zod schema. Moving from Zod Standard to Zod Mini gets you most of the size back for a fraction of that risk. Do the cheap thing first, measure, and only then decide whether the last few kilobytes are worth a rewrite.

FAQ

Is Valibot better than Zod?

Not universally. Valibot is smaller and prunes better, so it wins where bundle weight is the constraint. Zod wins on ecosystem, on how many libraries already speak its schemas, and on how quickly a new teammate becomes productive. Pick against your actual constraint rather than against a benchmark someone posted.

What changed in Zod v4?

String parsing got roughly 14x faster, and a new Zod Mini subpackage arrived that's built for tree shaking. Together those addressed the two reasons people were leaving: it was slow on hot paths and it was heavy in the browser. If you evaluated Zod before v4 and walked away, your evaluation is out of date.

Which TypeScript validation library has the smallest bundle?

Valibot, at roughly 1.37 kB minified and gzipped for the core package. Zod Mini sits at about 6.88 kB and Zod Standard at about 17.7 kB. Just make sure you compare against the build you'd actually ship, because quoting the Standard figure against Valibot overstates the gap by a lot.

The verdict

For most teams in 2026 the answer is Zod Mini. You keep the ecosystem, you keep the muscle memory, and you give up a handful of kilobytes to Valibot rather than the ten plus you were giving up before v4.

Reach for Valibot when you're starting fresh and the client bundle is genuinely the thing you're optimising. The old size argument still points at Valibot, it just doesn't win the whole debate by itself anymore.

If you want a deeper look at how validation fits into a production TypeScript stack, I cover it in more detail on my site.

If you want this wired up on your own site end to end, that is exactly the kind of work I take on.

Drop a comment if your setup looks different. Curious what people are actually running for schema validation in 2026, and whether anyone has migrated back.