My AI agent said "Forecast for undefined" and no test caught it, so I built a chaos testing tool

Saeed Allam — Sat, 07 Nov 2026 19:55:07 +0000

Every AI agent demo looks great. The tools answer instantly, the data is clean, and the agent writes a confident, correct reply.

Then you ship it, and the tools start doing what tools do in production: they time out, they rate limit, and sometimes they answer 200 OK with an empty body.

I wanted to know what my agent does in those moments. I couldn't find a simple way to test it in TypeScript, so I built one: agentQuake, an open source library that makes your agent's tools fail on purpose and tells you how the agent handled it.

The failure that scared me

Loud failures are mostly fine. If a tool throws a 503, the AI SDK passes the error to the model, and most models will say "sorry, I couldn't get that".

The quiet ones are the problem. Here's a Vercel AI SDK tool that passes the API's answer straight through:

getForecast: tool({
  description: 'Daily forecast for a city',
  inputSchema,
  execute: ({ city, days }) => getForecast(city, days),
}),

If the API returns {} with status 200, nothing throws. The model gets an empty object, and in my example the agent happily replies:

Forecast for undefined: no data.

No exception, no error log, every test green. And this isn't just my paranoia: a 2026 research paper on agent robustness (AgentChaos, ASE 2026) found that empty or truncated responses hurt agents about as much as crashes, and are much harder to notice after the fact.

Testing it like any other unit test

agentQuake runs inside Vitest or Jest. No server, no database:

npm i -D agentquake

import { quake } from 'agentquake';

it('survives a flaky weather API', async () => {
  const run = quake({ preset: 'silent', seed: 42 });

  await run.agent(() =>
    weatherAgent('What is the weather in Cairo?', {
      getWeather: run.tool('getWeather', getWeather),
    }),
  );

  expect(run.report().falseSuccess).toBe(false);
});

run.tool() wraps your tool function. Depending on the preset or scenario, a call might be slowed down, fail with a 503 or 429, time out, or come back as a 200 with an empty or cut-off body.

run.agent() wraps the whole run, so agentQuake sees both sides: what the tools did and what the agent finally said. That's what makes these two checks possible:

falseSuccess: a tool failed, nothing recovered after it, and the answer still reads like a success. A retry that works, or a fallback to another tool, counts as recovering.
leakedGarbage: a tool's answer was corrupted and undefined, null or NaN ended up in the reply.

If a tool really matters, mark it. Then an agent that lost its booking, called a few other tools and still said "Booked!" gets caught:

run.tool('bookFlight', bookFlight, { critical: true });

Same seed, same faults

Agents are already non-deterministic, so the chaos shouldn't be. Whether a fault fires depends only on the seed, the call number and the fault id. A failing test prints its seed, and you can replay exactly the same faults while you fix it.

What fixed my agent

The careful version of the example agent checks the tool's answer with zod before the model ever sees it, retries once, and otherwise tells the model the tool failed:

const forecastSchema = z.object({
  city: z.string().min(1),
  days: z.array(z.object({ day: z.number(), tempC: z.number() })).min(1),
});

async function checkedForecast(getForecast, city, days) {
  for (let attempt = 1; attempt <= 2; attempt += 1) {
    try {
      const forecast = forecastSchema.parse(await getForecast(city, days));
      if (forecast.city === city && forecast.days.length === days) {
        return forecast;
      }
    } catch {
      // try once more
    }
  }
  throw new Error('The forecast service gave an unusable answer');
}

The lesson surprised me: the fix wasn't a smarter model or a better prompt. It was treating tool output as untrusted input, the same way we treat user input.

Not only for functions

If your tools are HTTP APIs or MCP servers, the same package has a command line:

npx agentquake proxy --preset flaky --tool weather=https://api.weather.com
npx agentquake stdio --preset silent -- node weather-server.js   # MCP over stdio
npx agentquake report

Try it

The repo has a Vercel AI SDK example with a naive and a careful agent. It runs without an API key:

git clone https://github.com/saeedallam/agentquake
cd agentquake
npm ci && npm run example

It's early (0.4, Apache-2.0). The default false-success check looks for English failure words, so pass your own check for other languages. Scoring across many runs is next.

I'd really like to hear what failures you've seen agents mishandle in production. Those are the faults I want to add next.

saeedallam / agentQuake

Chaos testing for AI agents

Chaos testing for AI agents.
Break your agent before production does.

Quick start · Command line · Scenarios · Team server · Roadmap · Design decisions

agentQuake makes your agent's tools fail on purpose (slow answers, errors timeouts) and records what the agent does about it, so you can see whether it copes, retries, loops or claims a success that never happened.

The weather-agent example: npm ci && npm run example. No API key needed.

Quick start

Start inside your tests, with no server and no database:

npm i -D agentquake

import { quake } from 'agentquake';
it('survives a flaky weather API', async () => {
  const run = quake({ preset: 'flaky', seed: 42 });

  await run.agent(() =>
    weatherAgent('What is the weather in Cairo?', {
      getWeather: run

…

View on GitHub

DEV Community: Saeed Allam