My AI gate tests were green theater. The fix was to stub the wire — and nothing above it.

In a private multi-agent project, agent proposals go through a human approval gate. The Playwright tests for that gate were all green, and had been for a while.

They were green because they never once ran the thing they claimed to test.

Green theater

The gate tests staged model output straight into application state: write a pending proposal card, click approve, assert the card flips to approved. Clean, fast, deterministic — and a lie by construction. Those tests proved the consumer of a pending proposal works. They never exercised the path that produces one.

So when a refactor changed the producer to write proposals pre-approved — quietly dropping the approval guard on the way — the whole suite stayed green. The staged state still looked exactly like the state the tests expected, because the tests were the ones staging it.

That's the failure mode I now call green theater: a suite that manufactures the evidence it then inspects.

Stub the bytes, keep the SDK real

The honest fix was to move the fake down to the lowest layer that can hold it: the wire. That's what I extracted into llm-wire-stub — a scripted Anthropic Messages API at Playwright's network boundary.

The app under test runs unmodified: a real @anthropic-ai/sdk client, a real MessageStream, a real SSE decode, a real tool loop. The stub intercepts api.anthropic.com with context.route and answers with the documented streaming envelope:

message_start → (content_block_start → …delta… → content_block_stop)* → message_delta → message_stop

One property matters more than any other here: a wrong shape must fail loudly. A stub that emits a sloppy envelope which the app half-tolerates is just green theater one layer down. The SDK's own accumulator is the enforcer — it throws on the out-of-order stream if message_start is missing (the current SDK's message reads "Unexpected event order"), and finalMessage() rejects if message_stop never arrives. That claim is demonstrated, not asserted: tests/envelope.test.ts feeds the stub's bytes through the real SDK and shows exact reconstruction, then feeds it deliberately broken streams and shows the SDK throw.

Request bodies are evidence

The part of this that changed how I test: the stub records what the app sent, not just what it was shown. Every intercepted call becomes a RecordedRequest — model, system prompt, messages (including tool_result blocks), tool names, api key.

The response side of a test says "the app can render what it was given." The request side says "the app asked the right question." Two bugs from the private suite's history made me care — both invisible to any response-side assertion:

• The price lie. A UI chip displayed a cost figure that disagreed with what was actually going over the wire. Every response-side test passed, because the responses were fine; the lie was in the outbound traffic nobody was reading.
• The context leak. One agent's output was supposed to reach the next agent's prompt, and silently didn't. A node that never saw upstream output is provable in one assertion — a missing message in requests[n].messages — and in no other way I know of that doesn't involve staring at logs.

Those two anecdotes are private history; you can't reproduce them from the public repo. What you can check is the mechanism: the e2e spec a tool turn makes the SDK loop take a second request asserts on the second request body and shows the tool_result the app produced riding back up the wire.

Fixture expressiveness IS coverage

This one cost a release, so it gets its own section.

The private suite had a test asserting that the number of model requests matched the price quoted to the user — call it "requests == quoted." Correct assertion, sound idea. It passed for a full release cycle while the metering was wrong.

Why? The stub at the time could only produce plain text turns. It could not script a tool_use block, so the SDK's tool loop never fired, so no test run ever took a second request. Both sides of "requests == quoted" were trivially 1. The assertion was right and the fixture made it vacuous.

The lesson: your assertions can only be as strong as what your fixture can express. A fixture that cannot produce a second turn silently converts every multi-turn assertion into a tautology — no failure, no warning, nothing to review.

So ScriptedTurn grew a tool field that streams input_json_delta fragments the way the real API does, and — same lesson, other direction — an error field. A stub that can only succeed makes every consumer's error path green theater by omission. An error turn answers with the documented Anthropic error JSON, and the real SDK surfaces it as a catchable RateLimitError, exactly as in production:

const stub = await stubAnthropic(context, [
  { error: { status: 429, type: "rate_limit_error", message: "Rate limited." } },
]);
// …drive the UI; assert the app shows its rate-limit state, not a crash…

(One caveat that costs an afternoon if you skip it: the real SDK retries 429s and
5xxs by default, so an error-turn script either scripts the retries too or runs
the client with maxRetries: 0. Both in-repo demonstrations do the latter.)

Quickstart

npm install -D llm-wire-stub

The registry tarball ships dist/ prebuilt, so it needs no install scripts and no flags — which matters as of npm v12, where git dependencies and lifecycle scripts are off by default. (A github:egnaro9/llm-wire-stub install also works via the prepare hook, but on npm 12 it needs the new --allow-git and script allowances — the registry install is the clean path.)

import { test, expect } from "@playwright/test";
import { stubAnthropic } from "llm-wire-stub";

test("the agent answers from the scripted wire", async ({ context, page }) => {
  const stub = await stubAnthropic(context, [
    { text: "First scripted answer." },
    {
      text: "Filing a card now.",
      tool: { name: "create_card", input: { title: "prove the loop" } },
    },
    { text: "Card filed. Done." },
  ]);

  await page.goto("/");                       // your app, unmodified
  // …drive the UI; the app's real SDK client hits the stub…

  expect(stub.requests).toHaveLength(3);      // what the app actually sent
  expect(stub.overflow).toBe(0);              // no unscripted model calls
});

overflow counts requests that ran past the end of the script — an unexpected extra model call shows up in an assertion instead of hiding.

hold() / release(): concurrency observed, not assumed

const stub = await stubAnthropic(context, (req) =>
  JSON.stringify(req.messages).includes("alpha")
    ? { text: "for alpha" }
    : { text: "for beta" }
);

stub.hold();                 // responses now block
// …trigger two sends in the UI…
await expect.poll(() => stub.requests.length).toBe(2);  // both IN FLIGHT
stub.release();              // both complete

If the producer serialized its calls, the second request could never reach the wire while the first is still pending — so two recorded requests under hold is proof of concurrency, not a timing accident. Note the function-form script: an array keys answers to arrival order, which is a lottery under concurrency; a function keys them to who asked.

Limits

• Anthropic envelope only. One provider, tested end to end, over multi-provider support with one tested path. No OpenAI or Gemini framing.
• Playwright-oriented. stubAnthropic wants a Playwright BrowserContext. (sseBody and errorBody are framework-free; the vitest suite uses them with a plain custom fetch.)
• Messages API v1 streaming only. No batch API, no extended thinking, no citations, no server tool use. Success turns are text and/or one tool_use block; failures are the error variant. Nothing else is expressible, on purpose.
• Scripted, not simulated. The stub never invents behavior; if your script runs out, the overflow counter says so loudly.

The repo

github.com/egnaro9/llm-wire-stub — MIT, 9 vitest tests (envelope through the real SDK, including the fails-loudly demonstrations) and 10 Playwright tests (a browser fixture driving the real SDK's tool loop against the stubbed wire). The private-product bugs above are provenance; everything the stub is claimed to do is demonstrated by a test you can run.