DEV Community

Cover image for Testing AI Agents in TypeScript Without Calling the Model
Gabriel Anhaia
Gabriel Anhaia

Posted on

Testing AI Agents in TypeScript Without Calling the Model

#typescript #testing #agents #vitest

Ask someone how they test their agent and the answer is usually "we run it and
see." Which is fair, because the model is nondeterministic, but it quietly
concedes that the loop around the model is untested too, and that is where the
expensive bugs live.

The runaway loop, the missing tool_result, the budget check that runs after
the spend, the retry that re-sends a completed side effect: none of those are
model behaviour. They are your control flow, and control flow is testable.

The seam

You need exactly one: the thing that produces a model response.

export interface ModelClient {
  create(params: MessageCreateParams): Promise<Message>;
}

export async function runAgent(
  task: string,
  ctx: Ctx,
  limits: Limits,
  client: ModelClient = realClient,
) { /* ... */ }
Enter fullscreen mode Exit fullscreen mode

A default parameter keeps production call sites unchanged and makes every test
able to substitute. That is the entire refactor.

A fake that plays a script 

type Scripted =
  | { text: string }
  | { tools: { name: string; input: unknown }[] };

export function scriptedClient(script: Scripted[]) {
  let i = 0;
  const calls: MessageCreateParams[] = [];

  return {
    calls,
    get turns() { return i; },
    async create(params: MessageCreateParams): Promise<Message> {
      calls.push(params);
      const step = script[Math.min(i, script.length - 1)];
      i++;
      return "text" in step
        ? message([{ type: "text", text: step.text }], "end_turn")
        : message(
            step.tools.map((t, n) => ({
              type: "tool_use" as const,
              id: `tu_${i}_${n}`,
              name: t.name,
              input: t.input,
            })),
            "tool_use",
          );
    },
  };
}

const message = (content: ContentBlock[], stop_reason: string): Message => ({
  id: "msg_test", type: "message", role: "assistant", model: "test",
  content, stop_reason, stop_sequence: null,
  usage: { input_tokens: 100, output_tokens: 50 },
} as Message);
Enter fullscreen mode Exit fullscreen mode

Clamping to the last step (Math.min) is what makes infinite-loop tests
possible: a one-element script of a tool call repeats forever, exactly like a
stuck model.

Recording calls is what makes assertions about your behaviour possible —
what you sent, in what order, with what history.

The tests that earn their keep

The runaway loop. This is the failure that produces the incident reports,
and it takes four lines to cover.

it("stops at the turn cap when the model never finishes", async () => {
  const client = scriptedClient([{ tools: [{ name: "search", input: { q: "x" } }] }]);
  const out = await runAgent("go", ctx, { maxTurns: 5, maxCostUsd: 99 }, client);

  expect(client.turns).toBe(5);
  expect(out.status).toBe("turn_limit");
});
Enter fullscreen mode Exit fullscreen mode

The budget check happens before the spend. Ordering bugs here are
invisible in production until the bill.

it("does not call the model when the budget is already exhausted", async () => {
  const client = scriptedClient([{ text: "hi" }]);
  const out = await runAgent("go", ctx,
    { maxTurns: 10, maxCostUsd: 0 }, client);

  expect(client.turns).toBe(0);
  expect(out.status).toBe("budget");
});
Enter fullscreen mode Exit fullscreen mode

toBe(0) is the assertion. A loop that checks the budget at the bottom passes
a status check and fails this one.

Every tool_use gets a tool_result. A missing pair is a 400 from the API,
and it only shows up on multi-tool turns.

it("returns one result per tool call, matched by id", async () => {
  const client = scriptedClient([
    { tools: [
      { name: "search", input: { q: "a" } },
      { name: "get_order", input: { id: "bad" } },   // will fail
    ] },
    { text: "done" },
  ]);

  await runAgent("go", ctx, limits, client);

  const second = client.calls[1].messages.at(-1)!;
  const results = second.content as ToolResultBlockParam[];
  expect(results).toHaveLength(2);
  expect(results.map((r) => r.tool_use_id).sort())
    .toEqual(["tu_1_0", "tu_1_1"]);
  expect(results.some((r) => r.is_error)).toBe(true);
});
Enter fullscreen mode Exit fullscreen mode

Inspecting client.calls[1] is the trick — you are asserting on the request
your loop built, which is the thing that would have been malformed.

A scripted fake client driving the loop, with assertions on the requests the<br> loop<br> constructs.

A failing tool does not kill the run.

it("continues after a tool throws", async () => {
  const ctx = ctxWith({ search: async () => { throw new Error("upstream 503"); } });
  const client = scriptedClient([
    { tools: [{ name: "search", input: { q: "x" } }] },
    { text: "I could not search, but here is what I know." },
  ]);

  const out = await runAgent("go", ctx, limits, client);
  expect(out.status).toBe("complete");
  expect(client.turns).toBe(2);
});
Enter fullscreen mode Exit fullscreen mode

Internal errors do not reach the model. A stack trace in a tool_result
gets summarised and can be quoted to a user.

it("does not leak internal error detail to the model", async () => {
  const ctx = ctxWith({ get_order: async () => {
    throw new Error("pg: relation \"orders_v2\" does not exist at 10.0.0.4:5432");
  }});
  const client = scriptedClient([
    { tools: [{ name: "get_order", input: { id: "1" } }] },
    { text: "done" },
  ]);

  await runAgent("go", ctx, limits, client);

  const sent = JSON.stringify(client.calls[1]);
  expect(sent).not.toMatch(/10\.0\.0\.4|orders_v2|pg:/);
});
Enter fullscreen mode Exit fullscreen mode

Context does not grow without bound.

it("compacts once the window exceeds its budget", async () => {
  const client = scriptedClient([{ tools: [{ name: "big", input: {} }] }]);
  await runAgent("go", ctxWithHugeToolOutput(), { maxTurns: 8, ...limits }, client);

  const sizes = client.calls.map((c) => JSON.stringify(c.messages).length);
  expect(Math.max(...sizes)).toBeLessThan(200_000);
});
Enter fullscreen mode Exit fullscreen mode

That one catches the quadratic-cost bug — every turn resending an ever-larger
array, which is invisible in a two-turn manual test and expensive at eight.

Fake the tools too 

export function ctxWith(overrides: Partial<Record<string, ToolFn>>): Ctx {
  return {
    ...baseCtx,
    tools: { ...defaultFakeTools, ...overrides },
  };
}
Enter fullscreen mode Exit fullscreen mode

Tool fakes let you drive the interesting paths: empty results, 429s,
timeouts, malformed payloads. Those are the branches where loop logic goes
wrong, and they are hard to trigger against a real upstream on purpose.

Where this stops

These tests say nothing about answer quality, tool selection, or whether the
prompt is any good. That is what evals are for, and evals are slow, cost
money, and are statistical rather than binary.

The split that works: dozens of fast deterministic tests for the loop, running
on every commit; a small graded eval suite for behaviour, running on a
schedule or before release.

Most teams have the second and not the first, which is backwards — the loop is
where the bugs that cost money live, and it is the half you can actually pin
down.

Deterministic loop tests on every commit, graded evals on a slower<br> cadence.

The starting five

If you write nothing else: turn cap fires, budget checked before the call, one
tool_result per tool_use matched by id, a throwing tool does not end the
run, internal errors do not reach the model.

Half an hour, no API key, and they cover the failures that otherwise get
discovered in production.

If this was useful

AI That Acts builds the agent loop
with its seams in the right places — a substitutable client, tool fakes, and
the guards these tests are written against.

AI That Acts — Tool Calling in TypeScript

Graded eval suites are book five. The series is at
xgabriel.com/ai-in-typescript.

Top comments (0)