DEV Community

Cover image for Evaluating AI agents: small evals in TypeScript
Sascha Mirow-Stanitzok
Sascha Mirow-Stanitzok

Posted on Originally published at sascha-ms.de

Evaluating AI agents: small evals in TypeScript

#ai #testing #tutorial #typescript

Evaluating AI agents quickly leads to an uncomfortable question: how do you actually notice that your agent got worse after a change? A new system prompt, a model switch or a reworded tool description is enough, and suddenly the assistant reserves books that are sitting on the shelf. Without tests, you only find out when someone complains.

For regular code the answer is simple: unit tests. With agents that only half works, because the same model answers the same question differently from run to run. That is exactly what evals are for: repeatable checks that give an agent real tasks and grade the result. In this article I build a small eval suite in TypeScript with you, from an empty folder to a run in CI. I deliberately skip frameworks so you can see what happens under the hood.

In short

Evals give an AI agent real tasks in a fresh test environment and then grade the outcome, not the path that led there. To get started, 20 to 50 tasks taken from real failures, several trials per task and the pass^k metric for everything that has to work reliably are enough. The biggest pitfall is a broken grader, which is why you should read the transcripts regularly.

What you will learn in this article

  • What task, trial, grader and transcript mean in agent evals
  • How to build a small eval harness in TypeScript without a framework
  • When a code grader is enough and when a model judge makes sense
  • Why pass@k and pass^k tell such different stories
  • How regression evals in CI catch problems before a release

Why evaluating AI agents matters

An agent has three parts: the model, a loop that connects the model to its tools, and the tools themselves. You turn all three knobs in everyday work. You swap the model because a newer one is faster or cheaper. You refine the system prompt. On top of that you polish tool descriptions, as I showed in How to write tool descriptions for AI agents. Each of these changes can improve one thing and break another.

Without evals, all you have is the gut-feeling test: try three or four prompts by hand, the answer looks fine, done. Anthropic's engineering team sums up the value of evals in one sentence: “Evals make problems and behavioral changes visible before they affect users.” That is the whole point. You want to see the regression before your users feel it.

There is also a quirk of language models. A single run tells you little because answers vary. If a task succeeds once, you do not know whether it succeeds nine times out of ten or three. Evaluating AI agents therefore means running every task several times and computing metrics from the results. More on that later.

The key terms in agent evals

Before we write code, a shared vocabulary helps. I stick to the terms from Anthropic's article “Demystifying evals for AI agents” because they map well onto your own code. The table also shows where you will find each term in the example.

Term Meaning In the example
Task A single check: the input plus the criterion for passing it an entry in tasks.ts
Trial One single run of a task one call to runTrial()
Grader Logic that scores one aspect of the result functions in graders.ts
Transcript The full record of a trial with all messages and tool calls one line in results/*.jsonl
Outcome The state of the environment after the trial the reservations in the library
Eval harness The infrastructure that runs, grades and summarizes tasks runner.ts
Agent harness The loop that turns the model into an agent agent.ts
Eval suite A set of related tasks all tasks of the lending assistant

Keep two kinds of tasks apart from day one. Capability evals ask what the agent already does well. They may start with low pass rates, because they show you where you want to go. Regression evals, on the other hand, ask whether the agent can still do everything it could do before. They should sit close to 100 percent. Once a capability task reaches consistently high scores, it moves into the regression suite.

A lending assistant as the example

The example is a made-up lending assistant for a city library. It has three tools: search_books searches the catalog, check_availability shows free copies, and reserve_book creates a reservation. Two business rules come on top. A reservation is only made when no copy is available. And if the member number is missing, the agent asks for it instead of guessing.

What I like about this example: whether the agent did its job is visible in the state of the library. Is the right reservation there after the trial? Is there no wrong one? That is exactly the kind of outcome code can check reliably. This way you can evaluate AI agents without needing a human or a second model for every answer. The following diagram shows how a trial runs.

🎬 How one trial runs in the eval harness – this animated diagram only runs on my site: watch it on sascha-ms.de

Prerequisites

You need Node.js 22 or later and npm. For real runs you need either an API key for the Claude API or an AWS account with access to a Claude model in Amazon Bedrock. The tests for the harness itself run without a key and at no cost. Basic TypeScript is all you need; experience with agent frameworks is not required.

Building the eval suite step by step

1. Create the project

Create a new folder and install the Anthropic SDK, the Bedrock SDK and, as development tools, TypeScript and tsx. With tsx you run TypeScript files directly without compiling them first.

mkdir ai-agent-evals && cd ai-agent-evals
npm init -y
npm install @anthropic-ai/sdk@1.3.0 @anthropic-ai/bedrock-sdk@1.1.0
npm install -D typescript@7.0.2 tsx@4.23.15 @types/node@22
Enter fullscreen mode Exit fullscreen mode

Then replace package.json with this version. The important parts are "type": "module" and the three scripts.

{
  "name": "ai-agent-evals",
  "version": "1.0.0",
  "private": true,
  "type": "module",
  "scripts": {
    "typecheck": "tsc --noEmit",
    "test": "node --import tsx --test test/*.test.ts",
    "evals": "tsx src/run-evals.ts"
  },
  "dependencies": {
    "@anthropic-ai/bedrock-sdk": "1.1.0",
    "@anthropic-ai/sdk": "1.3.0"
  },
  "devDependencies": {
    "@types/node": "^22.20.0",
    "tsx": "4.23.15",
    "typescript": "7.0.2"
  }
}
Enter fullscreen mode Exit fullscreen mode 
// tsconfig.json
{
  "compilerOptions": {
    "target": "ES2023",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "noEmit": true,
    "skipLibCheck": true,
    "allowImportingTsExtensions": true,
    "types": ["node"]
  },
  "include": ["src", "test"]
}
Enter fullscreen mode Exit fullscreen mode

2. A test environment with a fresh state

The library lives purely in memory. That is on purpose: every trial gets a new, clean state from createLibrary(). If trials shared a database, a reservation from trial 1 could skew the result of trial 2. Errors like that are especially nasty because they look random.

// src/library.ts
// A small, made-up city library as the test environment for the agent.
// Every trial gets a fresh state from createLibrary().

export interface Book {
  id: string;
  title: string;
  author: string;
  copies: number;
}

export interface Reservation {
  bookId: string;
  memberId: string;
}

export interface LibraryState {
  books: Book[];
  loans: Record<string, number>; // copies on loan per book ID
  reservations: Reservation[];
}

export function createLibrary(): LibraryState {
  return {
    books: [
      { id: 'b-101', title: 'The Lighthouse on the Dyke', author: 'Jana Thiel', copies: 2 },
      { id: 'b-102', title: 'Cooking with Kids', author: 'Ole Brandt', copies: 1 },
      { id: 'b-103', title: 'TypeScript for Beginners', author: 'Mira Okafor', copies: 3 },
      { id: 'b-104', title: 'The Garden in Winter', author: 'Paul Seidel', copies: 1 },
    ],
    loans: { 'b-101': 2, 'b-102': 0, 'b-103': 1, 'b-104': 1 },
    reservations: [],
  };
}

export function availableCopies(state: LibraryState, bookId: string): number {
  const book = state.books.find((b) => b.id === bookId);
  if (!book) return 0;
  return book.copies - (state.loans[bookId] ?? 0);
}
Enter fullscreen mode Exit fullscreen mode

3. The agent's tools

The tools validate their own input and answer problems with an understandable error message. That way the agent can correct itself instead of getting stuck. Note that reserve_book refuses to reserve an available book. The eval still checks whether the agent even tries, because a tool in a real system is not always that strict.

// src/tools.ts
// Three tools the agent may use. Their descriptions are part of what
// your evals test as well.
import { availableCopies, type LibraryState } from './library.ts';

export interface ToolDefinition {
  name: string;
  description: string;
  input_schema: {
    type: 'object';
    properties: Record<string, unknown>;
    required: string[];
  };
}

export interface ToolResult {
  content: string;
  isError: boolean;
}

export const tools: ToolDefinition[] = [
  {
    name: 'search_books',
    description:
      'Searches the city library catalog by title or author. ' +
      'Returns book IDs (e.g. "b-101") that the other tools need.',
    input_schema: {
      type: 'object',
      properties: { query: { type: 'string', description: 'Search term, e.g. part of the title' } },
      required: ['query'],
    },
  },
  {
    name: 'check_availability',
    description: 'Shows how many copies of a book can be borrowed right now. Book ID from search_books.',
    input_schema: {
      type: 'object',
      properties: { bookId: { type: 'string' } },
      required: ['bookId'],
    },
  },
  {
    name: 'reserve_book',
    description:
      'Reserves a book for a member. Only use it when no copy is available ' +
      'and the user has given their member number (format "M-<number>"). Never guess.',
    input_schema: {
      type: 'object',
      properties: {
        bookId: { type: 'string' },
        memberId: { type: 'string', description: 'Member number, e.g. "M-17"' },
      },
      required: ['bookId', 'memberId'],
    },
  },
];

export function executeTool(state: LibraryState, name: string, input: unknown): ToolResult {
  const args = (input ?? {}) as Record<string, string>;

  switch (name) {
    case 'search_books': {
      const query = String(args.query ?? '').toLowerCase();
      const hits = state.books.filter(
        (b) => b.title.toLowerCase().includes(query) || b.author.toLowerCase().includes(query),
      );
      return { content: JSON.stringify(hits.map(({ id, title, author }) => ({ id, title, author }))), isError: false };
    }
    case 'check_availability': {
      if (!state.books.some((b) => b.id === args.bookId)) {
        return { content: `Unknown book ID "${args.bookId}". Use search_books first.`, isError: true };
      }
      return { content: JSON.stringify({ bookId: args.bookId, available: availableCopies(state, args.bookId) }), isError: false };
    }
    case 'reserve_book': {
      if (!state.books.some((b) => b.id === args.bookId)) {
        return { content: `Unknown book ID "${args.bookId}".`, isError: true };
      }
      if (!/^M-\d+$/.test(args.memberId ?? '')) {
        return { content: 'Invalid member number. Please ask the user.', isError: true };
      }
      if (availableCopies(state, args.bookId) > 0) {
        return { content: 'The book is available, no reservation needed.', isError: true };
      }
      const exists = state.reservations.some((r) => r.bookId === args.bookId && r.memberId === args.memberId);
      if (exists) {
        return { content: 'This member already has a reservation.', isError: false };
      }
      state.reservations.push({ bookId: args.bookId, memberId: args.memberId });
      return { content: 'Reservation created.', isError: false };
    }
    default:
      return { content: `Unknown tool "${name}".`, isError: true };
  }
}
Enter fullscreen mode Exit fullscreen mode

4. The agent loop

The loop is deliberately simple. It sends the history to the model, runs the requested tools, appends the results and repeats until the model stops calling tools or the maximum number of turns is reached. Along the way it collects everything for the transcript: messages, tool calls, turns and tokens.

The trick for testability lies in the ModelClient type. The loop only knows a function that takes a request and returns a response. Whether the Claude API, Amazon Bedrock or a test script sits behind it does not matter to the loop.

// src/agent.ts
// A lean agent loop: ask the model, run tools, return the results
// until the model is done. Everything ends up in the transcript.
import type { LibraryState } from './library.ts';
import { executeTool, tools, type ToolDefinition } from './tools.ts';

export type ContentBlock =
  | { type: 'text'; text: string }
  | { type: 'tool_use'; id: string; name: string; input: unknown }
  | { type: 'tool_result'; tool_use_id: string; content: string; is_error: boolean };

export interface Message {
  role: 'user' | 'assistant';
  content: string | ContentBlock[];
}

export interface ModelRequest {
  system: string;
  messages: Message[];
  tools: ToolDefinition[];
  max_tokens: number;
}

export interface ModelResponse {
  content: ContentBlock[];
  stop_reason: string | null;
  usage: { input_tokens: number; output_tokens: number };
}

// Any function with this signature can be the model: the Anthropic API,
// Amazon Bedrock or, in tests, a script with fixed answers.
export type ModelClient = (request: ModelRequest) => Promise<ModelResponse>;

export interface ToolCall {
  name: string;
  input: unknown;
  isError: boolean;
}

export interface Transcript {
  messages: Message[];
  toolCalls: ToolCall[];
  finalText: string;
  turns: number;
  inputTokens: number;
  outputTokens: number;
  stopReason: string | null;
}

export const SYSTEM_PROMPT =
  'You are the lending assistant of the city library. Help members find, ' +
  'check and reserve books. Use the tools instead of guessing. If information ' +
  'such as the member number is missing, ask for it. Answer briefly and kindly.';

export async function runAgent(
  client: ModelClient,
  state: LibraryState,
  prompt: string,
  maxTurns = 8,
): Promise<Transcript> {
  const messages: Message[] = [{ role: 'user', content: prompt }];
  const transcript: Transcript = {
    messages,
    toolCalls: [],
    finalText: '',
    turns: 0,
    inputTokens: 0,
    outputTokens: 0,
    stopReason: null,
  };

  while (transcript.turns < maxTurns) {
    const response = await client({ system: SYSTEM_PROMPT, messages, tools, max_tokens: 1024 });
    transcript.turns += 1;
    transcript.inputTokens += response.usage.input_tokens;
    transcript.outputTokens += response.usage.output_tokens;
    transcript.stopReason = response.stop_reason;
    messages.push({ role: 'assistant', content: response.content });

    const text = response.content
      .filter((b): b is Extract<ContentBlock, { type: 'text' }> => b.type === 'text')
      .map((b) => b.text)
      .join('\n')
      .trim();
    if (text) transcript.finalText = text;

    if (response.stop_reason !== 'tool_use') break;

    const results: ContentBlock[] = [];
    for (const block of response.content) {
      if (block.type !== 'tool_use') continue;
      const result = executeTool(state, block.name, block.input);
      transcript.toolCalls.push({ name: block.name, input: block.input, isError: result.isError });
      results.push({ type: 'tool_result', tool_use_id: block.id, content: result.content, is_error: result.isError });
    }
    messages.push({ role: 'user', content: results });
  }

  return transcript;
}
Enter fullscreen mode Exit fullscreen mode

The adapters for real models are short because both SDKs offer the same messages.create() method. The Bedrock client uses your regular AWS credentials and the region from AWS_REGION.

// src/model-clients.ts
// Adapters for real models. The agent loop only knows ModelClient.
import Anthropic from '@anthropic-ai/sdk';
import { AnthropicBedrock } from '@anthropic-ai/bedrock-sdk';
import type { ContentBlock, ModelClient } from './agent.ts';

// Both SDK clients offer messages.create() with the same shape.
interface MessagesApi {
  messages: {
    create(params: Anthropic.MessageCreateParamsNonStreaming): PromiseLike<Anthropic.Message>;
  };
}

export function messagesClient(sdk: MessagesApi, model: string): ModelClient {
  return async (request) => {
    const response = await sdk.messages.create({
      model,
      max_tokens: request.max_tokens,
      system: request.system,
      tools: request.tools,
      messages: request.messages as Anthropic.MessageParam[],
    });
    return {
      // We only pass on text and tool blocks; the loop ignores everything else.
      content: response.content.filter((b) => b.type === 'text' || b.type === 'tool_use') as ContentBlock[],
      stop_reason: response.stop_reason,
      usage: {
        input_tokens: response.usage.input_tokens ?? 0,
        output_tokens: response.usage.output_tokens ?? 0,
      },
    };
  };
}

// Anthropic API: reads ANTHROPIC_API_KEY from the environment.
export function anthropicClient(model: string): ModelClient {
  return messagesClient(new Anthropic(), model);
}

// Amazon Bedrock: uses your regular AWS credentials and the region from AWS_REGION.
export function bedrockClient(model: string): ModelClient {
  return messagesClient(new AnthropicBedrock(), model);
}
Enter fullscreen mode Exit fullscreen mode

5. Tasks and graders

Now comes the part that takes the most thinking. A grader receives the transcript and the final state of the library and returns “pass” or “fail” with a short reason. Most graders here are code graders. They are fast, free and always return the same verdict for the same input.

For soft criteria such as tone and clarity there is llmJudge. Here a model grades the answer against one criterion. The “unknown” option matters: a judge that is allowed to admit uncertainty invents verdicts less often. I count “unknown” as a fail, so it automatically ends up in manual review.

// src/graders.ts
// Graders check the outcome of a trial. Code graders are fast and
// reproducible; the model grader is meant for soft criteria such as tone.
import type { ModelClient, Transcript } from './agent.ts';
import type { LibraryState } from './library.ts';

export interface GradeResult {
  grader: string;
  pass: boolean;
  detail: string;
}

export type Grader = (run: Transcript, state: LibraryState) => GradeResult | Promise<GradeResult>;

// Check the outcome: does the reservation really exist in the library state?
export const reservationExists =
  (bookId: string, memberId: string): Grader =>
  (_run, state) => {
    const pass = state.reservations.some((r) => r.bookId === bookId && r.memberId === memberId);
    return { grader: `reservationExists(${bookId}, ${memberId})`, pass, detail: JSON.stringify(state.reservations) };
  };

export const reservationCount =
  (expected: number): Grader =>
  (_run, state) => ({
    grader: `reservationCount(${expected})`,
    pass: state.reservations.length === expected,
    detail: `found: ${state.reservations.length}`,
  });

// Counter-check: a tool must not be called at all in this task.
export const toolNotCalled =
  (name: string): Grader =>
  (run) => {
    const calls = run.toolCalls.filter((c) => c.name === name).length;
    return { grader: `toolNotCalled(${name})`, pass: calls === 0, detail: `calls: ${calls}` };
  };

export const toolCalled =
  (name: string): Grader =>
  (run) => {
    const calls = run.toolCalls.filter((c) => c.name === name && !c.isError).length;
    return { grader: `toolCalled(${name})`, pass: calls > 0, detail: `successful calls: ${calls}` };
  };

// Transcript metric: the agent should not run in circles forever.
export const maxTurns =
  (limit: number): Grader =>
  (run) => ({ grader: `maxTurns(${limit})`, pass: run.turns <= limit, detail: `turns: ${run.turns}` });

export const asksQuestion = (): Grader => (run) => ({
  grader: 'asksQuestion',
  pass: run.finalText.includes('?'),
  detail: run.finalText.slice(0, 120),
});

// Model as judge: grades a soft criterion and may answer "unknown".
// An "unknown" counts as a fail and belongs in manual review.
export const llmJudge =
  (judge: ModelClient, criterion: string): Grader =>
  async (run) => {
    const response = await judge({
      system:
        'You grade the answer of a library assistant. Reply with JSON only: ' +
        '{"verdict":"pass"|"fail"|"unknown","reason":"<one sentence>"}. Use "unknown" if you are unsure.',
      messages: [{ role: 'user', content: `Criterion: ${criterion}\n\nAssistant answer:\n${run.finalText}` }],
      tools: [],
      max_tokens: 200,
    });
    const text = response.content.map((b) => (b.type === 'text' ? b.text : '')).join('');
    try {
      const parsed = JSON.parse(text.slice(text.indexOf('{'), text.lastIndexOf('}') + 1)) as {
        verdict?: string;
        reason?: string;
      };
      return { grader: `llmJudge(${criterion})`, pass: parsed.verdict === 'pass', detail: `${parsed.verdict}: ${parsed.reason ?? ''}` };
    } catch {
      return { grader: `llmJudge(${criterion})`, pass: false, detail: `unreadable: ${text.slice(0, 80)}` };
    }
  };
Enter fullscreen mode Exit fullscreen mode

The tasks cover both directions. Some cases require the agent to reserve, others forbid it outright. If you only test positive cases, you end up with an agent that acts too eagerly. The task “two-books-at-once” is a real capability task: the agent has to figure out on its own how to search the whole catalog.

// src/tasks.ts
// The tasks of the eval suite. Each task has a clear request, a
// starting state and graders that check the outcome.
import type { LibraryState } from './library.ts';
import { asksQuestion, maxTurns, reservationCount, reservationExists, toolCalled, toolNotCalled, type Grader } from './graders.ts';

export interface Task {
  id: string;
  // capability: what should the agent be able to do? regression: what must never break again?
  kind: 'capability' | 'regression';
  prompt: string;
  setup?: (state: LibraryState) => void;
  graders: Grader[];
}

export const tasks: Task[] = [
  {
    id: 'reserve-when-unavailable',
    kind: 'regression',
    prompt: 'Hi, I am member M-17. Please reserve “The Lighthouse on the Dyke” for me.',
    graders: [reservationExists('b-101', 'M-17'), reservationCount(1), maxTurns(5)],
  },
  {
    id: 'no-reservation-when-available',
    kind: 'regression',
    prompt: 'Member M-17 here. Please reserve “Cooking with Kids” for me.',
    graders: [reservationCount(0), toolNotCalled('reserve_book')],
  },
  {
    id: 'member-number-missing',
    kind: 'regression',
    prompt: 'Can you reserve “The Garden in Winter” for me?',
    graders: [reservationCount(0), toolNotCalled('reserve_book'), asksQuestion()],
  },
  {
    id: 'unknown-book',
    kind: 'capability',
    prompt: 'I am M-8 and would like to reserve “Advanced Sailing”.',
    graders: [reservationCount(0), toolNotCalled('reserve_book')],
  },
  {
    id: 'availability-by-author',
    kind: 'capability',
    prompt: 'Do you have anything by Ole Brandt? Can I borrow it today?',
    graders: [toolCalled('check_availability'), reservationCount(0)],
  },
  {
    id: 'two-books-at-once',
    kind: 'capability',
    prompt: 'I am M-23. Please reserve everything that is fully on loan right now.',
    graders: [reservationExists('b-101', 'M-23'), reservationExists('b-104', 'M-23'), reservationCount(2)],
  },
];
Enter fullscreen mode Exit fullscreen mode

6. Metrics and harness

The harness runs every task several times, each time in a fresh library, with limited concurrency. That keeps you below the API rate limits. If a trial aborts with an error, for example a timeout, the harness flags it as an infrastructure error. This matters so you do not mistake a network error for an agent error.

Besides “pass” there is partial credit: the share of graders that passed. An agent that creates the reservation correctly but needs seven turns for it is better than one that gets nothing done. I explain the pass@k and pass^k metrics in detail in the next section.

// src/metrics.ts
// pass@k: how likely is at least one success in k trials?
// pass^k: how likely are all k trials successful?
// Both are estimated from n trials with c successes.

function binomial(n: number, k: number): number {
  if (k < 0 || k > n) return 0;
  let result = 1;
  for (let i = 1; i <= k; i++) result = (result * (n - k + i)) / i;
  return result;
}

export function passAtK(n: number, c: number, k: number): number {
  if (k > n) throw new Error(`k (${k}) must not be greater than n (${n})`);
  if (n - c < k) return 1;
  return 1 - binomial(n - c, k) / binomial(n, k);
}

export function passHatK(n: number, c: number, k: number): number {
  if (k > n) throw new Error(`k (${k}) must not be greater than n (${n})`);
  return binomial(c, k) / binomial(n, k);
}
Enter fullscreen mode Exit fullscreen mode 
// src/runner.ts
// The eval harness: runs every task several times in a fresh library,
// grades the trials and summarizes them.
import { runAgent, type ModelClient, type Transcript } from './agent.ts';
import type { Grader, GradeResult } from './graders.ts';
import { createLibrary } from './library.ts';
import { passAtK, passHatK } from './metrics.ts';
import type { Task } from './tasks.ts';

export interface TrialResult {
  taskId: string;
  trial: number;
  pass: boolean;
  score: number; // share of passed graders, for partial credit
  grades: GradeResult[];
  transcript?: Transcript;
  error?: string;
}

export interface TaskSummary {
  taskId: string;
  kind: Task['kind'];
  trials: number;
  successes: number;
  avgScore: number;
  passAtK: number;
  passHatK: number;
  avgTurns: number;
  avgTokens: number;
}

export interface EvalOptions {
  trials: number;
  k: number;
  concurrency: number;
  extraGraders?: Grader[];
}

async function runTrial(client: ModelClient, task: Task, trial: number, extra: Grader[]): Promise<TrialResult> {
  const state = createLibrary(); // every trial starts clean
  task.setup?.(state);
  try {
    const transcript = await runAgent(client, state, task.prompt);
    const grades: GradeResult[] = [];
    for (const grader of [...task.graders, ...extra]) grades.push(await grader(transcript, state));
    const passed = grades.filter((g) => g.pass).length;
    return { taskId: task.id, trial, pass: passed === grades.length, score: passed / grades.length, grades, transcript };
  } catch (error) {
    // An infrastructure error is not the agent's fault, so it is flagged separately.
    return { taskId: task.id, trial, pass: false, score: 0, grades: [], error: String(error) };
  }
}

async function pool<T>(jobs: Array<() => Promise<T>>, concurrency: number): Promise<T[]> {
  const results: T[] = new Array(jobs.length);
  let next = 0;
  const workers = Array.from({ length: Math.min(concurrency, jobs.length) }, async () => {
    while (next < jobs.length) {
      const index = next++;
      results[index] = await jobs[index]();
    }
  });
  await Promise.all(workers);
  return results;
}

export async function runEvals(client: ModelClient, tasks: Task[], options: EvalOptions) {
  const jobs = tasks.flatMap((task) =>
    Array.from({ length: options.trials }, (_, trial) => () => runTrial(client, task, trial + 1, options.extraGraders ?? [])),
  );
  const results = await pool(jobs, options.concurrency);

  const summaries: TaskSummary[] = tasks.map((task) => {
    const own = results.filter((r) => r.taskId === task.id);
    const successes = own.filter((r) => r.pass).length;
    const withTranscript = own.filter((r) => r.transcript);
    const avg = (values: number[]) => (values.length ? values.reduce((a, b) => a + b, 0) / values.length : 0);
    return {
      taskId: task.id,
      kind: task.kind,
      trials: own.length,
      successes,
      avgScore: avg(own.map((r) => r.score)),
      passAtK: passAtK(own.length, successes, options.k),
      passHatK: passHatK(own.length, successes, options.k),
      avgTurns: avg(withTranscript.map((r) => r.transcript!.turns)),
      avgTokens: avg(withTranscript.map((r) => r.transcript!.inputTokens + r.transcript!.outputTokens)),
    };
  });

  return { results, summaries };
}
Enter fullscreen mode Exit fullscreen mode

The command-line script ties everything together. It stores every trial as a line in a JSONL file, prints a table and sets an exit code. Exit code 1 means a regression task is below 90 percent. Exit code 2 means the run is invalid because trials failed on infrastructure.

// src/run-evals.ts
// Usage: EVAL_MODEL=<model-id> npm run evals
// Optional: EVAL_PROVIDER=bedrock, EVAL_TRIALS=5, EVAL_K=3, EVAL_JUDGE_MODEL=<model-id>
import { mkdir, writeFile } from 'node:fs/promises';
import { llmJudge } from './graders.ts';
import { anthropicClient, bedrockClient } from './model-clients.ts';
import { runEvals } from './runner.ts';
import { tasks } from './tasks.ts';

const model = process.env.EVAL_MODEL;
if (!model) {
  console.error('Please set EVAL_MODEL, e.g. EVAL_MODEL=claude-sonnet-5-5');
  process.exit(2);
}

const makeClient = process.env.EVAL_PROVIDER === 'bedrock' ? bedrockClient : anthropicClient;
const client = makeClient(model);
const trials = Number(process.env.EVAL_TRIALS ?? 5);
const k = Number(process.env.EVAL_K ?? 3);
const judgeModel = process.env.EVAL_JUDGE_MODEL;
const extraGraders = judgeModel
  ? [llmJudge(makeClient(judgeModel), 'The answer is friendly, factually correct and tells the member the next step.')]
  : [];

const { results, summaries } = await runEvals(client, tasks, { trials, k, concurrency: 3, extraGraders });

// Save all transcripts: reading them is the most important part of the analysis.
await mkdir('results', { recursive: true });
const file = `results/${new Date().toISOString().replace(/[:.]/g, '-')}.jsonl`;
await writeFile(file, results.map((r) => JSON.stringify(r)).join('\n') + '\n');

console.table(
  summaries.map((s) => ({
    Task: s.taskId,
    Kind: s.kind,
    Successes: `${s.successes}/${s.trials}`,
    Partial: s.avgScore.toFixed(2),
    [`pass@${k}`]: s.passAtK.toFixed(2),
    [`pass^${k}`]: s.passHatK.toFixed(2),
    Turns: s.avgTurns.toFixed(1),
    Tokens: Math.round(s.avgTokens),
  })),
);
console.log(`Transcripts: ${file}`);

const infraErrors = results.filter((r) => r.error).length;
if (infraErrors > 0) {
  // A run with API errors says nothing about the agent: better invalid than falsely red.
  console.error(`${infraErrors} trials with infrastructure errors, run invalid. See ${file}`);
  process.exit(2);
}

// Regression tasks must (almost) always pass, otherwise the run fails.
const broken = summaries.filter((s) => s.kind === 'regression' && s.successes / s.trials < 0.9);
if (broken.length > 0) {
  console.error(`Regression: ${broken.map((s) => s.taskId).join(', ')}`);
  process.exit(1);
}
Enter fullscreen mode Exit fullscreen mode

7. Testing the harness itself

An eval harness is code and can have bugs. A broken grader is even particularly dangerous, because it gives you false confidence. That is why I test the harness with a canned model: the scriptedClient returns predefined answers in order. With it you check that a good solution passes and a bad one fails. The tests for the metrics secure the calculations.

// test/helpers.ts
// A model for tests: returns predefined answers in order.
import type { ContentBlock, ModelClient, ModelRequest, ModelResponse } from '../src/agent.ts';

export function scriptedClient(steps: ContentBlock[][]): ModelClient & { requests: ModelRequest[] } {
  let index = 0;
  const requests: ModelRequest[] = [];
  const client = async (request: ModelRequest): Promise<ModelResponse> => {
    requests.push(structuredClone(request));
    const content = steps[Math.min(index++, steps.length - 1)];
    const usesTool = content.some((b) => b.type === 'tool_use');
    return { content, stop_reason: usesTool ? 'tool_use' : 'end_turn', usage: { input_tokens: 100, output_tokens: 20 } };
  };
  return Object.assign(client, { requests });
}

export const toolUse = (id: string, name: string, input: unknown): ContentBlock => ({ type: 'tool_use', id, name, input });
export const text = (value: string): ContentBlock => ({ type: 'text', text: value });
Enter fullscreen mode Exit fullscreen mode 
// test/graders.test.ts
import assert from 'node:assert/strict';
import { test } from 'node:test';
import { runAgent } from '../src/agent.ts';
import { llmJudge } from '../src/graders.ts';
import { createLibrary } from '../src/library.ts';
import { tasks } from '../src/tasks.ts';
import { scriptedClient, text, toolUse } from './helpers.ts';

const task = (id: string) => tasks.find((t) => t.id === id)!;

async function grade(id: string, steps: Parameters<typeof scriptedClient>[0]) {
  const state = createLibrary();
  task(id).setup?.(state);
  const run = await runAgent(scriptedClient(steps), state, task(id).prompt);
  return Promise.all(task(id).graders.map((g) => g(run, state)));
}

test('a good solution passes “reserve-when-unavailable”', async () => {
  const grades = await grade('reserve-when-unavailable', [
    [toolUse('t1', 'search_books', { query: 'Lighthouse' })],
    [toolUse('t2', 'reserve_book', { bookId: 'b-101', memberId: 'M-17' })],
    [text('Reserved.')],
  ]);
  assert.ok(grades.every((g) => g.pass), JSON.stringify(grades));
});

test('an agent that reserves an available book fails', async () => {
  const grades = await grade('no-reservation-when-available', [
    [toolUse('t1', 'reserve_book', { bookId: 'b-102', memberId: 'M-17' })],
    [text('Sorry, that did not work.')],
  ]);
  const failed = grades.filter((g) => !g.pass).map((g) => g.grader);
  assert.deepEqual(failed, ['toolNotCalled(reserve_book)']);
});

test('missing member number: asking back passes', async () => {
  const grades = await grade('member-number-missing', [[text('Sure! What is your member number?')]]);
  assert.ok(grades.every((g) => g.pass));
});

test('model judge: "unknown" does not count as a pass', async () => {
  const judge = scriptedClient([[text('{"verdict":"unknown","reason":"The answer is ambiguous"}')]]);
  const run = await runAgent(scriptedClient([[text('Hm.')]]), createLibrary(), 'Hello');
  const result = await llmJudge(judge, 'friendly')(run, createLibrary());
  assert.equal(result.pass, false);
  assert.match(result.detail, /^unknown/);
});
Enter fullscreen mode Exit fullscreen mode 
// test/metrics.test.ts
import assert from 'node:assert/strict';
import { test } from 'node:test';
import { passAtK, passHatK } from '../src/metrics.ts';

test('at k = 1, pass@k and pass^k equal the success rate', () => {
  assert.equal(passAtK(10, 7, 1), 0.7);
  assert.equal(passHatK(10, 7, 1), 0.7);
});

test('pass@k rises and pass^k falls as k grows', () => {
  assert.ok(passAtK(10, 7, 3) > passAtK(10, 7, 1));
  assert.ok(passHatK(10, 7, 3) < passHatK(10, 7, 1));
});

test('edge cases', () => {
  assert.equal(passAtK(5, 0, 3), 0);
  assert.equal(passHatK(5, 5, 3), 1);
  assert.equal(passHatK(5, 2, 3), 0);
  assert.throws(() => passAtK(2, 1, 3));
});
Enter fullscreen mode Exit fullscreen mode

Now check types and tests. Both run without an API key.

npm run typecheck
npm test
Enter fullscreen mode Exit fullscreen mode

Expected result: tsc reports nothing, and the test runner ends with # pass 7 and # fail 0.

8. The first real run

Set the API key and the model and start the evals. EVAL_TRIALS sets how often each task runs, EVAL_K sets k for the metrics.

export ANTHROPIC_API_KEY="your-key"
EVAL_MODEL=claude-sonnet-5-5 EVAL_TRIALS=5 EVAL_K=3 npm run evals
Enter fullscreen mode Exit fullscreen mode

With Amazon Bedrock the call looks like this. As the model, enter the model ID or inference profile that is enabled in your region.

export AWS_REGION=eu-central-1
EVAL_PROVIDER=bedrock EVAL_MODEL="<your-bedrock-model-id>" npm run evals
Enter fullscreen mode Exit fullscreen mode

You get a table with one row per task: successes, partial credit, pass@3, pass^3, average turns and tokens. Below it is the path to the JSONL file with all transcripts. I am not giving you numbers here, because they depend on the model, the day and the tool descriptions. I do have one expectation, though: the regression tasks should score very high with a current model, while a low value for “two-books-at-once” would not surprise me. That is exactly what a capability task is for.

If the key is missing, every trial fails as an infrastructure error and the script exits with code 2. That is intentional: it keeps you from reading a broken run as a bad agent.

Which metric counts, pass@k or pass^k?

Both metrics are based on k trials but answer opposite questions. pass@k is the probability that at least one of k trials succeeds. It rises with k and fits tools where one good solution is enough, for example when a developer picks from several suggestions. pass^k asks how likely it is that every single one of the k trials works. This value drops as k grows and fits everything customers use directly. A lending assistant that reserves nonsense every third time is useless in practice, even if it “mostly” works.

Anthropic works through an example: at 75 percent success per trial, the chance that three out of three trials succeed drops to about 42 percent (0.75 to the power of 3). The chart shows how quickly the two curves drift apart.

Line chart: pass@k rises from 75% to almost 100%, pass^k falls from 75% to 24% for k from 1 to 5

At 75% success per trial, pass@k quickly approaches 100%, while pass^k drops to about 42% at k = 3.

In metrics.ts I estimate both values from n trials with c successes. For pass@k I use the estimator from OpenAI's Codex paper: one minus the probability of drawing k trials only from the failures. For pass^k, which comes from the agent benchmark τ-bench, I draw k trials only from the successes. A practical rule follows: n must be at least as large as k, and the larger n, the more stable the estimate. With five trials per task and k = 3 you get a useful first impression.

Reading transcripts instead of just percentages

The table tells you that something is wrong. Why it is wrong is only in the transcript. Open the JSONL file and read at least the failed trials. You will find two kinds of errors, and you need to tell them apart.

The first kind are real agent errors. The agent calls reserve_book with a guessed member number or misses that a copy is available. A better tool description, a clearer system prompt or a different model helps here.

The second kind are errors in the eval itself. An example from my own code: asksQuestion passes as soon as there is a question mark anywhere. If the agent answers “Did you see our new opening hours?” and makes a wrong reservation at the same time, only the other two graders of that task catch it. That is why I always combine soft checks with a hard check of the outcome. If a task fails in almost every trial, look at the task and the grader first, then at the agent. Often the task is simply ambiguous.

Evaluating AI agents in CI

It gets really valuable once you are evaluating AI agents automatically, without having to remember it. I would hook the evals in at two points: on every pull request that changes prompts, tools or agent code, and once per working day against the current model. The daily run catches drift, meaning behavior changes that happen without a commit of your own, for example when a model alias points to a new version.

The following GitHub Actions workflow first runs the free tests and then the evals. The transcripts are attached to the run as an artifact, so you can read them even when it fails.

# .github/workflows/evals.yml
name: Agent-Evals

on:
  pull_request:
    paths: ['src/**']
  schedule:
    - cron: '17 3 * * 1-5' # early on weekdays so model drift shows up
  workflow_dispatch:

jobs:
  evals:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npm test
      - run: npm run evals
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          EVAL_MODEL: claude-sonnet-5-5
          EVAL_TRIALS: '5'
          EVAL_K: '3'
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: eval-transcripts
          path: results/
Enter fullscreen mode Exit fullscreen mode

Create the repository secret ANTHROPIC_API_KEY for this. Pull requests from forks do not get secrets on GitHub by default. The eval step fails there with exit code 2, and that is how it should be: outside code should not run with your API key.

Common mistakes when evaluating agents

Only one trial per task

One trial is an anecdote. Only with several trials can you see whether a result is stable. Five trials are a good start; use more for important regression tasks.

Checking the path instead of the outcome

If you prescribe that the agent must first search, then check and then reserve, you punish valid shortcuts. Check the final state. I make one exception for forbidden side effects: requiring that reserve_book is not called at all for an available book is a business rule, not a prescribed path.

Testing only positive cases

If you only test whether the agent reserves, sooner or later it reserves everything. Every “do X” task needs at least one “do not do X here” task. This balance keeps your agent from overreacting or underreacting.

Shared state between trials

If data from the previous trial is still lying around, you are measuring chance. Every trial starts clean, whether in memory like here or with a freshly provisioned test database.

Evals that no longer show anything

If a suite sits at 100 percent permanently, it no longer measures progress. For regressions that is the goal. For capability evals it means you need harder tasks.

Cost and security

Every run costs tokens. The math is simple: tasks times trials times turns per trial. With six tasks, five trials and three to four turns, you quickly end up with around a hundred model calls. Keep an eye on the “Tokens” column in the table, and feel free to use a smaller, cheaper model as the judge if it grades reliably. Check that first against a few cases you graded yourself.

On security: evals run against a test environment, never against real systems or real customer data. Write tools in particular can be called hundreds of times in an eval. The API key belongs in secrets, never in the code. And because rate limits can hit, the harness limits concurrency to three simultaneous trials.

My take

To me, evals are to agents what unit tests are to regular code: the precondition for changing things with confidence. If you can evaluate AI agents, you can afford to experiment. If you run an agent without evals, at some point you no longer dare to switch the model or touch the prompt. That is the opposite of what we want to achieve with AI.

At the same time, I do not think much of starting with a big framework and hundreds of tasks. Twenty tasks from real failures beat two hundred invented ones. And I would build code graders first. The model judge is handy, but it costs money, varies itself and has to be calibrated against human judgment. I use it where code really cannot go further.

Then there is the human side. The best tasks rarely come from the development team alone. They come from support, from the business side, from the people who know what goes wrong in daily use. When you introduce evals, invite these people to contribute tasks. That improves the agent and builds trust in what it does.

FAQ

What is the difference between evals and unit tests?

Unit tests check deterministic code and always return the same result for the same input. Evals check a system with a language model whose answers vary. That is why evals run several times and return rates instead of a simple green or red.

How many tasks do I need to start?

If you are evaluating AI agents for the first time, Anthropic recommends starting with roughly 20 to 50 simple tasks from real failures. You do not need hundreds of tasks at the beginning. What matters more is that every task is unambiguous and that two experts would independently reach the same verdict.

Do I need an eval framework?

Not to get started. A harness like the one in this article has a few hundred lines and shows you exactly what happens. A framework pays off once you need many suites, dashboards or comparisons between models.

Can I use a language model as a judge?

Yes, for soft criteria such as tone or completeness. Give the judge one clear criterion, allow the answer “unknown” and regularly compare its verdicts with human ratings. For hard facts such as “does the reservation exist”, code is more reliable.

How much does an eval run cost?

That depends on the model and the number of tasks, trials and turns. Multiply tasks by trials and average turns to get the number of model calls. The harness shows you the tokens per task after the first run.

If you want to not only test your agent but also run it on AWS, AgentCore CLI 1.0: Deploy an AI Agent with TypeScript shows the way into the AgentCore Runtime.

Your next step

Take the last three cases where your agent got it wrong and turn them into tasks with a grader that checks the outcome. Run them five times and read the transcripts. That is the smallest sensible start for evaluating AI agents. I bet at least one of the three cases leads you to a better tool description. You will find more about AI in everyday development on the blog.

Now I am curious: how do you check today whether your agent still does what it should after a change? Let me know in the comments.

Sources

This post was created with AI assistance.

👉 Next step: New here? My start page collects the best articles, tools and learning paths in one place.

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en

Top comments (0)