Your First Eval Suite in TypeScript (Vitest, Not a Notebook)

#typescript #vitest #ai #testing

Book: AI That Ships
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Evals usually live in a notebook someone ran once, or a spreadsheet
of prompts and vibes. Neither runs in CI, so neither catches the
regression when a prompt changes on a Tuesday.

They are tests. They belong in the test runner you already have, with
the fixtures, the reporting, and the CI integration you already have.
The only thing that differs is that some assertions are graded rather
than exact — and that difference is manageable if you separate the
two.

The dataset is fixtures

// evals/fixtures/support.ts
export type Case = {
  id: string;
  input: string;
  mustInclude?: string[];
  mustNotInclude?: string[];
  expectTool?: string;
  rubric?: string;
};

export const SUPPORT_CASES: Case[] = [
  {
    id: "refund-window",
    input: "Can I get a refund for an order from 40 days ago?",
    mustInclude: ["30 days"],
    mustNotInclude: ["yes, you can"],
    rubric: "States the 30-day window and that 40 days is outside it.",
  },
  {
    id: "order-lookup",
    input: "Where is order ord_a1b2c3d4e5?",
    expectTool: "get_order",
  },
];

Plain TypeScript, type-checked, reviewable in a pull request. When
someone adds a case they are adding data with a compiler behind it —
which is more than a spreadsheet gives you.

Twenty to fifty cases is enough to start. The value is in running
them every time, not in having thousands.

Separate the two kinds of assertion

This is the design decision that makes eval suites usable.

Deterministic assertions are exact: which tool got called, whether
the output parses, whether a required string is present, whether
latency and cost are within bounds. Same input, same result, every
time.

Graded assertions need judgement: is the answer correct, is the
tone right, did it avoid over-promising.

They belong in separate files with separate configs, because they
have different reliability and therefore different consequences.

// evals/deterministic.eval.ts — blocks the deploy
import { describe, it, expect } from "vitest";

describe("support agent — deterministic", () => {
  it.each(SUPPORT_CASES)("$id", async (c) => {
    const out = await runAgent(c.input, testCtx());

    if (c.expectTool) {
      expect(out.toolsCalled).toContain(c.expectTool);
    }
    for (const s of c.mustInclude ?? []) {
      expect(out.text.toLowerCase()).toContain(s.toLowerCase());
    }
    for (const s of c.mustNotInclude ?? []) {
      expect(out.text.toLowerCase()).not.toContain(s.toLowerCase());
    }
    expect(out.costUsd).toBeLessThan(0.05);
  });
});

Deterministic evals gate merges. They are as reliable as any other
test — the model is nondeterministic, but "did it call get_order"
is a question with a stable answer for a well-designed tool.

The judge, and why it does not block

// evals/graded.eval.ts — reports, does not block
const Verdict = z.object({
  pass: z.boolean(),
  score: z.number().min(0).max(5),
  reason: z.string().max(300),
});

async function judge(c: Case, answer: string) {
  const res = await client.messages.create({
    model: "claude-opus-5",
    max_tokens: 512,
    system:
      "You grade a support answer against a rubric. Be strict. " +
      "Judge only what the rubric asks. Do not reward style.",
    messages: [{
      role: "user",
      content: `Rubric: ${c.rubric}\n\nAnswer:\n${answer}`,
    }],
  });
  return Verdict.parse(JSON.parse(textOf(res.content)));
}

A judge is a model, so it is nondeterministic, and a nondeterministic
gate on your main branch teaches people to re-run CI until it passes.
Once that habit forms the suite is worthless.

So graded evals record a score and fail only on a drop against a
baseline:

it("graded score does not regress", async () => {
  const results = await Promise.all(
    SUPPORT_CASES.filter((c) => c.rubric).map(async (c) => {
      const out = await runAgent(c.input, testCtx());
      return { id: c.id, ...(await judge(c, out.text)) };
    }),
  );

  const mean = results.reduce((s, r) => s + r.score, 0) / results.length;
  await writeRun({ mean, results, commit: process.env.GIT_SHA });

  const baseline = await readBaseline();
  expect(mean).toBeGreaterThan(baseline.mean - 0.3);
});

A tolerance band rather than a fixed threshold. The judge varies by a
little; a regression that matters moves it by a lot.

Keep them out of the default run

Evals cost money and take minutes. Nobody should trigger them by
running npm test while writing a component.

// vitest.config.ts
export default defineConfig({
  test: {
    exclude: ["**/*.eval.ts", "**/node_modules/**"],
  },
});

// vitest.eval.config.ts
export default defineConfig({
  test: {
    include: ["evals/**/*.eval.ts"],
    testTimeout: 120_000,
    hookTimeout: 60_000,
    retry: 0,
    fileParallelism: false,
  },
});

{
  "scripts": {
    "test": "vitest run",
    "eval": "vitest run -c vitest.eval.config.ts",
    "eval:det": "vitest run -c vitest.eval.config.ts evals/deterministic"
  }
}

retry: 0 is deliberate. Vitest retries are for flaky infrastructure;
retrying an eval until it passes is measuring your patience rather
than the model.

fileParallelism: false keeps you from hitting provider rate limits
with a fan-out of concurrent runs.

Cache what does not change

Most of an eval run's cost is re-running cases untouched by your
change.

const key = createHash("sha256")
  .update([c.input, PROMPT_VERSION, MODEL, TOOLS_VERSION].join("\0"))
  .digest("hex");

const cached = await evalCache.get(key);
const out = cached ?? await runAgent(c.input, testCtx());
if (!cached) await evalCache.set(key, out);

Every input that affects the output is in the key, so a prompt edit
invalidates exactly the cases it could have changed. Change one tool
description and the suite re-runs; change a README and it does not.

This is what makes running evals on every pull request affordable
rather than a nightly job nobody reads.

Report as data, not console output

type EvalRun = {
  commit: string;
  promptVersion: number;
  model: string;
  at: string;
  cases: { id: string; pass: boolean; score?: number; costUsd: number }[];
};

Write one row per run to a table or an artifact. Then "did quality
drop when we changed the prompt" is a query, and the per-case history
tells you which cases are chronically borderline — those are the ones
whose rubric is ambiguous, not whose answers are bad.

Where the cases come from

Not from imagination. Three sources, in order of value:

Production failures. Every time someone reports a bad answer, it
becomes a case. This is the highest-value dataset you will ever have
and it builds itself.

Support tickets. Real phrasing, including the ambiguous and
badly-typed inputs nobody writes when inventing test data.

Edge cases from the domain. The refund exactly on day 30. The
order that does not exist. The user asking two questions at once.

A suite grown from real failures beats a synthetic one at any size.

Start here

Ten deterministic cases, run on every pull request. That alone
catches tool-selection regressions and forbidden-phrase leaks, which
between them are most of what breaks when a prompt changes.

Add the judge once the deterministic ones are stable. Add caching
once the bill is noticeable. In that order — a graded suite on top of
no deterministic suite is the version that gets abandoned.

If this was useful

AI That Ships covers evals as
engineering — dataset design, deterministic versus graded assertions,
judges that stay honest, caching, and wiring the whole thing into CI
without making it a nuisance.