Your AI eval is green because it never called the model

We run an eval suite on every PR. Seven fixtures, four metrics, deterministic, no API key, no network, costs nothing. It was green for weeks.

Then a code review found that two of the fixtures contained responses the live tool schema would have rejected outright. One had access nested inside a definition wrapper when the real pattern shape has it at the top level. Another declared a field as "json", which isn't a valid field type in our framework at all; the real one is "jsonb".

These are the canned responses the mock provider hands back. They were never checked against the schema the live model has to satisfy, because nothing in a mock run ever talks to a schema validator. So the suite was green on outputs that could not physically occur in production.

Every "test your LLM in CI" setup makes this trade, and it's worth stating plainly: a mocked eval tests your pipeline, never your model. Once you accept that, the mock gets more useful, because you stop asking it for something it can't give.

One runner, two providers

The entire trick is that the runner doesn't know which mode it's in. It takes a provider as a dependency and applies the same metric pipeline either way:

export type RunEvalOptions = {
  readonly provider: LLMProvider;
  readonly fixtures: readonly EvalFixture[];
  readonly metrics?: Readonly<Record<string, MetricFn>>;
  readonly mode: "mock" | "live";
  readonly generatedAt?: string;
};

mode is metadata. It lands in the report so you can tell a mock baseline from a live run later, and it changes no behavior. Mode selection lives in the script entry point, which is the only place that knows whether --live was passed.

For mock runs, the fixtures script themselves:

export async function runMockEval(options: RunMockEvalOptions): Promise<EvalReport> {
  const provider = createMockProvider();
  // Script every fixture's mockResponse in fixture-order. Runner
  // calls provider.chat() once per fixture, also in fixture-order,
  // so FIFO matches.
  for (const fixture of options.fixtures) {
    provider.script(fixture.mockResponse);
  }
  return runEval({ ...options, provider, mode: "mock" });
}

Each fixture carries its own mockResponse next to its expected outcome, so there's no separate mock-fixture directory to keep in sync with the real one. The provider is a FIFO queue and the runner is sequential, so the ordering holds without any matching logic.

Sequential is deliberate, by the way. Anthropic rate-limits, the eval set is seven fixtures, and parallelism would buy a few seconds in exchange for 429s in live mode.

Baseline diffs in review

A test suite that says "still green" after a prompt change is nearly worthless for this kind of work. Prompt and schema changes rarely break things outright, they shift them, and what you want to see is which fixture moved and by how much.

So the mock report goes into a checked-in JSON baseline, and a test compares a fresh mock run against it:

expect(live.summary.totalFixtures).toBe(baseline.summary.totalFixtures);
expect(live.summary.passing).toBe(baseline.summary.passing);
expect(live.summary.failing).toBe(baseline.summary.failing);
expect(live.summary.meanScore).toBeCloseTo(baseline.summary.meanScore, 6);

Plus per-fixture: same id set, same pass state, same aggregate score, same per-metric pass state.

Edit a fixture, a metric, the system prompt, or a tool definition, and this test goes red until you regenerate the baseline. The regenerated baseline is a file in your diff, so the reviewer sees parse-error flipping from true to false on one fixture, in the same PR that changed the tool schema. Quality changes become visible artifacts of review instead of something you notice three weeks later in production.

It also means "regenerate the baseline" is a normal, expected step, which is the part people resist. If regenerating feels like cheating, the baseline is doing its job: you have to look at the diff and decide the change was intended.

Two ways we got this wrong

The first was comparing the reason strings. Every metric result carries a human-readable message alongside its pass state and score. Comparing those in the drift test is tempting and wrong. The text drifts whenever someone improves an error message, and then the drift test fails for a reason that has nothing to do with model quality. After the third spurious failure people stop reading the diff, which kills the only thing the test is for. We compare structural data only: pass state, score, per-metric pass state. If a message needs to be load-bearing, it gets a dedicated metric test.

The second was letting the script pick the baseline filename. Live runs and mock runs write to different files, for the obvious reason that a live run must never clobber the deterministic baseline. That selection originally lived in a side-effectful script body, untested, where a swapped ternary would quietly destroy the mock baseline on the next live run. It's now one function with three tests, the third of which exists purely to state the invariant:

export function baselineFileName(isLive: boolean): string {
  return isLive ? "l2-eval-live-baseline.json" : "l2-eval-baseline.json";
}

test("the two modes never resolve to the same file", () => {
  expect(baselineFileName(true)).not.toBe(baselineFileName(false));
});

A test that asserts two constants differ looks silly right up until someone refactors the ternary.

Pass gates, score tracks

Every metric returns both a boolean and a 0..1 score, and the split matters more than it looks.

Pass is what gates CI. Score is what shows you a metric moving before it flips. Coverage of generated pattern kinds is a good example: four of five expected kinds found is 0.8, not zero, and not a pass.

const score = found / expected.length;
const pass = missing.length === 0;

The parse metric does the same, degrading instead of going binary. Clean parse is 1.0, N parse errors is max(0, 1 - N / 10), and a parser that throws is 0. So a change that takes a fixture from seven parse errors to two shows up as real movement even though it failed both times. Binary metrics hide exactly the progress you need to see while you're iterating on a prompt.

That metric parses the emitted source with the framework's real parser against an in-memory SourceFile. It's slower than a regex and it catches the case that matters: the model emits TypeScript that looks fine and our parser rejects, because it used factory style instead of the canonical form, or dropped the schema version header.

What the mock can't do

It cannot tell you the model got worse.

Everything above validates prompt assembly, tool definitions, metric logic, and report aggregation. All of that is code, all of it breaks in ordinary code ways, and testing it for free on every PR is worth doing. But the canned responses are ground truth by fiat. If the model starts emitting different shapes tomorrow, every one of these tests still passes.

That's what the live mode is for, and it's manual on purpose. A live run costs about $0.18 for the full set, which is nothing; the reason it isn't in CI is flakiness and rate limits, not money. It writes to its own baseline file and you fire it when you've changed something that could plausibly move model behavior. In the schema-tightening post the before-and-after numbers all came from live runs of this harness, for about $0.18 a round.

One detail that made live mode usable: a provider exception doesn't kill the run. The failing fixture gets a synthetic provider-error metric and the run continues, so a 429 on fixture three costs you one fixture instead of the whole report.

And the fixtures themselves need review like any other test data. Ours drifted because nobody was checking canned responses against the schema constraint that live mode enforces. That failure mode is intrinsic to mocking an LLM. You can only catch it by reading the fixtures, or by running live often enough that the gap shows up.

The short version

• A mocked eval tests your pipeline. It cannot test your model, and it will happily stay green on responses the real API would reject.
• Keep the runner mode-agnostic and pass the provider in. Both modes then share one metric pipeline, which is the thing you actually want identical.
• Put the value in the checked-in baseline diff, so quality changes show up in review as a file someone has to read.
• Compare structure, never human-readable messages, or the drift test becomes noise and people stop reading it.
• Report pass and score separately: one gates, the other shows movement before it flips.
• Review your fixtures against the constraints live mode enforces. Nothing else will.

Comments

[pm25coder]

The scripted response is keyed by call position, so one extra chat() during a fixture's evaluation desynchronises every fixture after it — and the report blames the wrong ones.

I replicated your loop (7 fixtures, response N carries kN, coverage metric) and injected one extra call during fx-2:

• clean: 7/7 judged on their own response
• +1 call at fx-2: 2/7 judged on their own, fx-3…fx-7 red — five failures for a perturbation that touched none of them, while fx-2 itself still reports green.

The report isn't noisy, it's misattributed, and the remedy records it: a regenerated baseline left 5 of 7 fixtures wrong.

Keying by fixture id instead (script(fixture.id, …), chat(fixture.id), throwing on an unknown or repeated id) confines it: the same injection gave one provider-error on fx-2 and 6/7 correct — the fixture that breaks is the one named.

[Mihai Perdum]

The schema-validation gap is the real finding here, not the eval bug itself. Ran into the same thing building an eval harness for CogniRunner - the mock provider had canned tool-call responses that were valid TypeScript objects but had drifted from the actual JSON schema the live model has to satisfy, because nothing in the mock path ever ran through a validator. Fixed it by pushing the schema check into the fixture layer itself so mock and live responses get validated the same way before scoring starts, adn it caught two stale fixtures immediately. The reason-string point is the one I'd flag loudest though - we made that exact mistake, comparing generated messages instead of structural pass/score, and it trained the team to stop reading eval diffs within a couple weeks.