Stop Eyeballing Prompt Changes: Eval-Driven Prompt Dev With a Matrix and a Regression Gate

The default way to iterate on a prompt is to tweak it, run the one input you happen to care about, skim the output, decide it "feels better," and ship. That works for exactly one case and forgets it the moment you move on. You never see the input the change broke, you can't compare two candidates fairly, and "better" is a vibe you can't defend — to a teammate or to yourself next week.

Prompt evals replace the vibe check with a measurement: pin a fixed test set, write a few candidate variants, run every variant against every test, and let deterministic graders turn each (variant × test) cell into a pass/fail and a score. This is the "promptfoo-style" loop, and the discipline layer under everything else.

Deterministic graders need no LLM

Most assertions are cheap, fast, and 100% repeatable: does the output match exactly, contain a string, match a regex, or parse as JSON? Each returns {pass, score} — for binary graders the score is just 1 or 0.

function grade(type, output, assertion) {
  const out = String(output);
  if (type === "exact_match") { const p = out.trim() === String(assertion).trim(); return { pass:p, score:p?1:0 }; }
  if (type === "contains")    { const p = out.toLowerCase().includes(String(assertion).toLowerCase()); return { pass:p, score:p?1:0 }; }
  if (type === "regex")       { let re; try { re = new RegExp(assertion); } catch { return { pass:false, score:0 }; } const p = re.test(out); return { pass:p, score:p?1:0 }; }
  if (type === "json_valid")  { try { JSON.parse(out); return { pass:true, score:1 }; } catch { return { pass:false, score:0 }; } }
  return rubric(out, assertion);
}

A rubric grader adds partial credit — the fraction of required keywords present — so avg score can separate two variants that share the same pass-rate.

Run the matrix, pick the winner, then guard

The harness loop is trivial: for every variant, run the prompt on every test, grade, and reduce each column to a pass-rate and an average score.

async function runEval(variants, tests, callModel) {
  return Promise.all(variants.map(async (v) => {
    const cells = await Promise.all(tests.map(async (t) => {
      const output = await callModel(v.system, t.input);   // ← a real LLM call here
      return grade(t.grader, output, t.assertion);
    }));
    const passes = cells.filter(c => c.pass).length;
    return { id: v.id, passRate: passes / tests.length,
             avgScore: cells.reduce((a, c) => a + c.score, 0) / tests.length, cells };
  }));
}

The winner is mechanical: highest pass-rate, tie-broken by avg score. But that's only half the job — and the less important half.

The part that actually saves you

A prompt change almost never strictly dominates. In the demo, V3 adds few-shot examples that finally catch the urgent cases V2 missed (pass-rate 60% → 80%) — but the same examples make it over-trigger urgent on a friendly thank-you, so it regressed a test V2 passed. The headline number went up while a specific behaviour got worse. A single average would have hidden that.

The fix is a cell-by-cell diff against a baseline that blocks on any pass → fail:

function regressions(base, cand, tests) {
  return tests.filter((t, i) => base.cells[i].pass && !cand.cells[i].pass);
}
const dropped = regressions(baseRow, candRow, tests);
if (dropped.length) throw new Error(`${dropped.length} regression(s) — blocked`);

Wire that into CI — hand-rolled or via a promptfooconfig.yaml with assert.type graders — and a net-positive change can no longer silently ship a net-negative surprise. Keep a holdout you never tune against so the score doesn't lie, report both pass-rate (the gate) and avg score (the dial), and quality can only go up.

Toggle graders and watch the matrix, pass-rates, winner, and regression diff recompute live: https://dev48v.infy.uk/prompt/day57-prompt-evals.html