We stopped writing eval cases by hand. Now every prod incident becomes one.

TL;DR: Hand-written eval cases test the failures you already imagined, which are never the ones that page you. The best eval cases we have did not come from a brainstorm, they came from production incidents. We wired the postmortem process to emit an eval case automatically, and our eval set started catching the next variant of last month's outage instead of the bugs we were already not making.

Hand-written eval sets have a blind spot shaped like your imagination

When you sit down to write eval cases, you write the failures you can think of, and by definition those are the ones you already defend against. The failure that takes down prod is the one nobody pictured, and it is not in your hand-written set, because if you had pictured it you would have fixed it. So a green eval run mostly tells you that you are still not making the mistakes you already knew about.

Mining cases from incidents

Every prod incident is a labeled example handed to you for free: an input, a wrong output, and a human who already decided it was wrong. We changed the postmortem template to capture the exact input envelope (prompt, retrieved context, tool outputs, model and params) and the corrected expected behavior, and a small script drops that into the eval set as a permanent case.

# at postmortem close, capture the incident as a permanent eval case
def incident_to_eval_case(incident):
    return {
        "input": incident.full_input_envelope,    # prompt + context + tool outputs + params
        "expected": incident.corrected_behavior,   # what a human said it should have done
        "source": f"incident:{incident.id}",
        "added": incident.closed_at,
    }

Two things changed. The eval set started reflecting how the system actually fails, not how we imagined it might. And "add the regression test" stopped being a separate task someone forgets, it became a field in the postmortem, so it happens every time.

The honest limit

This is reactive by construction: you only get the case after the incident, so it does nothing for the failure you have not had yet. We pair it with a small set of hand-written adversarial cases for the scary-but-unseen classes, but the bulk of the value, and the cases that actually catch repeat regressions, come from incidents.

Open question

An incident-derived case is one example. The real failure is usually a CLASS, every input shaped like that one, and turning a single incident into a generator for the whole class without hand-writing variations is the part I have not automated well. If you have a clean way to go from one incident to its equivalence class of eval cases, that is the comment I want.