Your edge-case suite is not a description of the task. It is a record of the ways one particular model surprised you. Point it at a different model and it will pass, and the passing will mean much less than it looks like it means.
The assumption the old suite encodes
Every case in a mature edge-case suite got there because something broke once. That history is the suite’s value and also its blind spot: the cases cluster around one model’s failure surface. A new model has a different surface. The overlap is real but partial, so a green suite after a migration tells you the target does not have the old model’s weaknesses, and nothing at all about whether it has its own.
Worse, some cases are actively wrong now. A case that asserts a specific hedging phrase, a case that asserts an exact output length, a case that asserts the model asks a clarifying question — each of those encodes an old model’s behaviour as a requirement. They will fail on the target, and the temptation is to relax them until they pass, which quietly deletes the coverage.
So the migration work is two-directional: add cases the target discovers, and rewrite cases that were never about the task.
Harvesting from a differential run
The generator of new cases is a differential run: the same input through both models, both outputs scored by the same validator, and the interesting rows are the ones where the two disagree. Concordant failures tell you about your prompt. Concordant passes tell you nothing. The disagreements are the harvest.
for (const input of sample) {
const oldOut = await callOld(input);
const newOut = await callNew(input);
const a = validate(oldOut); // per-requirement booleans
const b = validate(newOut);
if (equalScores(a, b)) continue; // no information
await candidates.insert({
input_hash: hash(input),
input_ref: store(input), // redacted; see below
old_score: a,
new_score: b,
direction: worse(b, a) ? "new_only_fails" : "old_only_fails",
seen_at: new Date().toISOString(),
});
}
Two operational notes. Redact before you store: a candidate case drawn from production traffic contains customer data and will end up in a repository, so run it through the same redaction path you use for logs and record the redacted form as the fixture. And deduplicate on a normalised hash of the input, not on the raw bytes — otherwise a single high-volume request shape floods the candidate pool and hides the rare cases you actually want.
Shadow traffic is the ideal source because it is free of user impact and representative by construction. If you cannot shadow, a stratified sample replayed offline works; it is worse only in that it misses whatever your traffic mix does that your sampling does not.
Triaging into three buckets
Every candidate goes into exactly one bucket, and the bucket determines what you do:
- A: a genuine target-specific edge case. The input is legitimate, the target’s behaviour on it is wrong by a standard that predates both models, and the old suite had nothing like it. Action: add it as a new case, with provenance pointing at the differential run.
- B: an assumption the old suite encoded. The target is not wrong; the case asserted an old-model behaviour. The classic examples are asserting exact wording, asserting length, and asserting an ask-versus-act choice. Action: rewrite the case to assert the property that actually matters — usually a schema constraint or a semantic check — and record that you rewrote it. Never delete it and never loosen it to pass.
- C: a bug in your adapter. The difference is caused by your own translation layer — a parameter that did not map, a stop condition you did not port, a system-message role that landed in the wrong place. Action: fix the code. Do not add a test case for it in the model suite; add one in the adapter’s own unit tests, where it runs in milliseconds and does not need a provider.
Bucket C is larger than anyone expects in the first week of a migration and it is worth stating loudly, because a team that skips this triage ends up with a model test suite full of cases that are really assertions about their own glue code, run against a paid API, slowly.
The storage shape
A case needs enough metadata that a future reader can decide whether it still earns its place. The fields that matter are the ones about where it came from:
{
"id": "ec-0412",
"input_ref": "fixtures/ec-0412.json",
"assert": { "type": "schema", "schema_ref": "schemas/triage.v3.json" },
"assert_extra": [
{ "type": "field_equals", "path": "outcome", "value": "tag_and_hold" }
],
"provenance": "differential-run",
"provenance_ref": "migration-2026-Q3/run-14/row-882",
"incident_ref": null,
"discovered_on": "2026-07-02",
"models_seen_failing": ["provider-b/model-x@2026-06"],
"models_seen_passing": ["provider-a/model-y@2026-01"],
"last_failed_on": "2026-07-11",
"bucket": "A",
"owner": "billing-platform"
}
provenance takes one of a small set of values — incident, differential-run, support-ticket, synthetic, speculative — and it is the field the retirement rule reads. A case with provenance incident and an incident_ref is a case somebody was paged for; a case with provenance speculative is one somebody imagined. Those deserve different treatment and today most suites cannot tell them apart.
models_seen_failing pins the model identifier and a version, which is what makes the suite readable a year later. “This case exists because model X at this version got it wrong” is a sentence a future maintainer can act on.
The retirement rule
A suite that only grows becomes a suite nobody runs. Write the retirement rule down before you start adding, and make it mechanical:
A case is retired when either (a) the property it guards is now enforced structurally — by the output schema, by an enum, by a code-side validator that cannot be bypassed — or (b) it has passed on every model in the current roster for two consecutive quarters and its provenance is not incident.
Condition (a) is the one that does most of the work, and it is the reward for the schema and decision-table moves elsewhere in this cluster: once a rule is an enum the validator enforces, the case that checked it by example is redundant, and deleting it is a strict improvement. Condition (b) is deliberately conservative and deliberately excludes incident-derived cases, because a case that came from a real outage should be argued about by a human before it goes.
Retirement means deletion, not a skip marker. A disabled test is a test that will be re-enabled by someone who does not know why it was disabled. Record the retirement in the suite’s changelog with the case id and the condition that fired, and let git hold the body.
Doing it
- Audit the existing suite for bucket-B cases first, before you run anything against the target. Grep for assertions on exact strings, on lengths, and on whether a question was asked. Rewrite them now, so that the differential run’s failures are signal rather than known noise.
- Stand up the differential runner over shadow traffic or a stratified replay, writing candidates to a table rather than straight into the suite.
- Run for at least one full traffic cycle — a week if your traffic has a weekly shape. Edge cases are rare by definition and a two-hour run finds the common ones you already know about.
- Triage the candidate table in one sitting with somebody who owns the prompt and somebody who owns the adapter in the room. Bucket C needs the second person.
- Add bucket-A cases with full provenance, then re-run the whole suite against both models. Both should now fail some cases, which is the correct state during dual-running.
- Apply the retirement rule once to the pre-existing suite, immediately, while you still have both models available to evaluate condition (b). This is the only cheap moment to do it.
- Schedule the next audit for the next migration, not for a date. The suite is a per-model artifact and the natural review trigger is a model change.
The differential runner needs one call path that can address two providers with the same request object — otherwise most of the effort goes into keeping two client integrations behaviourally identical, and half your candidates are bucket C. A gateway gives you that path for free, which is why Multigrid exists, but an adapter interface with two implementations and a shared contract test does the same job and is a reasonable afternoon’s work.
Top comments (0)