The prompt has a rule like: if the ticket is billing and the account is enterprise, escalate to the named account manager — unless the ticket is under 24 hours old, in which case tag it and hold. After the migration, forty sampled enterprise billing tickets under 24 hours old were all escalated. The unless never fired once.
The symptom: a branch that never fires
This failure has a distinctive signature and it is worth learning to recognise, because it does not look like a regression on any aggregate metric. The output is always well-formed. It is always one of the allowed outcomes. It is often defensible in isolation — escalating an enterprise billing ticket is not obviously wrong. Overall quality scores barely move.
What moves is the distribution over outcomes. Pull the counts per outcome from the last week on the old model and the first week on the new one, and one bucket has gone to zero while another has absorbed it. That is the alarm. A rule that used to fire and now never fires is not a quality question; it is a missing branch, and the fastest way to find it is to count outcomes rather than score them.
Why it is always the inner clause
A nested conditional written in prose asks the model to hold a predicate, then a second predicate, then an exception to the conjunction of both, and to resolve them in the right order while also attending to everything else in the prompt. How reliably a model does that is its instruction-following depth, and there is no parameter for it. It is a property of the model, it varies between models, and it degrades from the inside out: the outer conditions survive because they are stated first and repeated by the surrounding context, and the exception — stated last, attached to a conjunction, phrased as a negation of the rule just given — is the part that goes.
Two aggravating factors make it worse in migrated prompts specifically. Prose exceptions accumulate: a rule that started simple has usually acquired its unless clauses one incident at a time, so the deepest clause is also the newest and the least tested. And prompts grow during migrations, because people add clarifying instructions when something misbehaves, which pushes the original rules further from the end of the prompt.
The fix is not a stronger phrasing. Emphasis moves which clause wins when clauses compete; it does not increase how many clauses the model can resolve at once. The fix is to stop asking it to resolve them.
Flatten the prose into a decision table
Enumerate the combinations. A three-predicate rule has eight rows, which is a table small enough to read and small enough to test:
BEFORE (nested prose, one sentence, three predicates):
If the ticket is billing and the account is enterprise, escalate to
the named account manager - unless the ticket is under 24 hours old,
in which case tag it and hold. Non-enterprise billing tickets go to
the billing queue. Everything else follows the default routing.
AFTER (decision table, one row per combination):
| category | tier | age_hours | outcome |
|----------|------------|-----------|--------------------|
| billing | enterprise | < 24 | tag_and_hold |
| billing | enterprise | >= 24 | escalate_named_am |
| billing | standard | < 24 | billing_queue |
| billing | standard | >= 24 | billing_queue |
| other | enterprise | < 24 | default_routing |
| other | enterprise | >= 24 | default_routing |
| other | standard | < 24 | default_routing |
| other | standard | >= 24 | default_routing |
Return the outcome for the row that matches. Exactly one row matches
every input. If none matches, return "needs_review".
Three things changed. There is no nesting, so there is no inner clause to lose. There is no negation — unless has become a row rather than an exception. And the instruction is now a lookup, which is a task with a much shallower depth requirement than resolving a conjunction with an exception attached.
The obvious objection is that eight rows is more tokens than one sentence. It is, by perhaps sixty tokens, and it is the cheapest sixty tokens in the prompt. The objection that matters more is combinatorial growth: five predicates is thirty-two rows and is getting unwieldy. That is a signal, not a problem with the technique — see the split below.
Move what you can into the schema
The table names a closed set of outcomes, so put that set in the schema as an enum. Now a model that invents an outcome, or that describes what it would do in prose, produces a validation error at the boundary instead of a plausible-looking wrong answer downstream:
{
"type": "object",
"required": ["outcome", "matched_row"],
"properties": {
"outcome": {
"enum": ["tag_and_hold", "escalate_named_am", "billing_queue",
"default_routing", "needs_review"]
},
"matched_row": { "type": "integer", "minimum": 1, "maximum": 8 }
},
"additionalProperties": false
}
The matched_row field is worth the extra token. It makes the model state which rule it believes it applied, which turns debugging from “why did it choose that” into a comparison between two integers. When the outcome is right and the row is wrong, you have found a rule that is being applied for the wrong reason and will break on the next input, which is exactly the class of latent defect a migration otherwise ships silently.
The general treatment of how to lay out a prompt so its constraints land is in prompt structure; the point specific to migration is that anything expressible as an enum should be one before you move, because enum violations are the cheapest possible regression signal.
Compute the branch in code where you can
Look again at the three predicates. Account tier and ticket age are both things your system knows without asking a model — they are fields in your database. Only category requires judgement. So do not ask the model to evaluate them:
// Model call 1: the only genuinely fuzzy predicate.
const { category } = await classify(ticketText); // enum: billing | other
// Everything else is a lookup you already own.
const outcome = route({
category,
tier: account.tier,
ageHours: hoursSince(ticket.createdAt),
});
The table has not disappeared; it has moved into route, where it is unit-testable, deterministic, reviewable by someone who does not know what a prompt is, and immune to the next migration entirely. The rule of thumb: a predicate the model must evaluate is one it can only evaluate from the text. Every other predicate belongs in code, and the number of rows in your prompt table should shrink accordingly.
If the invariant genuinely must stay in the prompt — a safety rule that has to hold whatever else happens — state it once, plainly, as the last thing before the input. Position at the end is the one structural lever that is cheap and reliably helps.
One fixture per row
The table gives you your test suite for free: one fixture per row, asserting the exact outcome value. A branch that stops firing is now a named failing case — “row 1 (billing / enterprise / under 24h) expected tag_and_hold, got escalate_named_am” — rather than a quality discussion.
Add two fixtures the table does not cover: one input that sits exactly on a boundary (a ticket 24 hours old to the minute, which is where off-by-one bugs live) and one that matches no row, asserting needs_review. Then report outcome counts per week in production alongside the fixture results, because the fixtures prove the rule can fire and the counts prove it does.
Where the input genuinely underdetermines the answer — not a rule that was lost, but a question that cannot be answered from what was given — the fix is a different one: see handling underspecified requests, which adds a clarify path rather than a table row.
Top comments (0)