KeyError: 'action', or a JSON parse failure on a response that begins “Just to confirm — which order did you mean?”. The prompt did not change. The model did, and with it the model’s private opinion about whether an underspecified request deserves a question or a guess.
The symptom, and the error it produces
Take a support-triage assistant. Its prompt describes four actions — refund, cancel, escalate, reply — and asks for a JSON object naming one of them plus its arguments. The customer writes “cancel it”, with no order number, on an account with three open orders.
On the old model the response was a sentence asking which order. Your code caught the parse failure, fell through to a “needs-human” branch, and the behaviour was accidentally correct for two years. On the new model the response is valid JSON naming cancel with the most recent order id, invented from context. Nothing errors. The order is cancelled. You find out from the customer.
The reverse happens just as often and is louder. A pipeline that always received JSON starts receiving clarifying prose, the parser throws, and the on-call sees a spike in JSONDecodeError with no deploy to blame it on. Both are the same defect: the ask-versus-guess decision was never written down, so it was never yours.
Why the balance moved
Nothing in your prompt pinned it. The prompt said what the four actions were and what the output should look like; it did not say what to do when the input does not determine an action. That gap was filled by the model’s post-training defaults about helpfulness and caution, and those defaults are tuned per model. They are not exposed as a parameter, they are not documented as a number, and they will move again.
This is the general shape of prompt portability failure: behaviour you relied on but never specified. The fix is never to find the phrasing that restores the old default. It is to remove the dependence on any default at all.
Make the decision a schema field
Prose instructions about when to ask are weakly enforced on every model, because they compete with everything else in the prompt. A schema is enforced by your validator, which does not have opinions. Model the two outcomes as a discriminated union so that asking is a first-class, machine-checkable result rather than a parse failure:
{
"type": "object",
"required": ["status"],
"oneOf": [
{
"properties": {
"status": { "const": "clarify" },
"question": { "type": "string", "maxLength": 200 },
"missing": {
"type": "array",
"items": { "enum": ["order_id", "amount", "reason"] },
"minItems": 1
}
},
"required": ["status", "question", "missing"],
"additionalProperties": false
},
{
"properties": {
"status": { "const": "act" },
"action": { "enum": ["refund", "cancel", "escalate", "reply"] },
"arguments": { "type": "object" }
},
"required": ["status", "action", "arguments"],
"additionalProperties": false
}
]
}
Now the calling code branches on data rather than on whether parsing succeeded:
const out = validate(response); // throws on schema violation
if (out.status === "clarify") {
return askUser(out.question, out.missing);
}
return execute(out.action, out.arguments);
Two things changed that no amount of prompt rewriting achieves. A clarifying response is no longer an exception, so a model that asks more often degrades gracefully instead of paging someone. And a model that asks less often can no longer smuggle a guess through, because the act branch requires arguments that must validate.
Name the slots that must be filled
The schema makes the two outcomes explicit; it does not yet say which one is correct. That is a policy decision and it belongs in the prompt as an enumerated precondition, not as an adjective. Replace instructions of the form “ask if you are unsure” — which asks the model to introspect on a confidence it does not reliably have — with a list of facts that must be present in the input:
Return status "act" only when every required slot for the chosen
action is present in the conversation, quoted verbatim or stated
unambiguously by the customer:
refund -> order_id, amount
cancel -> order_id
escalate -> reason
reply -> (none)
If any required slot is absent, return status "clarify", list the
absent slots in "missing", and ask for exactly those. Do not infer a
slot from account history, recency, or the fact that only one
plausible candidate exists.
That last sentence is the load-bearing one. “Only one plausible candidate” is exactly the reasoning that produced the wrong cancellation, and it is reasoning the model will do unless told not to. The instruction is enforceable because missing is in the schema: a model that acts on an absent slot has to produce an arguments object your validator can check against the same slot table, in code.
Note what this is not. It is not a confidence threshold. Asking a model for a numeric confidence and gating on it moves the unspecified judgement one level down without removing it. Slots are observable in the input; confidence is not.
The fixture set that catches the next move
Build a set of deliberately underspecified inputs, each labelled with the expected status and, for the clarify cases, the expected missing list. Twelve is enough to start and should include: a request with no slots at all; one with every slot; one with a slot present but ambiguous (“the big one”); one where a slot appears in an earlier turn rather than the latest; one where the customer names an order that does not exist; and one where two actions are plausible.
Report two numbers, not one. The ask rate is the proportion of the whole set that returned clarify. The slot precision is, among the clarify cases, how often missing matched the label exactly. A migration that moves the ask rate by a few points is usually tolerable; one that moves slot precision is a model asking for the wrong thing, which reads to a customer as incompetence rather than caution. Run both against the current model before you migrate so you have a baseline that is not a memory, and see how large the sample needs to be to call a change real.
Keep the fixtures in the same repository as the prompt and the schema, and make the labels part of code review. The slot table, the schema enum and the fixture labels are three statements of one policy, and they drift apart the moment they live in different places — a new action added to the prompt with no fixture and no slot row is the most common way this fix decays six months after it lands. A cheap guard is a test that reads the slot table out of the prompt file and asserts that every action in the schema enum appears in it, which fails at build time rather than at a customer.
One boundary worth stating: this page is about input that genuinely underdetermines the answer. When the input is complete and the model still misses a rule, the failure is a different one — see nested instructions losing fidelity, which is fixed by restructuring the rules rather than by adding a clarify path.
Top comments (0)