Step one classifies, step two extracts, step three writes the record. You changed the model behind step one, the prompt is byte-identical, and step two now throws on roughly one call in nine. The exception is almost always one of four, and each one tells you which formatting default moved.
The errors you will actually see
-
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)— the response no longer starts with{. Either a fence appeared where there was none, or the model wrote a sentence of preamble first. -
SyntaxError: Unexpected token ` in JSON at position 0— the Node form of the same thing, and the backtick names the cause outright: the model wrapped the object in a triple-backtick fence. -
json.decoder.JSONDecodeError: Extra data: line 12 column 1 (char 431)— the object parsed and then something followed it. The model appended an explanation after the JSON, which the old one did not do. -
IndexError: list index out of rangefrom your own extractor — the regex that used to find the fence found nothing, because this model stopped emitting fences. This is the same defect as the first, inverted, and it is the one that catches teams migrating towards a terser model.
A fifth is quieter and worse: no exception at all, because the chain parses successfully and the field it wanted is now nested one level deeper, or the numeric field arrived as "1,240" with a thousands separator, or a mathematical expression came back wrapped in LaTeX delimiters. Nothing throws; the record is wrong.
Why the format changed when the prompt did not
Because the prompt never specified the format. It specified the content, and the format came from post-training — the fourth of the four layers described in the style guide migration page. Your chain works by parsing a habit.
Three habits do most of the damage. The first is fencing: whether the model volunteers a `json wrapper around structured output when not asked. The second is preamble and postamble: whether it narrates before producing the payload and explains afterwards. Anthropic documents both directions of this for its own models — current models are described as less verbose and may skip summaries after tool calls, which is a habit disappearing rather than appearing. The third is notation: the same documentation says current models default to LaTeX for mathematical expressions unless instructed otherwise, which turns a field your chain reads as a number into $1{,}240$.
None of these is a defect on either side. They are defaults, and your chain has an undeclared dependency on one particular set of them.
Patching the extractor
The immediate fix is to make extraction format-agnostic, and the almost-universal first attempt is wrong. A regex like /\{.*\}/s is greedy across the whole response and will swallow a trailing explanation that happens to contain a brace; the non-greedy version stops at the first inner object. Neither handles a brace inside a string value. Scan for balance instead, and ignore braces inside quoted strings:
`python
def first_json_object(text: str) -> str:
"""Return the first complete top-level JSON object in text.
Tolerates fenced blocks, preamble and postamble. Brace counting
is string-aware, so a "}" inside a value does not close the object.
"""
start = text.find("{")
if start == -1:
raise ValueError(f"no JSON object in response: {text[:120]!r}")
depth, in_string, escaped = 0, False, False
for i, ch in enumerate(text[start:], start):
if in_string:
if escaped:
escaped = False
elif ch == "\\":
escaped = True
elif ch == '"':
in_string = False
continue
if ch == '"':
in_string = True
elif ch == "{":
depth += 1
elif ch == "}":
depth -= 1
if depth == 0:
return text[start : i + 1]
raise ValueError("unterminated JSON object in response")
`
Two rules go with it. Raise an error that contains the first 120 characters of the offending response, because the single most expensive part of this failure is not being able to see what the model actually said. And log the extraction path taken — fenced, bare, preamble-stripped — as a counter, because the distribution of that counter is your early warning that a provider changed something. That is the same signal the library discusses under silent model updates.
Making the format a contract, not a habit
The extractor patch buys you the afternoon. The real fix is to stop parsing prose. Every current provider offers at least one mechanism that moves the format from the model’s discretion into the request: a JSON-schema-constrained response format, or a tool definition whose arguments are the structure you want. Both are described in the library’s JSON mode versus structured outputs page, and the choice between them is exactly the subject of function calling versus structured output.
What matters for a migration is that these mechanisms are not equivalent across providers even when both exist. Schema support differs in which JSON Schema keywords are honoured, whether constraints are enforced by constrained decoding or merely requested, and whether the response can still be truncated mid-object when the token limit is reached. A chain that moves from a strictly enforced schema to a best-effort one has silently reacquired the same class of bug, which is why the extractor above should stay in place even after the contract is added. Keep it, and alert when it has to do any work at all.
The middle step, if a schema is unavailable for that call, is to state the format positively in the prompt rather than forbidding alternatives. “Respond with a single JSON object and no other text” outperforms “do not use code fences”, for the same documented reason positive formatting instructions generally outperform negative ones.
Where else a chain leaks format
Extraction is the seam everyone finds because it throws. Three more leak silently and are worth auditing in the same pass.
- Few-shot examples written in the old house style. A chain step whose examples are formatted the way the old model liked is now teaching the new model a style neither of you wants — and because prompt formatting influences output formatting, the examples are a stronger instruction than the sentence above them.
- Stop sequences tuned to a habit. A stop sequence of
` works perfectly until the model stops emitting fences, at which point generation runs to the token limit and your latency and cost both jump with no error anywhere. - Length assumptions between steps. If step two truncates step one’s output to fit a window, a model with a different default verbosity changes what survives the truncation. That is a content bug wearing a formatting bug’s clothes, and it belongs with output length expectations.
Fix all four in one commit and add a contract test between every pair of chain steps that asserts on the schema rather than on a golden string. The next migration then fails in CI instead of in production.
One ordering note, because it decides how long the incident lasts. Fix the extractor first and ship it, then add the contract, then audit the silent seams. The extractor stops the bleeding without requiring a decision about schemas or tool definitions, and it stays useful afterwards as the layer that tells you when the contract itself has stopped being honoured. Doing it in the other order means arguing about structured output support while production keeps throwing.
Top comments (0)