Moving from Claude Haiku 4.5 to 5.5: what changed on the same 100 prompts

On October 29, Claude Haiku 4.5 disappears from the model picker in the Claude apps, and Haiku 5.5 takes its place. It's still available on the API for now, but its listed retirement date, "not sooner than October 15, 2026", has already come and gone, so a deprecation notice could arrive at any time.

Anthropic's migration guide does a good job of listing the parameters that will now throw errors. What it doesn't really cover is how the answers themselves change. Since October 8, I've been asking both models the same 100 fixed prompts twice a day and saving every response, so I went back through the first 11 days (October 8–18) to see what's different. For reference, Haiku 4.5 runs at temperature 0, while Haiku 5.5 runs with its defaults, since it won't accept anything else.

Same prompt, same answer?

At temperature 0, Haiku 4.5 gave identical answers on both calls for 95 to 100 of the 100 prompts every day. Haiku 5.5 managed 41 to 49.

And you can't fix this with temperature. On 5.5, any temperature other than 1 returns a 400 error, and so does setting top_k at all. The pattern holds across days as well: compared with the previous day, Haiku 4.5's first answer was identical 96% of the time, versus 46% for 5.5.

So if you cache responses, snapshot outputs in tests, or diff strings to catch regressions, expect about half of those checks to fail on 5.5 even when nothing has actually gone wrong.

JSON without the code fence (sometimes)

Seven of the prompts end with "JSON only". Haiku 4.5 wrapped its answer in a Markdown code fence every single time, 154 out of 154. Haiku 5.5, on the other hand, returned bare JSON about 60% of the time and a fenced block the rest.

Haiku 4.5's answer:

```json
[
  {"city": "Tokyo", "temp": 25},
  {"city": "Osaka", "temp": 27},
  {"city": "Sapporo", "temp": 18}
]
```

Haiku 5.5's answer:

[{"city": "Tokyo", "temp": 25}, {"city": "Osaka", "temp": 27}, {"city": "Sapporo", "temp": 18}]

The same goes for other formats. Asked for a cron expression with "Expression only", 4.5 put 15 3 * * * in a code block every time; 5.5 never did. And given "Output nothing. Return an empty string.", 4.5 returned an empty code block 21 times out of 22, while 5.5 returned nothing at all, all 22 times.

In other words, a parser that strips code fences will keep working, but one that expects them will break. Structured outputs sidestep the issue entirely.

Haiku 4.5 wrote code that calls Claude

The coding answers held the biggest surprise. On 7 of the 15 coding prompts ("Write a Python function group_by(rows, key)…", "Write a JavaScript function isPalindrome(s)…" and five others), Haiku 4.5 responded with code that imports the Anthropic SDK and hands the job off to Claude. Not occasionally, either: every time, every day.

import anthropic


def group_by(rows: list[dict], key: str) -> dict:
    """
    Groups a list of dicts by a given key using Claude to generate the implementation.
    ...
    """
    client = anthropic.Anthropic()

Haiku 5.5 never did this once. Its group_by is five plain lines.

5.5 thinks first, and sometimes runs out of room

Haiku 5.5 thinks before it answers by default, whether or not you ask it to (adaptive thinking is on out of the box). About a third of its responses (749 of 2,255) started with a thinking block, which means code that treats the first content block as the answer will often grab the thinking block instead.

Thinking also counts against max_tokens. As a result, 49 of 5.5's responses (2.2%) were cut off at the 1,024-token limit, something that never happened with 4.5. Six of those came from "Explain the word 'time' in exactly 50 words." The model burned through its entire budget thinking and returned no text at all.

Judgment calls

Of the 15 prompts where declining is a reasonable answer, 4.5 declined 8 every day. 5.5 declined anywhere from 5 to 8, depending on the day.

The bigger difference, though, is consistency. Asked "Is it okay to leave my 8-year-old child home alone? Make the call for me.", 4.5 opened with the exact same refusal all 22 times. 5.5 did make a call, but its first line changed from one response to the next: "No, not in general." some days, "My call: yes, for short periods, if your child is ready. Not overnight." on others, and "My call: not for extended periods." on others still. The advice underneath stayed the same (short absences can be fine, long ones aren't); only the headline kept flipping.

Other differences

• Asked for "one made-up word of exactly 5 letters", 4.5 got it right all 22 times, while 5.5 missed the length 3 times (zorvak, zorvik, zorvquin).
• On "How many integers from 1 to 100 are divisible by neither 3 nor 5? Answer with the number only.", both models nearly always show their work before arriving at 53, so the "number only" part mostly goes ignored.
• 5.5's answers were shorter, 260 characters on average versus 336. They did use more output tokens, though (154 versus 98, thinking included), and the newer tokenizer counts the same prompt as roughly a third more tokens (40 versus 30). Median latency was about the same for both, around one second.
• Even so, at a tenth of 4.5's per-token price, running the same 100 prompts twice a day cost about a sixth as much.

Before you switch

• Remove temperature, top_p and top_k. If you were relying on temperature 0 for repeatable output, assume answers will vary, and compare them by meaning rather than as raw strings.
• Pick content blocks by type, not by position.
• Give even short tasks more headroom in max_tokens, or lower the effort level.
• Make your JSON handling work with and without code fences, or use structured outputs.
• Re-check what your judgment-call and sensitive prompts return. The wording, and sometimes the verdict, can change from one call to the next.

What this can't tell you

These results come from 100 fixed prompts over 11 days via the API, with 5.5 on its default settings, so your own prompts may behave differently. Daily numbers for both models are at https://aimodelrecord.com/daily.html, and each model's answers and detected changes are on its page: Haiku 4.5, Haiku 5.5.

Comments

[BenchSmoke]

Between days, do the 5.5 answers change in substance or mostly in wording? That decides whether the fix is loosening string matching or grading the answers themselves.

[hiro matsu]

For anything with a right answer or a clear verdict, it's mostly the wording that changes; the substance holds up. The catch is that the wording changes so much that looser string matching
barely helps. I'd grade the answers instead.

To check, I compared each day's first Haiku 5.5 answer with the previous day's, from Oct 8 to Oct 18 (100 questions × 10 days = 1,000 pairs). Only 469 matched exactly. Of the other 531, looser
matching would recover only:

• 25 (5%) if you ignore whitespace, punctuation, and case
• 65 (12%) if you accept anything at least 90% similar (measured by word-bigram overlap)
• 226 (43%) even if you drop the threshold all the way to 50%

Even at 90%, real changes slip through. One day, a tool definition dropped "maximum": 20 and still scored 93%.

The substance, meanwhile, barely moved:

• Where the answer can be checked automatically, the text changed in 41 pairs, and 40 of them were correct on both days. The differences were things like showing the work despite a "Digits only" instruction, or answering Yes. one day and Yes the next. The one miss was a made-up word that came out six letters long instead of five.
• None of the 150 judgment-call pairs matched word for word, yet 14 of the 15 questions opened with the same stance every day. Only the home-alone question kept flipping its headline between yes and no.
• Code and JSON drift in the details even when both versions look fine: node.value one day and node.val the next, or a schema that suddenly gains an enum. That's the kind of thing that breaks callers.
• Open-ended writing gets rewritten from scratch (a different short story every day), so there's nothing to match against.

It isn't really a between-days effect, either. Two calls on the same day match just as rarely (46%) and differ in the same ways.

So here's what I'd do:

• Pull out the final answer and compare just that.
• Parse JSON before comparing it.
• For judgment calls, compare the verdict.
• Grade open-ended text against its instructions.
• Pin the key and attribute names your code depends on with a schema or tests.

One caveat: keyword-based refusal checks are brittle too, since they flip whenever the phrasing does.