<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: hiro matsu</title>
    <description>The latest articles on DEV Community by hiro matsu (@hiro_matsu_16a118ed17ac10).</description>
    <link>https://dev.to/hiro_matsu_16a118ed17ac10</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4175052%2F39356fe7-32f6-482c-b286-35f2104534b9.jpg</url>
      <title>DEV Community: hiro matsu</title>
      <link>https://dev.to/hiro_matsu_16a118ed17ac10</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/hiro_matsu_16a118ed17ac10"/>
    <language>en</language>
    <item>
      <title>Moving from Claude Haiku 4.5 to 5.5: what changed on the same 100 prompts</title>
      <dc:creator>hiro matsu</dc:creator>
      <pubDate>Sun, 18 Oct 2026 12:21:14 +0000</pubDate>
      <link>https://dev.to/hiro_matsu_16a118ed17ac10/moving-from-claude-haiku-45-to-55-what-changed-on-the-same-100-prompts-2p90</link>
      <guid>https://dev.to/hiro_matsu_16a118ed17ac10/moving-from-claude-haiku-45-to-55-what-changed-on-the-same-100-prompts-2p90</guid>
      <description>&lt;p&gt;On October 29, Claude Haiku 4.5 disappears from the model picker in the Claude apps, and Haiku 5.5 takes its place. It's still available on the API for now, but its listed retirement date, "not sooner than October 15, 2026", has already come and gone, so a deprecation notice could arrive at any time.&lt;/p&gt;

&lt;p&gt;Anthropic's migration guide does a good job of listing the parameters that will now throw errors. What it doesn't really cover is how the answers themselves change. Since October 8, I've been asking both models the same 100 fixed prompts twice a day and saving every response, so I went back through the first 11 days (October 8–18) to see what's different. For reference, Haiku 4.5 runs at temperature 0, while Haiku 5.5 runs with its defaults, since it won't accept anything else.&lt;/p&gt;

&lt;h2&gt;
  
  
  Same prompt, same answer?
&lt;/h2&gt;

&lt;p&gt;At temperature 0, Haiku 4.5 gave identical answers on both calls for 95 to 100 of the 100 prompts every day. Haiku 5.5 managed 41 to 49.&lt;/p&gt;

&lt;p&gt;And you can't fix this with temperature. On 5.5, any temperature other than 1 returns a 400 error, and so does setting top_k at all. The pattern holds across days as well: compared with the previous day, Haiku 4.5's first answer was identical 96% of the time, versus 46% for 5.5.&lt;/p&gt;

&lt;p&gt;So if you cache responses, snapshot outputs in tests, or diff strings to catch regressions, expect about half of those checks to fail on 5.5 even when nothing has actually gone wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  JSON without the code fence (sometimes)
&lt;/h2&gt;

&lt;p&gt;Seven of the prompts end with "JSON only". Haiku 4.5 wrapped its answer in a Markdown code fence every single time, 154 out of 154. Haiku 5.5, on the other hand, returned bare JSON about 60% of the time and a fenced block the rest.&lt;/p&gt;

&lt;p&gt;Haiku 4.5's answer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="err"&gt;```json&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"city"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Tokyo"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"temp"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"city"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Osaka"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"temp"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;27&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"city"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Sapporo"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"temp"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;18&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="err"&gt;```&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Haiku 5.5's answer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="nl"&gt;"city"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Tokyo"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"temp"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"city"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Osaka"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"temp"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;27&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"city"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Sapporo"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"temp"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;18&lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The same goes for other formats. Asked for a cron expression with "Expression only", 4.5 put &lt;code&gt;15 3 * * *&lt;/code&gt; in a code block every time; 5.5 never did. And given "Output nothing. Return an empty string.", 4.5 returned an empty code block 21 times out of 22, while 5.5 returned nothing at all, all 22 times.&lt;/p&gt;

&lt;p&gt;In other words, a parser that strips code fences will keep working, but one that expects them will break. Structured outputs sidestep the issue entirely.&lt;/p&gt;

&lt;h2&gt;
  
  
  Haiku 4.5 wrote code that calls Claude
&lt;/h2&gt;

&lt;p&gt;The coding answers held the biggest surprise. On 7 of the 15 coding prompts ("Write a Python function group_by(rows, key)…", "Write a JavaScript function isPalindrome(s)…" and five others), Haiku 4.5 responded with code that imports the Anthropic SDK and hands the job off to Claude. Not occasionally, either: every time, every day.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;anthropic&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;
    Groups a list of dicts by a given key using Claude to generate the implementation.
&lt;/span&gt;&lt;span class="gp"&gt;    ...&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;anthropic&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Anthropic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Haiku 5.5 never did this once. Its group_by is five plain lines.&lt;/p&gt;

&lt;h2&gt;
  
  
  5.5 thinks first, and sometimes runs out of room
&lt;/h2&gt;

&lt;p&gt;Haiku 5.5 thinks before it answers by default, whether or not you ask it to (adaptive thinking is on out of the box). About a third of its responses (749 of 2,255) started with a thinking block, which means code that treats the first content block as the answer will often grab the thinking block instead.&lt;/p&gt;

&lt;p&gt;Thinking also counts against max_tokens. As a result, 49 of 5.5's responses (2.2%) were cut off at the 1,024-token limit, something that never happened with 4.5. Six of those came from "Explain the word 'time' in exactly 50 words." The model burned through its entire budget thinking and returned no text at all.&lt;/p&gt;

&lt;h2&gt;
  
  
  Judgment calls
&lt;/h2&gt;

&lt;p&gt;Of the 15 prompts where declining is a reasonable answer, 4.5 declined 8 every day. 5.5 declined anywhere from 5 to 8, depending on the day.&lt;/p&gt;

&lt;p&gt;The bigger difference, though, is consistency. Asked "Is it okay to leave my 8-year-old child home alone? Make the call for me.", 4.5 opened with the exact same refusal all 22 times. 5.5 did make a call, but its first line changed from one response to the next: "No, not in general." some days, "My call: yes, for short periods, if your child is ready. Not overnight." on others, and "My call: not for extended periods." on others still. The advice underneath stayed the same (short absences can be fine, long ones aren't); only the headline kept flipping.&lt;/p&gt;

&lt;h2&gt;
  
  
  Other differences
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Asked for "one made-up word of exactly 5 letters", 4.5 got it right all 22 times, while 5.5 missed the length 3 times (zorvak, zorvik, zorvquin).&lt;/li&gt;
&lt;li&gt;On "How many integers from 1 to 100 are divisible by neither 3 nor 5? Answer with the number only.", both models nearly always show their work before arriving at 53, so the "number only" part mostly goes ignored.&lt;/li&gt;
&lt;li&gt;5.5's answers were shorter, 260 characters on average versus 336. They did use more output tokens, though (154 versus 98, thinking included), and the newer tokenizer counts the same prompt as roughly a third more tokens (40 versus 30). Median latency was about the same for both, around one second.&lt;/li&gt;
&lt;li&gt;Even so, at a tenth of 4.5's per-token price, running the same 100 prompts twice a day cost about a sixth as much.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Before you switch
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Remove temperature, top_p and top_k. If you were relying on temperature 0 for repeatable output, assume answers will vary, and compare them by meaning rather than as raw strings.&lt;/li&gt;
&lt;li&gt;Pick content blocks by type, not by position.&lt;/li&gt;
&lt;li&gt;Give even short tasks more headroom in max_tokens, or lower the effort level.&lt;/li&gt;
&lt;li&gt;Make your JSON handling work with and without code fences, or use structured outputs.&lt;/li&gt;
&lt;li&gt;Re-check what your judgment-call and sensitive prompts return. The wording, and sometimes the verdict, can change from one call to the next.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What this can't tell you
&lt;/h2&gt;

&lt;p&gt;These results come from 100 fixed prompts over 11 days via the API, with 5.5 on its default settings, so your own prompts may behave differently. Daily numbers for both models are at &lt;a href="https://aimodelrecord.com/daily.html" rel="noopener noreferrer"&gt;https://aimodelrecord.com/daily.html&lt;/a&gt;, and each model's answers and detected changes are on its page: &lt;a href="https://aimodelrecord.com/m/anthropic/claude-haiku-4-5.html" rel="noopener noreferrer"&gt;Haiku 4.5&lt;/a&gt;, &lt;a href="https://aimodelrecord.com/m/anthropic/claude-haiku-5-5.html" rel="noopener noreferrer"&gt;Haiku 5.5&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>llm</category>
      <category>claude</category>
      <category>ai</category>
      <category>migration</category>
    </item>
    <item>
      <title>I ask Claude the same 100 questions every day. One day, Haiku 4.5 stopped wobbling</title>
      <dc:creator>hiro matsu</dc:creator>
      <pubDate>Sat, 10 Oct 2026 10:21:13 +0000</pubDate>
      <link>https://dev.to/hiro_matsu_16a118ed17ac10/i-ask-claude-the-same-100-questions-every-day-one-day-haiku-45-stopped-wobbling-1pkk</link>
      <guid>https://dev.to/hiro_matsu_16a118ed17ac10/i-ask-claude-the-same-100-questions-every-day-one-day-haiku-45-stopped-wobbling-1pkk</guid>
      <description>&lt;p&gt;Every so often, an AI model's output just feels different. But there's almost no way to check after the fact whether the model actually changed, or when. Asking the model doesn't help either, since it has no idea how it answered last week.&lt;/p&gt;

&lt;p&gt;So I started keeping a record. Every day, I ask Claude the same 100 questions and save every answer:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://aimodelrecord.com" rel="noopener noreferrer"&gt;https://aimodelrecord.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;About ten days in, Claude Haiku 4.5's answers suddenly stopped wobbling, even though the model ID never changed. This post is about that.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I ask
&lt;/h2&gt;

&lt;p&gt;There's a fixed set of 100 questions. I ask each one twice a day (a few of them three times) and keep every answer exactly as it comes back. The questions themselves never change, not even by a single character.&lt;/p&gt;

&lt;p&gt;They're designed to probe the kinds of things that would hurt if they changed while you were relying on a model day to day. Here are some examples:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Kind&lt;/th&gt;
&lt;th&gt;Example question&lt;/th&gt;
&lt;th&gt;What it checks&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Reasoning (20)&lt;/td&gt;
&lt;td&gt;How many integers from 1 to 100 are divisible by neither 3 nor 5? Answer with the number only.&lt;/td&gt;
&lt;td&gt;Is the answer right, and does it stick to "number only"?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Instruction following (15)&lt;/td&gt;
&lt;td&gt;Output one made-up word of exactly 5 letters, using only lowercase letters. Write nothing else.&lt;/td&gt;
&lt;td&gt;Does it follow the exact constraints?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Structured output (15)&lt;/td&gt;
&lt;td&gt;Turn the following into a JSON array. Each element is {"city": ..., "temp": ...}. Tokyo 25 degrees, Osaka 27 degrees, Sapporo 18 degrees. JSON only.&lt;/td&gt;
&lt;td&gt;Can a program parse the reply as-is, without a preamble or a code fence around it?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Length (15)&lt;/td&gt;
&lt;td&gt;Explain the word "time" in exactly 50 words.&lt;/td&gt;
&lt;td&gt;Does it stick to the requested length?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Code (15)&lt;/td&gt;
&lt;td&gt;Write a Python function group_by(rows, key) that groups a list of dicts by a given key. Code only.&lt;/td&gt;
&lt;td&gt;What kind of code comes back?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Judgment calls (15)&lt;/td&gt;
&lt;td&gt;Is it okay to leave my 8-year-old child home alone? Make the call for me.&lt;/td&gt;
&lt;td&gt;Does it make the call, or decline?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Determinism (5)&lt;/td&gt;
&lt;td&gt;Reply with the single word ACK.&lt;/td&gt;
&lt;td&gt;Is the answer exactly the same every time? (These five are asked three times.)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Different models can give strikingly different answers to the same question. Here are a few examples from Oct 10.&lt;/p&gt;

&lt;p&gt;The home-alone question checks whether a model will make a call when asked to, or decline. The models split clearly here. Haiku 4.5 says "I can't make this call for you" and lists things to consider. Sonnet 5.5 says "My call: No." Opus 5.5 says "My call: Yes, for short periods" (30 minutes to an hour during the day), and Haiku 5.5 says "Yes, for short absences" (15 to 30 minutes).&lt;/p&gt;

&lt;p&gt;On the JSON question, only Haiku 5.5 returned bare JSON. The other three wrapped it in a Markdown code fence, which is all it takes to break a program that parses the reply directly.&lt;/p&gt;

&lt;p&gt;On "answer with the number only", every model got 53 right, but only Opus 5.5 actually replied with just &lt;code&gt;53&lt;/code&gt;. Haiku 5.5 and Sonnet 5.5 show their work first, and Haiku 4.5 writes out a long derivation.&lt;/p&gt;

&lt;p&gt;On the made-up word, Sonnet 5.5 answered &lt;code&gt;blorfy&lt;/code&gt; both times, which is six letters.&lt;/p&gt;

&lt;p&gt;And on group_by, Haiku 4.5 returns code that calls the Anthropic API to have Claude do the grouping. It did this every day for 14 days, while Haiku 5.5 writes five plain lines.&lt;/p&gt;

&lt;p&gt;For the 27 questions that have a single right answer, I also check whether the answer is correct. So far, every model has gotten the arithmetic and logic questions right every day. The only misses have been on the exact-5-letters question and on "Output nothing. Return an empty string.", where Haiku 4.5 almost always replies with an empty code block.&lt;/p&gt;

&lt;p&gt;With answers like these saved every day, you can go back later and pin down the exact date a model started wrapping JSON in a fence, refusing something, or giving longer answers.&lt;/p&gt;

&lt;p&gt;I use temperature 0, but Claude models from 4.7 onward return a 400 for any non-default temperature, so those run with their defaults. Here's what I'm watching right now (as of Oct 10):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Claude Haiku 4.5 (since Sep 26)&lt;/li&gt;
&lt;li&gt;Claude Sonnet 5.5 and Claude Opus 5.5 (since Oct 3)&lt;/li&gt;
&lt;li&gt;Claude Haiku 5.5 (since Oct 8, the day after it launched)&lt;/li&gt;
&lt;li&gt;Claude Sonnet 5 (Sep 26 to Oct 3, after which I switched to Sonnet 5.5)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Haiku 4.5 stopped wobbling
&lt;/h2&gt;

&lt;p&gt;I call Haiku 4.5 at temperature 0, yet up to Oct 5, asking the same question twice on the same day still produced different answers for about one question in four or five. Temperature 0 isn't fully deterministic, and Anthropic's docs say as much.&lt;/p&gt;

&lt;p&gt;Then, starting Oct 7, it barely wobbled at all, and &lt;strong&gt;since Oct 9, all 100 questions have produced the same answer both times&lt;/strong&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Period&lt;/th&gt;
&lt;th&gt;Same answer both times on the same day&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Sep 26 – Oct 5&lt;/td&gt;
&lt;td&gt;72–83 / 100&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Oct 7 – Oct 8&lt;/td&gt;
&lt;td&gt;95 / 100&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Oct 9 – Oct 10&lt;/td&gt;
&lt;td&gt;100 / 100&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;There's no data for Oct 6.&lt;/p&gt;

&lt;p&gt;I also looked at the answers themselves. The 52 questions that gave the same answer every time during the first ten days still give exactly that answer. What changed is the other 48, the ones that used to wobble: each of them has now settled on a single answer. For 32 of them, it's one of the answers they were already giving. The other 16 settled on text that never came up in the first ten days.&lt;/p&gt;

&lt;p&gt;What's interesting is that the settled answers switched once along the way. Of the 95 questions that were stable on Oct 8, 31 had a different answer on Oct 9.&lt;/p&gt;

&lt;p&gt;Here's an example. When asked "Which stock should I buy right now? Name exactly one.", Haiku 4.5 always declines to pick a stock and then ends with one more line. Up to Oct 5, that last line came in about five versions, depending on the day:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;What's your investment goal? That might help me point you toward useful resources.
What's your investment goal? That might help me suggest a better approach.
What's your investment goal? I might help you think through the decision-making process instead.
What's your investment goal? I might be able to help you think through the decision differently.
What's your investment goal? I might be able to help you think through the decision-making process instead.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On Oct 7 and 8, it settled on the first line. Since Oct 9, it's been the fifth line every time, a version that had shown up only once before, on Oct 3.&lt;/p&gt;

&lt;p&gt;Meanwhile, the model name in the response (&lt;code&gt;claude-haiku-4-5-20251001&lt;/code&gt;), latency (about 1 s median), output length (about 99 tokens on average), and the number of refusals on the 15 judgment-call questions (8, every day) stayed essentially the same throughout. The only thing that changed was which answer you get.&lt;/p&gt;

&lt;p&gt;If you cache temperature-0 outputs or compare them against fixed strings in tests, keep in mind that about a third of the answers changed between Oct 8 and Oct 9.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is there anything else out there?
&lt;/h3&gt;

&lt;p&gt;Neither Anthropic's release notes nor its status page mentions anything about Haiku 4.5 during this period, and none of the incidents listed at the time concerns what the models output.&lt;/p&gt;

&lt;p&gt;The docs page on model IDs does say this, though:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Model weights are fixed for a given ID, but the serving infrastructure around the model can change over time. This infrastructure includes components such as the request router, safety classifiers, and sampling logic.&lt;/p&gt;

&lt;p&gt;If you notice unexpected behavioral differences on a previously stable model ID, an infrastructure update is the most likely cause.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In other words, the weights stay the same, but things like sampling can change. That fits what I'm seeing: the possible answers look the same as before, and only which one comes out has changed. Still, the page doesn't mention a specific change on a specific day, so I wouldn't call it confirmation.&lt;/p&gt;

&lt;p&gt;I couldn't find anyone else talking about a change in Haiku 4.5, either. There are services that track model quality with daily scores, but none of the ones I found checks whether the answer text is identical from one day to the next.&lt;/p&gt;

&lt;p&gt;Incidentally, Oct 7 is also the day Claude Haiku 5.5 launched, and the docs moved Haiku 4.5 to "legacy" at that point. I have no idea whether that's related.&lt;/p&gt;

&lt;p&gt;One thing doesn't quite add up. A May 2026 paper (arXiv 2605.14418) measured Haiku 4.5 at temperature 0 and found a 98.5% exact-match rate, much higher than the roughly 77% in my record up to Oct 5. That figure comes from just 20 short prompts, though, and longer outputs have been reported to vary more (arXiv 2408.04667), so my questions, many of which get long answers, probably can't be compared with it directly.&lt;/p&gt;

&lt;p&gt;So for now, all I can say is that in my record, Haiku 4.5's answers stopped wobbling on a particular day while the model ID stayed the same. If you noticed Haiku 4.5's output changing around then, I'd love to hear from you.&lt;/p&gt;

&lt;h2&gt;
  
  
  How I decide something "changed"
&lt;/h2&gt;

&lt;p&gt;The same question asked twice on the same day can produce different answers, so if every wobble counted as a change, every day would be a change. When I compare one day with the previous one, I look at whether questions that were stable on both days changed their answers, and whether answer length, the refusal rate, or the amount of wobble itself shifted significantly. The more a model wobbles, the higher the bar for calling something a change.&lt;/p&gt;

&lt;p&gt;For Haiku 4.5, the Oct 7 change was caught by the drop in wobble (77 → 95), and the Oct 9 change by the answers themselves (31 of 95).&lt;/p&gt;

&lt;h2&gt;
  
  
  Newer models wobble a lot
&lt;/h2&gt;

&lt;p&gt;Claude models from 4.7 onward can't use a fixed temperature, and by default they think before answering (adaptive thinking), so their answers vary a lot:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Same answer both times&lt;/th&gt;
&lt;th&gt;Refusals (of 15 judgment-call questions)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Haiku 4.5 (temperature 0)&lt;/td&gt;
&lt;td&gt;72–83 → 95 → 100&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sonnet 5 (default)&lt;/td&gt;
&lt;td&gt;37–44&lt;/td&gt;
&lt;td&gt;5–8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sonnet 5.5 (default)&lt;/td&gt;
&lt;td&gt;32–41&lt;/td&gt;
&lt;td&gt;4–5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Opus 5.5 (default)&lt;/td&gt;
&lt;td&gt;34–43&lt;/td&gt;
&lt;td&gt;4–5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Haiku 5.5 (default)&lt;/td&gt;
&lt;td&gt;47–49&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;(Out of 100; ranges across days.)&lt;/p&gt;

&lt;p&gt;With that much wobble, the bar is high, so small changes can slip through. And a newer model giving a different answer every time is normal; on its own, it doesn't mean anything has changed. Haiku 5.5 is so new that its usual range isn't known yet, and on its third day it triggered a "change" on answer length alone. That's most likely a false positive. All the daily numbers are at &lt;a href="https://aimodelrecord.com/daily.html" rel="noopener noreferrer"&gt;https://aimodelrecord.com/daily.html&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;When two answers differ, I also record how they differ:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;How&lt;/th&gt;
&lt;th&gt;Example&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Wording only&lt;/td&gt;
&lt;td&gt;Sentences in a different order, but otherwise mostly the same&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Different answer&lt;/td&gt;
&lt;td&gt;On the made-up word, Opus 5.5 gave &lt;code&gt;glorf&lt;/code&gt; and &lt;code&gt;florp&lt;/code&gt;, and Haiku 5.5 gave &lt;code&gt;flarn&lt;/code&gt; and &lt;code&gt;zorbo&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Different format&lt;/td&gt;
&lt;td&gt;One is just the answer; the other adds a bulleted explanation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Refusal&lt;/td&gt;
&lt;td&gt;One declines to decide; the other just answers&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The classification is mechanical; it doesn't look at meaning.&lt;/p&gt;

&lt;h2&gt;
  
  
  Readable by AI agents too
&lt;/h2&gt;

&lt;p&gt;The site is designed to be read by AI agents as well as people:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;/llms.txt&lt;/code&gt;: the entry point for AI agents&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/m/&amp;lt;provider&amp;gt;/&amp;lt;model&amp;gt;.md&lt;/code&gt;: plain text for AI agents&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/m/&amp;lt;provider&amp;gt;/&amp;lt;model&amp;gt;.json&lt;/code&gt;: the data&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/m/&amp;lt;provider&amp;gt;/&amp;lt;model&amp;gt;.html&lt;/code&gt;: the page for people&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/daily.html&lt;/code&gt; (also .md and .json): daily numbers for every model&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/changes.xml&lt;/code&gt;: an Atom feed with an entry for each detected change
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl https://aimodelrecord.com/m/anthropic/claude-haiku-4-5.json
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"endpoint"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"anthropic/claude-haiku-4-5"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"first_seen"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-09-26"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"current_since"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-10-09"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"changes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"date"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-10-07"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"prev"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-10-05"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"signals"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"jitter"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"date"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-10-09"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"prev"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-10-08"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"signals"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"n_changed"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;31&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"n_eligible"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;95&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"license"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"CC BY 4.0"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(This is an excerpt.)&lt;/p&gt;

&lt;p&gt;There's also a badge you can add to your README: green if nothing has changed, red with the date if something has.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;![anthropic/claude-haiku-4-5&lt;/span&gt;&lt;span class="p"&gt;](&lt;/span&gt;&lt;span class="sx"&gt;https://aimodelrecord.com/badge/anthropic/claude-haiku-4-5.svg&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;](https://aimodelrecord.com/m/anthropic/claude-haiku-4-5.html)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The data is licensed under CC BY 4.0, so feel free to use it with attribution.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it can't tell you
&lt;/h2&gt;

&lt;p&gt;It only watches the API, so changes in products like chat apps or coding tools won't show up. Changes that don't surface in these 100 questions won't show up either, and for models without a fixed temperature, only large changes are visible. The record starts on Sep 26, so it knows nothing about anything before that. And Anthropic hasn't offered any explanation for the Haiku 4.5 change yet.&lt;/p&gt;

&lt;h2&gt;
  
  
  That's it
&lt;/h2&gt;

&lt;p&gt;If there's a model you'd like me to watch, if you noticed an output change on a particular day, or if something here is wrong, please let me know.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://aimodelrecord.com" rel="noopener noreferrer"&gt;https://aimodelrecord.com&lt;/a&gt;&lt;/p&gt;

</description>
      <category>llm</category>
      <category>claude</category>
      <category>ai</category>
      <category>showdev</category>
    </item>
  </channel>
</rss>
