<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Lukas Babaliauskas</title>
    <description>The latest articles on DEV Community by Lukas Babaliauskas (@lukas_babaliauskas).</description>
    <link>https://dev.to/lukas_babaliauskas</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4223055%2Fde45fcb6-70bd-4eb2-8bf1-0b9648b24265.png</url>
      <title>DEV Community: Lukas Babaliauskas</title>
      <link>https://dev.to/lukas_babaliauskas</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/lukas_babaliauskas"/>
    <language>en</language>
    <item>
      <title>How to replay agent traces against a different model</title>
      <dc:creator>Lukas Babaliauskas</dc:creator>
      <pubDate>Tue, 17 Nov 2026 22:23:33 +0000</pubDate>
      <link>https://dev.to/lukas_babaliauskas/how-to-replay-agent-traces-against-a-different-model-33ka</link>
      <guid>https://dev.to/lukas_babaliauskas/how-to-replay-agent-traces-against-a-different-model-33ka</guid>
      <description>&lt;p&gt;To replay recorded agent traces against a different model, use teacher-forced replay. Record each production run in full, including every tool result. Then send the new model the recorded context one round at a time, execute no tools, and compare the tool calls it makes in each round with the ones the original model made.&lt;/p&gt;

&lt;p&gt;This post covers why rerunning the agent does not work, the method step by step, a short Python script that does it, what to compare, and what the method cannot tell you.&lt;/p&gt;

&lt;p&gt;A disclosure: I build EvalShift, a tool for testing model changes on agents. It has a short section at the end, and everything before it works without it. This post is adapted from the trace-replay section of my longer post on &lt;a href="https://www.evalshift.dev/blog/llm-drift" rel="noopener noreferrer"&gt;LLM drift&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two kinds of replay
&lt;/h2&gt;

&lt;p&gt;"Replay" means two different things for agents, and they are opposites.&lt;/p&gt;

&lt;p&gt;In &lt;strong&gt;deterministic replay&lt;/strong&gt;, the model's recorded responses are served back, so the agent's code follows the same path again. It is a debugging technique for one past run. It cannot test a new model, because the model is the part that was replaced by a recording.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Replaying against a different model&lt;/strong&gt; is the reverse. The model call is live, and the tools are the part served from the recording. This post is about the second kind.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why you can't just rerun the agent
&lt;/h2&gt;

&lt;p&gt;An agent run is a loop: the model calls a tool, the tool result goes back in, the model calls another tool. Rerunning that loop live against a new model has two problems.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The tools have side effects.&lt;/strong&gt; A replay that executes &lt;code&gt;issue_refund&lt;/code&gt; issues a refund.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The runs stop being comparable.&lt;/strong&gt; Once the new model makes a different call in round one, everything after it is a different conversation, and the two runs can no longer be compared step by step.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Teacher-forced replay, step by step
&lt;/h2&gt;

&lt;p&gt;The name is borrowed from how sequence models are trained: at every step the model is given the recorded history, not its own earlier output.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Record each production run in full.&lt;/strong&gt; The input, the tools offered, every model call, every tool call with its arguments, and every tool result. Remove personal data before it is written to disk.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Replay round by round.&lt;/strong&gt; For round one, send the new model the recorded input and the recorded tools, and compare its tool calls with the recorded ones.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Feed back the recorded results, never live ones.&lt;/strong&gt; For round two, send the recorded conversation up to that point, including the recorded tool calls and their results, and compare again. No tool is executed, so the replay has no side effects.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Score each round on its own.&lt;/strong&gt; Both models saw identical context in every round, so each comparison isolates one model decision.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Do not follow the new model off the recording.&lt;/strong&gt; If it calls a tool the original run never called, there is no recorded result to feed back. Count that round as drift, and build the next round from the recorded history as before.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Take a support agent with two tools, &lt;code&gt;lookup_order&lt;/code&gt; and &lt;code&gt;issue_refund&lt;/code&gt;, and one recorded run in which a customer asks for a refund on order A-1042. The replay has three rounds:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Round&lt;/th&gt;
&lt;th&gt;What the new model is sent&lt;/th&gt;
&lt;th&gt;What its reply is compared with&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;The system prompt and the customer's message&lt;/td&gt;
&lt;td&gt;The recorded call to &lt;code&gt;lookup_order&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Round 1, plus the recorded &lt;code&gt;lookup_order&lt;/code&gt; call and its recorded result&lt;/td&gt;
&lt;td&gt;The recorded call to &lt;code&gt;issue_refund&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;Round 2, plus the recorded &lt;code&gt;issue_refund&lt;/code&gt; call and its recorded result&lt;/td&gt;
&lt;td&gt;The recorded reply, which called no tools&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Round 3 counts too. The recorded agent stopped calling tools there and answered, so a new model that keeps calling tools is caught in that round.&lt;/p&gt;

&lt;h2&gt;
  
  
  A replay script
&lt;/h2&gt;

&lt;p&gt;Here is the method as a script. It assumes each recorded run is one line of a JSONL file that holds the tools that were offered, the request settings, and the full message list of the run in OpenAI chat format. This is one run, pretty-printed and with the tool schemas left out:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"refund-a1042"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"tools"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="err"&gt;...&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"settings"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"tool_choice"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"auto"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"messages"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"system"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"You are a support agent. Look up an order before refunding it."&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Refund order A-1042, it arrived broken."&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"assistant"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"tool_calls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"call_1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"function"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"function"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"lookup_order"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"arguments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"{\"order_id\": \"A-1042\"}"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"tool"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"tool_call_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"call_1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"{\"order_id\": \"A-1042\", \"status\": \"delivered\", \"total\": 49.99}"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"assistant"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"tool_calls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"call_2"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"function"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"function"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"issue_refund"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"arguments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"{\"order_id\": \"A-1042\", \"amount\": 49.99}"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"tool"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"tool_call_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"call_2"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"{\"order_id\": \"A-1042\", \"refunded\": 49.99}"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"assistant"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"I've refunded order A-1042."&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every assistant message in that list is one round. The script walks the list, and at each assistant message it sends the new model everything recorded before it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="s2"&gt;"""Teacher-forced replay of recorded agent runs against another model.

usage: python replay.py traces.jsonl &amp;lt;new-model&amp;gt;
"""&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;

&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;OpenAI&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;OpenAI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;new_model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;names&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tool_calls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"function"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;call&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;tool_calls&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;trace&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;messages&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;trace&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"messages"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;round_no&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;recorded&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;recorded&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="s2"&gt;"assistant"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;  &lt;span class="c1"&gt;# every recorded assistant message is one round&lt;/span&gt;
        &lt;span class="n"&gt;round_no&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="n"&gt;reply&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;completions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;new_model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="c1"&gt;# the recorded context, tool results included&lt;/span&gt;
            &lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;trace&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"tools"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;trace&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"settings"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="c1"&gt;# tool_choice, parallel_tool_calls, ...&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;replayed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;reply&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;choices&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;model_dump&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"tool_calls"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="n"&gt;was&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;names&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;recorded&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"tool_calls"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;names&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;replayed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;verdict&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"same"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;was&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;"DRIFT"&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{trace['id']} round {round_no}: recorded {was}, new model {now}: {verdict}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For the run above and a new model that skips the lookup, it prints:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;refund-a1042 round 1: recorded ['lookup_order'], new model ['issue_refund']: DRIFT
refund-a1042 round 2: recorded ['issue_refund'], new model ['issue_refund']: same
refund-a1042 round 3: recorded [], new model []: same
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Round 1 is the finding: the new model went straight to the refund. Round 2 shows that once the lookup result is in front of it, it makes the same call the old model made. Its final reply can be the same sentence the old model wrote, which is why the comparison has to read the tool calls and not only the reply.&lt;/p&gt;

&lt;p&gt;Two things to do before you trust the output:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Run it against your current model first.&lt;/strong&gt; A model sampled twice on the same input does not always return the same thing. The rate at which your current model disagrees with its own recordings is the noise floor, and drift is what exceeds it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stay in one provider's format, or translate.&lt;/strong&gt; The script sends OpenAI-style messages. To replay against another provider, convert the messages and tool schemas to that provider's format first, or send them through a library that does the conversion, such as LiteLLM.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The script compares tool names only. A fuller comparison checks more.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to compare
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;What&lt;/th&gt;
&lt;th&gt;How to compare it&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Which tools were called&lt;/td&gt;
&lt;td&gt;The set of tool names in each round. Calls made in one round are requested together, so their order within it rarely matters.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The arguments&lt;/td&gt;
&lt;td&gt;Field by field: exact match for IDs, enums, booleans and dates, a tolerance for numbers, a similarity score for free text such as a search query.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;How many calls&lt;/td&gt;
&lt;td&gt;The number of calls in the run, with a tolerance of one. Models legitimately differ by a call: one fetches two records in one call, another in two.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The final reply&lt;/td&gt;
&lt;td&gt;Format checks in code, or an LLM judge with one pass-or-fail criterion.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A difference is drift, and drift is not yet a regression. Whether it is one depends on whether the old behavior was right. If the old model skipped a step it should have taken, the new model's difference is a fix.&lt;/p&gt;

&lt;p&gt;Report the drift rate with an interval. Three drifted runs out of 40 is 7.5%, and the 95% Wilson interval runs from 2.6% to 19.9%, which says almost nothing. Twelve out of 400 is 3%, with an interval of 1.7% to 5.2%.&lt;/p&gt;

&lt;h2&gt;
  
  
  Carry over the request settings
&lt;/h2&gt;

&lt;p&gt;A replay has to reproduce the request, not just the messages. Carry over the settings the original call used, such as &lt;code&gt;tool_choice&lt;/code&gt;, parallel tool calls and the response format. A replay that drops &lt;code&gt;tool_choice: required&lt;/code&gt; reports drift that is really a missing setting.&lt;/p&gt;

&lt;h2&gt;
  
  
  What teacher-forced replay cannot tell you
&lt;/h2&gt;

&lt;p&gt;It tests each decision given the original model's history, not the path the new model would take on its own. A new model that recovers well from its own mistakes, or compounds them, looks the same in a teacher-forced replay. That is the price of a comparison that stays paired and has no side effects. For the end-to-end path, run a small set of tasks against a sandbox with fake tools.&lt;/p&gt;

&lt;h2&gt;
  
  
  Doing this with EvalShift
&lt;/h2&gt;

&lt;p&gt;Disclosure again: I'm the founder. &lt;a href="https://www.evalshift.dev" rel="noopener noreferrer"&gt;EvalShift&lt;/a&gt; does the steps above with a capture SDK and a CLI. Both are open source and run on your machine with your own provider keys.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Record.&lt;/strong&gt; Wrap the provider client, and decorate the agent and its tools:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;OpenAI&lt;/span&gt;

&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;evalshift&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;capture&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;evalshift&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;adapters&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;openai&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;wrap_openai&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;wrap_openai&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;OpenAI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# records every model call made inside the agent&lt;/span&gt;


&lt;span class="nd"&gt;@capture&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tool&lt;/span&gt;  &lt;span class="c1"&gt;# records the call, its arguments and its result&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;lookup_order&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;order_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;


&lt;span class="nd"&gt;@capture&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;suite&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"support_agent"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;redact&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="no"&gt;TOOLS&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# TOOLS: your tool schemas&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;handle_ticket&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nothing is recorded unless &lt;code&gt;EVALSHIFT_CAPTURE=1&lt;/code&gt; is set, so the decorators can stay in production code. Captures go to local disk or to a bucket you own. &lt;code&gt;redact=True&lt;/code&gt; masks emails and common API key and bearer token patterns in process, before anything is written. That is a floor, not a full PII scrubber.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Replay.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip install evalshift                 &lt;span class="c"&gt;# the CLI (Python 3.11+); the capture SDK comes with it&lt;/span&gt;
evalshift init --provider openai      &lt;span class="c"&gt;# writes evalshift.yaml; set defaults.source_model to your production model&lt;/span&gt;
&lt;span class="nv"&gt;EVALSHIFT_CAPTURE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1 python agent.py   &lt;span class="c"&gt;# record real runs&lt;/span&gt;
evalshift capture sync --rounds all   &lt;span class="c"&gt;# promote them into a suite, keeping every round and its tool results&lt;/span&gt;
evalshift compare --suite-name support_agent --to &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;candidate-model&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;--rounds all&lt;/code&gt; keeps every round of each recorded run together with its tool results. &lt;code&gt;compare&lt;/code&gt; then replays each round teacher-forced, for both your current model and the candidate. The recorded tool calls and results are fed back, never a model's own calls, and no tool is executed. The recorded &lt;code&gt;tool_choice&lt;/code&gt; and parallel-call settings are replayed with each request.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Compare.&lt;/strong&gt; &lt;code&gt;capture sync&lt;/code&gt; also wires tool evaluators into each suite, based on what its recordings contain. &lt;code&gt;tool_selection&lt;/code&gt; compares which tools were called in each round. &lt;code&gt;tool_arguments&lt;/code&gt; compares arguments field by field, and picks exact, numeric or similarity matching from the tool's schema. A &lt;a href="https://www.evalshift.dev/docs/migration-policy" rel="noopener noreferrer"&gt;&lt;code&gt;migration_policy&lt;/code&gt;&lt;/a&gt; sets a budget for each, such as &lt;code&gt;max_tool_divergence&lt;/code&gt; and &lt;code&gt;max_tool_argument_drift&lt;/code&gt;. The verdict is pass, conditional pass, fail or inconclusive, and inconclusive means the suite was too small to decide. With &lt;code&gt;--policy-gate&lt;/code&gt; the command fails on a fail or a conditional pass, so it can gate a pull request.&lt;/p&gt;

&lt;p&gt;Traces recorded on one provider can be replayed against a model from another. Requests go out through LiteLLM, which maps them to each provider's format.&lt;/p&gt;

&lt;p&gt;What it does not do: run your tools, or watch production. The limit of teacher forcing described above applies to it too, and it has no dashboards or alerts on live traffic. The &lt;a href="https://www.evalshift.dev/docs/agents" rel="noopener noreferrer"&gt;agents docs&lt;/a&gt; cover replay in detail, and the code is on &lt;a href="https://github.com/evalshift/evalshift-cli" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Does replaying a trace run my tools?&lt;/strong&gt; No. The tool results come from the recording, so nothing is executed and the replay has no side effects.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What happens when the new model calls a different tool than the recording?&lt;/strong&gt; That round counts as drift. The next round is still built from the recorded history, so one early difference does not change what the later rounds test.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I replay traces recorded on one provider against a model from another?&lt;/strong&gt; Yes, if the messages and tool schemas are translated to the new provider's format. Carry the request settings over too: a forced tool choice has a different shape on each provider.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How many traces do I need?&lt;/strong&gt; Enough that the interval around the drift rate is narrow enough to decide. &lt;a href="https://www.evalshift.dev/blog/how-many-eval-cases-do-you-need" rel="noopener noreferrer"&gt;How many eval cases do you need&lt;/a&gt; works through the sizing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://www.evalshift.dev/blog/llm-drift" rel="noopener noreferrer"&gt;LLM drift: how to detect model drift in AI agents&lt;/a&gt;: the longer post this one is adapted from, with a CI gate script.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.evalshift.dev/blog/build-a-golden-suite-from-production-traffic" rel="noopener noreferrer"&gt;Build a golden suite from production traffic&lt;/a&gt;: turning recordings into test cases.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.evalshift.dev/blog/evaluating-agent-tool-calls" rel="noopener noreferrer"&gt;Evaluating agent tool calls&lt;/a&gt;: scoring the trace, not the reply.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>agents</category>
      <category>testing</category>
    </item>
    <item>
      <title>The new model was 59% cheaper and 75% faster. I still wouldn't ship it.</title>
      <dc:creator>Lukas Babaliauskas</dc:creator>
      <pubDate>Thu, 12 Nov 2026 21:05:10 +0000</pubDate>
      <link>https://dev.to/lukas_babaliauskas/the-new-model-was-59-cheaper-and-75-faster-i-still-wouldnt-ship-it-189n</link>
      <guid>https://dev.to/lukas_babaliauskas/the-new-model-was-59-cheaper-and-75-faster-i-still-wouldnt-ship-it-189n</guid>
      <description>&lt;p&gt;The migration looked like an obvious win on economics. In one EvalShift run, the candidate model was:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;58.8% cheaper&lt;/li&gt;
&lt;li&gt;75.2% faster&lt;/li&gt;
&lt;li&gt;0 failed or truncated calls&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe6amvl7lwcx6ule0akgr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe6amvl7lwcx6ule0akgr.png" alt="Run economics table: source and target each made 16 calls with no failures or truncations; the target cost $0.0741 against $0.1799 and averaged 773 ms against 3.1 s." width="799" height="115"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Run economics. Same 16 calls on both sides, no failures, and every number a finance reviewer cares about moved the right way.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Then the migration report returned:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;FAIL&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Not because the model crashed. Not because the API changed. Not because the output stopped parsing.&lt;/p&gt;

&lt;p&gt;It changed what the agent actually did.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftskk2lb96ywkzjzv8msw.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftskk2lb96ywkzjzv8msw.png" alt="Migration verdict card: FAIL, 5 of 7 budgets within policy, 61.4% equivalent, 25.0% improved, 13.6% regressed, cost −58.8%, latency −75.2%." width="799" height="187"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The verdict card. Five of seven budgets passed; the two that did not are the two that describe behavior.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What failed
&lt;/h2&gt;

&lt;p&gt;I replayed 16 real examples against the source and candidate models. Across the run, 13.6% of evaluated comparisons regressed.&lt;/p&gt;

&lt;p&gt;The overall regression-rate budget actually passed.&lt;/p&gt;

&lt;p&gt;Two other checks did not:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tool-selection divergence hit 25%, against a 10% limit.&lt;/li&gt;
&lt;li&gt;The overall equivalence rate fell to 61.4%, below the required 75%.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fb2qy9mjund9v6eyuhx8m.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fb2qy9mjund9v6eyuhx8m.png" alt="Report verdict and findings: the target breached max_tool_divergence at 25.0% against a 10% ceiling and fell below min_equivalence_rate at 61.4% against 75%; findings list searching instead of answering, confirming actions it never performed, and routing to the wrong tool." width="800" height="413"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The written verdict and its findings. The recommendation is not to migrate until the tool-selection and semantic regressions are fixed.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;That distinction matters. Looking only at latency, cost, successful requests, or even a few manually inspected outputs would have made this migration look pretty attractive.&lt;/p&gt;

&lt;p&gt;The behavioral diff told a different story.&lt;/p&gt;

&lt;h2&gt;
  
  
  The failures weren't cosmetic
&lt;/h2&gt;

&lt;p&gt;One example asked for an opinion on working late.&lt;/p&gt;

&lt;p&gt;The source model answered the question directly.&lt;/p&gt;

&lt;p&gt;The candidate instead issued a &lt;code&gt;search_web&lt;/code&gt; call and returned no text.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4wot0gloc3c9azu8p80r.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4wot0gloc3c9azu8p80r.png" alt="Per-example diff for the working-late question: the source called no tools and answered in two sentences; the target called search_web with the query &amp;quot;is working late worth it productivity well-being&amp;quot; and produced no final text. Flagged as TOOL_SELECTION_DRIFT." width="799" height="386"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The diff for that example. Source trace on the left, target trace on the right, and the reason it was flagged at the top.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Other cases had the same general problem in different forms:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;an action was reported as completed even though the corresponding tool was never called;&lt;/li&gt;
&lt;li&gt;a request was routed to a different tool;&lt;/li&gt;
&lt;li&gt;an unnecessary tool call appeared where the source answered directly.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These are not necessarily signs that the candidate model is "bad." They show that changing the model changed the application.&lt;/p&gt;

&lt;p&gt;For an agent, the model is part of the control flow.&lt;/p&gt;

&lt;h2&gt;
  
  
  Aggregate quality can hide this
&lt;/h2&gt;

&lt;p&gt;The evaluator breakdown made the tradeoff clearer.&lt;/p&gt;

&lt;p&gt;Routing arguments and routing conformance were broadly equivalent in this run, while tool-selection divergence, semantic similarity, and the pairwise equivalence judge showed regressions.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzojvj0tegyd7h9q2h1x4.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzojvj0tegyd7h9q2h1x4.png" alt="Per-evaluator table over 16 examples: routing divergence regressed (score delta −0.167, likely), routing args equivalent (+0.179, unclear), routing conformance equivalent (+0.047, unclear), the LLM equivalence judge regressed critically (−0.438, certain), and semantic similarity regressed (−0.214, likely)." width="800" height="348"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Overall, by evaluator. Two rows say equivalent, three say regressed, and each comes with its own effect size, confidence interval and confidence label.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A single average score would flatten all of that into one number.&lt;/p&gt;

&lt;p&gt;For a migration, I care more about the question:&lt;/p&gt;

&lt;p&gt;&lt;em&gt;What changed, on which examples, and is that change acceptable for this application?&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  This is why I built EvalShift
&lt;/h2&gt;

&lt;p&gt;I kept running into model migrations that were tested roughly like this:&lt;/p&gt;

&lt;p&gt;change model → try several prompts → outputs look fine → ship.&lt;/p&gt;

&lt;p&gt;That works until the difference is something subtle like an extra tool call, a missing action, or different routing behavior.&lt;/p&gt;

&lt;p&gt;EvalShift replays the same suite against the source and target model, compares the behavior, and produces the migration decision and individual diffs before the model string gets changed in production. The CLI is local-first and produces a self-contained HTML report; hosted upload is optional. How that differs from an observability tool or a prompt-eval runner is on the &lt;a href="https://www.evalshift.dev/compare" rel="noopener noreferrer"&gt;comparison pages&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The interesting result from this run wasn't that the candidate was worse.&lt;/p&gt;

&lt;p&gt;It was that "59% cheaper and 75% faster" wasn't enough information to decide whether the migration was safe.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>testing</category>
      <category>python</category>
    </item>
  </channel>
</rss>
