DEV Community

Lukas Babaliauskas
Lukas Babaliauskas

Posted on Originally published at evalshift.dev AI-assisted

How to replay agent traces against a different model

To replay recorded agent traces against a different model, use teacher-forced replay. Record each production run in full, including every tool result. Then send the new model the recorded context one round at a time, execute no tools, and compare the tool calls it makes in each round with the ones the original model made.

This post covers why rerunning the agent does not work, the method step by step, a short Python script that does it, what to compare, and what the method cannot tell you.

A disclosure: I build EvalShift, a tool for testing model changes on agents. It has a short section at the end, and everything before it works without it. This post is adapted from the trace-replay section of my longer post on LLM drift.

Two kinds of replay

"Replay" means two different things for agents, and they are opposites.

In deterministic replay, the model's recorded responses are served back, so the agent's code follows the same path again. It is a debugging technique for one past run. It cannot test a new model, because the model is the part that was replaced by a recording.

Replaying against a different model is the reverse. The model call is live, and the tools are the part served from the recording. This post is about the second kind.

Why you can't just rerun the agent

An agent run is a loop: the model calls a tool, the tool result goes back in, the model calls another tool. Rerunning that loop live against a new model has two problems.

  • The tools have side effects. A replay that executes issue_refund issues a refund.
  • The runs stop being comparable. Once the new model makes a different call in round one, everything after it is a different conversation, and the two runs can no longer be compared step by step.

Teacher-forced replay, step by step

The name is borrowed from how sequence models are trained: at every step the model is given the recorded history, not its own earlier output.

  1. Record each production run in full. The input, the tools offered, every model call, every tool call with its arguments, and every tool result. Remove personal data before it is written to disk.
  2. Replay round by round. For round one, send the new model the recorded input and the recorded tools, and compare its tool calls with the recorded ones.
  3. Feed back the recorded results, never live ones. For round two, send the recorded conversation up to that point, including the recorded tool calls and their results, and compare again. No tool is executed, so the replay has no side effects.
  4. Score each round on its own. Both models saw identical context in every round, so each comparison isolates one model decision.
  5. Do not follow the new model off the recording. If it calls a tool the original run never called, there is no recorded result to feed back. Count that round as drift, and build the next round from the recorded history as before.

Take a support agent with two tools, lookup_order and issue_refund, and one recorded run in which a customer asks for a refund on order A-1042. The replay has three rounds:

Round What the new model is sent What its reply is compared with
1 The system prompt and the customer's message The recorded call to lookup_order
2 Round 1, plus the recorded lookup_order call and its recorded result The recorded call to issue_refund
3 Round 2, plus the recorded issue_refund call and its recorded result The recorded reply, which called no tools

Round 3 counts too. The recorded agent stopped calling tools there and answered, so a new model that keeps calling tools is caught in that round.

A replay script

Here is the method as a script. It assumes each recorded run is one line of a JSONL file that holds the tools that were offered, the request settings, and the full message list of the run in OpenAI chat format. This is one run, pretty-printed and with the tool schemas left out:

{
  "id": "refund-a1042",
  "tools": [...],
  "settings": {"tool_choice": "auto"},
  "messages": [
    {"role": "system", "content": "You are a support agent. Look up an order before refunding it."},
    {"role": "user", "content": "Refund order A-1042, it arrived broken."},
    {"role": "assistant", "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "lookup_order", "arguments": "{\"order_id\": \"A-1042\"}"}}]},
    {"role": "tool", "tool_call_id": "call_1", "content": "{\"order_id\": \"A-1042\", \"status\": \"delivered\", \"total\": 49.99}"},
    {"role": "assistant", "tool_calls": [{"id": "call_2", "type": "function", "function": {"name": "issue_refund", "arguments": "{\"order_id\": \"A-1042\", \"amount\": 49.99}"}}]},
    {"role": "tool", "tool_call_id": "call_2", "content": "{\"order_id\": \"A-1042\", \"refunded\": 49.99}"},
    {"role": "assistant", "content": "I've refunded order A-1042."}
  ]
}
Enter fullscreen mode Exit fullscreen mode

Every assistant message in that list is one round. The script walks the list, and at each assistant message it sends the new model everything recorded before it:

"""Teacher-forced replay of recorded agent runs against another model.

usage: python replay.py traces.jsonl <new-model>
"""

import json
import sys

from openai import OpenAI

client = OpenAI()
path, new_model = sys.argv[1], sys.argv[2]


def names(tool_calls):
    return sorted(call["function"]["name"] for call in tool_calls or [])


for line in open(path):
    trace = json.loads(line)
    messages = trace["messages"]
    round_no = 0
    for i, recorded in enumerate(messages):
        if recorded["role"] != "assistant":
            continue  # every recorded assistant message is one round
        round_no += 1
        reply = client.chat.completions.create(
            model=new_model,
            messages=messages[:i],  # the recorded context, tool results included
            tools=trace["tools"],
            **trace.get("settings", {}),  # tool_choice, parallel_tool_calls, ...
        )
        replayed = reply.choices[0].message.model_dump()["tool_calls"]
        was, now = names(recorded.get("tool_calls")), names(replayed)
        verdict = "same" if was == now else "DRIFT"
        print(f"{trace['id']} round {round_no}: recorded {was}, new model {now}: {verdict}")
Enter fullscreen mode Exit fullscreen mode

For the run above and a new model that skips the lookup, it prints:

refund-a1042 round 1: recorded ['lookup_order'], new model ['issue_refund']: DRIFT
refund-a1042 round 2: recorded ['issue_refund'], new model ['issue_refund']: same
refund-a1042 round 3: recorded [], new model []: same
Enter fullscreen mode Exit fullscreen mode

Round 1 is the finding: the new model went straight to the refund. Round 2 shows that once the lookup result is in front of it, it makes the same call the old model made. Its final reply can be the same sentence the old model wrote, which is why the comparison has to read the tool calls and not only the reply.

Two things to do before you trust the output:

  • Run it against your current model first. A model sampled twice on the same input does not always return the same thing. The rate at which your current model disagrees with its own recordings is the noise floor, and drift is what exceeds it.
  • Stay in one provider's format, or translate. The script sends OpenAI-style messages. To replay against another provider, convert the messages and tool schemas to that provider's format first, or send them through a library that does the conversion, such as LiteLLM.

The script compares tool names only. A fuller comparison checks more.

What to compare

What How to compare it
Which tools were called The set of tool names in each round. Calls made in one round are requested together, so their order within it rarely matters.
The arguments Field by field: exact match for IDs, enums, booleans and dates, a tolerance for numbers, a similarity score for free text such as a search query.
How many calls The number of calls in the run, with a tolerance of one. Models legitimately differ by a call: one fetches two records in one call, another in two.
The final reply Format checks in code, or an LLM judge with one pass-or-fail criterion.

A difference is drift, and drift is not yet a regression. Whether it is one depends on whether the old behavior was right. If the old model skipped a step it should have taken, the new model's difference is a fix.

Report the drift rate with an interval. Three drifted runs out of 40 is 7.5%, and the 95% Wilson interval runs from 2.6% to 19.9%, which says almost nothing. Twelve out of 400 is 3%, with an interval of 1.7% to 5.2%.

Carry over the request settings

A replay has to reproduce the request, not just the messages. Carry over the settings the original call used, such as tool_choice, parallel tool calls and the response format. A replay that drops tool_choice: required reports drift that is really a missing setting.

What teacher-forced replay cannot tell you

It tests each decision given the original model's history, not the path the new model would take on its own. A new model that recovers well from its own mistakes, or compounds them, looks the same in a teacher-forced replay. That is the price of a comparison that stays paired and has no side effects. For the end-to-end path, run a small set of tasks against a sandbox with fake tools.

Doing this with EvalShift

Disclosure again: I'm the founder. EvalShift does the steps above with a capture SDK and a CLI. Both are open source and run on your machine with your own provider keys.

Record. Wrap the provider client, and decorate the agent and its tools:

from openai import OpenAI

from evalshift import capture
from evalshift.adapters.openai import wrap_openai

client = wrap_openai(OpenAI())  # records every model call made inside the agent


@capture.tool  # records the call, its arguments and its result
def lookup_order(order_id: str) -> dict: ...


@capture.agent(suite="support_agent", redact=True, tools=TOOLS)  # TOOLS: your tool schemas
def handle_ticket(query: str) -> str: ...
Enter fullscreen mode Exit fullscreen mode

Nothing is recorded unless EVALSHIFT_CAPTURE=1 is set, so the decorators can stay in production code. Captures go to local disk or to a bucket you own. redact=True masks emails and common API key and bearer token patterns in process, before anything is written. That is a floor, not a full PII scrubber.

Replay.

pip install evalshift                 # the CLI (Python 3.11+); the capture SDK comes with it
evalshift init --provider openai      # writes evalshift.yaml; set defaults.source_model to your production model
EVALSHIFT_CAPTURE=1 python agent.py   # record real runs
evalshift capture sync --rounds all   # promote them into a suite, keeping every round and its tool results
evalshift compare --suite-name support_agent --to <candidate-model>
Enter fullscreen mode Exit fullscreen mode

--rounds all keeps every round of each recorded run together with its tool results. compare then replays each round teacher-forced, for both your current model and the candidate. The recorded tool calls and results are fed back, never a model's own calls, and no tool is executed. The recorded tool_choice and parallel-call settings are replayed with each request.

Compare. capture sync also wires tool evaluators into each suite, based on what its recordings contain. tool_selection compares which tools were called in each round. tool_arguments compares arguments field by field, and picks exact, numeric or similarity matching from the tool's schema. A migration_policy sets a budget for each, such as max_tool_divergence and max_tool_argument_drift. The verdict is pass, conditional pass, fail or inconclusive, and inconclusive means the suite was too small to decide. With --policy-gate the command fails on a fail or a conditional pass, so it can gate a pull request.

Traces recorded on one provider can be replayed against a model from another. Requests go out through LiteLLM, which maps them to each provider's format.

What it does not do: run your tools, or watch production. The limit of teacher forcing described above applies to it too, and it has no dashboards or alerts on live traffic. The agents docs cover replay in detail, and the code is on GitHub.

FAQ

Does replaying a trace run my tools? No. The tool results come from the recording, so nothing is executed and the replay has no side effects.

What happens when the new model calls a different tool than the recording? That round counts as drift. The next round is still built from the recorded history, so one early difference does not change what the later rounds test.

Can I replay traces recorded on one provider against a model from another? Yes, if the messages and tool schemas are translated to the new provider's format. Carry the request settings over too: a forced tool choice has a different shape on each provider.

How many traces do I need? Enough that the interval around the drift rate is narrow enough to decide. How many eval cases do you need works through the sizing.

Keep reading

Top comments (0)