On 23 April 2026, a study of 1,140 agent traces put a plain question to six production-grade models: run the same agent on the same task twice, and does it do the same thing? Abel Yagubyan's answer is that agents usually pick the same tools in the same order — and when they don't, the split happens almost immediately. 60% of first-divergence events land inside the first two pipeline steps, at a mean divergence point of 2.2.
That number is the reason agent postmortems keep stalling. The standard reliability loop — reproduce the failure, bisect it, fix it — assumes the second run is the first run. For a tool-calling agent it frequently isn't, and the place it stops being the same run is step one or step two, upstream of whatever you were actually investigating.
The debugging loop has a hidden precondition
The operational practices engineers carry over from distributed systems share an unstated assumption: given the same input, the system takes the same path. Tracing, bisection, canaries, regression tests, "works on my machine" arbitration — each rests on re-execution being faithful. When it is, a trace is a convenience. When it isn't, the trace is the only evidence that will ever exist of that particular run.
Rasheed Mudasiru's April 2026 paper on deterministic replay for agent systems names four sources of non-determinism that, in its framing, collectively prevent any prior agent run from being faithfully re-executed: LLM sampling variance, external API state, CDN infrastructure headers, and execution-environment noise. Re-running a failed agent task therefore does not retrieve the failure. It draws a fresh sample from the same distribution, which may or may not land in the same place.
This is an architectural property rather than a maturity gap. It does not resolve as tooling improves, because the named causes include the state of upstream APIs and the infrastructure sitting between the agent and them.
The divergence lands where it does the most damage
Yagubyan's benchmark ran 19 tasks, ten times each, against six models — GPT-4o, GPT-4o-mini, GPT-4.1, GPT-4.1-mini, Claude Sonnet 4 and Llama 3.3 70B — using ten deterministic simulated tools, at temperature 1.0. Holding the tools fixed isolates model-generated variance from environmental noise, which makes these figures a floor rather than a ceiling; the paper says as much in its limitations, noting that simulated tools "may overestimate real-world consistency."
Three findings matter operationally.
Agents are structurally consistent and parametrically variable. Tool Sequence Similarity averaged 0.87, argument consistency 0.69 — a large and highly significant gap. Agents reliably reach for the same recipe and vary in how they fill it in.
Of the two layers, only the structural one predicts whether the run worked. Runs in the top band of sequence similarity were correct 90.2% of the time. Runs in the bottom band were correct 61.2%. Argument-level variance showed no measurable relationship with correctness. Different phrasing of a search query is benign. A different tool, or a missing one, is where failure concentrates.
And that structural split happens early. 60% of first divergences occur in steps one and two, at a mean divergence point of 2.2. Yagubyan draws the constructive conclusion — comparing only a run's first two tool calls against a reference sequence catches most of the variance cheaply. The postmortem corollary is less comfortable: by the time an engineer is reading a trace at step seven, the run in front of them may have branched away from the run they are trying to explain five steps earlier, and the artefact will not necessarily flag the branch.
There is a second trap in the same data. Final natural-language responses matched exactly less than 5% of the time even when the underlying tool sequences were identical — the paper's own conclusion is that output text is not a reliability signal. A postmortem that reasons from what the agent said, rather than from what it called, is reading the noisiest available layer.
Fidelity is manufactured before the run, not after it
The agrepl framework in Mudasiru's paper is instructive less as a product than as a proof of what faithful replay actually costs. It intercepts every external interaction at the transport layer through a man-in-the-middle proxy, serialises them as structured execution traces, and replays them in an environment with zero outbound network access. Across five workloads and 250 replay instances it reports replay fidelity of 1.0 and a median per-step latency reduction of 98.3%.
Read the mechanism carefully. Fidelity comes from substituting a recording for the outside world. The paper's stated motivation for building it is that existing observability platforms capture execution logs but, in its words, cannot reproduce a run in isolation. The recording is the load-bearing part, and it has to have been made while the run was happening. Information that was never captured — which candidate tools the model was weighing at step two, what an upstream API returned before it changed — is not recoverable later by any amount of analysis.
Which reframes the AgentOps question. "Can we replay this run?" is downstream of "did we record enough, at the right granularity, before we knew we would need it?" The first question is about tooling. The second is about the execution environment, and it has to be answered before the incident.
How Waxell handles this
Waxell Observe is built around the granularity the structural layer requires. It auto-instruments 200+ Python frameworks, LLMs and vector databases in two lines of code, and from that point captures every LLM call, tool invocation and agent decision inside the instrumented process — including, per the product page, routing decisions with the options considered and the choice made, and tool calls with inputs, outputs and timing. Multi-agent work is recorded as execution trees with parent-child span relationships in OpenTelemetry, linked by session and lineage. That is the record a sequence comparison needs: the tool names, in order, with their arguments, rather than the final text.
For workflows built on the Waxell SDK, Runtime's durable execution model writes the record at the step boundary. Every tool call is a durable step: the runtime records that a step is starting before it runs, and checkpoints the result after it completes. The docs are explicit about the limit, and it is worth quoting rather than glossing — work done between tool calls is not checkpointed. If an expensive computation matters to the reconstruction, it belongs inside a tool so its result is saved.
Three of Runtime's terminal run states separate questions a single "failed" flag collapses. BLOCKED means the run was stopped by governance — a policy or a budget. FAILED means an error inside the agent. INTERRUPTED means the process died before finishing. The documentation calls that last distinction deliberate: an agent that errored is a different problem from a process killed under it, and collapsing them hides infrastructure issues. In a postmortem, that is the difference between a bug, a policy working as designed, and an infrastructure fault — read off recorded state rather than inferred afterwards.
Waxell Runtime gates each step against the same 50+ policy categories before it executes, with kill switches at the agent, workflow and session level, and isolated execution per run.
One practical constraint belongs in the same paragraph as the capability: a record you cannot reach is not evidence. Waxell's published plan limits set trace retention at 14 days on the Free tier, 30 on Team, 90 on Business and 365+ on Enterprise. Set the window against how long your incidents actually take to surface. The Yagubyan data argues for the shortest useful comparison — first two tool calls against a reference — precisely because it is cheap enough to keep.
FAQ
What is AI agent reproducibility?
AI agent reproducibility is the property that running the same agent on the same input produces the same behaviour — the same tools, in the same order, with equivalent arguments. It is distinct from output reproducibility, which for language models is near-zero: Yagubyan's 2026 study found final responses matched exactly less than 5% of the time even when the underlying tool sequences were identical. Reproducibility in the operational sense is about the action chain, not the text.
Why can't I just re-run a failed agent task to debug it?
Because the second run is not guaranteed to be the first run. Mudasiru's 2026 paper identifies four independent sources of divergence: LLM sampling variance, external API state, CDN infrastructure headers, and execution-environment noise. Re-running draws a new sample rather than retrieving the old one. Where the runs differ, they tend to differ early — 60% of first divergences occur in the first two steps — so the branch usually happens before the step you are investigating.
Does behavioural variance actually predict failure?
At the structural layer, yes. In Yagubyan's benchmark, runs whose tool sequence closely matched a reference were correct 90.2% of the time, against 61.2% for runs whose sequence diverged. Argument-level variance showed no measurable relationship with correctness. So a run that picked different tools warrants investigation; a run that phrased a query differently generally does not.
What should an agent run record contain to be usable in a postmortem?
At minimum: the ordered sequence of tool names, the arguments passed at each step, tool outputs and timing, the model calls with their parameters, and parent-child relationships across sub-agents. Recording the decision points — which options were available and which was taken — is what makes a later comparison against a reference sequence possible. Final natural-language output is the least informative layer to reason from.
How long should agent traces be retained?
Long enough to cover the gap between an agent failure occurring and someone noticing. Waxell's published plan limits run 14 days on Free, 30 on Team, 90 on Business and 365+ on Enterprise. Incidents that surface through cost reviews or customer reports rather than alerts tend to have the longest lag, so retention should be set against that path, not against the alerting path.
Is deterministic replay of an agent run possible at all?
Under specific conditions, yes — but the conditions are demanding. Mudasiru reports replay fidelity of 1.0 across 250 replay instances, achieved by intercepting every external interaction at the transport layer during the original run and replaying it in an environment with no outbound network access. The fidelity comes from the recording, which must be made while the run is happening. It cannot be reconstructed after the fact.
Sources
- arXiv (Abel Yagubyan), "How Consistent Are LLM Agents? Measuring Behavioral Reproducibility in Multi-Step Tool-Calling Pipelines", 23 April 2026.
- arXiv (Rasheed Mudasiru), "Deterministic Replay for AI Agent Systems", 30 April 2026.
- Waxell, "Waxell Observe — AI Agent Observability & Governance", accessed 28 August 2026.
- Waxell, "Durable Execution", accessed 28 August 2026.
- Waxell, "Governed AI Agent Runtime & Execution", accessed 28 August 2026.
- Waxell, "Pricing", accessed 28 August 2026.
The run you most need to explain is the one you cannot ask to happen again — unless you were recording while it did.
Originally published on the Waxell blog.
Start free with Waxell Observe and one governed MCP upstream — pip install waxell, two lines to initialise, 10,000 traced executions a month on the Free tier. Create your workspace →
For workflows where a step must be checkpointed before it runs, see Waxell Runtime, included on Business.
Top comments (0)