Part 2 of Your Agent Failed in Prod. Good Luck Reproducing It.
This work was presented at the AI Engineer World's Fair 2026 by Susheem Koul and Ti...
For further actions, you may consider blocking this person and/or reporting abuse
The principle of freezing the run rather than the model is something I think about a lot. In Opportunity Skill, impressions work on a similar logic. You capture what actually happened in a collaboration, the exact preferences expressed, the precise boundaries stated, and commit it as an immutable semantic unit. No editing in place, only create and prune. That recorded signal stays accurate over time, exactly like your incident fixture staying green in CI. The parallel between deterministic replay and a trustworthy professional record is stronger than most people realize.
yes its an effort to make our MAS workflows more deterministic for better reliability.
"Freeze the run, not the model" is the sentence people should steal from this. Replaying the exact prompt, sampled completion, retrieved chunks, and tool outputs turns an anecdote into a regression artifact, which is the only way fixes stay honest once the model backend moves under you. The other production lesson is to make redaction part of the capture path, not a cleanup step later. Otherwise the runs that are most valuable for debugging become the ones nobody can safely share.
Agreed!
Hi Tisha,
Can we contact for each other?
sure!
I just emailed to you. Please kindly check
Hey! What mail did you use?
The cut-point replay pattern solves the exact problem I kept running into when testing LangGraph-based agents: a fix that looked correct in a fresh re-run would fail again in production because the re-run sampled a different decision path than the original incident. Freezing everything upstream of the changed boundary and running only the modified node live against the recorded inputs is structurally the same as what LangGraph's checkpoint replay gives you when you pin a trace and re-enter at a specific node, though it requires you to discipline the fixture pipeline the way you describe. The redaction gate before committing fixtures is the piece most teams skip and then discover when a PII incident shows up in their git history.
Glad this helps!
The replay plan keys off call order (
agent@1,delete_file@1), so I'm curious what happens when the fix changes how many times a boundary runs. A guard that makes the agent retry once would shift every index after it, and I can't tell from the example whether that fails loudly or just quietly stubs the wrong call.Good question.
The plan keys each call by the boundary name plus a per name counter, not by flat position. In replay the wrapper keeps one counter for each boundary name and looks up the fixture by name and index. So when your fix adds a retry, only that boundary's counter moves. agent@1 and agent@2 keep their identity, and the retry just appends delete_file@2 instead of shifting everything after it.
When a call has no stub entry in the plan, the wrapper runs it live. It does not raise, and it does not reuse another recorded output. So your appended delete_file@2 runs the gate for real instead of getting @1's envelope. That specific case is safe.
The one real gap is that nothing compares the recorded call count to the live count. So a wrong stub is only possible if a fix inserts an extra call of a boundary you already stubbed, and inserts it before a later recorded call of that same name. Then the indexes shift and the earlier call quietly gets its neighbor's envelope. The cut point demos avoid this by stubbing only the upstream call and running everything downstream live.
So to answer directly: it does not fail loudly, but it also does not silently stub the wrong call in the retry example you gave. The only silent mismatch is the stub before a later same name call case above, and adding a check that recorded and live counts match per boundary would close it.
Source code : github.com/theagentplane/chronicle
Cut point replay is the right instinct. The failure mode I keep hitting is different though. It is not the retest that is broken, it is that most teams never actually define the cut point in the first place. When an agent misbehaves, the fix usually lands as a system prompt tweak or a retry wrapper, and nobody isolates which function actually changed. I have started treating any agent fix the way I treat a diff review, asking what specific code path moved and what stayed frozen before I even look at whether the new output looks better. Freezing the recording and replaying only the changed function is the only way I have found to tell a real fix from a lucky sample.
This matches what I see, and I think there is a structural reason the cut point never gets defined. The two fixes you named are exactly the ones that do not map to a single function. A prompt tweak changes what you feed the boundary, so it lives upstream in the input assembly. A retry wrapper changes how often boundaries fire, so it lives in the orchestration between them. Neither is "change this function and freeze the rest," so the isolation step gets skipped. If you cannot name what stays frozen, you do not have a fix yet. You have a new distribution and one sample from it, and naming the frozen set is the part people skip because it is harder than editing the prompt.
The ordering is the part I would underline. Judging whether the output got better first is the lucky sample trap, because a live rerun lets the model reroll everything and you cannot separate your diff from the noise. Freezing the recording and replaying only the changed path holds the model constant, so any difference is attributable to your code and not a good roll.
"Freeze the run, not the model" is the cleanest statement of this I've read — and the trap you name, that re-running to verify a fix is just drawing one fresh sample from the same distribution, is exactly where I've watched teams fool themselves. The hard part in practice is the boundary of what you replay: the prompt and sampled completion are easy to pin, but tool calls that hit live state (a DB that's moved on since 9:04) are where the frozen replay starts to leak. Do you stub those tool responses from the recording too, or draw the line at the model boundary and let the tools re-execute? That choice seems to decide whether "everything frozen except the one thing you changed" actually holds.
Think of it as changed versus unchanged, not model versus tools. The one thing you changed runs live. Everything else plays back its recorded output, tools included. So that DB tool gets stubbed from the recording, because if you let it run live it reads today's data, and now the agent changes for two reasons at once, your fix and the moved-on world. You cannot tell which caused it, which is the exact thing you were trying to avoid. A test of a past incident should use the world as it was at 9:04, not now.
The one exception is when the tool itself is what you changed. Then you run its new code, but still on the recorded inputs, and check what it decided rather than letting it touch anything real.
Freezing everything upstream of the fix boundary and only running the changed code live is what makes this actually provable, because re-running the whole agent just resamples and you can pass without having touched the original failure. The cut-point also turns into a natural place to inject counterfactuals, so once the fixture captures the exact prior context you can ask whether the guard still holds under a slightly reworded input. Are you storing the full trace capture, or just the boundary inputs?
Both, and I think they are the same thing at different zoom levels. The unit we store is the boundary, and for each one we capture its inputs and its output, not just the inputs. The full trace is just those per boundary envelopes linked by parent and order.
You need both sides for this to work. The recorded outputs are what let you freeze everything upstream, and the inputs are what let you run the changed boundary live and reword them for counterfactuals. Each envelope holds the assembled prompt, graph state, and retrieved chunks going in, and the completion or tool result coming out, plus the model version and sampling params so a stub returns exactly what was seen.
That is also why the cut point doubles as a counterfactual spot. The exact prior context is already pinned in the envelope, so you can reword one input, keep every other boundary frozen, and see whether the guard still holds.
Repo - github.com/theagentplane/chronicle