You Recorded the Incident. Now Prove Your Fix Actually Works.

Part 2 of Your Agent Failed in Prod. Good Luck Reproducing It.

This work was presented at the AI Engineer World's Fair 2026 by Susheem Koul and Tisha Chawla.

---

In Part 1 the agent deleted the wrong record at 9:04, and you could not make it happen again. The fix was not to force the model to be deterministic. It was to record the run: the exact prompt, the sampled completion, the tool calls, the retrieved chunks, the pinned model version. Freeze the run, not the model.

So you did that. The incident is now a file on disk. You can open it and see, precisely, what the agent decided that night.

Now comes the part nobody talks about. You still have to fix the bug. And you have to prove the fix works. This is where most teams quietly fall back into the swamp they just climbed out of.

The trap, one more time

You change the code. Now you want to confirm it is fixed. So you run the agent again and watch.

Stop. You just walked back into the trap.

The moment you re-run to check your fix, you are regenerating. The model samples a fresh path. The retrieval returns slightly different chunks. The batch shape on the endpoint is different from the batch shape at 9:04. Your "it works now" is one draw from a distribution, and so was the failure. You have proven nothing, and you know it, because the failure never reproduced on demand in the first place.

To verify a fix you need the exact opposite of a re-run. You need everything about that incident to stay frozen, except the one thing you changed.

A scalpel, not a re-run

That is the whole idea, and it has a name: cut-point replay.

Take the recorded incident. Replay it. But mark one boundary as the cut-point. Everything upstream of it is served from the recording, byte for byte the same inputs the agent saw that night. The boundary you changed runs your new code, live. Everything downstream runs live too, so you can watch what your fix does to the rest of the run.

You are not re-running the agent. You are dropping your new code into the middle of a frozen incident and asking one question: given exactly what happened up to this point, does my change do the right thing now?

No model call. No API cost. No flakiness. The same incident, every time, forever.

Walking through it

Here is the deletion agent from Part 1. Two boundaries: the model decides, the tool acts.

from chronicle import boundary, reset_session, ReplayPlan
from chronicle.envelope.store import EnvelopeStore

@boundary("agent", kind="llm")
def agent(state: dict) -> dict:
    ...  # calls the model, returns a decision and tool calls

@boundary("delete_file", kind="tool")
def delete_file(path: str, environment: str) -> dict:
    ...  # the ungated tool that wiped prod at 9:04

You already recorded the incident and froze it as a fixture:

session = reset_session()
session.store = EnvelopeStore(".chronicle/runs/incident.jsonl")
session.begin_trace("deletion-incident-001")
run_agent(...)                                    # the bad run, captured
session.export_trace("fixtures/traces/deletion-incident-001/")

The recorded graph is exactly what you would expect:

agent@1  ->  delete_file@1 (deleted prod)  ->  agent@2

Now the fix. One guard, in the tool:

@boundary("delete_file", kind="tool")
def delete_file(path: str, environment: str) -> dict:
    if environment == "production":
        return {"blocked": True, "reason": "guard: destructive op refused in prod"}
    ...

And here is the test that proves it, against the real incident:

session = reset_session()
session.load_trace("fixtures/traces/deletion-incident-001/")
session.enable_replay(
    ReplayPlan()
    .stub("agent", 1)         # upstream: the exact decision the model made at 9:04
    .live("delete_file", 1)   # cut-point: your new gated code runs for real
    .live("agent", 2)         # downstream: watch what the agent does after the block
)
run_agent(...)

assert session.captured_result("delete_file", 1)["blocked"] is True

Read the plan again, because it is the entire point. agent@1 is stubbed: the model does not run, you replay the decision it actually made that night. delete_file@1 is live: your new guard executes against those exact arguments. agent@2 is live: you get to see how the agent reacts to a refused deletion instead of a successful one.

You changed one boundary and held the rest of history still. If the guard blocks the delete, the fix works. Not "worked once." Works, deterministically, on the recorded incident, in CI, with no API key.

The two layers, put to work

Part 1 argued for testing in two layers. Cut-point replay is Layer 1 in action: structural, deterministic, about control flow and tool safety. Did the right tool get called? Were the arguments shaped correctly? Was the destructive action refused? None of that needs the model, so none of it flakes.

Layer 2 is for the questions structure cannot answer. If your fix was a prompt rewrite or a model bump, "is the output still correct" is a judgment, not an equality check. That is where an LLM-as-judge scores the new completion against the recorded gold one for meaning, not for bytes. Use Layer 1 to prove the machinery is right. Use Layer 2 to prove the words are still good.

The incident becomes a regression test

Here is the quiet payoff. That fixture under fixtures/traces/ is committed to git. It is now a permanent test. Six months from now, when someone refactors the tool router and the guard silently stops firing, this test goes red on their pull request, not on a customer's production database.

The failure that was unreproducible at 9:04 becomes a green check that runs on every commit. That is the difference between an incident and a regression test: one is a story you tell, the other is a thing your CI enforces.

Before you commit anything: redact

A recorded run is a faithful copy of production. It contains the assembled prompt, the retrieved chunks, the tool arguments. That means it can contain customer names, emails, API keys, internal URLs, whatever your agent touched.

You cannot commit that to git raw. Security and legal are right to block it, and a leaked secret in a fixture is a real incident of its own.

So redaction is not a nice-to-have on the recording path, it is a gate. Scrub secrets and PII out of the envelope before it is written, keep the shape and the structure that your tests assert on, and drop the sensitive values. A recording you cannot safely commit is a recording you will not use.

What this does not fix

Be honest with yourself about the boundary of the technique.

Cut-point replay fixes bugs in your code: routing, guards, argument assembly, tool safety, orchestration. It reproduces those perfectly and lets you verify a fix cheaply.

It does not fix a bad generation. If the model hallucinated a refund amount, replay will faithfully serve that hallucination back to you. Fixing that lives in Layer 2 and in prompt and model work, not in deterministic replay.

Fixtures also drift. Treat them like snapshot tests: when the prompt or the schema changes on purpose, the fixture has to be re-recorded on purpose. And hosted model drift is still outside your control, which is exactly why you pin the version in the envelope so you at least know when it moved.

TLDR;

1. Record at the boundary, the full run, not just the prompt. (Part 1.)
2. Reproduce the incident by replaying the recording, no model call.
3. Pick the boundary you changed. That is your cut-point.
4. Stub everything upstream from the recording. Run the cut-point live. Watch downstream.
5. Assert on the cut-point result, the blocked flag, the tool call, the argument, not on the final prose.
6. Commit the trace as a regression test so the incident can never quietly return.
7. Redact before you commit. Always.

If you want the tooling instead of building it yourself, our team has put the record, cut-point replay, and two-layer verification into an open-source library called Chronicle: github.com/theagentplane/chronicle. It is early and honest about its limits. Issues and war stories welcome.

Official PyPi Release Here

Install to record and replay your agentic workflow

pip install agent-chronicle

Comments

[Raju Dandigam]

"Freeze the run, not the model" is the sentence people should steal from this. Replaying the exact prompt, sampled completion, retrieved chunks, and tool outputs turns an anecdote into a regression artifact, which is the only way fixes stay honest once the model backend moves under you. The other production lesson is to make redaction part of the capture path, not a cleanup step later. Otherwise the runs that are most valuable for debugging become the ones nobody can safely share.

[Tisha]

Agreed!