This is the next how-to guide in the series. The earlier guides covered individual functions, such as lineage, replay, drift, at(), forget(), SHA-256 checksums, and audit preparation. This one walks through a single scenario every team running agents eventually faces: something went wrong in production, and you need to find out what happened, why, and what to do about it. It is written for engineering leads, on-call engineers, and heads of risk at vertical AI companies and banks who want a repeatable process instead of a scramble.
Before you start: what a good investigation produces
A good incident investigation answers five questions, in this order:
Did something actually change, or is this a one-off? (detection)
What exactly happened in the affected session? (reconstruction)
Why did the agent take the step that went wrong? (causation)
What did the system look like at the time? (context)
Can you prove the record is intact and hand it to someone else? (evidence)
Most teams jump straight to causation, guess, and fix the wrong thing. The steps below keep the order honest.
Step 1: Define the incident in one sentence
Write down what was reported, by whom, and when. For example: a customer says the agent quoted the wrong refund amount on a specific date. Note the session ID if you have it, or the customer and time window if you do not.
Resist the urge to write a theory yet. At this stage you only record the symptom.
Step 2: Check whether behavior has drifted
Open the Agent Behavior view and compare recent sessions against the baseline. You are looking for two things: whether the overall drift score has moved, and whether any specific events or transitions shifted around the time of the incident, for example a verification step being skipped more often than before.
Read the drift score alongside error rate and session length. Drift tells you something changed, not whether it is good or bad. Two outcomes are common. If a relevant shift lines up with the incident, you may be dealing with a systemic issue, and the ranked deltas give you a first hypothesis. If nothing moved, the incident is more likely a one-off or an edge case, which changes how urgent the response is.
Step 3: Reconstruct the session with Session Replay
Open the affected session and step through it in order, from the first user message to the final response. Do not stop at the step where the problem became visible. Many failures begin two or three steps earlier, such as a tool returning a stale result that a later decision relied on.
Mark the first step where the session diverges from what should have happened. That is the step you will investigate in the next stage.
Step 4: Explain the step with causal lineage
Run why() on the step you marked. It returns the chain of inputs, retrieved context, tool results, and decision points that produced it. Check each link:
Was the user input interpreted correctly?
Was the retrieved context accurate and current?
Did the tool return what it should have?
Did the decision follow reasonably from those inputs?
This usually narrows the cause to one of a few categories: bad input, bad context, a failing tool, or a reasoning error. Write down which one, and the evidence for it.
Step 5: Confirm the system state with at()
Query the agent’s state at the time of the incident: which prompt version, tools, and model were live. This protects you from a common mistake, debugging against the current system instead of the system as it was. If a prompt was changed, a tool swapped, or a model upgraded since the incident, the current configuration may not reproduce the problem.
Compare the historical state with the current one and note every difference that could plausibly matter.
Step 6: Check scope
One bad session is rarely the whole story. Using what you learned in Steps 4 and 5, look for other sessions that share the same cause: same tool, same prompt version, same transition pattern. Decide whether this affected one customer, a segment, or everyone, and whether any other customers need to be told.
Be honest about the limits of what you found. If you can only confirm the cause in some sessions, say so in the report instead of generalizing.
Step 7: Verify the integrity of the evidence
Before you share anything outside the team, verify the SHA-256 checksums on the events involved. If they match, the records are exactly as written. If any do not, stop and investigate why before relying on that evidence.
Remember what this does and does not show. Checksums show the record has not changed since it was written. They do not prove the logged content was correct at the time.
Step 8: Decide on the fix
Match the fix to the cause you found:
Bad input or context: improve validation or retrieval, and add checks before the agent acts.
Failing tool: fix or replace the tool, and add a fallback so the agent does not route around it silently.
Reasoning error: adjust the prompt or add a human review step for this class of decision.
Change that caused drift: consider rolling back, then reintroduce it with safeguards.
After shipping the fix, watch drift and error rate for the following days to confirm the shift reverses and nothing new appears.
Step 9: Write the case file
Capture the investigation in a short, standard format so anyone can read it later:
Symptom and timeline
Replay findings, including the first divergent step
Lineage explanation and the cause category
System state at the time, and relevant differences from now
Scope and affected customers
Checksum verification result
Fix, owner, and follow-up date
Where the incident involves a customer dispute or a regulated decision, this file is what you hand to the complaints team, auditor, or ombudsman.
Step 10: Learn from it
Add the failing session to your evaluation set so the same failure is caught before release next time. Decide whether a drift alert threshold, a review rule, or a monitoring check should change. Review the process itself: where did the investigation take longest, and what evidence was missing?
What the business case looks like
These figures are modeled assumptions, not measured results. Replace them with your own.
Assume 20 agent related incidents a month at 100 euro per engineering hour. Without a structured process and connected tooling, each takes about 7 hours, mostly spent reconstructing what happened from scattered logs, costing 14,000 euro a month. With drift detection, replay, lineage, and point in time state in one place, each takes about 1.5 hours, costing 3,000 euro a month. That saves roughly 11,000 euro a month, or 132,000 euro a year.
Add the cost of misdiagnosis. If 1 in 5 investigations leads to fixing the wrong thing and costs about 3 hours of rework, that is 4 cases a month, around 1,200 euro. Debugging against the real historical state and the real causal chain removes most of it.
The larger value is usually outside engineering: faster answers for customers, fewer disputes that drag on, and evidence you can hand over with confidence.
Common mistakes to avoid
Jumping to causation. Skipping reconstruction means you explain a step you have not fully seen.
Debugging the current system. The configuration that caused the incident may no longer exist, so check the historical state.
Fixing one session. If you do not check scope, the same cause keeps producing incidents.
Sharing unverified evidence. Verify integrity before the record leaves the team.
Skipping the case file. Without it, the same investigation gets repeated from scratch next time.
Where to go from here
Start by running this process on your next real incident, and time each step. The steps that take longest show you where your current tooling and procedures are weakest. After three or four incidents, you will have a baseline of your own, which is worth more than any modeled number.
If you tell me your sector and roughly how many incidents you handle a month, I can tailor this process and the numbers to your case.
Want to test ZizkaDB on your agent? Try our open source version here: https://github.com/ZIZKA-AI-SL/ZizkaDB
Interested in a design partnership? Fill in the form on our site or reach me directly at founder@zizka.ai
The article is originally published in Medium and can be viewed here : https://medium.com/@MirArshadTalpur/how-to-investigate-an-agent-incident-end-to-end-with-zizkadb-28c2f328f05f?postPublishedType=initial
Top comments (0)