DEV Community

Cover image for How to Upgrade Your LLM Without Breaking Production: A Replay Testing Guide with ZizkaDB
Mir Arshad Ali Talpur
Mir Arshad Ali Talpur

Posted on

How to Upgrade Your LLM Without Breaking Production: A Replay Testing Guide with ZizkaDB

Sooner or later every team running AI agents in production faces the same decision. The model provider is retiring the version you use. A cheaper model has appeared that might do the job. Latency is too high, or a customer wants their data processed in a different region. Someone edits the prompt to fix one complaint. In each case the question is identical: will the agent still behave the way our customers rely on?

Most teams answer it with a handful of manual tests and a hopeful release. Then, a week later, approval rates shift, reviewers start correcting a new kind of mistake, and nobody can say whether the change came from the upgrade or from something else.

This guide shows a better way, using something you already have if you record your agents properly: a history of real cases. You will turn recorded production history into a replay test, run a candidate model or prompt in shadow mode against it, compare the results with evidence, release in stages, and keep a sign-off record that you can show a customer or auditor. The audit layer in the examples is ZizkaDB, an open source causal audit database for AI agents, but the method applies wherever you have causally linked history.

A note on scope. ZizkaDB is not an evaluation framework and does not judge whether an answer is good. It records and links what happened, so you can build a trustworthy test set and compare runs. Code parameter names are illustrative and should be checked against the SDK reference. Nothing here is legal advice, and ZizkaDB does not by itself make any system compliant with the EU AI Act or any other law.

Why upgrades are riskier than they look

Agents built on language models are probabilistic. Two versions of a model can both be good and still behave differently on the cases your customers care about. A new version may be more verbose, more cautious, quicker to call a tool, or slightly different in how it follows your formatting rules. None of this throws an error. It shows up as agent drift: a slow shift in outcomes that no alert catches.

Three business realities make this a real problem for vertical AI companies and any enterprise running agents.

Forced upgrades are routine. Model versions are retired on the provider's schedule, not yours. If your only test is a few examples someone remembers, you are gambling on a deadline.

Cost pressure invites change. A cheaper or faster model can cut your bill sharply, and the temptation to switch is strong. Without a way to show that quality held, you cannot make the switch with confidence, and you cannot justify it to a customer who asks.

Customers care about change. Enterprise buyers ask how you manage changes to the AI in your product, and they notice when behaviour changes without notice. A documented, evidence-based change process is part of what they buy.

The common failure is not a dramatic outage. It is a quiet degradation on a slice of cases, found by a customer instead of by you.

The idea: replay real cases against the candidate

The core idea is simple. If you have recorded real cases, with their inputs, retrieved context and outcomes, you can run the same inputs through a candidate configuration and compare. You are not guessing what customers send, because you have what they sent.

For this to work, your record needs a few things:

The inputs the agent received, with sensitive values redacted where appropriate.
The context it retrieved, so the candidate can be given the same material.
The output and decision of the production version.
The versions: model, prompt and tool configuration on each call.
The human outcome: whether a reviewer accepted, edited or rejected the result, and what they changed.
The customer and task type, so you can sample fairly.

If you log events with causal links, as described in our other guides, you already have this. If not, start there: add model and prompt versions and human review events to your instrumentation first, because replay is only as good as the record.

Step 1: build a test set from production history

Do not replay everything. Build a deliberate sample that reflects the risks.

Stratify. Take cases across customers, task types and time periods, so that one busy customer does not dominate. Include recent cases, since they reflect current behaviour.

Include the hard ones. Add cases where a human reviewer corrected or rejected the agent. These are your most valuable tests, because they show where the production version already struggles, and a candidate that fixes them is a real improvement. Add cases that triggered errors, long inputs and unusual formats.

Include the boring ones. The bulk of the set should be ordinary cases the agent handles well. A candidate that fails on these is not ready.

Keep it sensitive-data safe. Redact before storage and before replay. Before you send customer data to any new model provider, check your contracts and data processing terms, because a different provider may be a new subprocessor that customers must be told about. This is easy to forget and expensive to get wrong.

You can pull the set with ZizkaDB's export and iteration functions. An illustrative sketch:

python
cases = []
async for ev in db.iter_events(agent="claims-agent", event="decision",
since="2026-09-01"):
cases.append(ev)

pair each decision with its human review and the retrieved context

test_set = build_test_set(cases, include_corrected=True, per_customer_cap=50)

Freeze the test set once built. A moving test set makes comparisons meaningless.

Step 2: run the candidate in shadow mode

Shadow mode means the candidate processes the same inputs, but its output never reaches a customer. You record what it would have done and compare.

Log the shadow run under a separate agent name, and link each shadow decision back to the production decision it mirrors. That link is the key to everything that follows. Illustrative code:

python
for case in test_set:
out = await run_candidate(case.input, case.context,
model="candidate-model", prompt_v="v15")

await db.log(agent="claims-agent-shadow", event="decision",
    data={"outcome": out.outcome, "reason": out.reason,
          "model": "candidate-model", "prompt_v": "v15",
          "tokens": out.tokens, "latency_ms": out.latency_ms},
    parent_id=case.production_decision_id)
Enter fullscreen mode Exit fullscreen mode

Because the shadow decision points to the production one, you can walk from either side to the other. You can also use batching and idempotency keys when logging many shadow results, so that a retry does not duplicate events.

Two practical points. First, run the candidate more than once on a sample if outputs vary, because a single run of a probabilistic model can mislead. Second, keep the shadow agent's record separate from production metrics, so shadow runs do not pollute your drift baselines for the live agent.

Step 3: compare with evidence, not impressions

Now you have paired results. Compare them on what matters to your business, and write down the criteria before you look at the numbers, so you do not shape them to fit the result.

Agreement. How often does the candidate reach the same outcome as production? High agreement is reassuring, but the disagreements are where the information is.

Performance on the corrected cases. Among cases where a human reviewer overrode production, how often does the candidate land on the human's answer? This is the closest thing to ground truth in your data, and it tells you whether the candidate is better, equal or worse where it counts.

Regressions. Among cases production got right and the human accepted, how often does the candidate differ? Each is a potential regression to review.

Cost. Tokens and spend per case, from the recorded usage. This is where a cheaper model earns its keep, but only if quality holds.

Speed. Latency per case.

Behaviour mix. Tool calls per case, refusal rate, output length, and the share of cases where the candidate cites its sources. A model that suddenly calls twice as many tools may be fine, or may be wasting money or taking risks.

By segment. Break results down by customer, task type and language. An average can hide a slice where the candidate fails badly.

ZizkaDB's baseline() can help here by comparing the shadow agent's behaviour against the production agent's reference period on the metrics you choose. It does not decide whether a difference is good. You do.

Step 4: review disagreements with lineage

Averages tell you how much differs. Lineage tells you why. For each meaningful disagreement, open both sides and read the chains.

python
(await db.why(shadow_decision.id)).print()
(await db.why(production_decision.id)).print()

Look for patterns. Did the candidate ignore a retrieved rule? Did it weigh a document differently? Did it fail on a particular input format? Group disagreements into categories and count them. Often the result is not accept or reject but fix the prompt for this category, then rerun. Because the shadow run is recorded with its prompt version, you can compare version 15 against version 16 the same way, and the history of your experiments becomes part of your record.

A human review of a sample of disagreements is worthwhile. Have a domain expert, not only an engineer, look at them. Record their verdicts as events linked to the cases, so that your conclusions rest on documented judgments.

Step 5: release in stages and watch for drift

Passing the replay test does not guarantee success in production, so release gradually.

Canary. Route a small share of live traffic, by customer or by case, to the candidate. Record the model and prompt version on every call, so that you can separate canary and baseline in your measurements.

Watch the signals. Compare approval rates, human edit rates, review time, error rates, cost and latency between canary and control. Watch the share of reviewer edits in particular, because a rise means humans are fixing more mistakes.

Set rollback criteria in advance. Decide before the canary what result triggers rollback, for example a rise in reviewer overrides beyond a threshold, or any case in a critical category going wrong. Having the rule written down removes the temptation to rationalise.

Expand. If the canary holds, widen it in steps, and keep measuring. When the change is complete, set a new baseline period for the new version so that future drift checks compare against it.

If something does go wrong, ZizkaDB's time travel features can help you see the agent's state as of a given moment from logged state events, and the impact route can list the cases that descended from a particular input, which helps you scope who was affected and tell them precisely.

Step 6: keep a sign-off record

The last step is the one that turns an engineering exercise into evidence. Record the decision to change as an event.

python
await db.log(agent="change-control", event="model_change_approved",
data={"from": {"model": "model-a", "prompt_v": "v14"},
"to": {"model": "candidate-model", "prompt_v": "v15"},
"test_set_size": 400, "agreement": "recorded in report",
"approver": "head-of-ml", "date": "2026-11-17"},
parent_id=None)

Link it to the summary of test results. Keep the report with the record. Store your chain head outside the database so that the history cannot be quietly rewritten.

This does three useful things. It answers customers who ask how you manage changes to the AI in your product. It gives your own team a record of why a change was made. And it supports the kind of record keeping that rules such as the EU AI Act point toward, for systems that fall under them. Article 12 of the Act concerns automatic recording of events over the lifetime of high risk systems. Whether a substantial change to your system triggers further obligations depends on your system and your role, so take legal advice on that.

Three scenarios

The following scenarios are illustrative composites, not real customers.

Scenario 1: a forced upgrade with a deadline

A provider announces that a model version will be retired in ninety days. A claims automation company has recorded eight months of cases. It builds a test set of 400 cases across five customers, including 60 where reviewers overrode the agent. The candidate replacement agrees with production on most cases, but on the overridden cases it matches the reviewer less often than the current model, in one customer's category only. Lineage shows the new model ignores a formatting rule in that customer's retrieved policy. A prompt adjustment fixes it, the replay is rerun and the canary proceeds. The deadline is met, and the team can show the customer how the change was tested.

Scenario 2: switching to a cheaper model

A legal tech company wants to cut costs. The replay shows the candidate costs far less per case and agrees with production on most cases, but the regression review finds that it misses a certain type of indemnity clause more often. The team decides to use the cheaper model for low-risk document types and keep the current one for the rest. The decision, the evidence and the routing rule are all recorded. The company saves money without taking a risk it cannot explain.

Scenario 3: a prompt change to fix one complaint

A bank's credit pre-assessment vendor edits a prompt to fix a customer complaint. The replay shows the edit fixes the complaint but raises the decline rate for a different segment. Without replay, this would have appeared weeks later as a drift signal and an angry client. With it, the vendor catches the side effect in an afternoon and revises the edit.

Common pitfalls
No versions recorded. If you did not log model and prompt versions, you cannot attribute a change to anything.
A test set full of easy cases. Include the corrected and failed cases, or the test proves nothing.
Judging after seeing the numbers. Write criteria first.
One run of a probabilistic model. Repeat on samples where variance matters.
Sending customer data to a new provider without checking contracts. A new provider may be a new subprocessor.
Mixing shadow and production records. Keep separate agent names so baselines stay clean.
Skipping the canary. Replay reduces risk. It does not remove it.
No sign-off record. If the decision is not recorded, the next question from a customer or auditor is hard to answer.
A one week plan

Day 1: check that the record includes inputs, context, outputs, versions and human outcomes. Fix gaps. Day 2: build and freeze the test set, including corrected cases, with redaction applied. Day 3: run the candidate in shadow mode and log results with links to production decisions. Day 4: compare agreement, corrected case performance, regressions, cost and latency, by segment. Day 5: review disagreements with lineage, group them and adjust the prompt if needed, then rerun. Day 6: plan the canary, set rollback criteria and check contracts for any new subprocessor. Day 7: record the sign-off and begin the canary.

Limits to remember

ZizkaDB records, links and measures. It does not run your model, judge answers, or guarantee that a candidate will behave in production as it did in replay. Replay shows how the candidate handles past cases, and the future may differ. Outputs vary between runs. The corrected cases in your history reflect your reviewers' judgment, which may itself contain mistakes. The open source edition records one agent per install, so multi-customer and multi-agent setups use the Cloud edition. Confirm the specific features you rely on, such as iteration, export, baseline() and time travel, against the current documentation before you build on them.

Get started

Pick one agent and one upcoming change, whether a model upgrade, a cost saving or a prompt edit. Make sure versions and human outcomes are recorded, build a small test set that includes corrected cases, and run your first shadow replay. You will learn more from one afternoon of comparison than from a month of hoping.

ZizkaDB runs self hosted in a single Docker container, with a managed Cloud edition at db.zizka.ai for multi-agent setups.

Corrections: if you represent a party named here and believe a statement is inaccurate, contact us and we will review it promptly.

Sources: ZizkaDB documentation; Regulation (EU) 2024/1689, the EU AI Act, Article 12.

Top comments (0)