DEV Community

Ujjwal Dubey
Ujjwal Dubey

Posted on

Replay Testing: Run Last Week's Real Events Through the New Version of Your Workflow

Eval sets catch obvious mistakes. Replays catch the quiet ones: a changed rule that routes a refund to the wrong queue, or a prompt edit that stops tagging leads.

This is the pattern we use when working as a custom AI workflow agency on changes to a live workflow: record the inputs, replay them through both versions, and diff the decisions, not the wording.

1. Record inputs, not outputs

Store each incoming event with its timestamp and an anonymised payload. You need what came in, not what the old version said.

2. Replay through both versions

import json

def replay(events, old_flow, new_flow):
    diffs = []
    for e in events:
        a, b = old_flow(e), new_flow(e)
        keys = ("route", "needs_approval", "crm_stage")
        if any(a.get(k) != b.get(k) for k in keys):
            diffs.append({"id": e["id"], "old": a, "new": b})
    return diffs

events = [json.loads(line) for line in open("last_week.jsonl")]
# print(replay(events, flow_v1, flow_v2))
Enter fullscreen mode Exit fullscreen mode

3. Diff decisions, not prose

Model replies vary between runs, so comparing text is noise. Compare the structured decisions: which queue, whether approval was required, which CRM stage was set.

4. Review every diff by hand

A diff is not a failure. Some are the fix you wanted. The rule is that nobody ships a version whose diffs nobody has read.

5. Keep the replay set fresh

Add new unusual events weekly and drop ones that no longer reflect how the business works.

If you are still mapping which steps deserve automation at all, the openly documented Vibe Checker scoring shows how we write a method down for anyone to check.

Top comments (0)