Eval sets catch obvious mistakes. Replays catch the quiet ones: a changed rule that routes a refund to the wrong queue, or a prompt edit that stops tagging leads.
This is the pattern we use when working as a custom AI workflow agency on changes to a live workflow: record the inputs, replay them through both versions, and diff the decisions, not the wording.
1. Record inputs, not outputs
Store each incoming event with its timestamp and an anonymised payload. You need what came in, not what the old version said.
2. Replay through both versions
import json
def replay(events, old_flow, new_flow):
diffs = []
for e in events:
a, b = old_flow(e), new_flow(e)
keys = ("route", "needs_approval", "crm_stage")
if any(a.get(k) != b.get(k) for k in keys):
diffs.append({"id": e["id"], "old": a, "new": b})
return diffs
events = [json.loads(line) for line in open("last_week.jsonl")]
# print(replay(events, flow_v1, flow_v2))
3. Diff decisions, not prose
Model replies vary between runs, so comparing text is noise. Compare the structured decisions: which queue, whether approval was required, which CRM stage was set.
4. Review every diff by hand
A diff is not a failure. Some are the fix you wanted. The rule is that nobody ships a version whose diffs nobody has read.
5. Keep the replay set fresh
Add new unusual events weekly and drop ones that no longer reflect how the business works.
If you are still mapping which steps deserve automation at all, the openly documented Vibe Checker scoring shows how we write a method down for anyone to check.
Top comments (0)