TL;DR: Your agent said "done." A dependency bumped last week. You cannot actually prove the agent still does what it did before, because you have no record of what it did before. Here is how I started recording real runs and diffing every future run against them, at zero LLM cost.
The moment this became my problem
I merged a green Dependabot PR. Small patch bump on an MCP tool server, SemVer said safe, CI was green, I clicked the button.
Three days later a user flagged that the agent was skipping a step it used to run. Nothing in my test suite caught it, because my "tests" were me running the agent by hand, reading the transcript, and deciding it looked right. The agent had told me "done" every single time. It was not lying on purpose. It just had no way to know, and I had no way to check.
The real question underneath "how do I prove my agent did what it claimed" is this: compared to what? You cannot prove a run is correct if you have no honest record of a run that was correct.
Why the transcript is not proof
The natural instinct is to re-read the transcript. That does not scale past a handful of runs, and worse, the transcript is the agent's own summary. If the agent says it wrote the file and it did not, the transcript still says it wrote the file.
What you actually want is the ground truth: the exact tool calls it made, the arguments it passed, and the outputs it got back. Not the narration. The receipt.
Record the run that worked
The fix that finally held for me was boring in the best way. Record one real run while it is behaving. That capture becomes the baseline everything else is measured against.
capture a real, known-good run of your agent
reelier record -- node agent.js "onboard a new client"
writes a recording: every tool call, args, and result
The recording is not a mock and not a hand-written fixture. It is what the agent genuinely did, tool call by tool call. That is the receipt you were missing.
Replay it, and diff against the baseline
Now the useful part. When anything changes (a dependency bump, an MCP tool-server patch, a refactor), you replay the recorded run and diff the result against the original.
replay the recorded tool behavior, no model calls
reelier run --max-level 0
compare this run against the recorded baseline
reelier diff
--max-level 0 replays the recorded tool behavior without calling the LLM at all. That means the replay costs 0 LLM tokens. In reelier's own runs, 1,000 out of 1,000 replays came back byte-identical, which is what makes the diff trustworthy: if something changed, it changed because of your bump, not because the model felt different today.
If the bump changed nothing, reelier diff is empty and you merge with confidence. If it changed a tool call, an argument, or a result, the diff shows you exactly what and where, before it reaches a user.
Assert on the steps that matter
A full diff is great for a human. For CI you usually want to fail on specific things: this tool must be called, this argument must be present, this step must not disappear. Per-step asserts let you pin the parts of the run you actually care about, so a meaningful drift turns into a red check instead of a shrug.
Wire it into the PR (the check Dependabot lacks)
This is the piece Dependabot cannot give you. Dependabot tells you a version changed and that SemVer says it is safe. It does not tell you whether your agent still behaves the same. SemVer-safe is not behavior-safe.
So put the replay on the pull request. A GitHub Action re-runs your recorded workflows against the bumped dependency and diffs them, and the check goes red on the PR when behavior drifts. Now "is this bump safe" has an answer you can read instead of a hope you carry into prod.
.github/workflows/agent-drift.yml (sketch)
name: replay recorded runs against the bump
run: reelier run --max-level 0 && reelier diff --check
Because the replay is 0 tokens, this runs on every Dependabot PR for free. The bump-check recipe walks through the full setup here: https://www.reelier.com/docs/dependabot-bump-check
An honest caveat about model upgrades
One thing I want to be straight about: the 0-token story is for replaying recorded tool behavior, like dependency and MCP-tool bumps. If you are testing a model upgrade, you have to actually re-run against the new model, and that costs tokens, because the model is the thing you are checking. The record then diff pattern still applies, you just cannot do it for free in that case. Different problem, same discipline.
The mental model that stuck
Treat the agent's "done" as a claim, not a fact. A recorded run is the receipt that either backs the claim up or exposes it. Once every run has a receipt, "prove it" stops being a code-review argument and becomes a diff.
reelier is MIT-licensed and open source, so you can read exactly how the record, replay, and diff work rather than trust a black box.
Top comments (0)