DEV Community: Maxime Houle

Proving your agent did what it claimed: a receipts-based approach

Maxime Houle — Sat, 25 Jul 2026 11:44:43 +0000

TL;DR: Your agent said "done." A dependency bumped last week. You cannot actually prove the agent still does what it did before, because you have no record of what it did before. Here is how I started recording real runs and diffing every future run against them, at zero LLM cost.

The moment this became my problem

I merged a green Dependabot PR. Small patch bump on an MCP tool server, SemVer said safe, CI was green, I clicked the button.

Three days later a user flagged that the agent was skipping a step it used to run. Nothing in my test suite caught it, because my "tests" were me running the agent by hand, reading the transcript, and deciding it looked right. The agent had told me "done" every single time. It was not lying on purpose. It just had no way to know, and I had no way to check.

The real question underneath "how do I prove my agent did what it claimed" is this: compared to what? You cannot prove a run is correct if you have no honest record of a run that was correct.

Why the transcript is not proof

The natural instinct is to re-read the transcript. That does not scale past a handful of runs, and worse, the transcript is the agent's own summary. If the agent says it wrote the file and it did not, the transcript still says it wrote the file.

What you actually want is the ground truth: the exact tool calls it made, the arguments it passed, and the outputs it got back. Not the narration. The receipt.

Record the run that worked

The fix that finally held for me was boring in the best way. Record one real run while it is behaving. That capture becomes the baseline everything else is measured against.

capture a real, known-good run of your agent reelier record -- node agent.js "onboard a new client" writes a recording: every tool call, args, and result

The recording is not a mock and not a hand-written fixture. It is what the agent genuinely did, tool call by tool call. That is the receipt you were missing.

Replay it, and diff against the baseline

Now the useful part. When anything changes (a dependency bump, an MCP tool-server patch, a refactor), you replay the recorded run and diff the result against the original.

replay the recorded tool behavior, no model calls reelier run --max-level 0

compare this run against the recorded baseline reelier diff

--max-level 0 replays the recorded tool behavior without calling the LLM at all. That means the replay costs 0 LLM tokens. In reelier's own runs, 1,000 out of 1,000 replays came back byte-identical, which is what makes the diff trustworthy: if something changed, it changed because of your bump, not because the model felt different today.

If the bump changed nothing, reelier diff is empty and you merge with confidence. If it changed a tool call, an argument, or a result, the diff shows you exactly what and where, before it reaches a user.

Assert on the steps that matter

A full diff is great for a human. For CI you usually want to fail on specific things: this tool must be called, this argument must be present, this step must not disappear. Per-step asserts let you pin the parts of the run you actually care about, so a meaningful drift turns into a red check instead of a shrug.

Wire it into the PR (the check Dependabot lacks)

This is the piece Dependabot cannot give you. Dependabot tells you a version changed and that SemVer says it is safe. It does not tell you whether your agent still behaves the same. SemVer-safe is not behavior-safe.

So put the replay on the pull request. A GitHub Action re-runs your recorded workflows against the bumped dependency and diffs them, and the check goes red on the PR when behavior drifts. Now "is this bump safe" has an answer you can read instead of a hope you carry into prod.

.github/workflows/agent-drift.yml (sketch) name: replay recorded runs against the bump run: reelier run --max-level 0 && reelier diff --check

Because the replay is 0 tokens, this runs on every Dependabot PR for free. The bump-check recipe walks through the full setup here: https://www.reelier.com/docs/dependabot-bump-check

An honest caveat about model upgrades

One thing I want to be straight about: the 0-token story is for replaying recorded tool behavior, like dependency and MCP-tool bumps. If you are testing a model upgrade, you have to actually re-run against the new model, and that costs tokens, because the model is the thing you are checking. The record then diff pattern still applies, you just cannot do it for free in that case. Different problem, same discipline.

The mental model that stuck

Treat the agent's "done" as a claim, not a fact. A recorded run is the receipt that either backs the claim up or exposes it. Once every run has a receipt, "prove it" stops being a code-review argument and becomes a diff.
reelier is MIT-licensed and open source, so you can read exactly how the record, replay, and diff work rather than trust a black box.

You Snapshot-Test Your UI. Your AI Agent's Workflows? Nothing. Here's the Fix (0 Tokens per Replay)

Maxime Houle — Tue, 21 Jul 2026 20:10:03 +0000

Your agent runs the same "pull the metrics, check the deploy, write the report" loop on a schedule. Every single run, the model re-derives those steps from scratch — at full token cost, with no guarantee it does the same thing twice.

You'd never ship UI like that. You snapshot-test it. Your agent's workflows have no equivalent. 🫠

That's the gap Reelier (github.com/seldonframe/reelier) fills: record a working run once, compile it to a SKILL.md, replay it deterministically at 0 tokens, and diff runs to catch drift — with an exit code your CI understands.

The numbers first, because you should demand them

From the repo's published benchmark:

1,000/1,000 replays byte-identical
0 tokens per replay (verified from run records — there's no model in the loop to bill)
**48ms vs 2,842ms **against the agent doing the same work

And the limits, stated just as plainly:

Replay handles the deterministic core; judgment steps still need a model
Read-only by default — writes never re-fire without --allow-writes

Here's what a replay actually hands you back:

`▶ nightly-deploy-check (level 0, read-only)
  1. github.get_latest_deployment .... ok  4/4 asserts  31ms
  2. http.get ........................ ok  3/3 asserts  17ms

PASS  2 steps · 7/7 asserts · 0 tokens · 48ms
receipt: .reelier/runs/2026-07-21T06-00-12Z.json`

A receipt 🧾, not a vibe. Let's build one. All commands work as-is in PowerShell and bash — one command per line, no &&.

Step 1 — Install and init

npm i -g reelier
reelier init

reelier init looks for agent session history already on your machine — Claude Code, Codex, Windsurf, and OpenClaw all write sessions to disk — and offers to lift a replayable workflow out of work you've already paid for.

No API key, no signup. Everything runs and stays local.

Want to record fresh instead? Wrap any MCP server in Reelier's recording proxy and use your agent normally:
reelier mcp --wrap "npx -y @your/mcp-server"

Every tool call and result flowing through the proxy gets captured. Or convert a specific past session directly:

reelier from-session
reelier scan

Step 2 — What a compiled SKILL.md looks like

Compilation makes zero LLM calls — it's a deterministic transform from the recording. Here's the shape (trimmed from a real recording; your field names will differ):

# Skill: nightly-deploy-check

Recorded from: claude-code session 2026-07-14
Replay level: 0 (deterministic, read-only)

## Steps

### 1. github.get_latest_deployment
args: { "repo": "acme/storefront", "env": "production" }
assert:
  - status: ok
  - type: result.sha == string
  - regex: result.sha ~ ^[0-9a-f]{40}$
  - contains: result.state in ["active"]

### 2. http.get
args: { "url": "https://storefront.acme.com/api/health" }
assert:
  - status: ok
  - range: result.latency_ms in 0..2000
  - contains: result.body has "\"db\":\"up\""

## Open questions
- Step 1 `result.created_at` is a timestamp — excluded from byte
  comparison. Confirm this is intentional.

Two design choices worth noticing

1.The assertion grammar is small on purpose. Five checks — status, type, regex, range, contains. They survive expected variation (a fresh timestamp doesn't fail a type check) but catch structural change: a renamed field, an auth error, HTML where JSON used to be.
Assertions, not vibes. No embedding similarity, no "close enough" thresholds.

The compiler doesn't guess. Values it can't prove stable — dates, UUIDs, cursors — land in an honest ## Open questions section for you to resolve, instead of being silently baked in and breaking replay #2.

Step 3 — Replay it

reelier run nightly-deploy-check.skill.md

That prints the receipt you saw up top. The 0 tokens isn't an estimate — no model ran.

One more thing in that receipt: if a step had recorded a write (a POST, a create_* tool), it would be skipped with a warning unless you passed --allow-writes. That default exists because re-firing a recorded write against a changed world is the one mistake this tool refuses to make for you.

Step 4 — reelier diff as a CI gate

diff compares runs step by step, reports SAME or DRIFTED, and exits 1 on drift. That exit code is the whole trick for CI:

reelier run nightly-deploy-check.skill.md
reelier diff nightly-deploy-check
nightly-deploy-check: 2026-07-20 → 2026-07-21
  1. github.get_latest_deployment .. SAME
  2. http.get ...................... DRIFTED
     assert failed: contains result.body has "\"db\":\"up\""
     got: {"db":"degraded", ...}
exit 1

When something drifts, Reelier proposes — it never auto-heals. A silently self-patching test is not a test.

Wire it into GitHub Actions

Minimal workflow (there's also a published action in the repo if you prefer uses: over managing the install):

name: agent-drift-check
on:
  schedule:
    - cron: "0 6 * * *"
  workflow_dispatch:

jobs:
  replay:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm i -g reelier
      - run: reelier run skills/nightly-deploy-check.skill.md
      - run: reelier diff nightly-deploy-check
        # diff exits 1 on DRIFTED → the job fails → you get pinged

Now compare the economics with re-running the agent itself on that schedule.

A 15-minute cron is 2,880 model calls a month. I've worked that math — with OpenClaw's documented heartbeat costs as the worst case, it lands at $1,728/month at Opus prices — at reelier.com/blog/openclaw-cron-costs. Replay makes the recurring cost effectively zero, which is the entire point of reelier.com/blog/scheduled-agents-zero-token.

Step 5 — MCP-native: let your agent record itself

Reelier is also an MCP server, so your agent can freeze and replay its own workflows mid-session. For Claude Code:

claude mcp add reelier -- npx -y reelier serve

For Cursor / Windsurf / anything that takes an mcpServers JSON block:

{
  "mcpServers": {
    "reelier": {
      "command": "npx",
      "args": ["-y", "reelier", "serve"]
    }
  }
}

This exposes reelier_scan, reelier_from_session, reelier_replay, and reelier_diff to the agent itself.

The workflow I actually use: finish a task that was mostly tool calls, then tell the agent "freeze that as a skill." Next time it replays instead of re-reasoning — and hands me the receipt.

What I'd tell you before you try it ⚠️

Honest limits, so you don't find out the hard way:

Judgment steps still need a model. Replay covers the deterministic core — fetch, check, transform, report. Triage and synthesis don't replay; there's an opt-in BYOK escalation for those steps.
Writes are gated. Read-only by default; --allow-writes is per-skill and deliberate.
Non-repeating workflows gain nothing. If every run is genuinely novel, there's nothing to snapshot.
The first run costs normal tokens. You pay once to record; the win amortizes over every scheduled re-run.

TL;DR

Three takeaways, same as promised at the top:

Record a working agent run once → compile to SKILL.md with zero LLM calls
Replay deterministically: 0 tokens, 48ms, 1,000/1,000 byte-identical in the published benchmark
reelier diff exits 1 on drift → your CI already knows what to do with that

Repo, benchmark data, and the run records behind the 0-token claim: github.com/seldonframe/reelier. The cost math lives at reelier.com/blog/openclaw-cron-costs and reelier.com/blog/scheduled-agents-zero-token.

Disclosure: I'm the author of Reelier — it's AGPL, local-first, and free.

Your turn: what's the agent workflow you re-pay for most often — a deploy check, a metrics pull, a report loop? Tell me in the comments and I'll tell you whether replay would actually cover it (including when the honest answer is "it wouldn't"). 👇

I built an open-source "CI for AI agents", and here is why I went open-source

Maxime Houle — Mon, 20 Jul 2026 15:14:39 +0000

Hi, my name is Max. I built Reelier, an open-source tool that gives AI agent workflows the thing normal code has had for decades: snapshot tests and CI.

Here is the itch.

My coding agent kept re-deriving the same tool-call workflow on every run.
Fetch this, transform that, post the result.
Same tokens, same seconds, for a task with exactly one correct answer.

And every so often it did it a little differently, and nothing caught it. That second part is the scary one.

Why open-source it

Agent tooling is flooded right now. New closed tools ship every day, and most of them ask you to trust a black box with your workflows and your keys.

Open-source is how a small tool earns trust in that noise: you can read exactly what it does, run it on your own machine, and keep your data as plain files.

What it actually does

You record one agent run that worked.
Reelier compiles it into a readable SKILL.md file, a recipe with typed inputs, typed outputs, and an assertion on every step.
Then it replays that file deterministically: no model call, 0 tokens, byte-identical, in milliseconds.
Every replay is a receipt.

And reelier diff compares two runs and exits 1 when a step drifts, so a silent regression fails your build instead of reaching production.

The honest part

It only replays deterministic tool calls, HTTP and MCP.

It does not replay file edits or free-form reasoning, and it never fabricates a step it cannot reproduce. It is for the deterministic half of what agents do.

I say that up front because the version that would go viral (record my whole agent) is the version I will not claim, since it would not be true.

Numbers, from the benchmark in the repo: 1000 of 1000 replays byte-identical, 0 tokens per replay, about 50x cheaper and 59x faster than the agent re-running the same task.

It is AGPL-3.0, BYOK, and runs entirely on your machine.

Any help you can give me would be super tremendous! 🥹
I would be happy to get a star if possible ❤️https://github.com/seldonframe/reelier.