DEV Community

Cover image for You Snapshot-Test Your UI. Your AI Agent's Workflows? Nothing. Here's the Fix (0 Tokens per Replay)
Maxime Houle
Maxime Houle

Posted on

You Snapshot-Test Your UI. Your AI Agent's Workflows? Nothing. Here's the Fix (0 Tokens per Replay)

#ai #testing #mcp #agents

Your agent runs the same "pull the metrics, check the deploy, write the report" loop on a schedule. Every single run, the model re-derives those steps from scratch — at full token cost, with no guarantee it does the same thing twice.

You'd never ship UI like that. You snapshot-test it. Your agent's workflows have no equivalent. 🫠

That's the gap Reelier (github.com/seldonframe/reelier) fills: record a working run once, compile it to a SKILL.md, replay it deterministically at 0 tokens, and diff runs to catch drift — with an exit code your CI understands.

The numbers first, because you should demand them

From the repo's published benchmark:

  • 1,000/1,000 replays byte-identical
  • 0 tokens per replay (verified from run records — there's no model in the loop to bill)
  • **48ms vs 2,842ms **against the agent doing the same work

And the limits, stated just as plainly:

  1. Replay handles the deterministic core; judgment steps still need a model
  2. Read-only by default — writes never re-fire without --allow-writes

Here's what a replay actually hands you back:

`▶ nightly-deploy-check (level 0, read-only)
  1. github.get_latest_deployment .... ok  4/4 asserts  31ms
  2. http.get ........................ ok  3/3 asserts  17ms

PASS  2 steps · 7/7 asserts · 0 tokens · 48ms
receipt: .reelier/runs/2026-07-21T06-00-12Z.json`
Enter fullscreen mode Exit fullscreen mode

A receipt 🧾, not a vibe. Let's build one. All commands work as-is in PowerShell and bash — one command per line, no &&.

Step 1 — Install and init 

npm i -g reelier
reelier init
Enter fullscreen mode Exit fullscreen mode

reelier init looks for agent session history already on your machine — Claude Code, Codex, Windsurf, and OpenClaw all write sessions to disk — and offers to lift a replayable workflow out of work you've already paid for.

No API key, no signup. Everything runs and stays local.

Want to record fresh instead? Wrap any MCP server in Reelier's recording proxy and use your agent normally:
reelier mcp --wrap "npx -y @your/mcp-server"

Every tool call and result flowing through the proxy gets captured. Or convert a specific past session directly:

reelier from-session
reelier scan
Enter fullscreen mode Exit fullscreen mode

Step 2 — What a compiled SKILL.md looks like

Compilation makes zero LLM calls — it's a deterministic transform from the recording. Here's the shape (trimmed from a real recording; your field names will differ):

# Skill: nightly-deploy-check

Recorded from: claude-code session 2026-07-14
Replay level: 0 (deterministic, read-only)

## Steps

### 1. github.get_latest_deployment
args: { "repo": "acme/storefront", "env": "production" }
assert:
  - status: ok
  - type: result.sha == string
  - regex: result.sha ~ ^[0-9a-f]{40}$
  - contains: result.state in ["active"]

### 2. http.get
args: { "url": "https://storefront.acme.com/api/health" }
assert:
  - status: ok
  - range: result.latency_ms in 0..2000
  - contains: result.body has "\"db\":\"up\""

## Open questions
- Step 1 `result.created_at` is a timestamp — excluded from byte
  comparison. Confirm this is intentional.
Enter fullscreen mode Exit fullscreen mode

Two design choices worth noticing

1.The assertion grammar is small on purpose. Five checks — status, type, regex, range, contains. They survive expected variation (a fresh timestamp doesn't fail a type check) but catch structural change: a renamed field, an auth error, HTML where JSON used to be.
Assertions, not vibes. No embedding similarity, no "close enough" thresholds.

  1. The compiler doesn't guess. Values it can't prove stable — dates, UUIDs, cursors — land in an honest ## Open questions section for you to resolve, instead of being silently baked in and breaking replay #2.

Step 3 — Replay it

reelier run nightly-deploy-check.skill.md

That prints the receipt you saw up top. The 0 tokens isn't an estimate — no model ran.

One more thing in that receipt: if a step had recorded a write (a POST, a create_* tool), it would be skipped with a warning unless you passed --allow-writes. That default exists because re-firing a recorded write against a changed world is the one mistake this tool refuses to make for you.

Step 4 — reelier diff as a CI gate

diff compares runs step by step, reports SAME or DRIFTED, and exits 1 on drift. That exit code is the whole trick for CI:

reelier run nightly-deploy-check.skill.md
reelier diff nightly-deploy-check
nightly-deploy-check: 2026-07-20 → 2026-07-21
  1. github.get_latest_deployment .. SAME
  2. http.get ...................... DRIFTED
     assert failed: contains result.body has "\"db\":\"up\""
     got: {"db":"degraded", ...}
exit 1
Enter fullscreen mode Exit fullscreen mode

When something drifts, Reelier proposes — it never auto-heals. A silently self-patching test is not a test.

Wire it into GitHub Actions

Minimal workflow (there's also a published action in the repo if you prefer uses: over managing the install):

name: agent-drift-check
on:
  schedule:
    - cron: "0 6 * * *"
  workflow_dispatch:

jobs:
  replay:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm i -g reelier
      - run: reelier run skills/nightly-deploy-check.skill.md
      - run: reelier diff nightly-deploy-check
        # diff exits 1 on DRIFTED → the job fails → you get pinged
Enter fullscreen mode Exit fullscreen mode

Now compare the economics with re-running the agent itself on that schedule.

A 15-minute cron is 2,880 model calls a month. I've worked that math — with OpenClaw's documented heartbeat costs as the worst case, it lands at $1,728/month at Opus prices — at reelier.com/blog/openclaw-cron-costs. Replay makes the recurring cost effectively zero, which is the entire point of reelier.com/blog/scheduled-agents-zero-token.

Step 5 — MCP-native: let your agent record itself

Reelier is also an MCP server, so your agent can freeze and replay its own workflows mid-session. For Claude Code:

claude mcp add reelier -- npx -y reelier serve

For Cursor / Windsurf / anything that takes an mcpServers JSON block:

{
  "mcpServers": {
    "reelier": {
      "command": "npx",
      "args": ["-y", "reelier", "serve"]
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

This exposes reelier_scan, reelier_from_session, reelier_replay, and reelier_diff to the agent itself.

The workflow I actually use: finish a task that was mostly tool calls, then tell the agent "freeze that as a skill." Next time it replays instead of re-reasoning — and hands me the receipt.

What I'd tell you before you try it ⚠️

Honest limits, so you don't find out the hard way:

  1. Judgment steps still need a model. Replay covers the deterministic core — fetch, check, transform, report. Triage and synthesis don't replay; there's an opt-in BYOK escalation for those steps.
  2. Writes are gated. Read-only by default; --allow-writes is per-skill and deliberate.
  3. Non-repeating workflows gain nothing. If every run is genuinely novel, there's nothing to snapshot.
  4. The first run costs normal tokens. You pay once to record; the win amortizes over every scheduled re-run.

TL;DR

Three takeaways, same as promised at the top:

  1. Record a working agent run once → compile to SKILL.md with zero LLM calls
  2. Replay deterministically: 0 tokens, 48ms, 1,000/1,000 byte-identical in the published benchmark
  3. reelier diff exits 1 on drift → your CI already knows what to do with that

Repo, benchmark data, and the run records behind the 0-token claim: github.com/seldonframe/reelier. The cost math lives at reelier.com/blog/openclaw-cron-costs and reelier.com/blog/scheduled-agents-zero-token.

Disclosure: I'm the author of Reelier — it's AGPL, local-first, and free.

Your turn: what's the agent workflow you re-pay for most often — a deploy check, a metrics pull, a report loop? Tell me in the comments and I'll tell you whether replay would actually cover it (including when the honest answer is "it wouldn't"). 👇

Top comments (1)

Collapse
 
mads_hansen_27b33ebfee4c9 profile image
Mads Hansen •

The deterministic core is a useful boundary, but byte-identical replay can still be wrong if the world around it changed. I'd put a manifest beside every skill: tool/server version or schema digest, normalized endpoint identity, policy version, credential scope, and freshness assumptions. Before replay, fail closed on incompatible tool schemas or expired assumptions instead of treating a structurally valid response as success.

For writes, --allow-writes is too broad as the final safety boundary. Require operation-specific approval plus an idempotency key bound to the exact normalized arguments and target environment, then record the resulting external resource/version in the receipt. A dry-run or shadow mode should validate preconditions without side effects. Also distinguish three tests: replay against recorded responses (workflow determinism), replay against mocked failures (recovery behavior), and replay against live read-only dependencies (environment drift). That makes “0 tokens” a cost property without confusing it with current correctness.